Types of Neural Networks

Feedforward Neural Networks

The simplest neural network type. Data flows in one direction from input through hidden layers to output. Every neuron in one layer connects to every neuron in the next (fully connected). Feedforward networks make no assumptions about data structure, making them versatile but parameter-heavy.

Best for tabular data (spreadsheets, databases) and as the final classification or regression head on top of other architectures. For tabular data specifically, gradient boosted trees (XGBoost, LightGBM) often outperform feedforward networks with less tuning, so neural networks are not always the first choice even when they are applicable.

A typical feedforward network has 2 to 5 hidden layers with 64 to 1,024 neurons per layer, ReLU activations, and dropout for regularization. The perceptron (single-layer, no hidden layers) is a special case that can only learn linear decision boundaries. Multi-layer perceptrons (MLPs) with at least one hidden layer and nonlinear activation can approximate any continuous function.

Convolutional Neural Networks (CNNs)

Designed for spatial data, primarily images. CNNs use small learnable filters (typically 3x3) that slide across the input, detecting local patterns at every position. This weight sharing makes them dramatically more parameter-efficient than dense networks for images. A CNN with 25 million parameters can process images that would require billions of parameters in a feedforward network.

The key property is translation equivariance: a feature detected in one part of the image is detected by the same filter in any other part. Stacked convolutional layers build hierarchical features from edges (layer 1) to textures (layers 2-3) to object parts (layers 4-6) to complete objects (layers 7+).

Landmark architectures include AlexNet (2012, 8 layers), VGGNet (2014, 16-19 layers), ResNet (2015, 50-152 layers with skip connections), and EfficientNet (2019, balanced scaling). CNNs are also used for audio processing (on spectrograms), video analysis (3D convolutions), and some scientific data with spatial structure.

Recurrent Neural Networks (RNNs)

Designed for sequential data. RNNs process input one step at a time, maintaining a hidden state that carries information from previous steps. The same weights are applied at every time step, and the hidden state provides context from earlier in the sequence.

Vanilla RNNs suffer from vanishing gradients over long sequences, limiting their effective memory to roughly 10-20 steps. LSTMs (Long Short-Term Memory) solve this with a gated cell state that can maintain information over hundreds of steps. GRUs (Gated Recurrent Units) are a simpler variant with similar performance and faster training.

RNNs were the dominant architecture for language tasks from 2013 to 2018 before transformers replaced them. They remain useful for streaming applications (real-time audio processing, sensor data) where the entire sequence is not available upfront and for situations where the O(n) computational cost of RNNs is preferred over the O(n^2) cost of transformer attention.

Transformers

The dominant architecture for language and increasingly for vision, audio, and multimodal tasks. Transformers replace recurrence with self-attention, allowing every position in a sequence to attend to every other position simultaneously. This provides better parallelization during training and superior modeling of long-range dependencies.

Encoder-only transformers (BERT) excel at understanding tasks. Decoder-only transformers (GPT, Claude, LLaMA) excel at generation and have become the default for general-purpose AI. Encoder-decoder transformers (T5, BART) handle sequence-to-sequence tasks like translation and summarization.

Vision transformers (ViT) split images into patches, treat each patch as a token, and process them with standard transformer blocks. They match or beat CNNs on image tasks when trained on sufficient data, demonstrating that the transformer architecture is not inherently language-specific.

Generative Adversarial Networks (GANs)

GANs consist of two networks trained simultaneously: a generator that creates synthetic data and a discriminator that tries to distinguish synthetic from real. The adversarial training process drives both networks to improve. The generator learns to produce increasingly realistic outputs, while the discriminator becomes increasingly discerning.

GANs have produced photorealistic face generation (StyleGAN), image-to-image translation (pix2pix, CycleGAN), super-resolution (ESRGAN), and data augmentation. They are notoriously difficult to train (mode collapse, training instability) and have been partially replaced by diffusion models for image generation tasks since 2022.

Autoencoders

Autoencoders learn compressed representations by training the network to reconstruct its own input through a bottleneck. The encoder compresses the input into a low-dimensional latent representation, and the decoder reconstructs the input from this representation. The bottleneck forces the network to learn the most important features of the data.

Variational autoencoders (VAEs) learn a probability distribution in the latent space rather than a single point, enabling them to generate new data by sampling from the distribution. VAEs produce smoother, more controllable outputs than standard autoencoders. Applications include image generation, anomaly detection, denoising, and dimensionality reduction.

Graph Neural Networks (GNNs)

Designed for data that is naturally represented as graphs: social networks, molecular structures, transportation networks, citation networks. GNNs operate by passing messages between connected nodes, with each node aggregating information from its neighbors to update its representation.

Message passing neural networks (MPNNs), graph convolutional networks (GCNs), and graph attention networks (GATs) are the main variants. GNNs are particularly important in chemistry and drug discovery, where molecules are naturally graphs with atoms as nodes and bonds as edges.

Diffusion Models

The current state of the art for image generation. Diffusion models learn to reverse a gradual noising process: given an image progressively corrupted with Gaussian noise over many steps, the model learns to predict and remove the noise at each step. Generation starts from pure noise and iteratively denoises until a clean image emerges.

DALL-E 2, Stable Diffusion, and Midjourney all use diffusion models. They produce higher-quality, more diverse images than GANs and are easier to train. The main drawback is slow generation: producing an image requires hundreds of denoising steps, each a full forward pass through a large neural network.

Spiking Neural Networks (SNNs)

The most biologically realistic neural network type. Instead of passing continuous values between neurons, SNNs communicate with discrete spikes, similar to how biological neurons fire action potentials. Information is encoded in the timing and frequency of spikes rather than in continuous activation values.

SNNs are more energy-efficient than conventional neural networks when run on neuromorphic hardware (like Intel's Loihi), making them attractive for edge computing and battery-powered devices. However, they are harder to train than conventional networks (backpropagation does not directly apply to discrete spikes), and they lag behind conventional networks on most benchmarks.

Choosing the Right Architecture

The decision tree is straightforward for most tasks. For images, start with a pre-trained CNN or ViT. For text, start with a pre-trained transformer. For tabular data, try gradient boosted trees first, then a feedforward network if needed. For sequences (time series, audio), try transformers first, then LSTMs if the sequence length makes attention too expensive. For graph-structured data, use a GNN. For generation, use diffusion models for images or a decoder transformer for text.

In nearly every case, starting with a pre-trained model and fine-tuning is better than training from scratch. The pre-trained model has already learned general features from millions of examples, and fine-tuning adapts those features to your specific task with far less data and compute.