Neural Network Layers Explained

The Three Layer Types

Input layers receive raw data and pass it to the network without transformation. The input layer's size matches the data dimensions. For a 224x224 color image, the input layer has 224 * 224 * 3 = 150,528 values (height times width times three color channels). For text processed by a transformer, the input layer receives token embeddings, typically vectors of 768 to 4,096 dimensions per token. The input layer is the interface between the external world and the network's internal computations.

Hidden layers are where the network's learning happens. Each hidden layer takes the output of the previous layer and produces a new representation through weighted sums and nonlinear activations. The term "hidden" reflects that these layers are internal to the network; their values are not directly observed as inputs or outputs. A network can have anywhere from one hidden layer (a shallow network) to hundreds (a deep network). The representations in hidden layers are not designed by humans; they emerge from the training process.

Output layers produce the network's final prediction in a format appropriate to the task. For multi-class classification, the output layer uses softmax activation to produce a probability distribution over classes. For binary classification, a single neuron with sigmoid activation outputs a probability between 0 and 1. For regression, a single neuron with no activation (linear output) produces a continuous value. For language generation, the output layer produces a probability distribution over the entire vocabulary of tokens.

How Layers Build Representations

The power of layered architectures comes from their ability to build increasingly abstract representations. Each layer transforms its input into a representation that is slightly more useful for the task than the representation it received.

In an image classification network, this progression is well understood through visualization research. Layer 1 detects edges and color gradients, the simplest patterns that differ from random noise. Layer 2 combines edges into textures and corners. Layers 3 and 4 combine textures into recognizable parts like eyes, wheels, or window frames. The final hidden layers represent complete objects and scenes. The network has learned to decompose visual recognition into a hierarchy of reusable parts, all without being told that such a hierarchy exists.

In a language model, the progression is less visually intuitive but equally structured. Lower layers encode token-level information: word identity, part of speech, and local context. Middle layers encode syntactic relationships: subject-verb agreement, dependency structure, coreference. Upper layers encode semantic meaning: topic, intent, factual associations, and the abstract representations needed for the specific prediction task.

Each layer builds on the previous one, which means the depth of the network determines the complexity of the representations it can construct. A single-layer network can only combine raw input features linearly (plus a nonlinearity). A two-layer network can combine first-layer features into more complex patterns. A hundred-layer network can build enormously abstract concepts from sequences of increasingly refined transformations.

Common Layer Types

Dense (fully connected) layers connect every neuron in the layer to every neuron in the previous layer. A dense layer with 512 neurons receiving input from a layer with 1,024 neurons has 512 * 1,024 = 524,288 weights plus 512 biases. Dense layers make no assumptions about the structure of the input data, which makes them versatile but parameter-hungry. They are typically used in the final stages of a network after specialized layers have extracted relevant features.

Convolutional layers apply small learnable filters across spatial positions. A 3x3 convolutional filter has only 9 weights (times the number of input channels), but it is applied at every position in the image, detecting the same feature pattern wherever it occurs. A convolutional layer might have 64 different filters, each detecting a different pattern, producing 64 feature maps. Stacking convolutional layers builds the hierarchical feature representation that makes CNNs so effective for images.

Pooling layers reduce the spatial dimensions of feature maps by summarizing local regions. Max pooling takes the maximum value in each region (typically 2x2), reducing the feature map to half its width and height. This makes the representation more compact, reduces computation in subsequent layers, and provides a degree of translation invariance (the exact position of a feature matters less after pooling). Pooling layers have no learnable parameters.

Attention layers compute dynamic connections between all positions in a sequence (or image). Each position produces a query, key, and value vector. Attention weights are computed from the dot product of queries and keys, then used to weight the values. Multi-head attention runs this process multiple times in parallel with different learned projections, allowing different heads to capture different types of relationships. Attention layers are the core building block of transformers.

Normalization layers stabilize the distribution of values flowing through the network. Batch normalization normalizes activations across the batch dimension, keeping the mean near zero and variance near one. Layer normalization normalizes across the feature dimension for each individual example. These layers dramatically improve training stability and speed, especially in deep networks where the distribution of activations can shift unpredictably as parameters are updated.

Dropout layers randomly set a fraction of neuron outputs to zero during training. This prevents the network from relying on any specific set of neurons and forces it to learn redundant, distributed representations that generalize better. Dropout rates of 0.1 to 0.5 are common. At inference time, dropout is disabled and all neurons are active.

Embedding layers convert discrete tokens (words, categories, item IDs) into dense numerical vectors. An embedding layer is essentially a lookup table where each row is a learnable vector. When the network processes token ID 5,437, the embedding layer returns the 5,437th row, a vector of perhaps 768 dimensions. These vectors are learned during training so that semantically similar tokens end up with similar vectors.

Depth vs. Width

A network can increase its capacity by adding more layers (depth) or more neurons per layer (width). Both increase the total number of parameters, but they have different effects on learning.

Depth increases the abstraction hierarchy. Each additional layer can combine features from the previous layer into higher-level representations. This is particularly valuable for problems with natural hierarchical structure, like images (edges, textures, parts, objects) and language (characters, words, phrases, meaning). Deep networks learn more abstract, compositional features.

Width increases each layer's representational capacity. More neurons per layer means the network can detect more features at each level of abstraction. However, adding width without depth does not increase the number of abstraction levels. A very wide, shallow network can memorize many patterns but may not build the hierarchical representations needed for generalization.

In practice, modern architectures use both depth and width, but depth matters more. ResNet demonstrated that networks with 50, 101, or even 152 layers significantly outperform shallower, wider networks with similar parameter counts. The key innovation that enabled this depth was skip connections, which we cover in the residual connections section below.

Residual Connections and Very Deep Networks

Training very deep networks (more than about 20 layers) was difficult before 2015 because of the degradation problem: adding more layers actually decreased accuracy, not because of overfitting, but because gradients vanished or exploded during backpropagation through many layers.

Residual connections (introduced in ResNet by Kaiming He et al., 2015) solved this by adding the input of each layer directly to its output. Instead of learning a complete transformation y = f(x), the layer learns a residual y = x + f(x). If the optimal transformation is close to the identity (which is common in deep networks), learning the small residual f(x) is much easier than learning the full transformation from scratch.

Residual connections also provide a gradient highway. During backpropagation, the gradient can flow directly through the skip connection without being transformed by intermediate layers. This prevents gradient vanishing and allows training networks with hundreds of layers. ResNet-152 (152 layers), DenseNet-264 (264 layers), and transformer models with 96+ layers all rely on residual connections.

Modern transformer blocks standardize this pattern: each block consists of an attention layer followed by a feedforward layer, with residual connections around each and layer normalization to stabilize training. This block is repeated dozens or hundreds of times, and the residual connections ensure that gradients flow effectively through the entire stack.

How Many Layers Do You Need?

The answer depends entirely on the task and data complexity. Simple patterns in tabular data might need only 2 to 4 layers. Image classification typically uses 18 to 152 layers. Large language models use 32 to 120 transformer blocks. Adding layers beyond what the task requires wastes computation without improving performance and can make training harder.

A practical approach is to start with an architecture known to work for your data type (ResNet for images, BERT or GPT for text, XGBoost for tabular data) and adjust from there. Architecture search, which automates the process of finding optimal layer configurations, is an active research area but is expensive and typically reserved for applications where even small accuracy improvements have large value.