Math Behind Neural Networks

Linear Algebra: The Language of Neural Networks

Every neural network operation reduces to linear algebra at its core. Data is represented as vectors and matrices, layers are matrix multiplications, and the entire forward pass is a sequence of matrix operations with nonlinear functions interleaved.

Vectors are ordered lists of numbers. An input to a neural network is a vector: a 784-dimensional vector for an MNIST image (one value per pixel), a 768-dimensional vector for a BERT token embedding. Vector operations like addition and scalar multiplication are used at every layer.

Matrices are 2D arrays of numbers. The weights of a dense layer are stored as a matrix. A layer with 1,024 inputs and 512 outputs has a weight matrix of shape 512 x 1,024. The forward pass through this layer is a matrix-vector multiplication: output = W * input + bias, producing a 512-dimensional output vector from a 1,024-dimensional input vector.

Matrix multiplication is the single most important operation in neural networks. The forward pass through a dense layer, the application of a convolutional filter, and the attention computation in a transformer all reduce to matrix multiplication. GPUs are optimized for matrix multiplication, which is why they are so much faster than CPUs for neural network training.

Dot products measure similarity between two vectors. The attention mechanism computes dot products between query and key vectors to determine how relevant each position is to every other. Word2Vec embeddings encode semantic similarity as dot product proximity. The dot product of two unit vectors is the cosine of the angle between them, ranging from -1 (opposite) to +1 (identical).

Tensors generalize vectors (1D) and matrices (2D) to arbitrary dimensions. A color image is a 3D tensor (height x width x channels). A batch of images is a 4D tensor (batch x height x width x channels). Neural network frameworks (PyTorch, TensorFlow) operate on tensors, and understanding tensor shapes is essential for building and debugging networks.

Calculus: How Networks Learn

Calculus provides the machinery for training. The derivative of the loss function with respect to each weight tells the optimizer how to adjust that weight to reduce the error. Backpropagation is a systematic application of the chain rule to compute these derivatives efficiently.

Derivatives measure the rate of change of a function. The derivative of the loss L with respect to a weight w, written dL/dw, tells you how much the loss would change if you increased the weight by a tiny amount. If dL/dw is positive, increasing the weight increases the loss (bad), so gradient descent decreases the weight. If dL/dw is negative, increasing the weight decreases the loss (good), so gradient descent increases the weight.

Partial derivatives extend derivatives to functions of many variables. A neural network's loss is a function of billions of weights, and we need the partial derivative with respect to each weight individually. The partial derivative dL/dw_i measures the effect of changing weight w_i while holding all other weights constant.

The chain rule is the mathematical foundation of backpropagation. If y = f(g(x)), then dy/dx = df/dg * dg/dx. A neural network is a composition of many functions (layers), and the chain rule lets you compute the derivative of the loss with respect to any weight by multiplying the derivatives along the path from the loss to that weight. This is exactly what backpropagation does, one layer at a time, working backward from the output.

The gradient is the vector of all partial derivatives. For a network with N weights, the gradient is an N-dimensional vector pointing in the direction of steepest loss increase. Gradient descent moves in the opposite direction (steepest decrease) by subtracting a fraction of the gradient from each weight.

This simple update rule, applied to every weight after every batch, is the complete training algorithm. Everything else (Adam, momentum, learning rate scheduling) is refinement of this basic operation.

Probability: Interpreting Outputs and Defining Losses

Probability theory provides the framework for interpreting neural network outputs and defining what "good" predictions mean.

Probability distributions describe the likelihood of different outcomes. A classification network's output is a probability distribution over classes: P(cat) = 0.85, P(dog) = 0.12, P(bird) = 0.03. The softmax function converts raw logits into a valid probability distribution (non-negative values that sum to 1).

Cross-entropy loss measures the difference between the predicted probability distribution and the true distribution (a one-hot vector with 1 at the correct class and 0 elsewhere). The formula is: L = -sum(y_true * log(y_predicted)). For a correctly predicted example where the model assigns probability 0.95 to the true class, the loss is -log(0.95) = 0.051. For a badly predicted example where the model assigns probability 0.05, the loss is -log(0.05) = 3.0. Cross-entropy penalizes confident wrong predictions much more heavily than uncertain ones.

Maximum likelihood estimation is the principle behind most loss functions. Training a neural network is equivalent to finding the parameters that maximize the probability of the training data under the model's predicted distribution. Minimizing cross-entropy loss is mathematically identical to maximizing the likelihood of the correct labels. This connection to statistics gives neural network training a principled theoretical foundation.

Bayes' theorem underlies probabilistic interpretations of neural networks. Bayesian neural networks maintain probability distributions over weights rather than point estimates, using Bayes' theorem to update beliefs about parameter values as more data is observed. While full Bayesian inference is computationally expensive, approximations like Monte Carlo dropout provide practical uncertainty estimates.

Information theory provides the concept of entropy, which measures uncertainty in a probability distribution. A uniform distribution (equal probability for all classes) has maximum entropy, high uncertainty. A peaked distribution (one class with probability near 1) has low entropy, high confidence. The entropy of the model's output distribution at each prediction step is a direct measure of its uncertainty.

Connecting the Math to Network Operations

With these three foundations, every neural network operation has a clear mathematical interpretation:

Forward pass = a sequence of matrix multiplications (linear algebra) interleaved with nonlinear activation functions. For a 3-layer network: output = f3(W3 * f2(W2 * f1(W1 * input + b1) + b2) + b3).

Loss computation = measuring the probability of the correct answer under the model's predicted distribution (probability). Cross-entropy for classification, MSE for regression.

Backward pass = applying the chain rule (calculus) to compute the gradient of the loss with respect to every weight, working backward through the composition of functions.

Weight update = subtracting the gradient scaled by the learning rate (calculus). The optimizer may modify this with momentum, adaptive rates, or weight decay.

Attention = computing dot products between query and key vectors (linear algebra), normalizing with softmax (probability), and computing a weighted sum of value vectors (linear algebra).

Frameworks like PyTorch abstract all of this. You define the forward computation, and automatic differentiation handles the backward pass. You specify the loss function and optimizer, and the framework handles the gradient computation and weight updates. But understanding the underlying math lets you diagnose problems (why are gradients vanishing?), make informed architecture choices (why does this layer need more neurons?), and interpret the model's behavior (what does this output probability mean?).