Autoencoders Explained

The Encoder-Decoder Structure

An autoencoder has two halves. The encoder maps the high-dimensional input to a low-dimensional latent representation (also called the bottleneck or code). The decoder maps the latent representation back to the original input dimensions. The network is trained to minimize the reconstruction error, the difference between the input and the reconstructed output.

For a 784-dimensional input (a 28x28 image), the encoder might compress through layers of 512, 256, and 128 neurons down to a latent space of 32 dimensions. The decoder mirrors this structure, expanding from 32 through 128, 256, 512 back to 784. The entire network is trained end-to-end with mean squared error between input and output pixels as the loss function.

The bottleneck is the critical design choice. If it is too large (close to the input dimension), the autoencoder can simply copy the input through without learning meaningful compression. If it is too small, the autoencoder cannot preserve enough information for accurate reconstruction. The optimal bottleneck size depends on the intrinsic dimensionality of the data, how many independent factors of variation the data actually has.

What Autoencoders Learn

The latent representation is a learned summary of the input. For images of faces, the latent dimensions might correspond to high-level attributes like face orientation, lighting direction, expression, skin tone, and hair style. The autoencoder discovers these factors without being told they exist, purely from the objective of reconstructing faces accurately through a bottleneck.

This learned representation is often more useful than the raw data for downstream tasks. Classification, clustering, and retrieval all benefit from working with a compact, meaningful representation rather than raw pixels. The autoencoder's representation removes noise, normalizes variation, and captures the essential structure.

Autoencoders are related to principal component analysis (PCA), a classical dimensionality reduction method. In fact, a linear autoencoder (no activation functions) with mean squared error loss learns the same subspace as PCA. Nonlinear autoencoders with deep architectures learn more powerful representations because they can capture nonlinear relationships that PCA misses.

Denoising Autoencoders

A denoising autoencoder is trained to reconstruct clean inputs from corrupted versions. The input is deliberately noised (pixels randomly zeroed, Gaussian noise added, or random patches masked) and the network learns to recover the original clean data. This forces the autoencoder to learn robust features rather than merely copying the input.

Denoising autoencoders produce better representations than standard autoencoders because they must understand the underlying structure of the data well enough to infer the missing or corrupted parts. They are also directly useful for practical denoising: remove noise from photographs, clean up audio recordings, or fill in missing values in tabular data.

The masked autoencoder approach used in BERT's pre-training (predicting masked tokens from context) and MAE's pre-training for vision (predicting masked image patches) are conceptually denoising autoencoders. The "noise" is the masking, and the model learns rich representations by learning to reconstruct the masked portions.

Variational Autoencoders (VAEs)

Standard autoencoders learn a mapping from input to a specific point in the latent space. This means the latent space may have gaps, regions where no training example maps, and sampling random points from the latent space will not produce realistic outputs. Variational autoencoders solve this by learning a probability distribution in the latent space rather than a point.

In a VAE, the encoder outputs two vectors for each input: a mean and a variance, defining a Gaussian distribution in the latent space. During training, the latent code is sampled from this distribution (using the reparameterization trick, which allows gradients to flow through the sampling operation). The decoder reconstructs the input from this sampled code.

The VAE loss function has two terms. The reconstruction loss (same as a standard autoencoder) encourages accurate reconstruction. The KL divergence loss encourages the learned distributions to stay close to a standard normal distribution. This regularization prevents the encoder from producing extremely narrow distributions (which would reduce to a standard autoencoder) and ensures the latent space is smooth and continuous.

The smooth latent space is what makes VAEs generative. You can sample random points from the standard normal distribution, pass them through the decoder, and get realistic outputs. You can interpolate between two encoded inputs and get smooth transitions. You can manipulate specific latent dimensions to change specific attributes of the generated data. These properties make VAEs useful for controlled generation, data augmentation, and exploring the latent structure of datasets.

VAEs generate blurrier images than GANs because the reconstruction loss (typically MSE) penalizes pixel-level errors uniformly, which encourages the model to produce the average of all plausible outputs rather than committing to a specific sharp output. Perceptual losses and adversarial training components can mitigate this, and the latent diffusion models behind Stable Diffusion use a VAE's latent space as the compressed representation in which diffusion operates.

Sparse and Contractive Autoencoders

Sparse autoencoders add a sparsity penalty to the latent representation, encouraging most latent dimensions to be near zero for any given input. This forces each input to be represented by a small number of active latent features, producing a code similar to a sparse coding or dictionary learning. Sparse representations are more interpretable (each active dimension corresponds to a specific feature) and often transfer better to downstream tasks.

Contractive autoencoders add a penalty on the derivative of the encoder's output with respect to its input. This makes the learned representation insensitive to small input perturbations, encoding only the significant features and ignoring noise. Contractive autoencoders learn smoother, more robust representations than standard autoencoders.

Applications

Anomaly detection. Train an autoencoder on normal data only. When presented with anomalous data, the reconstruction error will be high because the autoencoder has not learned to represent anomalies. This is used for fraud detection, manufacturing defect identification, and network intrusion detection.

Dimensionality reduction. Compress high-dimensional data to a manageable number of dimensions for visualization, clustering, or as preprocessing for other models. Autoencoders handle nonlinear relationships that PCA and other linear methods cannot capture.

Image compression. Learned image compression using autoencoders achieves better quality at the same file size compared to traditional codecs (JPEG, WebP) for certain image types. The encoder compresses, the latent code is quantized and stored, and the decoder reconstructs.

Drug discovery. Autoencoders compress molecular fingerprints or molecular graphs into latent representations that capture chemical similarity. Exploring the latent space generates novel molecular candidates with desired properties.