Generative AI Models Explained: How Machines Create New Content

What Makes a Model Generative

A discriminative model learns the boundary between categories. Given an image, it outputs "cat" or "dog." It learns P(label|data), the probability of a label given the data. A generative model learns the data distribution itself: P(data), the probability of any particular piece of data occurring. Once you have a model of P(data), you can sample from it to create new data points that are statistically similar to the training data but are not copies of any specific training example.

The distinction matters practically. A discriminative model can tell you whether an email is spam but cannot write you a new email. A generative model can do both: it models the distribution of all possible emails, which lets it generate new ones and also assess how likely any given email is (spam emails and legitimate emails occupy different regions of the distribution). Generative models are strictly more powerful in what they can represent, but they are also harder to train because modeling the full data distribution is a much more complex task than learning a decision boundary.

The quality of generated content depends on how faithfully the model captures the true data distribution. If the model misses important modes of the distribution, it will never generate certain types of content. If it assigns too much probability density to a narrow region, it will produce repetitive outputs. Evaluating generative model quality is itself a difficult problem, because you need to assess both the quality of individual samples (do they look realistic?) and the diversity of the distribution (does the model produce the full range of possible outputs?).

Autoregressive Models

Autoregressive models generate data one element at a time, with each element conditioned on all previously generated elements. For text, this means generating one token at a time: given "The cat sat on," the model predicts the probability distribution over possible next tokens and samples one (perhaps "the"), then conditions on "The cat sat on the" to generate the next token, and so on. Every large language model, from GPT-4 to Claude to LLaMA, is autoregressive.

The training process is elegant in its simplicity. Given a corpus of text, the model learns to predict the next token at every position. The training data provides the correct answer at every step: the actual next token in the text. The loss function is cross-entropy between the predicted probability distribution and the actual next token. No special training procedure or adversarial setup is needed. The same approach works for images (predicting the next pixel or patch), audio (predicting the next waveform sample), and any data that can be serialized into a sequence.

The quality of autoregressive generation depends heavily on the sampling strategy. Greedy decoding, which always picks the most likely next token, produces bland, repetitive text. Temperature sampling scales the logits before softmax, with higher temperature producing more random outputs. Top-k sampling restricts the choice to the k most likely tokens, and top-p (nucleus) sampling restricts to the smallest set of tokens whose cumulative probability exceeds a threshold p. Finding the right balance between coherence and creativity is one of the key practical challenges in deploying autoregressive generators.

Generative Adversarial Networks (GANs)

GANs, introduced by Ian Goodfellow in 2014, use a two-network setup: a generator that creates fake data and a discriminator that tries to distinguish fake from real. The generator starts by producing random noise and gets feedback from the discriminator about how convincing its outputs are. The two networks train simultaneously in a competitive game. The generator improves at producing realistic outputs, and the discriminator improves at detecting fakes, until (ideally) the generator's outputs are indistinguishable from real data.

The game-theoretic framework produces remarkably sharp, detailed outputs. StyleGAN, the most famous GAN architecture for images, generates faces so realistic that humans cannot reliably identify them as synthetic. The progressive growing technique starts by generating low-resolution images and gradually adds detail, producing 1024x1024 faces with consistent identity, realistic hair, skin texture, and lighting. BigGAN extended this approach to diverse categories beyond faces, generating convincing images of dogs, landscapes, food, and hundreds of other categories.

Training GANs is notoriously difficult. The generator and discriminator must stay in balance: if the discriminator becomes too strong, the generator receives no useful gradient signal; if the generator becomes too strong, the discriminator provides no meaningful feedback. Mode collapse, where the generator learns to produce only a few types of outputs and ignores the rest of the data distribution, is a persistent problem. Training instability, where the loss oscillates without converging, requires careful hyperparameter tuning and architectural choices. These difficulties are the main reason diffusion models have largely replaced GANs for image generation.

Variational Autoencoders (VAEs)

Variational autoencoders combine an encoder that compresses data into a low-dimensional latent space with a decoder that reconstructs data from latent vectors. The key innovation over standard autoencoders is that the encoder outputs a probability distribution (typically a Gaussian with a learned mean and variance) rather than a fixed point. During training, a sample is drawn from this distribution and passed to the decoder. A regularization term in the loss function (the KL divergence from the prior distribution) ensures that the latent space is smooth and continuous.

The smooth latent space is what makes VAEs generative. Because the encoder maps similar data to nearby regions and the regularization prevents gaps, you can sample random points from the latent space and decode them to produce new, plausible data. Interpolating between two points in latent space produces a smooth transition between two data points. For faces, this might mean a gradual transition from one person's face to another, passing through intermediate faces that look realistic at every step.

VAEs tend to produce slightly blurry outputs compared to GANs because the reconstruction loss (typically pixel-wise mean squared error) averages over multiple plausible outputs rather than committing to a sharp, specific one. The VQ-VAE (Vector Quantized VAE) addresses this by using a discrete rather than continuous latent space, which produces sharper reconstructions. The latent diffusion approach used in Stable Diffusion combines a VQ-VAE encoder/decoder with a diffusion model operating in the latent space, getting the computational efficiency of a compressed representation with the generation quality of diffusion.

Diffusion Models

Diffusion models generate data by learning to reverse a gradual noise-addition process. The forward process takes a real image and adds small amounts of Gaussian noise over many steps (typically 1000) until the image becomes pure noise. The model learns the reverse: given a noisy image at step t, predict the slightly less noisy version at step t-1. At generation time, the model starts from pure random noise and iteratively denoises it, producing a clean image after all steps are reversed.

The mathematical framework is grounded in score-based generative modeling. At each noise level, the model learns the gradient of the log probability density (the "score"), which points in the direction of higher data probability. Following these gradients from noise toward data produces samples from the data distribution. The denoising score matching objective trains the model by asking it to predict the noise that was added at each step, which is equivalent to learning the score function.

Diffusion models have largely replaced GANs for image generation because they are much easier to train, do not suffer from mode collapse, and produce higher quality and more diverse outputs. DALL-E 2, Stable Diffusion, Midjourney, and Imagen all use diffusion. Text-conditional generation works by incorporating a text encoder (typically CLIP or T5) that guides the denoising process toward images matching the text description. Classifier-free guidance strengthens the influence of the text condition, producing images that more closely match the prompt at the cost of reduced diversity.

The main disadvantage of diffusion models is speed. Generating a single image requires running the denoising network hundreds of times, each time producing a slightly cleaner image. At the standard 1000 steps, this takes orders of magnitude longer than a single forward pass through a GAN. Distillation techniques, consistency models, and fewer-step schedulers have reduced this to 4 to 50 steps with minimal quality loss, making real-time generation feasible for some applications.

Comparing the Approaches

Each generative approach has distinct strengths. Autoregressive models produce the highest quality text and handle arbitrary-length sequences naturally, but they are slow because they generate one element at a time and cannot revise earlier outputs. GANs produce the sharpest images with the fastest inference (a single forward pass), but they are hard to train and prone to mode collapse. VAEs have smooth, well-structured latent spaces ideal for controlled generation and interpolation, but their outputs tend to be blurrier. Diffusion models produce the highest quality images with the best mode coverage, but they require many iterative steps to generate.

In practice, the boundaries between these approaches are blurring. Stable Diffusion operates a diffusion process in the latent space of a VAE. Some recent models combine autoregressive and diffusion approaches, generating an image coarse-to-fine with an autoregressive model providing the structure and a diffusion model adding the details. Consistency models attempt to match diffusion quality with single-step generation. The field is moving toward hybrid architectures that combine the best properties of multiple paradigms.