How AI Generates Images

What Generative Models Learn

A generative image model learns the probability distribution of images in its training dataset. This is a conceptually simple idea with staggering mathematical complexity. Consider the space of all possible 512x512 RGB images: each pixel can take 256 values in each of 3 color channels, so the total number of possible images is 256 raised to the power of 786,432 (512 x 512 x 3). This number is so large that it makes the number of atoms in the observable universe look negligible by comparison. The overwhelming majority of these possible images are meaningless noise. Only an infinitesimally thin surface within this space corresponds to images that look like photographs, paintings, or anything recognizable to a human.

A generative model learns to identify and sample from this thin surface of meaningful images. It does this by studying millions of real images and learning the correlations, structures, and regularities that distinguish real images from random noise. It learns that sky pixels tend to be blue, that eyes appear in pairs on faces, that shadows fall opposite to light sources, that grass has a particular texture, that buildings have straight edges. This knowledge is encoded in the model's parameters, which number in the hundreds of millions to billions, forming a compressed representation of the visual world's statistical structure.

When the model generates a new image, it is not copying or recombining fragments of training images. It is sampling from the learned distribution, producing a novel image that is consistent with the patterns it learned. This is analogous to how a person who has read thousands of novels can write a new sentence that has never been written before: the sentence is original, but it follows the statistical patterns of language that the author has internalized. Generative image models do the same with visual patterns.

Generative Adversarial Networks: The Competition Approach

GANs (Generative Adversarial Networks), introduced by Ian Goodfellow in 2014, use a training framework where two neural networks compete against each other. The generator network takes random noise as input and produces images. The discriminator network examines images and tries to determine whether each one is a real image from the training set or a fake image produced by the generator. The generator is trained to fool the discriminator, and the discriminator is trained to catch the generator's fakes. This adversarial dynamic pushes both networks to improve continuously.

Early GANs produced blurry, small images with obvious artifacts. Progressive GAN (2017) introduced the technique of growing both networks gradually, starting with tiny 4x4 images and progressively adding layers to reach higher resolutions, producing the first GAN images at 1024x1024 resolution that looked convincingly photorealistic. StyleGAN (2018) and its successors StyleGAN2 and StyleGAN3 introduced a style-based generator architecture that separates high-level attributes (pose, face shape) from fine details (hair texture, skin pores), enabling unprecedented control over generated content. StyleGAN2 face images fooled human evaluators at rates approaching 50%, meaning people could not reliably distinguish generated faces from photographs.

Despite producing spectacular results, GANs suffer from fundamental training instabilities. Mode collapse occurs when the generator learns to produce only a narrow range of outputs that fool the discriminator, ignoring the diversity present in the training data. Training divergence happens when the adversarial balance between generator and discriminator breaks down, causing one network to overwhelm the other. These instabilities make GANs notoriously difficult to train reliably, requiring careful hyperparameter tuning and monitoring. The practical difficulties of GAN training are a major reason why the field shifted toward diffusion models starting in 2020.

Diffusion Models: The Denoising Revolution

Diffusion models, the architecture behind Stable Diffusion, DALL-E 2/3, Midjourney, and Imagen, work by learning to reverse a gradual noise corruption process. The forward process takes a real image and progressively adds Gaussian noise to it over a sequence of steps (typically 1000 steps during training), eventually transforming it into pure random noise. The model is trained to predict and remove the noise at each step, learning to denoise images at every level of corruption from barely noisy to completely random.

Image generation works by running the learned denoising process in reverse. Start with a sample of pure random noise and repeatedly apply the denoising network, with each step removing some noise and adding some coherent image structure. After 20 to 50 steps of iterative refinement, the initial noise has been transformed into a clean, detailed image. The quality and diversity of the generated images depends on how well the model learned the denoising process during training. Each denoising step is guided by the model's internal representation of what real images look like, pulling the noisy intermediate result toward the manifold of natural images.

Text-guided generation adds a conditioning mechanism. During training, the model is shown image-caption pairs and learns to denoise images while conditioning on the text embedding of the associated caption. During generation, the user provides a text prompt ("a red panda sitting in a bamboo forest at sunset"), which is encoded by a text encoder (typically CLIP or T5) into a numerical representation. This text embedding guides the denoising process at each step, steering the random noise toward an image that matches the description. Classifier-free guidance, a technique where the model is trained with and without text conditioning and the two predictions are combined during generation, dramatically improves the alignment between the generated image and the text prompt.

Latent diffusion models, the architecture used by Stable Diffusion, add an efficiency optimization. Instead of applying the diffusion process to full-resolution pixel images (which requires enormous computation), they first encode images into a compressed latent space using a variational autoencoder (VAE). The diffusion process operates in this latent space, which is typically 8x smaller in each spatial dimension than the pixel space. After denoising in latent space, the result is decoded back to pixel space by the VAE decoder. This compression reduces the computational cost by roughly a factor of 50 while preserving image quality, making high-resolution image generation feasible on consumer GPUs.

Variational Autoencoders and Flow Models

Variational autoencoders (VAEs), introduced by Kingma and Welling in 2013, were among the first deep generative models for images. A VAE consists of an encoder network that maps images to a compact latent space and a decoder network that reconstructs images from latent vectors. The model is trained to minimize reconstruction error while keeping the latent space smooth and continuous by imposing a Gaussian prior. New images can be generated by sampling random points from the Gaussian prior and decoding them. VAEs tend to produce blurrier images than GANs or diffusion models because they optimize a reconstruction loss that averages over possible outputs, but they provide a mathematically principled framework with useful properties like smooth interpolation between images in latent space.

Flow-based models, including RealNVP, Glow, and more recently Flow Matching, learn an invertible transformation between the image distribution and a simple base distribution (typically Gaussian). Because the transformation is invertible, these models can both generate images (by sampling from the base distribution and applying the forward transformation) and compute exact likelihoods (by applying the inverse transformation and evaluating the base distribution). Flow matching, which has become popular since 2023, simplifies the training objective compared to earlier flow models and produces results competitive with diffusion models while often generating images in fewer steps.

The distinction between these architectures has become less sharp as the field matures. Consistency models, developed by OpenAI, distill diffusion models into single-step generators. Rectified flows combine ideas from diffusion and flow matching. The trend is toward methods that combine the training stability and quality of diffusion models with the speed of single-step generators, approaching the point where high-quality image generation requires only 1 to 4 forward passes through the network.

Image-to-Image Generation and Editing

Beyond creating images from text alone, generative models excel at image transformation tasks. Inpainting fills in missing or selected regions of an image with content that seamlessly matches the surrounding context. A user can select a person in a photograph, delete them, and the model fills the hole with plausible background content including correct perspective, lighting, and texture continuation. Outpainting extends images beyond their original boundaries, generating new content that naturally continues the existing scene.

Style transfer applies the visual style of one image to the content of another. ControlNet, released in 2023, enables precise spatial control over generated images by conditioning the diffusion process on additional inputs like edge maps, depth maps, pose skeletons, or segmentation masks. A user can provide a rough sketch of a room layout and a text prompt describing "a modern minimalist living room," and the model generates a photorealistic image that follows the spatial structure of the sketch while matching the textual description. This level of controllability has made generative models practical for design workflows where creative professionals need specific compositions rather than random outputs.

Image super-resolution uses generative models to upscale low-resolution images, synthesizing plausible high-frequency detail that was not present in the input. A 256x256 image can be upscaled to 1024x1024 with convincing fine detail, though the added detail is hallucinated by the model rather than recovered from the original data. This distinction matters for applications like medical imaging and forensics where the accuracy of fine details is critical, but is acceptable for creative and presentation purposes.

Quality, Limitations, and Open Problems

Modern generative image models produce outputs of remarkable quality, but they have consistent failure modes that reveal the limits of current technology. Hands remain a challenge, with models frequently generating the wrong number of fingers, anatomically impossible joint angles, or blurred hand regions. Text rendering within generated images is improving but still unreliable, with misspellings and garbled characters common in models that were not specifically trained for text accuracy. Complex spatial relationships described in prompts ("a red ball on top of a blue cube to the left of a green cylinder") are often interpreted incorrectly, with objects placed in wrong spatial configurations.

Training data raises significant ethical and legal questions. Most image generation models were trained on datasets scraped from the internet that include copyrighted images, and the legal status of using copyrighted material for model training varies by jurisdiction and is actively being litigated. Some artists argue that models trained on their work without permission constitute a form of copyright infringement, while model developers argue that training constitutes fair use analogous to a human artist studying existing works. The resolution of these legal questions will shape the future of generative AI.

Deepfake detection has become an active research area in response to the photorealism of generated images. Detectors analyze statistical artifacts, inconsistent lighting, and metadata signatures to distinguish generated images from photographs. However, the arms race between generators and detectors favors the generators: each new model generation eliminates the artifacts that previous detectors relied upon, requiring continuous development of new detection methods. Watermarking approaches, where generators embed invisible signatures in their outputs, offer a more robust path to provenance tracking.