Diffusion Models
The mathematical elegance behind Stable Diffusion, DALL-E 3, Midjourney, and Sora. Learn how destroying data with noise, then learning to reverse the destruction, yields the most powerful generative models we've ever built.
Learning Objectives
- Understand the forward diffusion process: progressively adding noise
- Understand the reverse process: learning to denoise step by step
- Connect score matching to denoising and the score function ∇log p(x)
- Explain how conditioning (text, images) enables controllable generation
- Understand key innovations: latent diffusion, classifier-free guidance, DDIM
- Compare diffusion models to other generative approaches
- Explore emerging paradigms: Flow Matching and Consistency Models for faster generation
The Core Insight: Destruction is Easy, Creation is Hard
A class of generative models that learn to generate data by learning to reverse a gradual noising process. Given any data point, progressively add Gaussian noise until it becomes pure noise; then train a neural network to reverse each step.
Imagine dropping ink into water. It gradually diffuses until the pattern is completely lost. Now imagine learning to reverse that process—given diffused ink, reconstruct where it started. If you can do that, you can start from any random diffusion state and "reverse" it into a plausible ink pattern.
Diffusion models do exactly this with images: learn to undo noise, one small step at a time, until random static becomes a coherent image.
Forward process (fixed): Gradually add noise over T steps
Reverse process (learned): A neural network predicts the reverse
To generate an image: Start with pure Gaussian noise (x_T), run the learned reverse process for T steps, end up with a realistic image (x_0).
To train: Take a real image, add noise at a random timestep t, ask the model to predict what noise was added (or predict the clean image). This simple training objective produces remarkably high-quality generators.
The Forward Process: Structured Destruction
The forward process is completely fixed—no learning required. We simply define how to gradually corrupt any data point into noise.
The Noise Schedule
The sequence {β₁, β₂, ..., β_T} controls how quickly we add noise. This "noise schedule" is a crucial design choice:
- Linear schedule: β increases linearly from β₁≈0.0001 to β_T≈0.02
- Cosine schedule: Produces a smoother transition, often works better
- Learned schedule: Some approaches learn optimal schedules
The Closed-Form Shortcut
We don't need to iterate through all t steps to get x_t. Define ᾱ_t = ∏ᵢ₌₁ᵗ (1-βᵢ). Then:
This lets us jump directly to any noise level during training—sample a random t, compute x_t in one step, and train.
The Reverse Process: Learning to Denoise
The reverse process is where all the learning happens. Given a noisy image x_t and timestep t, the model must predict either the noise that was added, the clean image x₀, or the "score" ∇log p(x_t).
Three Equivalent Parameterizations
ε-prediction (DDPM)
Predict the noise ε that was added
Most common. Model learns: "what noise made this?"
x₀-prediction
Predict the clean data directly
More intuitive but can be harder to train for large t
v-prediction
Predict v = √ᾱ_t · ε - √(1-ᾱ_t) · x₀
Velocity parameterization, used in some SOTA models
The Connection to Score Matching
There's a deep connection between denoising and score functions. The score is the gradient of log probability: s(x) = ∇_x log p(x).
Remarkably, predicting the noise ε is equivalent to predicting the score:
This score-based view explains why diffusion models work: they learn to follow the gradient of the data distribution, step by step, from noise to data.
The Architecture: U-Net and Beyond
The neural network that predicts ε (or x₀) is typically a U-Net—an architecture originally designed for image segmentation that works remarkably well for denoising.
+ Time embedding
+ Text embedding
Key Components
- Time embedding: The timestep t is encoded (usually with sinusoidal positional encoding, then MLP) and added to the network. This tells the model how noisy the input is.
- Residual blocks: Standard ResNet-style blocks with GroupNorm, SiLU activation, and skip connections.
- Attention layers: Self-attention and (for conditioning) cross-attention. This is where text embeddings are injected in text-to-image models.
- Skip connections: The U-shape passes high-resolution features directly from encoder to decoder, preserving detail.
DiT: Diffusion Transformers
Recent work (including Sora) replaces the U-Net with a pure Transformer architecture. Images are patchified, treated as sequences, and processed with standard transformer blocks. DiT scales better with compute.
Latent Diffusion: The Stable Diffusion Revolution
Instead of running diffusion in pixel space, first compress images into a lower-dimensional latent space using a VAE, then run diffusion there. This is the key innovation behind Stable Diffusion.
Image (512×512×3) → Latent (64×64×4)
64× compression, preserving semantic content
Add noise / denoise in the compact latent space
Much faster and cheaper than pixel-space diffusion
Latent (64×64×4) → Image (512×512×3)
Decode back to full resolution
Why Latent Diffusion Works
Advantages
- ~64× fewer pixels to process
- Train on consumer GPUs (vs clusters for pixel-space)
- Faster inference (fewer FLOPs per step)
- Latent space removes imperceptible details
Trade-offs
- VAE introduces slight blurriness
- Small details may be lost in compression
- Two-stage training is more complex
Conditioning: From Noise to Controlled Generation
The real power of diffusion models emerges when we add conditioning— generating not just any image, but images that match a text prompt, input image, or other constraints.
Text Conditioning
Encode text prompt with CLIP or T5, inject via cross-attention
Stable Diffusion, DALL-E, Imagen
Image Conditioning
img2img: start from noised version of input image
Style transfer, variations, inpainting
ControlNet
Add spatial control: edges, poses, depth maps
Trainable copy of encoder, zero-convolutions
Pose-guided, edge-guided generation
Class Conditioning
Simple: add class embedding to time embedding
ImageNet class-conditional generation
Classifier-Free Guidance (CFG)
The key technique for high-quality conditional generation. Instead of using a separate classifier, train a single model that can do both conditional and unconditional generation. Then at inference:
Higher guidance scale w (e.g., 7-15) → images follow the prompt more closely but may lose diversity. Lower w → more diverse but less prompt-adherent.
Sampling: From 1000 Steps to 20
The original DDPM required 1000 denoising steps—impractically slow. Modern methods achieve similar quality in 20-50 steps through smarter samplers.
DDPM (Original)
Original sampler. Add noise at each step. Very slow but high quality.
DDIM
Skip steps via non-Markovian process. Same trained model, deterministic sampling. Enables interpolation.
DPM-Solver
Treats reverse process as ODE, uses higher-order solvers. Very fast with good quality.
Consistency Models
Learn to map any noise level directly to x₀. Can generate in a single step.
The ODE/SDE Perspective
Diffusion can be viewed as a continuous process. The forward process is a stochastic differential equation (SDE); the reverse is another SDE, or a probability flow ODE. This view enables using numerical ODE solvers for faster, more flexible sampling.
Why Diffusion Models Win
| Property | GANs | VAEs | Diffusion |
|---|---|---|---|
| Sample quality | High | Medium (blurry) | Highest |
| Mode coverage | Mode collapse risk | Good | Excellent |
| Training stability | Unstable | Stable | Very stable |
| Sampling speed | Fast (1 forward) | Fast (1 forward) | Slow (many steps) |
| Likelihood | Not available | ELBO bound | ELBO bound |
| Conditioning | Requires tricks | Possible | Natural & flexible |
The Key Advantages
- No adversarial training: Simple MSE loss, no minimax games, no discriminator to balance. Training just works.
- Mode coverage: The denoising objective naturally covers the full distribution—no mode collapse.
- Flexible conditioning: Cross-attention makes it easy to add any conditioning signal.
- Scaling laws: Quality improves predictably with more compute and data—essential for large-scale training.
Beyond Diffusion: Emerging Paradigms
While diffusion models dominate today, the field is rapidly evolving. Two promising directions address diffusion's key limitation: slow sampling requiring many iterative steps.
Flow Matching
The insight: Instead of learning to reverse a noise process, directly learn a velocity field that transports samples from noise to data along straight paths.
Diffusion (Curved Paths)
Forward: Add Gaussian noise incrementally
Reverse: Learn complex curved trajectories
Steps: 20-1000 typically needed
Flow Matching (Straight Paths)
Forward: Define optimal transport path
Reverse: Learn simple velocity field
Steps: Often 10-50 sufficient
Key advantage: Simulation-free training—no need to simulate the full ODE during training. Used in Stable Diffusion 3, Flux, and OpenAI's latest image models.
Consistency Models
The goal: Generate high-quality samples in just 1-2 steps by learning a function that maps any point on a diffusion trajectory directly to the clean data endpoint.
Noise (t=T) Clean Data (t=0)
x_T ─────────────────────────────────────────────► x_0
↘ ↗
x_t ───────────────────────────────────►
↘ ↗
x_s ─────────────────────────►
f_θ(x_T, T) = f_θ(x_t, t) = f_θ(x_s, s) = x_0
All points on the same trajectory map to the same output!
Training Approaches
- Consistency Distillation: Train from a pre-trained diffusion model
- Consistency Training: Train from scratch using consistency loss
- Latent Consistency Models: Apply to latent space (LCM-LoRA)
📚 Key Literature
- Lipman et al. (2023) — "Flow Matching for Generative Modeling"
- Song et al. (2023) — "Consistency Models"
- Esser et al. (2024) — "Scaling Rectified Flow Transformers for High-Resolution Image Synthesis" (SD3)
- Luo et al. (2023) — "Latent Consistency Models"
🔮 The Frontier
These methods are actively converging: Flow Matching provides cleaner theory, Consistency Models provide speed, and hybrid approaches combine both. Expect 1-4 step high-quality generation to become standard within 1-2 years.
Common Misconceptions
"Diffusion models memorize and copy training images"
While memorization can occur (especially for frequently duplicated data), diffusion models primarily learn statistical patterns. Studies show most generations are novel interpolations.
The accurate framing: Diffusion models learn the data distribution, not individual images. Memorization is a training data issue (duplicates, small datasets), not inherent to diffusion.
"The model removes noise like a photo denoiser"
Photo denoisers try to recover the original signal. Diffusion models "denoise" random noise into images that never existed. They're not recovering anything—they're creating.
The accurate framing: The model learns the gradient of the data distribution, following it from noise toward the manifold of real images. "Denoising" is the mechanism, but generation is the result.
"More diffusion steps always means better quality"
With modern samplers (DDIM, DPM-Solver), 20-30 steps often match 1000-step DDPM quality. Beyond a point, more steps waste compute without improving quality.
The accurate framing: The optimal step count depends on the sampler. With good samplers, diminishing returns hit quickly. The frontier is now about efficient sampling, not more steps.
Interactive Lab: The Diffusion Process
Visualize the forward (noising) and reverse (denoising) diffusion processes. See how an image dissolves into noise and how a trained model could reverse it.
Noise Schedule: α̅_t over time
α̅_t starts at ~1 (clean) and approaches 0 (pure noise)
Key Observations
- Early steps add imperceptible noise; details disappear later
- Cosine schedule preserves structure longer than linear
- At t=T, no information remains—just Gaussian noise N(0,I)
- The reverse process must learn to recover structure from nothing
Check Your Understanding
What does a diffusion model actually learn to predict during training?
What is the key innovation of Latent Diffusion Models (Stable Diffusion)?
What does Classifier-Free Guidance (CFG) allow you to control?
How is the "score function" ∇log p(x) related to diffusion models?
Why do diffusion models avoid the mode collapse problem that plagues GANs?