Generative AI→Module 4.1
Generative AI ~45 min

Generative vs Discriminative Models

The fundamental divide in machine learning: models that learn to classify versus models that learn to create. Understanding this distinction is key to grasping modern AI systems from GANs to GPT.

Learning Objectives

  • Distinguish generative from discriminative modeling at a mathematical level
  • Understand when to use each approach and their relative strengths
  • Explain the major generative model families: VAEs, GANs, autoregressive, flow-based, diffusion
  • Connect generative modeling to modern applications: image synthesis, language models, drug discovery
  • Understand the likelihood-based vs likelihood-free training distinction

Two Fundamentally Different Questions

The Core Distinction

Discriminative models learn P(y|x)—the probability of a label given input. Generative models learn P(x) or P(x,y)—the probability distribution over the data itself, enabling them to create new samples.

Intuition

Discriminative: Given a photo, is this a cat or a dog? The model learns to draw boundaries between classes without understanding what cats or dogs actually look like.

Generative: What does a cat look like? The model learns the full distribution of cat images—their shapes, textures, poses—well enough to generate new, realistic cats that never existed.

Mathematical Framing

Both approaches relate through Bayes' theorem:

P(y|x) = P(x|y)P(y) / P(x)

Discriminative: Model P(y|x) directly. Simpler, often more accurate for classification when you have labeled data.

Generative: Model P(x|y) and P(y), or P(x) directly. More powerful—can generate samples, detect anomalies, handle missing data—but harder to train well.

In Practice

Discriminative examples: Logistic regression, SVMs, most neural network classifiers, BERT for classification

Generative examples: GPT (generates text), Stable Diffusion (generates images), VAEs, GANs, language models

Discriminative Models: Learning Boundaries

Discriminative models focus entirely on the decision: given input x, what's the most likely output y? They don't need to understand the data's full structure—just enough to classify it.

What Discriminative Models Learn

P(y|x) = f(x; θ)
Map input directly to output probability

Training objective: maximize P(y|x) for training examples. The model learns whatever features help distinguish classes, ignoring everything else.

Strengths

  • Higher accuracy: Directly optimizes for the classification task
  • Sample efficient: Doesn't need to model irrelevant features
  • Simpler: Modeling P(y|x) is easier than modeling P(x)
  • Well-understood: Decades of theory and practice

Limitations

  • Can't generate: No way to sample new data points
  • Needs labels: Requires supervised data
  • Poor with missing data: Can't infer missing features
  • Overconfident: May give high confidence on OOD inputs

Discriminative Model Examples

Logistic Regression Linear decision boundary, P(y|x) via sigmoid
Random Forests Ensemble of decision trees for classification
Neural Classifiers Deep networks ending in softmax
BERT (fine-tuned) Transformer encoder for classification

Generative Models: Learning the Data Distribution

Generative models learn the underlying distribution of the data itself. This is fundamentally harder—but enables sampling, density estimation, and understanding data structure.

What Generative Models Learn

P(x) = ∫ P(x|z)P(z) dz
Model the full data distribution, often with latent variables z

Training objective: maximize P(x) for training examples (likelihood-based), or match generated samples to real data distribution (likelihood-free).

◈

The Generative Model Promise

If you truly understand a data distribution, you can: (1) generate new samples, (2) compute how likely any sample is, (3) fill in missing parts, (4) detect anomalies, (5) learn useful representations without labels.

Strengths

  • Generation: Create new, realistic samples
  • Unsupervised: Learn from unlabeled data
  • Density estimation: Know how likely any sample is
  • Missing data: Can impute missing features
  • Representation learning: Learn meaningful latent spaces

Challenges

  • Harder to train: Modeling P(x) is complex
  • Evaluation is hard: How do you measure generation quality?
  • Mode collapse: May miss parts of the distribution
  • Computationally expensive: Often slow to train/sample

The Generative Model Zoo

There are many approaches to generative modeling, each with different trade-offs. Understanding these families is crucial for choosing the right tool.

Autoregressive Models

GPT, LLaMA, PixelCNN, WaveNet

Key idea: Factor P(x) as a product of conditionals. Generate one element at a time, each conditioned on previous elements.

P(x) = P(x₁)P(x₂|x₁)P(x₃|x₁,x₂)...
✓ Exact likelihood ✓ High quality ✗ Sequential (slow) ✗ Unidirectional

This is how GPT works: predict next token given all previous tokens. Extremely successful for language but slow for images.

Variational Autoencoders (VAEs)

VAE, β-VAE, VQ-VAE

Key idea: Learn a latent space z that captures data structure. Encoder maps x→z, decoder maps z→x.

max E[log P(x|z)] - KL(Q(z|x) || P(z))
✓ Learned latent space ✓ Stable training ✓ Fast sampling ✗ Blurry outputs

Great for learning representations. The latent space often has meaningful structure (interpolation, arithmetic). Limited by blurriness.

Generative Adversarial Networks (GANs)

StyleGAN, BigGAN, CycleGAN

Key idea: Two networks compete—generator creates fakes, discriminator tries to detect them. Both improve through competition.

min_G max_D E[log D(x)] + E[log(1-D(G(z)))]
✓ Sharp, realistic outputs ✓ Fast sampling ✗ Training instability ✗ Mode collapse ✗ No likelihood

Dominated image generation 2014-2021. Produces sharp images but training is notoriously difficult. Largely superseded by diffusion models.

Normalizing Flows

RealNVP, Glow, Neural Spline Flows

Key idea: Learn an invertible transformation from a simple distribution (Gaussian) to the data distribution.

P(x) = P(f⁻¹(x)) |det(∂f⁻¹/∂x)|
✓ Exact likelihood ✓ Exact inference ✓ Invertible ✗ Architectural constraints ✗ Lower quality

Theoretically elegant: exact likelihood and exact latent inference. But invertibility constraints limit expressiveness.

Diffusion Models

DDPM, Stable Diffusion, DALL-E 2/3, Sora

Key idea: Gradually add noise until data becomes pure noise, then learn to reverse the process. Generate by denoising random noise.

Learn to predict ε from x_t = √α_t x + √(1-α_t) ε
✓ State-of-the-art quality ✓ Stable training ✓ Mode coverage ✗ Slow sampling

The current dominant paradigm for image/video generation. Combines VAE stability with GAN quality. Covered in depth in Module 4.2.

Likelihood-Based vs Likelihood-Free Training

Likelihood-Based

Directly maximize P(data)

Objective: Maximize log P(x) for training data. Requires being able to compute (or bound) the likelihood.

Examples: Autoregressive models, VAEs, normalizing flows, diffusion models (via ELBO)

Advantage: Principled, comparable across models (bits/dim), covers all modes

Limitation: High likelihood ≠ high quality samples. Can assign high probability to blurry averages.

Likelihood-Free

Match distributions without explicit likelihood

Objective: Make generated samples indistinguishable from real samples, without computing P(x).

Examples: GANs (adversarial loss), energy-based models (contrastive divergence)

Advantage: Can produce sharper samples, more flexible architectures

Limitation: Training instability, mode collapse, hard to evaluate, no density estimation

When to Use Each Approach

Task Best Approach Reasoning
Image classification Discriminative Only need P(label|image); discriminative is more accurate
Spam detection Discriminative Binary classification with labeled data
Image generation Generative (Diffusion) Need to sample from P(image); discriminative can't generate
Text generation Generative (Autoregressive) Need to produce coherent sequences; LLMs are autoregressive
Anomaly detection Generative Need P(x) to identify unlikely samples
Semi-supervised learning Hybrid Generative can leverage unlabeled data
Drug molecule design Generative (VAE/Diffusion) Need to generate novel molecules with desired properties
Missing data imputation Generative Need to model joint distribution to infer missing features

Common Misconceptions

✗

"Generative models are always better because they learn more"

Learning P(x) is harder and requires more data. For classification, a discriminative model that ignores irrelevant features often outperforms a generative one.

The accurate framing: "Better" depends on the task. For classification with labeled data, discriminative usually wins. For generation, anomaly detection, or semi-supervised learning, generative is necessary.

✗

"GANs are the best generative models"

GANs dominated 2014-2021, but diffusion models now achieve better quality, more stable training, and better mode coverage. The field evolves rapidly.

The accurate framing: Different generative families have different strengths. Diffusion models currently lead for images; autoregressive models lead for text. The "best" depends on the application.

✗

"High likelihood means high-quality samples"

A model can assign high likelihood to blurry, averaged images. Likelihood measures coverage of the distribution, not perceptual quality.

The accurate framing: Likelihood and sample quality are different objectives that can conflict. Evaluation requires multiple metrics: likelihood, FID, human evaluation, downstream task performance.

⚗ Interactive Lab: Generative vs Discriminative

Explore the difference between discriminative and generative approaches visually. See how each model type learns from the same data.

Discriminative (P(y|x))

Learns the boundary between classes

Generative (P(x|y) for each class)

Learns the distribution of each class

Watch how different generative model types produce samples. Each has a distinct generation process.

Autoregressive

Generates pixel by pixel, left-to-right, top-to-bottom

VAE

Samples latent z, decodes to image (often blurry)

Diffusion

Starts with noise, gradually denoises

Key Observations

  • Discriminative models only learn the boundary—they don't "understand" the data distribution.
  • Generative models learn the full shape of each class, enabling sampling.
  • Autoregressive is sequential; VAE is parallel but blurry; diffusion iteratively refines.

✓ Check Your Understanding

1

What is the key mathematical difference between generative and discriminative models?

2

Which generative model family does GPT belong to?

3

Why might a discriminative model outperform a generative model for classification?

4

What is a key advantage of diffusion models over GANs?

5

Which task requires a generative model rather than a discriminative one?

0 / 5

Previous ← Large Language Models Next Module Diffusion Models →