Model Evaluation

Why Evaluation Is Harder Than It Looks

Model Evaluation

The systematic process of measuring how well a model performs its intended task, using metrics and methodologies that honestly reflect real-world performance and generalization capability.

The Problem

A model that achieves 99% accuracy sounds impressive—until you learn the dataset was 99% one class. A language model with low perplexity might still generate nonsense. An image classifier that works perfectly on ImageNet might fail catastrophically on real-world photos.

Evaluation isn't just measurement—it's the art of asking the right questions about your model's behavior.

Core Challenges

Distribution shift: Test data may not reflect deployment
Metric mismatch: What you optimize ≠ what you care about
Data leakage: Test data contaminating training
Class imbalance: Rare events distort simple metrics
Goodhart's Law: When a metric becomes a target, it ceases to be a good metric

Real Consequences

Poor evaluation has led to: medical AI that works on hospital A but fails at hospital B (distribution shift), hiring algorithms that discriminate (proxy metrics), and self-driving cars that can't handle edge cases (insufficient coverage).

Classification Metrics: Beyond Accuracy

😰 Feeling Overwhelmed by Metrics? Start Here

You don't need to memorize every metric! Here's the beginner's shortcut:

Balanced dataset? Use Accuracy
Imbalanced dataset? Use F1 Score
False positives are costly? Focus on Precision
Missing positives is costly? Focus on Recall
Need to compare models? Use ROC-AUC

Read the rest when you need it. This section is a reference—not a checklist to memorize.

Accuracy—the percentage of correct predictions—is often the worst metric to use. Here's why, and what to use instead.

The Confusion Matrix: Foundation of Classification Metrics

Predicted Negative

Predicted Positive

Actual Negative

TN
True Negative

FP
False Positive
(Type I Error)

Actual Positive

FN
False Negative
(Type II Error)

TP
True Positive

Accuracy

(TP + TN) / (TP + TN + FP + FN)

Percentage correct overall.

✓ Use when: Classes are balanced, all errors equally costly

✗ Avoid when: Class imbalance (99% one class = 99% accuracy by predicting majority)

Precision

TP / (TP + FP)

Of all positive predictions, how many were correct?

✓ Use when: False positives are costly (spam filter, fraud detection)

"When I say yes, am I right?"

Recall (Sensitivity)

TP / (TP + FN)

Of all actual positives, how many did we find?

✓ Use when: Missing positives is costly (disease screening, security threats)

"Did I find all the real positives?"

F1 Score

2 × (Precision × Recall) / (Precision + Recall)

Harmonic mean of precision and recall.

✓ Use when: You need a single number balancing both

Punishes extreme imbalance between P and R

Specificity

TN / (TN + FP)

Of all actual negatives, how many did we correctly identify?

✓ Use when: False alarms are costly (medical tests, security alerts)

"How well do I identify true negatives?"

Matthews Correlation Coefficient

(TP×TN - FP×FN) / √[(TP+FP)(TP+FN)(TN+FP)(TN+FN)]

Balanced measure even with imbalanced classes.

✓ Use when: Severe class imbalance; ranges -1 to +1

Gold standard for imbalanced binary classification

◈

The Precision-Recall Trade-off

By adjusting your classification threshold, you can trade precision for recall and vice versa. A lower threshold catches more positives (higher recall) but also more false positives (lower precision). There's no free lunch—the right balance depends on the cost of each error type.

ROC Curves and AUC: Threshold-Independent Evaluation

ROC Curve

A plot of True Positive Rate (Recall) vs False Positive Rate (1-Specificity) at every possible classification threshold. Shows how well the model separates classes regardless of threshold choice.

Perfect Model

AUC = 1.0. Curve goes straight up then right. Perfect separation.

Good Model

AUC = 0.8-0.9. Curve bows toward upper-left. Strong discrimination.

Random Guessing

AUC = 0.5. Diagonal line. Model has no predictive power.

Worse Than Random

AUC < 0.5. Below diagonal. Flip predictions to improve!

When to Use AUC vs Precision-Recall AUC

ROC-AUC

Balanced classes
Care about both FP and FN equally
Comparing models across thresholds

PR-AUC

Severe class imbalance
Positive class is rare and important
ROC-AUC can be misleadingly high

Regression Metrics

When predicting continuous values, we need different metrics that capture how far off our predictions are.

Mean Squared Error (MSE)

MSE = (1/n) Σ(yᵢ - ŷᵢ)²

Average squared difference between predictions and actuals.

✓ Properties: Penalizes large errors heavily; differentiable

✗ Limitation: Sensitive to outliers; units are squared

Root Mean Squared Error (RMSE)

RMSE = √MSE

Square root of MSE—same units as the target variable.

✓ Properties: Interpretable scale; still penalizes large errors

Mean Absolute Error (MAE)

MAE = (1/n) Σ|yᵢ - ŷᵢ|

Average absolute difference.

✓ Properties: Robust to outliers; intuitive

✗ Limitation: Not differentiable at zero

R² (Coefficient of Determination)

R² = 1 - (SS_res / SS_tot)

Proportion of variance explained by the model.

✓ Properties: Scale-independent; 1.0 = perfect fit

✗ Limitation: Can be negative; doesn't measure calibration

Mean Absolute Percentage Error (MAPE)

MAPE = (100/n) Σ|yᵢ - ŷᵢ| / |yᵢ|

Percentage error—scale independent.

✓ Properties: Easy to interpret as percentage

✗ Limitation: Undefined when y=0; asymmetric

Quantile Loss

L_q = q(y - ŷ)⁺ + (1-q)(ŷ - y)⁺

For probabilistic forecasts and prediction intervals.

✓ Properties: Captures uncertainty; asymmetric loss

Evaluating Generative Models

Generative models present unique evaluation challenges: what does it mean for generated content to be "good"? There's no single ground truth to compare against.

Language Model Metrics

Perplexity

PPL = exp(-1/N Σ log P(wᵢ|w₁...wᵢ₋₁))

How "surprised" the model is by the test data. Lower = better. Perplexity of k means the model is as uncertain as choosing uniformly among k options per token.

Caveats: Only measures likelihood, not quality. A model can have low perplexity but generate repetitive or nonsensical text.

BLEU Score

N-gram overlap between generated and reference text. Used for translation, summarization. Ranges 0-100.

Caveats: Penalizes valid paraphrases; correlates poorly with human judgment for open-ended generation.

ROUGE

Recall-oriented n-gram overlap. ROUGE-L uses longest common subsequence. Better for summarization.

BERTScore

Semantic similarity using BERT embeddings. Captures meaning beyond exact word matches.

Image Generation Metrics

Fréchet Inception Distance (FID)

Distance between feature distributions of real and generated images using InceptionNet. Lower = better.

FID = ||μ_r - μ_g||² + Tr(Σ_r + Σ_g - 2√(Σ_rΣ_g))

Caveats: Requires many samples (~50k); sensitive to image preprocessing; InceptionNet biases toward ImageNet-like images.

Inception Score (IS)

Measures quality (confident classifications) and diversity (varied classifications across samples).

Caveats: Doesn't compare to real data; mode dropping can still achieve high IS.

CLIP Score

Measures alignment between generated image and text prompt using CLIP embeddings.

Caveats: Only measures text alignment, not image quality or realism per se.

Human Evaluation

Often the gold standard, but expensive, slow, and subjective. Common approaches:

A/B testing: Which output is better?
Likert scales: Rate quality 1-5
Turing tests: Is this human or AI?
Task success: Can humans use the output?
Red teaming: Adversarial probing for failures

◈

LLM Evaluation Is an Open Problem

Evaluating large language models remains deeply challenging. Benchmarks like MMLU, HumanEval, and BigBench capture narrow capabilities. Models can be excellent at benchmarks but fail in deployment. The field is moving toward: capability-specific evals, adversarial testing, and LLM-as-judge (using one LLM to evaluate another).

Data Splits and Cross-Validation

The Golden Rule

Never evaluate on data the model has seen during training or hyperparameter tuning. The test set must be a true hold-out that simulates deployment.

Train / Validation / Test Split

Train (70%)

Val (15%)

Test (15%)

Train: Fit model parameters
Validation: Tune hyperparameters, early stopping
Test: Final evaluation only—never peek!

K-Fold Cross-Validation

TestTrainTrainTrainTrain

TrainTestTrainTrainTrain

TrainTrainTestTrainTrain

TrainTrainTrainTestTrain

TrainTrainTrainTrainTest

Every data point gets to be in the test set once. Average performance across folds. More reliable estimate, especially for small datasets.

Stratified Splits

Maintain class proportions in each split. Critical for imbalanced data to ensure rare classes appear in all folds.

Time-Series Splits

Never use future data to predict the past. Train on past, test on future. Rolling window or expanding window approaches.

Group Splits

When data points are correlated (e.g., multiple samples from same patient), ensure groups don't span train/test. GroupKFold prevents leakage.

Common Evaluation Pitfalls

⚠

Data Leakage

Information from test set "leaks" into training. Examples: scaling before splitting, feature engineering on full data, target leakage (features that encode the label).

Fix: Split first, then preprocess. Use pipelines.

⚠

Test Set Contamination

For LLMs, test benchmarks may appear in training data (internet scale). GPT-4 might have "seen" your eval questions.

Fix: Use held-out dates, contamination detection, novel test sets.

⚠

Overfitting to Validation Set

Running hundreds of experiments, selecting the best on validation, can overfit to validation randomness.

Fix: Final test set is sacred. Report confidence intervals. Use fresh test sets for final claims.

⚠

Cherry-Picking Results

Only reporting the metric that looks good. Running until you get a good random seed. Selective ablations.

Fix: Pre-register metrics. Report all relevant metrics. Multiple seeds with error bars.

⚠

Distribution Mismatch

Test data doesn't match deployment. Clean lab data vs messy real world. Geographic or demographic differences.

Fix: Evaluate on production-like data. Domain adaptation. Monitor post-deployment.

⚠

Wrong Metric for the Task

Optimizing accuracy when false negatives are catastrophic. Using BLEU for creative writing. Perplexity for helpfulness.

Fix: Start with the business/safety goal. Work backwards to metrics that align.

Statistical Significance and Confidence

"Model A got 85.2% and Model B got 84.9%"—is A actually better, or is this just noise? Statistical rigor separates real improvements from random fluctuation.

Confidence Intervals

Report 95% CI: "Accuracy = 85.2% ± 1.3%". If CIs overlap substantially, the difference may not be meaningful.

Methods: Bootstrap (resample with replacement), Wilson score interval (for proportions)

Paired Tests

When comparing models on the same test set, use paired tests (McNemar's test, paired t-test, Wilcoxon signed-rank) for more statistical power.

Multiple Comparisons

Testing many hypotheses inflates false positives. If comparing 10 models, apply Bonferroni correction or control false discovery rate.

Effect Size

Statistical significance ≠ practical significance. A 0.01% improvement can be "significant" with enough data but useless in practice.

Reporting Best Practices

Report mean and standard deviation across multiple runs/seeds
Include confidence intervals or significance tests
Report multiple relevant metrics, not just the best one
Describe the test set clearly (size, source, potential biases)
Be transparent about hyperparameter search scope
Compare to strong, well-tuned baselines

Calibration: When Confidence Lies

Critical Safety Issue

Calibration measures whether predicted probabilities reflect actual frequencies. A model is well-calibrated if, among all predictions with 80% confidence, approximately 80% are actually correct.

⚠

Neural Networks Are Typically Overconfident

This isn't a minor problem—it's a fundamental failure mode. Modern deep networks can be 99% confident while wrong 50% of the time. Cross-entropy loss training drives softmax outputs toward extreme probabilities (near 0 or 1), even when uncertainty should be high.

Understanding Calibration Plots

A calibration plot (reliability diagram) plots predicted confidence against actual accuracy. Perfect calibration = diagonal line.

Well-Calibrated

Points follow the diagonal. 70% confidence → 70% accurate.

Overconfident (Common)

Curve below diagonal. Model says 90% confident but only 60% accurate.

Underconfident (Rare)

Curve above diagonal. Model says 60% confident but actually 80% accurate.

Expected Calibration Error (ECE)

ECE = Σ (|Bₘ|/n) × |acc(Bₘ) - conf(Bₘ)|

Weighted average of calibration gap across confidence bins. Lower is better. Quantifies overall miscalibration in a single number.

Temperature Scaling

p = softmax(z/T)

Post-hoc fix: divide logits by temperature T > 1 before softmax. Softens probabilities, reducing overconfidence. Simple, effective, doesn't change predictions.

Why This Matters for Deployment

In high-stakes domains, trust in probabilities is essential:

Medical diagnosis: A 95% cancer prediction should mean something
Autonomous vehicles: Uncertainty triggers human takeover
Decision support: Humans calibrate trust based on stated confidence

Miscalibrated models lead to automation bias (trusting overconfident wrong predictions) or unnecessary caution (distrusting correct predictions with understated confidence).

Hyperparameter Tuning

Key Distinction

Parameters are learned from data (weights, biases). Hyperparameters are set by the practitioner and control the learning process itself (learning rate, regularization strength, architecture choices).

Common Hyperparameters

Learning Rate

Step size for gradient descent. Too high → divergence. Too low → slow/stuck.

Typical: 10⁻⁴ to 10⁻¹

Batch Size

Samples per gradient update. Affects noise, speed, memory, generalization.

Typical: 16 to 512

Regularization (λ)

Penalty strength for weight magnitudes. Trades training fit for generalization.

Typical: 10⁻⁵ to 10⁻¹

Architecture

Number of layers, units per layer, activation functions, dropout rate.

Problem-dependent

Search Strategies

Grid Search

Try all combinations of specified values. Exhaustive but exponential in number of hyperparameters. Works for ≤3 hyperparameters.

Random Search

Sample random combinations. Often finds good solutions faster than grid search because it explores more unique values per hyperparameter.

Bayesian Optimization

Build a probabilistic model of the objective function. Choose next point to maximize expected improvement. More efficient for expensive evaluations.

Learning Rate Schedules

Start high, decay over time. Warmup schedules, cosine annealing, reduce on plateau. Often more important than finding a single optimal LR.

◈

The Meta-Validation Problem

Hyperparameters are tuned on the validation set, not test set. But extensive tuning can overfit to validation data too. Best practice: limit total number of experiments, use cross-validation for stability, and report the hyperparameter search budget honestly.

Common Misconceptions

✗

"Higher accuracy is always better"

In a cancer screening with 1% disease prevalence, always predicting "healthy" gives 99% accuracy while missing every cancer case.

The accurate framing: Accuracy is only meaningful with balanced classes. For imbalanced data, use precision, recall, F1, MCC, or AUC-PR depending on what errors matter most.

✗

"Good test performance means the model will work in production"

Test sets are static snapshots. The real world has distribution shift, adversarial inputs, edge cases, and evolving data patterns.

The accurate framing: Test performance is necessary but not sufficient. Production needs monitoring, graceful degradation, human-in-the-loop fallbacks, and continuous evaluation.

✗

"FID is the definitive measure of image generation quality"

FID measures distributional similarity, not individual image quality. It's computed on InceptionNet features trained on ImageNet—biased toward that domain. It requires 50k+ samples for stability.

The accurate framing: FID is one useful signal among many. Combine with CLIP score for text alignment, human evaluation for subjective quality, and diversity metrics for mode coverage.

⚗ Interactive Lab: Confusion Matrix Explorer

Explore how changing the confusion matrix values affects different classification metrics. See the precision-recall trade-off in action.

Adjust Confusion Matrix

Pred -

Pred +

Actual -

TN

FP

Actual +

FN

TP

Try These Scenarios

Computed Metrics

Accuracy 93.0%

Precision 61.5%

Recall 80.0%

F1 Score 69.6%

Specificity 94.4%

MCC 0.67

Key Observations

Imbalanced data: High accuracy can coexist with terrible recall
Precision vs Recall: Reducing FP improves precision; reducing FN improves recall
F1 Score: Harmonic mean punishes large imbalances
MCC: More robust to class imbalance than accuracy

Learning Objectives