Backpropagation & Learning
The algorithm that makes deep learning possible: how neural networks compute gradients efficiently and use them to learn from data through iterative weight updates.
Learning Objectives
- Understand backpropagation as efficient gradient computation via the chain rule
- Trace how error signals flow backward through a network
- Explain the complete training loop: forward pass, loss, backward pass, update
- Identify common issues: vanishing/exploding gradients
- Connect backpropagation to modern automatic differentiation frameworks
The Credit Assignment Problem
When a neural network makes an error, which of its millions of parameters are responsible? Backpropagation solves this credit assignment problem by efficiently computing how much each weight contributed to the error.
Imagine a factory assembly line where the final product is defective. To fix it, you need to trace backward through each station: which worker made an error? How much did each station contribute to the final defect?
Backpropagation does this for neural networks. Starting from the output error, it traces backward layer by layer, computing exactly how much each weight contributed—then adjusting each weight proportionally.
Backpropagation is an application of the chain rule from calculus. If y = f(g(x)), then dy/dx = (dy/dg) × (dg/dx). For a network with many composed functions:
Each factor can be computed locally—the network computes ∂L/∂y, then propagates it backward, multiplying by local derivatives at each layer.
Modern frameworks (PyTorch, TensorFlow) implement automatic differentiation—you define the forward pass, and the framework automatically computes gradients. But understanding backpropagation helps debug training problems and design better architectures.
The Chain Rule in Action
Compositional Derivatives
Neural networks are compositions of functions: output = f₃(f₂(f₁(input))). The chain rule lets us compute the derivative of this composition by multiplying the derivatives of each component.
Forward Pass
Compute output and loss
Backward Pass
Propagate gradients backward
Simple Example: 2-Layer Network
For a single neuron: y = σ(wx + b), with loss L = (y - target)²
Forward:
z = wx + b = 0.5 × 1.0 + 0.1 = 0.6
y = σ(z) = 1/(1+e⁻⁰·⁶) ≈ 0.646
L = (0.646 - 1.0)² ≈ 0.125
Backward (chain rule):
∂L/∂y = 2(y - target) = 2(0.646 - 1.0) = -0.708
∂y/∂z = σ(z)(1 - σ(z)) = 0.646 × 0.354 ≈ 0.229
∂z/∂w = x = 1.0
∂L/∂w = ∂L/∂y × ∂y/∂z × ∂z/∂w = -0.708 × 0.229 × 1.0 ≈ -0.162
The gradient tells us: to reduce loss, increase w (since gradient is negative).
The Complete Training Loop
Sample Batch
Select a mini-batch of training examples (x, y) pairs
Forward Pass
Compute predictions ŷ = f(x; θ) and loss L(ŷ, y)
Backward Pass
Compute gradients ∇θL using backpropagation
Update Parameters
θ ← θ - η∇θL (gradient descent step)
Repeat
Continue until convergence or stopping criterion
One complete pass through all training data is called an epoch. Training typically runs for many epochs, with the loss gradually decreasing as the network learns.
Gradient Flow Problems
In deep networks, gradients must flow through many layers. Two pathological behaviors can prevent learning: gradients that shrink to zero (vanishing) or grow explosively (exploding).
Vanishing Gradients
Gradients shrink exponentially through layers
If each layer's gradient is < 1 (e.g., 0.5), then after 10 layers: 0.5¹⁰ ≈ 0.001. Early layers receive near-zero gradients and barely learn.
Causes:
- Sigmoid/tanh saturating at extremes
- Weight initialization too small
Solutions:
- ReLU activations (gradient = 1 for positive inputs)
- Residual connections (skip connections)
- Batch normalization
- Careful initialization (Xavier, He)
Exploding Gradients
Gradients grow exponentially through layers
If each layer's gradient is > 1 (e.g., 2.0), then after 10 layers: 2¹⁰ = 1024. Updates become huge, causing NaN values or divergence.
Causes:
- Weight initialization too large
- Very deep networks (especially RNNs)
Solutions:
- Gradient clipping (cap gradient magnitude)
- Proper initialization
- Layer normalization
- LSTM/GRU for sequences (gated architectures)
Common Misconceptions
"Backpropagation is how the brain learns"
Biological neurons don't have access to the error signal from the output or the weights of downstream neurons—making exact backprop biologically implausible.
The accurate framing: Backpropagation is an efficient algorithm for computing gradients, not a model of biological learning. How brains implement credit assignment remains an open research question.
"Each neuron is updated independently"
Gradients are computed considering the entire network—each weight's gradient depends on all downstream weights through the chain rule.
The accurate framing: Updates are computed globally but applied locally. Each weight's gradient captures its global impact on the loss through the entire computational graph.
"Larger gradients mean faster learning"
Very large gradients can cause instability—overshooting minima, oscillating, or diverging entirely. Stable training requires gradients of appropriate magnitude.
The accurate framing: Gradient magnitude should be "just right"—large enough to make progress but small enough for stability. Normalization and careful learning rate selection help maintain this balance.
Interactive Lab: Backpropagation Visualizer
Watch gradients flow backward through a network in real-time. See how each weight's gradient is computed and how updates affect the network.
Key Observations
- Gradient magnitude decreases as you move backward—especially with sigmoid activations.
- Weights with large inputs have larger gradients—they're more "responsible" for the output.
- Learning rate matters: Too high causes oscillation; too low causes slow convergence.
Check Your Understanding
What mathematical principle underlies backpropagation?
What causes vanishing gradients?
In the training loop, what happens immediately after the backward pass?
Why do residual connections (skip connections) help with deep networks?
What is gradient clipping used for?