📐 Prerequisite • Mathematics

Calculus for AI

Understanding derivatives is the key to understanding how neural networks learn. Every time a model improves, calculus is working behind the scenes. This guide covers exactly what you need to understand AI optimization—explained intuitively with real examples.

⏱ Estimated reading time: 20-25 minutes

🎯 What You'll Learn

What derivatives measure and why they matter for optimization
How to compute derivatives using key rules (power, sum, product, chain)
How partial derivatives handle functions with multiple inputs
What gradients are and how they guide neural network learning
Why the chain rule is the foundation of backpropagation

1 Why Calculus Matters for AI

Machine learning is fundamentally about optimization—finding the best possible parameters for a model. Imagine you're blindfolded on a hilly landscape, trying to find the lowest valley. How do you know which direction to walk?

Calculus answers this question. Specifically, derivatives tell us the slope of the ground beneath our feet. If we're on a slope going up to the right, we should walk left. If it's going up to the left, we walk right. This simple idea—"follow the downhill slope"—is the core of how neural networks learn.

🤖 The AI Connection

Every neural network has millions (or billions!) of parameters. During training, the model computes how changing each parameter would affect its errors. This computation uses derivatives. The famous "backpropagation" algorithm is essentially clever application of the chain rule to compute all these derivatives efficiently.

💡 The Big Picture

Think of training a neural network as adjusting millions of knobs on a massive control panel. Each knob affects the output. Calculus tells us: "Turn this knob clockwise to reduce the error" or "Turn that knob counterclockwise." Without calculus, we'd be randomly guessing which way to turn each knob.

2 Derivatives: Measuring Change

A derivative measures how fast something changes. If you have a function f(x), the derivative f'(x) tells you the rate of change at any point x.

The Formal Definition

f'(x) = lim_h→0 [f(x+h) - f(x)] / h

The derivative is the slope of the tangent line at point x

In plain English: take a tiny step h, see how much f changes, divide by the step size. As h gets infinitely small, you get the instantaneous rate of change.

💡 Real-World Intuition

Imagine driving a car. Your position changes over time. The derivative of position is velocity (how fast position changes). The derivative of velocity is acceleration (how fast velocity changes). Derivatives describe "how fast something is changing."

What Does the Derivative Tell Us?

Positive derivative: The function is increasing (going uphill)
Negative derivative: The function is decreasing (going downhill)
Zero derivative: The function is flat (at a peak, valley, or plateau)
Large magnitude: The function is changing rapidly (steep slope)
Small magnitude: The function is changing slowly (gentle slope)

        f(x)
         │         ╱ ← Derivative positive (increasing)
         │        ╱
         │       ╱
         │      •  ← Derivative = 0 (peak or valley)
         │     ╱
         │    ╱ ← Derivative negative (decreasing)  
         │   ╱
         └──────────────── x

The derivative describes the slope at each point on the curve

📝 Worked Example

For the function f(x) = x², let's compute f'(2):

f(x) = x²
f(2) = 4
f(2.001) = 4.004001

Approximate derivative:
f'(2) ≈ (4.004001 - 4) / 0.001 = 4.001

Exact answer: f'(x) = 2x, so f'(2) = 4

At x = 2, the function x² is increasing at a rate of 4 units of output per 1 unit of input.

3 Essential Derivative Rules

Rather than computing limits every time, mathematicians have derived rules for common functions. These rules let you quickly compute derivatives of any combination of basic functions.

Rule 1: Power Rule

d/dx[xⁿ] = n · x^n-1

Bring down the exponent, reduce it by one

📝 Examples

d/dx[x³] = 3x²
d/dx[x²] = 2x
d/dx[x]  = 1 · x⁰ = 1
d/dx[x⁻¹] = -1 · x⁻² = -1/x²
d/dx[√x] = d/dx[x^½] = ½ · x^(-½) = 1/(2√x)

Rule 2: Constant Rule

d/dx[c] = 0

The derivative of any constant is zero

This makes sense—a constant doesn't change, so its rate of change is zero.

Rule 3: Sum & Difference Rules

d/dx[f(x) + g(x)] = f'(x) + g'(x)

Derivative of a sum = sum of derivatives

📝 Example

d/dx[x³ + 2x² - 5x + 7]
= d/dx[x³] + d/dx[2x²] - d/dx[5x] + d/dx[7]
= 3x² + 4x - 5 + 0
= 3x² + 4x - 5

Rule 4: Constant Multiple Rule

d/dx[c · f(x)] = c · f'(x)

Constants can be "pulled out" of derivatives

Rule 5: Product Rule

d/dx[f(x) · g(x)] = f'(x) · g(x) + f(x) · g'(x)

"First times derivative of second, plus second times derivative of first"

💡 Mnemonic

Think of it as: "Keep one function the same, differentiate the other, then switch roles and add."

📝 Example

d/dx[x² · sin(x)]
= (2x) · sin(x) + x² · cos(x)
= 2x·sin(x) + x²·cos(x)

Rule 6: Quotient Rule

d/dx[f(x)/g(x)] = [f'(x)·g(x) - f(x)·g'(x)] / [g(x)]²

"Low d-high minus high d-low, over low squared"

Key Derivatives to Memorize

Basic Functions

d/dx[e^x] = e^x
d/dx[ln(x)] = 1/x
d/dx[sin(x)] = cos(x)
d/dx[cos(x)] = -sin(x)

AI-Specific Functions

d/dx[sigmoid(x)] = σ(x)·(1-σ(x))
d/dx[ReLU(x)] = 1 if x>0, else 0
d/dx[tanh(x)] = 1 - tanh²(x)
d/dx[softplus(x)] = sigmoid(x)

🤖 Why These Matter for AI

Activation functions like ReLU, sigmoid, and tanh are used in every neural network. Their derivatives determine how gradients flow during training. ReLU is popular partly because its derivative is simple (0 or 1), making training efficient. Sigmoid's derivative can get very small (near 0 or 1), causing "vanishing gradients"—a real problem in deep networks.

4 Partial Derivatives & Gradients

So far, we've looked at functions with one input. But neural networks have millions of parameters! Partial derivatives let us handle functions with multiple inputs by looking at how the output changes with respect to one variable at a time, while treating others as constants.

∂f/∂x = rate of change of f as x changes (holding y, z, ... constant)

The ∂ symbol (curly d) denotes partial derivatives

Computing Partial Derivatives

To find ∂f/∂x: treat all other variables as constants and differentiate normally with respect to x.

📝 Detailed Example

f(x, y) = x² + 3xy + y²

To find ∂f/∂x:
  - Treat y as a constant
  - d/dx[x²] = 2x
  - d/dx[3xy] = 3y  (y is constant, so 3xy is "3y times x")
  - d/dx[y²] = 0   (y² is a constant w.r.t. x)
  - ∂f/∂x = 2x + 3y

To find ∂f/∂y:
  - Treat x as a constant
  - d/dy[x²] = 0
  - d/dy[3xy] = 3x
  - d/dy[y²] = 2y
  - ∂f/∂y = 3x + 2y

The Gradient: All Partials Together

The gradient bundles all partial derivatives into a single vector. It points in the direction of steepest increase.

∇f = [∂f/∂x₁, ∂f/∂x₂, ..., ∂f/∂xₙ]

The gradient is a vector pointing "uphill"

📝 Example

For f(x, y) = x² + 3xy + y²:

∇f = [∂f/∂x, ∂f/∂y] = [2x + 3y, 3x + 2y]

At point (1, 2):
∇f(1,2) = [2(1) + 3(2), 3(1) + 2(2)]
        = [8, 7]

This vector [8, 7] points in the direction of 
steepest increase from point (1, 2).

Gradient Descent: The Learning Algorithm

Gradient descent is the core algorithm for training neural networks. The idea is simple: to minimize a function, move opposite to the gradient (downhill).

Start with initial parameters

Initialize your model's weights (often randomly).

Compute the gradient

Calculate ∇L with respect to all parameters, where L is the loss function.

Update parameters

Move in the opposite direction: w_new = w_old - η · ∇L, where η is the learning rate.

Repeat

Continue until the loss stops decreasing (convergence).

w := w - η · ∇L(w)

The gradient descent update rule

        Loss Surface (2D visualization)
        
             High Loss
                ╱╲
               ╱  ╲
              ╱    ╲
             ╱      ╲
            ╱  • ←───── Starting point
           ╱   ↓        
          ╱    ↓ ←───── Gradient points uphill,
         ╱     •        we move OPPOSITE (downhill)
        ╱      ↓        
       ╱       ↓        
      ╱        • ←───── Getting closer to minimum
     ╱         ↓        
    ╱          ★ ←───── Minimum (goal!)
   ──────────────────── 
         Parameters (weights)

Gradient descent follows the downhill direction to find the minimum loss

🤖 How This Works in Practice

In a neural network with 1 billion parameters, we compute 1 billion partial derivatives—one for each parameter. This tells us how to adjust each weight to reduce the error. Modern frameworks like PyTorch and TensorFlow do this automatically using "autograd" (automatic differentiation).

⚠️ Common Pitfalls

Learning rate too high: Steps are too big, you might "jump over" the minimum.
Learning rate too low: Training takes forever, might get stuck.
Local minima: You might find a valley that's not the deepest (though this is less problematic in high dimensions than you might think).

5 The Chain Rule & Backpropagation

The chain rule is arguably the most important calculus concept for AI. It tells us how to compute derivatives of composed functions—functions inside functions—which is exactly what a neural network is!

The Basic Chain Rule

d/dx[f(g(x))] = f'(g(x)) · g'(x)

"Derivative of outer × derivative of inner"

💡 Intuition

If x affects y, and y affects z, then to find how x affects z, multiply the effects together:

(rate z changes per y) × (rate y changes per x) = (rate z changes per x)

📝 Step-by-Step Example

Find the derivative of h(x) = (3x + 1)²

Step 1: Identify inner and outer functions
  Inner: g(x) = 3x + 1
  Outer: f(u) = u²

Step 2: Find their derivatives
  g'(x) = 3
  f'(u) = 2u

Step 3: Apply chain rule
  h'(x) = f'(g(x)) · g'(x)
        = 2(3x + 1) · 3
        = 6(3x + 1)
        = 18x + 6

Chain Rule with Multiple Variables

When functions have multiple inputs and outputs, the chain rule extends naturally. This is exactly what happens in neural networks.

∂L/∂w = ∂L/∂y · ∂y/∂z · ∂z/∂w

Chain the derivatives through each layer

Backpropagation: The Chain Rule in Action

A neural network is just a composition of functions: input → layer 1 → layer 2 → ... → output → loss. Backpropagation computes derivatives by applying the chain rule from output back to input.

🔢 Numerical Example First (No Scary Notation!)

Let's trace through backprop with real numbers before introducing symbols:

Simple network: y = 2x, then square it → output = (2x)²
Input: x = 3

FORWARD PASS (calculate step by step):
  Step 1: z = 2 × 3 = 6
  Step 2: output = 6² = 36

BACKWARD PASS (how does changing x affect the output?):
  If x increases by 1 → z increases by 2 (because z = 2x)
  If z increases by 1 → output increases by 12 (derivative of z² at z=6 is 2×6=12)
  
  Combined: x↑1 → z↑2 → output↑24
  
So: changing x by 1 changes output by 24. That's the gradient!
Check: d/dx[(2x)²] = d/dx[4x²] = 8x = 8×3 = 24 ✓

That's all backpropagation is: multiply how much each step affects the next, backwards through the chain.

FORWARD PASS (compute output):
────────────────────────────────────────────────→

  Input     Layer 1     Layer 2     Output    Loss
    x    →    z₁    →     z₂    →    ŷ    →    L
          W₁        W₂         W₃


BACKWARD PASS (compute gradients via chain rule):
←────────────────────────────────────────────────

∂L/∂W₁ = ∂L/∂ŷ · ∂ŷ/∂z₂ · ∂z₂/∂z₁ · ∂z₁/∂W₁
         ─────   ──────   ──────   ────────
           │       │        │         │
           └───────┴────────┴─────────┘
             Multiply these "local gradients"

Backpropagation applies the chain rule layer by layer

Forward pass

Compute the output by feeding input through all layers. Store intermediate values.

Compute loss

Compare output to target, compute loss (e.g., squared error, cross-entropy).

Backward pass

Starting from the loss, compute gradients layer by layer using the chain rule.

Update weights

Use gradient descent to adjust all weights in the direction that reduces loss.

📝 Concrete Example: Simple 2-Layer Network

Network: x → [Linear] → [ReLU] → [Linear] → ŷ
                     w₁        w₂

Forward:
  z₁ = w₁ · x        (linear layer 1)
  a₁ = ReLU(z₁)      (activation)
  ŷ  = w₂ · a₁       (linear layer 2)
  L  = (ŷ - y)²      (squared error loss)

Backward (chain rule):
  ∂L/∂ŷ  = 2(ŷ - y)
  ∂ŷ/∂w₂ = a₁
  ∂ŷ/∂a₁ = w₂
  ∂a₁/∂z₁ = 1 if z₁ > 0, else 0  (ReLU derivative)
  ∂z₁/∂w₁ = x

Final gradients:
  ∂L/∂w₂ = ∂L/∂ŷ · ∂ŷ/∂w₂ = 2(ŷ-y) · a₁
  ∂L/∂w₁ = ∂L/∂ŷ · ∂ŷ/∂a₁ · ∂a₁/∂z₁ · ∂z₁/∂w₁
         = 2(ŷ-y) · w₂ · ReLU'(z₁) · x

🤖 Why Backpropagation is Efficient

Computing gradients naively would require O(n²) operations for n parameters. Backpropagation is clever: it reuses intermediate computations from the forward pass, achieving O(n) complexity. This efficiency is why we can train billion-parameter models. Modern frameworks (PyTorch, TensorFlow, JAX) implement this automatically—you just define the forward pass, and gradients are computed for free!

6 Practice Problems

Test your understanding with these problems. Try to solve them before revealing the solutions!

✏️ Problem 1: Basic Derivatives

Find d/dx[3x⁴ - 2x² + 5x - 7]

Click to reveal solution ▼

d/dx[3x⁴ - 2x² + 5x - 7]
= 3·4x³ - 2·2x + 5·1 - 0
= 12x³ - 4x + 5

✏️ Problem 2: Chain Rule

Find d/dx[sin(x²)]

Click to reveal solution ▼

Using chain rule with:
  Outer: f(u) = sin(u), f'(u) = cos(u)
  Inner: g(x) = x², g'(x) = 2x

d/dx[sin(x²)] = cos(x²) · 2x = 2x·cos(x²)

✏️ Problem 3: Partial Derivatives

For f(x,y) = x²y + 3xy², find ∂f/∂x and ∂f/∂y

Click to reveal solution ▼

∂f/∂x: treat y as constant
  ∂/∂x[x²y] = 2xy
  ∂/∂x[3xy²] = 3y²
  ∂f/∂x = 2xy + 3y²

∂f/∂y: treat x as constant
  ∂/∂y[x²y] = x²
  ∂/∂y[3xy²] = 6xy
  ∂f/∂y = x² + 6xy

✏️ Problem 4: Gradient

For f(x,y) = x² + y² (a bowl shape), compute ∇f at point (3, 4)

Click to reveal solution ▼

∂f/∂x = 2x
∂f/∂y = 2y

∇f = [2x, 2y]
∇f(3,4) = [6, 8]

This points away from the origin (uphill).
To minimize f, move in direction [-6, -8].

🎯 Key Takeaways

Derivatives measure rate of change—they tell us the slope at each point
Partial derivatives handle multiple variables by varying one at a time
The gradient combines all partials into a vector pointing uphill
Gradient descent minimizes loss by moving opposite to the gradient
The chain rule lets us differentiate composed functions (neural networks!)
Backpropagation efficiently computes gradients via the chain rule

You're Ready!

With these concepts, you have the calculus foundation needed for the AI course. You'll encounter these ideas throughout:

Module 1.4: Optimization & Gradients
Module 2.3: Backpropagation & Learning
Module 2.4: Loss Functions & Optimization

Next Prerequisite Linear Algebra → Ready to Learn? Start the Course →