Neural Networks as Function Approximators
The theoretical foundation of deep learning: understanding neural networks as universal function approximators and how architecture choices affect what functions can be learned.
Learning Objectives
- Understand neural networks as parameterized function families
- Explain the Universal Approximation Theorem and its implications
- Trace how data flows through a network (forward propagation)
- Understand the role of activation functions in creating non-linearity
- Connect network architecture to function expressiveness
The Function Approximation View
A neural network is a parameterized function f(x; θ) that maps inputs x to outputs y. Training adjusts the parameters θ (weights and biases) so that f approximates some target function—the relationship between inputs and desired outputs.
Think of a neural network as a very flexible curve-fitting tool. If you give it input-output pairs (like photos and their labels), it adjusts its internal knobs until its outputs match the targets as closely as possible.
Unlike fitting a specific curve (linear, quadratic), neural networks can fit almost any shape. The "curve" can have millions of degrees of freedom, allowing it to capture extraordinarily complex patterns.
Formally, we seek a function f* that minimizes expected loss:
Where F is the family of functions representable by our network architecture, D is the data distribution, and L is the loss function. Training with gradient descent finds a specific f within F that (approximately) minimizes empirical loss on training data.
The architecture defines F—what functions are possible to represent. Training finds which specific function in F to use.
An image classifier learns a function from pixel arrays to class probabilities. The "true" function maps any possible image to the correct label—a function so complex we could never write it down explicitly. The neural network approximates this function by learning from examples.
The remarkable fact: given enough data and the right architecture, neural networks can approximate functions we have no other way to describe.
Anatomy of a Neural Network
Neurons (Units)
Each neuron computes a weighted sum of its inputs plus a bias, then applies an activation function:
The weights w determine how much each input contributes. The bias b shifts the activation threshold. The activation σ introduces non-linearity.
Layers
Neurons are organized into layers. Each layer transforms its input into a new representation:
- Input layer: Raw features (pixels, words, numbers)
- Hidden layers: Learned representations
- Output layer: Final predictions (classes, values)
"Deep" learning means many hidden layers—more opportunities to build complex representations.
Weights & Biases
The parameters θ of a network are all its weights and biases. A layer with n inputs and m outputs has n×m weights plus m biases.
📐 Rusty on matrix math? The n×m weights form a matrix that multiplies the input vector. See: Linear Algebra prerequisite
Modern networks have millions to billions of parameters. GPT-3: 175 billion. GPT-4: estimated >1 trillion. These are all learned through optimization.
Activation Functions
Without activation functions, stacking layers just multiplies matrices—the whole network would compute a linear function. Activations introduce the non-linearity needed to approximate complex functions.
Common activations: ReLU, sigmoid, tanh, GELU. Each has different properties affecting gradient flow and expressiveness.
The Universal Approximation Theorem
A Foundational Guarantee
The Universal Approximation Theorem (Cybenko 1989, Hornik 1991) proves that a feedforward network with a single hidden layer containing enough neurons can approximate any continuous function on a compact domain to arbitrary accuracy.
For any continuous function f : [0,1]ⁿ → ℝ and any ε > 0, there exists a neural network g with one hidden layer such that |g(x) - f(x)| < ε for all x.
Neural networks are, in principle, capable of representing any reasonable function. The limitation isn't can they represent it—it's whether we can find the right parameters with training.
Think of it like clay: you can sculpt almost any shape, but that doesn't mean sculpting a perfect replica is easy. The theorem says neural networks are sufficiently expressive; it doesn't say training will succeed.
The theorem has important limitations:
- It doesn't say how many neurons are needed—potentially exponentially many
- It doesn't guarantee that gradient descent will find the right weights
- It says nothing about sample efficiency (how much data is needed)
- The approximation may not generalize beyond the training domain
In practice, depth often beats width—deeper networks can represent functions more efficiently than wide shallow ones.
Activation Functions Deep Dive
The choice of activation function significantly affects training dynamics, expressiveness, and what kinds of functions the network can efficiently approximate.
ReLU (Rectified Linear Unit)
Pros: Simple, fast, doesn't saturate for positive values, sparse activations. The default choice for most architectures.
Cons: "Dead neurons"—if a neuron always outputs 0, gradients stop flowing and it never recovers.
Sigmoid
Pros: Smooth, bounded (0,1), interpretable as probability.
Cons: Saturates at extremes (gradients vanish), outputs not zero-centered. Rarely used in hidden layers today.
Tanh
Pros: Zero-centered (outputs in [-1,1]), stronger gradients than sigmoid.
Cons: Still saturates at extremes. Used in RNNs and some specialized applications.
GELU (Gaussian Error Linear Unit)
Pros: Smooth approximation of ReLU with probabilistic interpretation. Used in transformers (BERT, GPT).
Cons: Computationally more expensive than ReLU.
Why Non-Linearity Matters
Without non-linear activations, layer composition just multiplies matrices: W₂(W₁x) = (W₂W₁)x = Wx. No matter how many layers, the result is a single linear transformation—unable to represent XOR, circles, or any non-linear pattern.
Forward Propagation
Forward propagation is the process of computing the network's output for a given input by passing data through each layer sequentially, applying weights, biases, and activations at each step.
Input
Raw features enter the network: x = [x₁, x₂, ..., xₙ]
Linear Transformation
Multiply by weight matrix, add bias: z = Wx + b
Activation
Apply non-linearity element-wise: h = σ(z)
Repeat
Hidden activations become input to next layer
Output
Final layer produces predictions (softmax for classification, linear for regression)
Concrete Example: 2-Layer Network
Input x = [0.5, -0.3], hidden layer with 2 neurons, output layer with 1 neuron:
Layer 1 (hidden):
z₁ = W₁x + b₁ = [[0.2, 0.4], [0.5, -0.1]] · [0.5, -0.3] + [0.1, 0.2]
z₁ = [0.1 - 0.12 + 0.1, 0.25 + 0.03 + 0.2] = [0.08, 0.48]
h₁ = ReLU(z₁) = [0.08, 0.48]
Layer 2 (output):
z₂ = W₂h₁ + b₂ = [0.7, 0.3] · [0.08, 0.48] + 0.1
z₂ = 0.056 + 0.144 + 0.1 = 0.3
output = sigmoid(0.3) ≈ 0.574
Common Misconceptions
"Neural networks model how the brain works"
Despite the name, artificial neural networks are very different from biological brains. Real neurons have complex temporal dynamics, chemical signaling, and structural plasticity that ANNs don't capture.
The accurate framing: Neural networks are loosely inspired by biological neurons but are better understood as parameterized function approximators. They're mathematical tools, not brain simulations.
"More layers always means better performance"
Very deep networks can suffer from vanishing gradients, optimization difficulties, and overfitting. Depth must be matched to problem complexity and available data.
The accurate framing: More layers increase capacity but also increase optimization difficulty and data requirements. Techniques like residual connections and normalization make deep training possible, but depth should match the task.
"The Universal Approximation Theorem means neural networks can solve any problem"
The theorem only guarantees existence of a good approximation—not that training will find it, that it will generalize, or that it's computationally feasible.
The accurate framing: The theorem tells us neural networks are expressive enough in principle. Success in practice depends on architecture, training data, optimization, and regularization—none of which the theorem addresses.
Interactive Lab: Network Playground
Build and visualize neural networks in real-time. See how changing architecture and activations affects what functions the network can approximate.
Trace data flowing through a network. Enter input values and watch how each layer transforms them.
ReLU Properties
- Range: [0, ∞)
- Zero-centered: No
- Gradient: 0 or 1
- Issue: Dead neurons (gradient = 0 for x < 0)
Key Observations
- More neurons = more flexibility: With few neurons, the network can only produce simple shapes. More neurons allow finer detail.
- Activation matters: ReLU creates piecewise linear functions; tanh creates smooth curves. Match activation to your function's character.
- Some functions are harder: Smooth functions like sine are easier to approximate than discontinuous ones like square waves.
Check Your Understanding
What does the Universal Approximation Theorem guarantee?
Why are activation functions necessary in neural networks?
What is a potential problem with ReLU activation?
What happens during forward propagation?
How many parameters does a fully-connected layer with 100 inputs and 50 outputs have?