Foundations of AI
Before Transformers dominated, CNNs conquered vision and RNNs ruled sequences. Understanding these architectures reveals why structure-aware design matters—and when to use each approach today.
You've learned that MLPs are universal function approximators (Module 2.2). But MLPs treat every input dimension independently—they don't "know" that pixels next to each other are related, or that words form sequences.
CNNs and RNNs solved this by building structure into the architecture. Understanding their design choices prepares you for Transformers, which take a different approach: learning structure from data rather than hard-coding it.
The inductive bias of an architecture is the set of assumptions it makes about data structure. These assumptions guide learning, reducing the amount of data needed—but can hurt performance when assumptions don't match reality.
A Convolutional Neural Network is a neural network that uses convolution operations to exploit spatial structure in data—particularly images. It assumes that patterns are local (nearby pixels are related) and translation-invariant (a cat is a cat regardless of where it appears).
Imagine looking at a photo through a small sliding window. As you move the window across the image, you look for specific patterns: edges, textures, shapes. You don't need to see the whole image at once—local patterns combine to form global understanding.
This is exactly what CNNs do. Small "filters" slide across the image, detecting local patterns. Early layers find edges; deeper layers combine edges into shapes; final layers recognize objects.
A convolution applies a small filter (kernel) to every location in the input:
Output[i,j] = Σₘ Σₙ Input[i+m, j+n] × Filter[m,n]
The same filter is applied everywhere (weight sharing), meaning:
A typical CNN stacks: Conv → ReLU → Pool → Conv → ReLU → Pool → ... → Flatten → Dense
Pooling (max or average) reduces spatial dimensions, creating a hierarchical representation: pixel → edge → texture → part → object.
import torch.nn as nn
class SimpleCNN(nn.Module):
def __init__(self, num_classes=10):
super().__init__()
self.features = nn.Sequential(
# Conv Block 1
nn.Conv2d(3, 64, kernel_size=3, padding=1), # [B, 3, H, W] → [B, 64, H, W]
nn.ReLU(),
nn.MaxPool2d(2), # [B, 64, H/2, W/2]
# Conv Block 2
nn.Conv2d(64, 128, kernel_size=3, padding=1),
nn.ReLU(),
nn.MaxPool2d(2), # [B, 128, H/4, W/4]
# Conv Block 3
nn.Conv2d(128, 256, kernel_size=3, padding=1),
nn.ReLU(),
nn.AdaptiveAvgPool2d(1) # Global average pooling → [B, 256, 1, 1]
)
self.classifier = nn.Linear(256, num_classes)
def forward(self, x):
x = self.features(x)
x = x.view(x.size(0), -1) # Flatten
return self.classifier(x)
# Key insight: Conv2d has ~3×3×C_in×C_out parameters
# Much smaller than Dense: H×W×C_in × H×W×C_outWon ImageNet by a huge margin. Proved deep learning works at scale.
Showed that depth matters. Simple 3×3 convolutions stacked deep.
Introduced skip connections. Enabled training of 100+ layer networks.
Balanced depth, width, resolution. State-of-the-art efficiency.
A Recurrent Neural Network processes sequences by maintaining a hidden state that accumulates information from previous time steps. It assumes that data has sequential structure where earlier elements influence later ones.
Reading a sentence word-by-word, you build up context. When you see "The dog chased the ___", your understanding of the sentence so far influences your expectation for the next word.
RNNs formalize this: a hidden state vector is updated at each time step, carrying forward information from the past. The same weights process each step—the network learns a reusable "update rule" for any sequence length.
At each time step t, the RNN updates its hidden state:
h_t = tanh(W_h · h_{t-1} + W_x · x_t + b)
The same weights (W_h, W_x, b) are used at every step—this is weight sharing across time, analogous to CNN weight sharing across space.
Problem: During backpropagation through many time steps, gradients shrink exponentially (vanishing) or explode. Long-range dependencies are hard to learn.
Solution: LSTM (Long Short-Term Memory) and GRU (Gated Recurrent Unit) add gates that control information flow, allowing gradients to flow unchanged over many steps.
Same weights at every step. Hidden state h carries information forward.
LSTMs add a separate "cell state" C that runs through time like a conveyor belt. Three gates control what to forget, what to add, and what to output:
f_t = σ(W_f · [h_{t-1}, x_t])
What to remove from cell state
i_t = σ(W_i · [h_{t-1}, x_t])
What new info to store
o_t = σ(W_o · [h_{t-1}, x_t])
What to output from cell state
import torch.nn as nn
class LSTMClassifier(nn.Module):
def __init__(self, vocab_size, embed_dim, hidden_dim, num_classes):
super().__init__()
self.embedding = nn.Embedding(vocab_size, embed_dim)
self.lstm = nn.LSTM(
input_size=embed_dim,
hidden_size=hidden_dim,
num_layers=2,
batch_first=True,
dropout=0.3,
bidirectional=True # Process both directions
)
# Bidirectional doubles the output dimension
self.classifier = nn.Linear(hidden_dim * 2, num_classes)
def forward(self, x):
# x: [batch, seq_len] token indices
embedded = self.embedding(x) # [batch, seq_len, embed_dim]
# LSTM returns: output (all hidden states), (h_n, c_n)
lstm_out, (h_n, c_n) = self.lstm(embedded)
# Use final hidden state for classification
# h_n: [num_layers*2, batch, hidden_dim] for bidirectional
# Concatenate forward and backward final states
final_hidden = torch.cat([h_n[-2], h_n[-1]], dim=1)
return self.classifier(final_hidden)
# Key limitation: Sequential processing = no parallelization
# This is why Transformers replaced RNNs for most NLP tasksUnsupervised learning finds structure in data without explicit labels. The algorithm discovers patterns, groupings, or representations that weren't provided by humans—essential for understanding data before building models.
Most real-world data is unlabeled—labels are expensive. Unsupervised learning lets you:
Partitions data into K clusters by minimizing within-cluster variance. Each point belongs to the cluster with the nearest centroid.
from sklearn.cluster import KMeans
import numpy as np
# Cluster customer data
X = load_customer_features() # [n_customers, n_features]
kmeans = KMeans(n_clusters=5, random_state=42)
cluster_labels = kmeans.fit_predict(X)
# Find cluster centers
centers = kmeans.cluster_centers_
# Evaluate clustering quality
from sklearn.metrics import silhouette_score
score = silhouette_score(X, cluster_labels)
print(f"Silhouette Score: {score:.3f}") # Higher is betterBuilds tree of clusters. No need to specify K upfront.
Density-based. Finds clusters of arbitrary shape. Handles outliers.
Soft clustering. Points have probabilities of belonging to each cluster.
Finds directions of maximum variance in data and projects onto them. The first few principal components often capture most of the information.
from sklearn.decomposition import PCA
import matplotlib.pyplot as plt
# Reduce dimensions for visualization
pca = PCA(n_components=2)
X_2d = pca.fit_transform(X)
# How much variance is explained?
print(f"Explained variance: {pca.explained_variance_ratio_.sum():.1%}")
# Visualize
plt.scatter(X_2d[:, 0], X_2d[:, 1], c=labels, cmap='viridis')
plt.xlabel('PC1')
plt.ylabel('PC2')
plt.title('Data in Principal Component Space')| Method | Type | Preserves | Best For |
|---|---|---|---|
| PCA | Linear | Global structure, variance | Fast exploration, preprocessing |
| t-SNE | Non-linear | Local neighborhoods | Visualization, finding clusters |
| UMAP | Non-linear | Local + some global | Visualization + downstream ML |
An autoencoder learns to compress input into a small latent code, then reconstruct the original. The bottleneck forces it to learn the most important features.
class Autoencoder(nn.Module):
def __init__(self, input_dim, latent_dim):
super().__init__()
self.encoder = nn.Sequential(
nn.Linear(input_dim, 256),
nn.ReLU(),
nn.Linear(256, latent_dim)
)
self.decoder = nn.Sequential(
nn.Linear(latent_dim, 256),
nn.ReLU(),
nn.Linear(256, input_dim)
)
def forward(self, x):
z = self.encoder(x) # Compress
return self.decoder(z) # Reconstruct
# Training: minimize reconstruction loss
loss = F.mse_loss(model(x), x)Transformers are increasingly used everywhere—Vision Transformers for images, Audio Transformers for speech. But CNNs and RNNs remain valuable:
Explore how different architectures process data. See the convolution operation in action, watch an RNN accumulate state, and visualize clustering.
Watch how the hidden state changes as each word is processed. Information from earlier words influences the representation of later words.
What is the main advantage of weight sharing in CNNs?
Why were LSTMs developed to replace vanilla RNNs?
In K-Means clustering, what does the algorithm minimize?
When should you prefer a CNN over a Transformer for image tasks?