Foundations of AI
How neural networks learn to represent discrete concepts (words, users, items) as continuous vectors—enabling semantic reasoning and forming the foundation of modern language models.
The inductive bias of a model is the set of assumptions it makes about the structure of the problem. These assumptions guide learning when data is limited, but can hurt performance when they don't match the actual data structure.
Imagine learning to play chess. A weak prior assumes nothing—you must discover every rule from scratch. A strong prior says "pieces move in specific patterns, capture by replacement, king must be protected." The strong prior helps you learn faster, but if you're actually learning Go, it hurts.
Weak Bias
No structural assumptions. Must learn all patterns from data. Need lots of data.
Strong Spatial Bias
Assumes translation invariance (a cat is a cat anywhere in the image) and local structure (nearby pixels are related). Excellent for images.
Strong Sequential Bias
Assumes sequential dependence (position matters, earlier tokens affect later ones). Good for text, time series.
Flexible Bias
Learns which positions relate to which via attention. No hard-coded locality or sequence assumptions. Can handle anything—with enough data.
In the 2010s, CNNs dominated vision and RNNs dominated language—their biases matched the data well. But as datasets grew enormous (billions of examples), Transformers' flexibility became an advantage. They could learn the "right" inductive bias from data rather than having it hard-coded.
Key insight: Transformers didn't prove CNNs/RNNs "wrong"—they showed that with enough data, learned biases outperform hand-designed ones.
Module 3 follows the path from MLPs to Transformers: 3.1 Embeddings (how to represent discrete data), 3.2 Attention (how to learn which positions matter), 3.3 Transformers (putting it together), and 3.4 LLMs (what happens at scale).
While Transformers dominate modern AI, understanding CNNs and RNNs is essential—not for history, but because their inductive biases encode fundamental concepts that recur everywhere.
Output[i,j] = Σ Input[i+m, j+n] × Filter[m,n]
The same Filter slides across all positions
h_t = f(W_h · h_{t-1} + W_x · x_t + b)
Hidden state h_t carries information from all past steps
| Aspect | CNNs | RNNs | Transformers |
|---|---|---|---|
| Receptive Field | Local (grows with depth) | All past tokens | All tokens (via attention) |
| Parallelism | Fully parallel | Sequential | Fully parallel |
| Position Handling | Implicit (spatial) | Implicit (order) | Explicit (positional encoding) |
| Inductive Bias | Locality, translation invariance | Sequential, recency | Minimal (learned via attention) |
Neural networks operate on continuous vectors, but many real-world entities are discrete: words, users, products, genes. Embeddings bridge this gap by mapping discrete items to dense, low-dimensional vectors where similarity has meaning.
Think of a map: cities exist as discrete points on Earth, but by assigning coordinates (latitude, longitude), we can measure distances and directions. Paris is "close to" Lyon in a way that matches geographic reality.
Embeddings do this for abstract concepts. Words like "king" and "queen" get coordinates in a vector space where proximity reflects semantic similarity. The network learns these coordinates by observing how concepts are used.
An embedding is a learned lookup table E ∈ ℝ^(V×d) where V is vocabulary size and d is embedding dimension. For item i with one-hot encoding e_i:
The embedding matrix E is trained end-to-end with the rest of the network. Gradients flow back through the embedding lookup, updating only the row corresponding to each input item.
Pre-trained embeddings (Word2Vec, GloVe, BERT) capture general linguistic knowledge and can be used as starting points for specific tasks. Modern LLMs learn embeddings as their first layer, mapping tokens to the vectors that feed into transformer layers.
A vocabulary of 50,000 words represented as one-hot vectors means each word is a 50,000-dimensional vector with a single 1 and 49,999 zeros. This is wasteful and captures no similarity—"cat" and "dog" are as different as "cat" and "quantum".
The breakthrough insight: words that appear in similar contexts have similar meanings. By training to predict context from words (or vice versa), the network learns embeddings where semantically related words cluster together.
Given a center word, predict surrounding context words. The sentence "The cat sat on the mat" trains to predict "sat", "on" from "cat".
Forces the embedding to capture what concepts typically appear together.
Continuous Bag of Words: given context words, predict the center word. From "The ___ sat on", predict "cat".
Often faster to train; good for frequent words.
Instead of full softmax over vocabulary, contrast real context pairs against random "negative" pairs. Much more efficient.
Makes training tractable for large vocabularies.
After training on billions of words, embeddings capture syntactic and semantic patterns—without any labeled data.
"King" and "queen" cluster; "run" and "ran" cluster.
Embeddings learn to cluster similar concepts together—but how do we verify this, and what techniques let us explore high-dimensional spaces? This is the domain of unsupervised learning.
Learning to find structure, patterns, or groupings in data without explicit labels. The algorithm discovers organization that wasn't provided by humans.
Groups data points into K clusters based on similarity (distance in embedding space).
from sklearn.cluster import KMeans
# Cluster word embeddings
kmeans = KMeans(n_clusters=10)
clusters = kmeans.fit_predict(embeddings)
# Words in same cluster are semantically related!Why it matters: If embeddings are good, K-Means on word vectors will produce sensible semantic groups (animals, verbs, places) automatically.
Embeddings live in high dimensions (300-768+). To visualize them, we project to 2D/3D while preserving structure.
from sklearn.manifold import TSNE
import matplotlib.pyplot as plt
# Reduce 512-dim embeddings to 2D
tsne = TSNE(n_components=2, perplexity=30)
embeddings_2d = tsne.fit_transform(embeddings)
# Plot—semantic clusters should be visible!
plt.scatter(embeddings_2d[:, 0], embeddings_2d[:, 1])Good embeddings should pass the "clustering test": run t-SNE + K-Means on your embeddings. If semantically similar items cluster together without any label information, your embeddings have captured meaningful structure.
king - man + woman ≈ queen
This isn't cherry-picked magic—it's a consequence of how embeddings encode relationships. The vector from "man" to "king" captures "royalty", and adding it to "woman" lands near "queen".
The "capital-of" relationship is encoded as a consistent direction in embedding space.
Grammatical relations (present → present participle) also have consistent vector offsets.
Comparative forms are encoded systematically.
These patterns emerge from statistics—the network was never explicitly told about capitals, grammar, or comparatives. Regularities in language use become regularities in embedding geometry.
The most common similarity measure in embedding spaces is cosine similarity—the cosine of the angle between vectors, ignoring magnitude.
| Measure | Formula | Properties | Use Case |
|---|---|---|---|
| Cosine Similarity | A·B / (||A||||B||) | Direction-only, scale-invariant | Text, semantic search |
| Euclidean Distance | ||A - B|| | Considers magnitude | Spatial data, clustering |
| Dot Product | A · B | Unbounded, considers magnitude | Recommendation, attention |
Static embeddings: one vector per word, regardless of context. "Bank" has the same embedding in "river bank" and "bank account".
Global Vectors: combines local context (like Word2Vec) with global co-occurrence statistics. Often performs better on analogy tasks.
Contextual embeddings: different vectors for the same word in different contexts. "Bank" now depends on surrounding words.
Deep contextual embeddings: each token's representation is computed by attention over all other tokens. The foundation of modern LLMs.
Embeddings capture statistical co-occurrence patterns, not true understanding. They can produce the "right" answer to analogies while encoding problematic biases or failing on reasoning tasks.
The accurate framing: Embeddings are useful statistical summaries of language use. They capture patterns humans created but don't represent human-like comprehension or reasoning.
The king-queen analogy is famous because it works well. Many other analogies fail or give nonsensical results. Results depend heavily on training data and are often brittle.
The accurate framing: Vector arithmetic works for some relationships that are consistently expressed in training data. It's a useful property, not a universal reasoning mechanism.
Very high dimensions increase computational cost and can lead to overfitting. There's typically diminishing returns beyond a certain dimension.
The accurate framing: Optimal embedding dimension depends on vocabulary size, task complexity, and available data. Common choices range from 50-1024, with diminishing returns beyond task-specific optima.
Explore word embeddings in 2D. See how similar words cluster together and experiment with vector arithmetic.
Note: This is a simplified 2D projection. Real embeddings exist in 50-1000+ dimensions.
What is the main advantage of embeddings over one-hot encoding?
How does Word2Vec learn word embeddings?
Why does "king - man + woman ≈ queen" work in embedding space?
What is the key difference between Word2Vec and BERT embeddings?
Why is cosine similarity commonly used for embeddings?