Attention Mechanisms
The breakthrough that enabled transformers: how attention allows models to dynamically focus on relevant parts of the input, enabling long-range dependencies and the foundation of modern language models.
Learning Objectives
- Understand attention as dynamic, input-dependent weighting
- Explain the Query-Key-Value framework
- Compute scaled dot-product attention step by step
- Understand self-attention and why it enables transformers
- Distinguish single-head from multi-head attention
Why Attention?
Before attention, sequence models (RNNs, LSTMs) processed inputs step-by-step, compressing everything into a fixed-size hidden state. Long-range dependencies were lost—the model couldn't "look back" at earlier tokens when needed.
When you read "The cat sat on the mat because it was tired", understanding "it" requires looking back at "cat"—not the immediately preceding words. Attention lets the model directly access any relevant earlier token.
Think of attention as a spotlight: instead of fixed processing, the model learns to shine its spotlight on the most relevant parts of the input for each output it produces.
Attention computes a weighted average of values, where weights are determined by how well each key matches the current query:
The softmax normalizes weights to sum to 1, creating a probability distribution over positions. Higher weights mean "pay more attention to this position."
Attention is the core of transformers. In GPT, each token attends to all previous tokens to predict the next. In BERT, each token attends to all other tokens bidirectionally. This enables parallelization (unlike RNNs) and direct long-range connections.
Query, Key, Value: The QKV Framework
A Database Analogy
Think of attention like a fuzzy database lookup: the query is what you're searching for, keys are what you search against, and values are what you retrieve. Unlike exact lookup, attention returns a weighted combination of all values.
Query (Q)
"What am I looking for?"
The query represents the current position's "question" to the rest of the sequence. It's computed by projecting the input through a learned weight matrix W_Q.
Key (K)
"What do I contain?"
Keys represent what each position "offers" for matching. The dot product Q·K measures how well the query matches each key—high values mean high relevance.
Value (V)
"What do I return?"
Values are the actual content to aggregate. The attention weights determine how much of each value contributes to the output.
Why Separate Q, K, V?
Having separate projections gives the model flexibility. What makes a good "query" might differ from what makes a good "key" or what content should be retrieved. The three matrices (W_Q, W_K, W_V) are learned to optimize this.
In practice, each has dimension d_model × d_k (or d_v for values), where d_k is typically d_model / number_of_heads.
Scaled Dot-Product Attention
The most common attention variant, used in transformers. "Scaled" refers to dividing by √d_k to prevent dot products from growing too large.
Compute Attention Scores
Dot product of query with all keys:
Shape: (seq_len, seq_len) for self-attention
Scale
Divide by √d_k to stabilize gradients:
Prevents softmax saturation for large d_k
Softmax
Normalize to get attention weights:
Each row sums to 1 (probability distribution)
Weighted Sum of Values
Aggregate values using attention weights:
Each position gets a weighted combination of all values
Self-Attention
In self-attention, the queries, keys, and values all come from the same sequence. Each position attends to all other positions (including itself) in the same input.
Why Self-Attention is Powerful
- Direct connections: Any two positions can interact in one step (vs. O(n) for RNNs)
- Parallelizable: All positions computed simultaneously (unlike sequential RNNs)
- Dynamic: Connections are input-dependent, not fixed by architecture
- Interpretable: Attention weights show what the model focuses on
Self-Attention in Action
For the sentence "The cat sat on the mat":
When computing the representation for "sat", self-attention might assign:
- "The": 0.05 (determiner, low relevance)
- "cat": 0.40 (subject doing the sitting)
- "sat": 0.30 (the word itself)
- "on": 0.15 (preposition, related)
- "the": 0.02 (determiner)
- "mat": 0.08 (location)
Multi-Head Attention
Instead of single attention, transformers use multiple attention heads in parallel. Each head learns to focus on different types of relationships.
Why Multiple Heads?
A single attention head might learn syntactic relationships (subject-verb). With multiple heads, one might learn syntax, another semantics, another coreference, etc. The model combines these diverse perspectives.
Typical configurations: GPT-2 small uses 12 heads with d_k = 64 each, totaling 768 dimensions. GPT-3 uses 96 heads with d_k = 128, totaling 12,288 dimensions.
Common Misconceptions
"Attention is like human attention"
The name is metaphorical. Human attention is selective and exclusive; neural attention computes weighted combinations of everything simultaneously.
The accurate framing: Attention is a differentiable mechanism for computing dynamic weighted averages. It's "soft" attention that includes all positions with varying weights, not human-like focus.
"Attention weights show what the model 'understands'"
Attention weights show where information flows, but this doesn't mean high attention implies understanding or that low attention means irrelevance.
The accurate framing: Attention weights are useful for visualization but should be interpreted carefully. The model's "understanding" is distributed across all weights and activations, not localized to attention patterns.
Interactive Lab: Attention Visualizer
See attention in action. Watch how attention weights distribute across a sequence and how queries match with keys.
Key Observations
- Attention is sparse in practice: Most weights are near zero; only a few positions get high attention.
- Context matters: The same word gets different attention patterns depending on surrounding words.
- Positional relationships: Adjacent words often attend to each other, but long-range connections also appear.
Check Your Understanding
In attention, what do the Query, Key, and Value represent?
Why do we scale the dot product by √d_k?
What is special about self-attention?
Why do transformers use multiple attention heads?
What is the output of the softmax operation in attention?