Modern Architectures→Module 3.4
Modern Architectures ~55 min

Large Language Models

Understanding the systems behind GPT, Claude, and Gemini: how scale, pre-training objectives, and alignment techniques produce the most capable AI systems ever created.

Learning Objectives

  • Understand what makes a language model "large" and why scale matters
  • Explain pre-training objectives: next-token prediction and masked language modeling
  • Distinguish pre-training, fine-tuning, and alignment (RLHF, Constitutional AI)
  • Understand emergent capabilities and why they appear at scale
  • Recognize limitations: hallucinations, context windows, reasoning failures
  • Explain Chain-of-Thought prompting and Process Reward Models for reasoning
  • Understand efficient architectures: Mixture of Experts (MoE) and sparse attention

What is a Large Language Model?

Definition

A Large Language Model (LLM) is a neural network trained on massive text corpora to predict text. "Large" refers to both parameter count (billions) and training data (trillions of tokens). The scale enables capabilities that don't appear in smaller models.

What They Actually Do

At their core, LLMs predict the next token given previous tokens. That's it. GPT-4 answering your question is fundamentally running: "Given this prompt, what token is most likely next?" repeated thousands of times.

The remarkable finding: this simple objective, at sufficient scale, produces systems that appear to "understand" language, "reason" about problems, and "know" factsβ€”even though they have no explicit world model.

Scale Numbers
GPT-2 (2019) 1.5B parameters
GPT-3 (2020) 175B parameters
GPT-4 (2023) ~1.8T parameters (est.)
Training tokens ~10T+ tokens
Why This Matters

Scale isn't just more of the sameβ€”it unlocks qualitatively new capabilities. A 100M parameter model can't follow complex instructions. A 100B parameter model can. Understanding why requires understanding scaling laws and emergence.

The Training Pipeline

Modern LLMs go through multiple training phases, each with different objectives and data. This multi-stage approach produces models that are both capable and (relatively) aligned with human preferences.

1

Pre-training

Objective: Predict next token

Train on internet-scale text (books, websites, code, papers). The model learns language structure, facts, reasoning patternsβ€”everything encoded in its training data.

Data: Trillions of tokens, diverse sources

Cost: Millions of dollars, weeks of compute

2

Supervised Fine-Tuning (SFT)

Objective: Learn to follow instructions

Train on human-written examples of helpful responses. Converts a "text predictor" into an "assistant" that understands prompts and generates appropriate responses.

Data: ~100K high-quality instruction/response pairs

3

Alignment (RLHF / Constitutional AI)

Objective: Match human preferences

Use human feedback or AI feedback to further refine behavior. Train a reward model on human preferences, then optimize the LLM to maximize reward while staying close to the SFT model.

Data: Human comparisons of response quality

Pre-training: The Foundation

β—ˆ

Next-Token Prediction

The pre-training objective is deceptively simple: given tokens [t₁, tβ‚‚, ..., tβ‚™], predict tβ‚™β‚Šβ‚. The model is trained on every position in every document, learning to predict each token from its context.

L = -Ξ£ log P(tα΅’ | t₁, ..., tᡒ₋₁)
Cross-entropy loss: negative log probability of the correct next token

Why Next-Token Prediction Works

To predict the next word accurately, the model must:

  • Learn syntax: Grammar, word order, agreement
  • Learn semantics: Word meanings, relationships, context
  • Learn facts: "The capital of France is..." β†’ Paris
  • Learn reasoning: "If A implies B, and A, then..." β†’ B

The objective forces the model to build internal representations that capture everything relevant to predictionβ€”which turns out to be a lot.

Scaling Laws and Emergence

A remarkable discovery: LLM capabilities follow predictable scaling laws. Loss decreases smoothly with more parameters, data, and compute. But some capabilities appear suddenly at specific scalesβ€”these are emergent abilities.

Scaling Laws

Predictable improvement

Test loss follows a power law: L ∝ N^(-α) where N is parameters. Doubling parameters gives consistent (though diminishing) improvement.

Chinchilla scaling: Optimal to scale parameters and data together. A 70B model trained on 1.4T tokens beats a 280B model trained on 300B tokens.

Emergent Abilities

Sudden capability jumps

Some capabilities appear suddenly at scale thresholds:

  • Multi-step arithmetic: ~10B parameters
  • Chain-of-thought reasoning: ~100B parameters
  • In-context learning: ~1B parameters

Debate: Are these true emergent abilities or artifacts of how we measure? Recent work suggests many "emergent" abilities may be gradual improvements that cross evaluation thresholds.

Alignment: Making LLMs Helpful and Safe

The Alignment Problem

Pre-trained LLMs predict likely textβ€”but likely text isn't always helpful, truthful, or safe. Alignment refers to techniques that shape model behavior to match human values and intentions.

β—ˆ

Why Reinforcement Learning?

Unlike supervised learning (learning from labeled examples), alignment requires learning from preferences. RL provides the framework: an agent (the LLM) takes actions (generating tokens) in an environment (the conversation) to maximize a reward signal (human preference scores).

Reinforcement Learning Fundamentals

Agent The LLM that generates responses
Environment The prompt and conversation context
Action Each token generated (or full response)
Reward Score from reward model (trained on human preferences)
Policy The LLM's strategy for generating responses

Goal: Adjust the policy (LLM weights) to maximize expected reward while staying close to the pre-trained model (to preserve capabilities).

RLHF (Reinforcement Learning from Human Feedback)

The standard technique for aligning LLMs (Ouyang et al., 2022, "Training language models to follow instructions with human feedback").

1
Collect Comparisons Human raters rank responses: "Response A is better than B"
2
Train Reward Model Learn to predict human preferences from comparison data
3
RL Optimization (PPO) Fine-tune LLM to maximize reward model scores
Conceptual PyTorch
# RLHF training loop (simplified)
for prompt in prompts:
    response = llm.generate(prompt)
    reward = reward_model(prompt, response)
    
    # PPO update: increase probability of 
    # high-reward responses
    loss = -reward * log_prob(response)
    loss.backward()
    optimizer.step()

Used by: ChatGPT, GPT-4, Claude 1.x, Gemini

Constitutional AI (CAI)

Anthropic's approach to reduce reliance on human labeling by using AI feedback.

1
Define Constitution Set of principles (be helpful, honest, harmless)
2
Self-Critique Model critiques and revises its own outputs using constitution
3
RLAIF RL from AI Feedbackβ€”train on revised outputs

Used by: Claude 2, Claude 3. Scales better than human labeling.

DPO (Direct Preference Optimization)

A simpler alternative to RLHF that skips the reward model entirely.

1
Collect Comparisons Same as RLHF: pairs of (better, worse) responses
2
Direct Optimization Directly increase probability of preferred response vs rejected

Advantage: Simpler, more stable training. No separate reward model needed.

The Alignment Tax

Alignment often reduces raw capabilities. A model optimized for "being helpful" may be less willing to engage with edge cases, leading to overly cautious behavior. Balancing safety with capability is an active research area.

Limitations and Failure Modes

Despite impressive capabilities, LLMs have fundamental limitations. Understanding these is crucial for appropriate use and for research progress.

Hallucinations

LLMs confidently generate false informationβ€”invented facts, fake citations, incorrect reasoning. This occurs because the training objective is "predict likely text," not "predict true text."

Severity: High in factual domains

Limited Context Window

Models can only "see" a fixed number of tokens (4K-200K typically). Long documents, extended conversations, or complex codebases exceed this limit.

Mitigation: RAG, extended context methods

Reasoning Limitations

LLMs can fail on multi-step reasoning, especially novel problems not similar to training data. They often "pattern match" rather than reason.

Mitigation: Chain-of-thought, tool use

No True Understanding

LLMs learn correlations in text, not grounded world knowledge. They can describe how to ride a bike without understanding balance or motion.

Open question: How much does this matter?

Training Data Cutoff

Knowledge is frozen at training time. Events after cutoff are unknown unless provided in context or via retrieval augmentation.

Mitigation: RAG, fine-tuning, tool use

Prompt Sensitivity

Small changes in phrasing can dramatically change outputs. Models aren't robust to paraphrase in the way humans are.

Mitigation: Prompt engineering, ensembles

Reasoning Models: Beyond Pattern Matching

Standard LLMs generate tokens auto-regressively without explicit "thinking." Recent advances introduce inference-time computeβ€”models that reason step-by-step before producing answers, sometimes called "System 2" thinking.

Chain-of-Thought (CoT) Prompting

The breakthrough: Simply asking a model to "think step by step" dramatically improves performance on reasoning tasks. This was formalized by Wei et al. (2022).

Standard Prompting

Q: Roger has 5 tennis balls. He buys 2 more cans of 3. How many does he have? A: 11 ❌

Chain-of-Thought

Q: Roger has 5 tennis balls. He buys 2 more cans of 3. How many does he have? A: Roger starts with 5 balls. 2 cans Γ— 3 balls = 6 balls. 5 + 6 = 11 βœ“

Why It Works

CoT decomposes complex problems into simpler sub-problems that each fall within the model's capabilities. The intermediate steps serve as a form of working memory, reducing the effective reasoning depth.

Process Reward Models (PRMs)

The limitation of outcome-based evaluation: Standard reward models score only final answers. PRMs instead evaluate each reasoning step, providing fine-grained feedback.

Outcome vs Process Reward
ORM: Solution β†’ Final Answer β†’ βœ“/βœ—
PRM: Step 1 βœ“ β†’ Step 2 βœ“ β†’ Step 3 βœ— β†’ Backtrack

Modern reasoning systems (like OpenAI o1, DeepSeek-R1) combine:

  • Test-time compute scaling: More inference computation β†’ better answers
  • Tree search: Exploring multiple reasoning paths
  • PRMs: Evaluating which paths are promising
  • Self-correction: Identifying and fixing errors mid-reasoning

πŸ“š Key Literature

  • Wei et al. (2022) β€” "Chain-of-Thought Prompting Elicits Reasoning in LLMs"
  • Lightman et al. (2023) β€” "Let's Verify Step by Step" (PRMs)
  • OpenAI (2024) β€” "Learning to Reason with LLMs" (o1 technical report)

Efficient Architectures: Scaling Sustainably

As models grow to hundreds of billions of parameters, efficiency becomes critical. Two key innovations enable practical deployment: Mixture of Experts (MoE) and Sparse Attention.

Mixture of Experts (MoE)

The core idea: Instead of activating all parameters for every input, route tokens to specialized "expert" subnetworks. This enables massive total parameters with constant compute per token.

MoE Layer Architecture
Input Token
     β”‚
     β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚   Router    β”‚ β†’ Softmax over experts
β”‚  (Gating)   β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
     β”‚ Top-K selection
     β–Ό
β”Œβ”€β”€β”€β”€β”¬β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”
β”‚ E1 β”‚ E2 β”‚ E3 β”‚ E4 β”‚  ← Experts (only K activated)
β””β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”˜
     β”‚ Weighted combination
     β–Ό
  Output
8Γ— Total params vs dense
~2Γ— Active params per token
Top-2 Typical expert selection

Examples: GPT-4 (rumored), Mixtral 8x7B, Switch Transformer, Grok, DeepSeek-MoE. Mixtral 8x7B has 47B total parameters but only 13B active per forward pass, matching 70B dense model quality.

Sparse Attention

The bottleneck: Standard attention is O(nΒ²) in sequence length, limiting context windows. Sparse patterns reduce this to O(n) or O(n log n).

Sliding Window

Each token attends only to local neighbors. Used in Mistral, Longformer.

O(n Γ— w)

Dilated/Strided

Attend to every k-th token, expanding receptive field exponentially.

O(n Γ— w)

Global + Local

Some tokens (CLS, delimiters) attend globally; others locally.

O(n Γ— (w + g))

Linear Attention

Reformulate attention to avoid explicit NΓ—N matrix. Used in RWKV, Mamba.

O(n Γ— d)

Modern Hybrid Approaches

State-of-the-art models often combine techniques: sliding window for most layers, full attention at key layers, and hardware-optimized implementations (FlashAttention) that make full attention practical up to 128K tokens.

πŸ“š Key Literature

  • Shazeer et al. (2017) β€” "Outrageously Large Neural Networks: The Sparsely-Gated MoE Layer"
  • Fedus et al. (2022) β€” "Switch Transformers: Scaling to Trillion Parameter Models"
  • Dao et al. (2022) β€” "FlashAttention: Fast and Memory-Efficient Exact Attention"
  • Gu & Dao (2023) β€” "Mamba: Linear-Time Sequence Modeling with Selective State Spaces"

Common Misconceptions

βœ—

"LLMs understand and think like humans"

LLMs have no consciousness, goals, or genuine understanding. They are sophisticated pattern matchers that produce plausible-sounding text.

The accurate framing: LLMs capture useful statistical patterns that often correlate with meaning and reasoning. Whether this constitutes "understanding" is a philosophical question, but their internal mechanisms are fundamentally different from human cognition.

βœ—

"If an LLM says it, it must be true"

LLMs are optimized for plausibility, not truth. They confidently produce false statements, especially for rare facts or complex reasoning.

The accurate framing: Treat LLM outputs as suggestions requiring verification, not authoritative answers. Use them for brainstorming, drafting, and assistanceβ€”but verify important claims independently.

βœ—

"Bigger models are always better"

Scale helps, but efficiency matters too. Smaller, well-trained models often outperform larger poorly-trained ones. Cost, latency, and deployment constraints matter in practice.

The accurate framing: Optimal model size depends on training data, task requirements, and deployment constraints. Chinchilla scaling suggests data-efficient training often beats parameter scaling.

βš— Interactive Lab: LLM Concepts Explorer

Explore key LLM concepts interactively. Understand tokenization, temperature sampling, and how context affects generation.

Tokenization splits text into subword units. Common words are single tokens; rare words are split. This is why LLMs struggle with character-level tasks.

1.0
Temperature controls randomness. Low = deterministic, high = creative/random.

Key Observations

  • Tokenization matters: "ChatGPT" might be 1 token while "Llama" is 2.
  • Temperature = 0: Always picks most likely token (deterministic).
  • Temperature > 1: Flattens distribution, more random choices.

βœ“ Check Your Understanding

1

What is the fundamental objective during LLM pre-training?

2

What is RLHF used for?

3

Why do LLMs hallucinate (make up false information)?

4

What does higher "temperature" in sampling do?

5

What is an "emergent ability" in LLMs?

0 / 5

Previous ← Transformer Architecture Continue Learning Back to Curriculum β†’