AI systems reason under uncertainty. When a language model says "I'm 80% confident," when a classifier outputs class probabilities, when a diffusion model generates imagesโprobability theory is the foundation. Understanding it transforms AI from magic to mathematics.
The real world is uncertain. Sensors are noisy, data is incomplete, and the future is unpredictable. AI systems must handle this gracefullyโexpressing confidence levels rather than false certainty.
Image Classifier Output:
P(cat) = 0.82
P(dog) = 0.15
P(rabbit) = 0.03
Language Model (next token):
P("the") = 0.23
P("a") = 0.18
P("an") = 0.07
P(".") = 0.05
... 50,000 more tokens ...
Machine learning is fundamentally about learning probability distributions from data:
โข Classification: Learn P(class | input)
โข Language Models: Learn P(next token | context)
โข Generative Models: Learn P(data) to sample new examples
Every prediction, every generation, every decision involves probability.
Hard decisions lose information. If a medical AI says "cancer: yes" vs "cancer: 51% likely," the doctor makes very different decisions. Probabilities let downstream systems make informed choicesโa self-driving car should slow down at "30% chance of pedestrian" even if it wouldn't fully brake.
Probability measures how likely an event is to occur, on a scale from 0 (impossible) to 1 (certain). It formalizes our intuitive notion of "chance."
For equally likely outcomes:
P(rolling 4) = 1/6 โ 0.167 (one favorable, six total)
P(rolling even) = 3/6 = 0.5 (2, 4, 6 are favorable)
P(rolling < 7) = 6/6 = 1.0 (certainโall outcomes work)
P(rolling 7) = 0/6 = 0.0 (impossible with standard die)
P(not rolling 6) = 1 - P(rolling 6) = 1 - 1/6 = 5/6
P(rolling 1 or 6) = P(1) + P(6) - P(1 and 6)
= 1/6 + 1/6 - 0 (can't roll both)
= 2/6 = 1/3
A probability distribution assigns probabilities to all possible outcomes. The key constraint: all probabilities must sum to 1 (something must happen).
DISCRETE Distribution CONTINUOUS Distribution
(e.g., dice, classifications) (e.g., heights, model outputs)
P(x) p(x)
โ โ
1/6 โโโโฌโโโฌโโโฌโโโฌโโโฌโโโ โฑโฒ
โโโโโโโโโโโโโโโโโโโ โฑ โฒ
โโโโดโโโดโโโดโโโดโโโดโโโดโ x โฑ โฒ
1 2 3 4 5 6 โฑ โฒ
โฑ โฒ
All bars sum to 1.0 โฑโโโโโโโโโโโฒโโโ x
ฮผ (mean)
Area under curve = 1.0
Discrete vs continuous distributionsโboth sum/integrate to 1
Neural networks output raw numbers ("logits") that can be any value. The softmax function converts these to a valid probability distribution (positive, sums to 1). Every classification and language model uses softmax at the output.
Conditional probability is perhaps the most important concept for AI. It answers: "What's the probability of A, given that we know B happened?"
P(A|B) restricts our view to only scenarios where B is true, then asks: "What fraction of those have A?" It's like zooming in on a subset of possibilities.
All Possible Outcomes
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ โ
โ โโโโโโโโโ โ
โ โ A โ P(A) = A/total
โ โ โโโโโโผโโโโโ โ
โ โ โ AโฉBโ โ โ
โ โโโโผโโโโโ โ โ
โ โ B โ P(B) = B/total
โ โโโโโโโโโโโ โ
โ โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
P(A|B) = P(AโฉB) / P(B) โ Focus only on B region
"Of the B outcomes, what fraction are also A?"
Conditional probability zooms in on the B region
A deck of 52 cards. Draw one card.
Event A: Card is a King
Event B: Card is a face card (J, Q, K)
P(A) = 4/52 = 1/13 (4 kings in deck)
P(B) = 12/52 = 3/13 (4 each of J, Q, K)
P(A and B) = 4/52 (kings are face cards)
P(King | face card) = P(A|B) = P(A and B)/P(B)
= (4/52) / (12/52)
= 4/12 = 1/3
If we know it's a face card, there's a 1/3 chance it's a King.
Two events are independent if knowing one tells you nothing about the other.
Flip two fair coins.
P(first is heads) = 1/2
P(second is heads) = 1/2
P(both heads) = 1/2 ร 1/2 = 1/4 โ Multiply because independent
The first coin doesn't affect the second.
Many real-world events seem independent but aren't! In ML data, features are often correlated. Height and weight aren't independent. Previous words affect next-word probabilities. The "Naive" in Naive Bayes refers to assuming independence when it's not strictly true.
When a classifier outputs "P(cat | image) = 0.85," it's computing a conditional probability: "Given this image, what's the probability the label is 'cat'?" The model learned this from data where P(label | features) was observed.
Bayes' Theorem is one of the most important formulas in AI. It lets us "flip" conditional probabilitiesโif we know P(B|A), we can find P(A|B).
Bayes' Theorem Components:
P(hypothesis | data) = P(data | hypothesis) ร P(hypothesis)
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
P(data)
โ โ โ โ
POSTERIOR LIKELIHOOD PRIOR EVIDENCE
What we want: How likely is What we Normalizing
Updated belief this data if believed constant
after seeing data hypothesis true before (same for all H)
Bayes' Theorem updates our beliefs based on new evidence
Bayes' Theorem formalizes rational belief updating: Start with a prior (what you believed before). See new evidence. Update to a posterior (what you believe now). The more evidence, the more your posterior dominates your prior.
Setup:
- A disease affects 1% of the population: P(disease) = 0.01
- A test is 90% accurate:
- P(positive | disease) = 0.90 (true positive rate)
- P(positive | healthy) = 0.10 (false positive rate)
Question: If you test positive, what's P(disease | positive)?
Intuition might say 90%... but let's compute:
Step 1: Find P(positive)
P(positive) = P(pos|disease)รP(disease) + P(pos|healthy)รP(healthy)
= 0.90 ร 0.01 + 0.10 ร 0.99
= 0.009 + 0.099
= 0.108
Step 2: Apply Bayes
P(disease|positive) = P(positive|disease) ร P(disease) / P(positive)
= (0.90 ร 0.01) / 0.108
= 0.009 / 0.108
โ 0.083 (about 8%!)
Only 8% chance of disease despite positive test!
Why? The disease is rare (1%). Even a 90% accurate test produces many false positives from the large healthy population. The prior matters!
Naive Bayes classifiers directly apply Bayes' theorem.
Regularization can be seen as encoding a prior (small weights are more likely).
Bayesian neural networks maintain probability distributions over weights.
Calibration ensures that when a model says "90% confident," it's right 90% of the time.
The expected value (or expectation) is the "average" outcome you'd expect over many trials. It's the probability-weighted sum of all possible values.
E[X] = 1ร(1/6) + 2ร(1/6) + 3ร(1/6) + 4ร(1/6) + 5ร(1/6) + 6ร(1/6)
= (1 + 2 + 3 + 4 + 5 + 6) / 6
= 21/6
= 3.5
You can't roll 3.5, but over many rolls, the average converges to 3.5.
E[aX + b] = a ร E[X] + b
E[X + Y] = E[X] + E[Y]
Works even if X, Y dependent!
E[X ร Y] = E[X] ร E[Y]
Only if X, Y are independent!
Neural networks are trained to minimize expected loss: E[L(model, data)]. Since we can't compute this exactly over all possible data, we approximate using mini-batchesโaveraging the loss over a random sample. This is called Monte Carlo estimation. More batches = better estimate of true expected loss.
Variance measures how spread out values are. Low variance = values cluster near the mean. High variance = values are spread out. The standard deviation is โVar(X), which has the same units as X.
| Distribution | Use Case | Key Property |
|---|---|---|
| Bernoulli | Binary outcomes | P(success) = p, P(failure) = 1-p |
| Categorical | Multi-class classification | Generalized Bernoulli, k classes |
| Gaussian (Normal) | Continuous values, noise | Bell curve, defined by ฮผ (mean) and ฯ (std dev) |
| Uniform | Random initialization | All values equally likely in range |
The most important continuous distribution. Many natural phenomena are approximately Gaussian due to the Central Limit Theorem.
The Normal Distribution
p(x)
โ
โ โญโโโฎ
โ โฑ โฒ
โ โฑ โฒ
โ โฑ โฒ
โ โฑ 68% โฒ
โ โฑ โโโโโโโค โฒ
โ โฑ โฒ
โโโโฑโโโโโโโโโโโโโโโโโฒโโโ x
ฮผ-ฯ ฮผ ฮผ+ฯ
68% of data within 1ฯ of mean
95% of data within 2ฯ of mean
99.7% within 3ฯ of mean
The Gaussian "bell curve" is defined by mean and standard deviation
Softmax converts a vector of real numbers (logits) into a probability distribution. It's used in nearly every classification model.
Logits (raw model output): [2.0, 1.0, 0.1]
Step 1: Exponentiate
exp([2.0, 1.0, 0.1]) = [7.39, 2.72, 1.11]
Step 2: Normalize (divide by sum)
sum = 7.39 + 2.72 + 1.11 = 11.22
softmax = [7.39/11.22, 2.72/11.22, 1.11/11.22]
= [0.66, 0.24, 0.10]
Interpretation: 66% class 0, 24% class 1, 10% class 2
Cross-entropy measures how different two probability distributions are. It's the standard loss function for classification.
Cross-entropy heavily penalizes confident wrong predictions. If the true label is class A and you predict P(A) = 0.01 (very confident it's NOT A), your loss is -log(0.01) = 4.6 (high!). If you predict P(A) = 0.99 (correctly confident), loss is -log(0.99) = 0.01 (low!).
The -log(x) Penalty Curve
Loss โ
5 โ โ
โ โ
4 โ โ
โ โ
3 โ โ โ Wrong predictions = HIGH loss
โ โ
2 โ โ
โ โ
1 โ โ
โ โ
0 โโโโโโโโโโโโโโโโโโโ โ
0.0 0.2 0.4 0.6 0.8 1.0
โ โ
Confidence in correct answer
predict 0.01 โ loss = 4.6 (disaster!)
predict 0.50 โ loss = 0.7 (uncertain)
predict 0.99 โ loss = 0.01 (great!)
Loss explodes when you're confidently WRONG, drops to near-zero when confidently RIGHT
For binary classification (label y โ {0, 1}):
BCE = -[y ร log(p) + (1-y) ร log(1-p)]
Where p is the predicted probability of class 1.
If true label y=1: loss = -log(p) โ want p close to 1
If true label y=0: loss = -log(1-p) โ want p close to 0
Example:
- True label: 1, Prediction: 0.9 โ loss = -log(0.9) = 0.105
- True label: 1, Prediction: 0.1 โ loss = -log(0.1) = 2.303 (much higher!)
Almost every classification model follows this pattern:
Input โ Neural Network โ Logits โ Softmax โ Probabilities โ Cross-Entropy โ Loss
Language models use the same setupโpredicting the next token is multi-class classification
over the vocabulary!
Total balls = 3 + 5 + 2 = 10
P(green) = 2/10 = 0.2
P(not green) = 1 - P(green) = 1 - 0.2 = 0.8
Or directly: P(not green) = P(red or blue) = (3+5)/10 = 0.8
P(A|B) = P(A and B) / P(B)
= 0.12 / 0.3
= 0.4
Note: P(A|B) = P(A), so A and B are independent!
(Knowing B doesn't change probability of A)
P("free") = P("free"|spam)รP(spam) + P("free"|not spam)รP(not spam)
= 0.8 ร 0.3 + 0.1 ร 0.7
= 0.24 + 0.07
= 0.31
P(spam|"free") = P("free"|spam) ร P(spam) / P("free")
= (0.8 ร 0.3) / 0.31
= 0.24 / 0.31
โ 0.774 (77.4%)
An email containing "free" has 77% chance of being spam.
Possible outcomes (after paying $5 to play):
- Net gain of $15 (win $20 - $5 cost) with P = 0.2
- Net gain of -$5 (win nothing, lose entry fee) with P = 0.8
E[gain] = 15 ร 0.2 + (-5) ร 0.8
= 3 - 4
= -$1
On average, you lose $1 per game. (House always wins!)
P(A) โ [0, 1]
P(not A) = 1 - P(A)
P(A or B) = P(A) + P(B) - P(A and B)
P(A and B) = P(A|B) ร P(B) = P(B|A) ร P(A)
P(A|B) = P(A and B) / P(B)
Bayes: P(A|B) = P(B|A) ร P(A) / P(B)
Independence: P(A|B) = P(A) โน P(A and B) = P(A) ร P(B)
Expected Value: E[X] = ฮฃ xแตข ร P(xแตข)
Variance: Var(X) = E[Xยฒ] - E[X]ยฒ
With these probability foundations, you're prepared for the AI course. You'll see these concepts throughout: