Develop intuition for how numbers behaveβfrom basic types to growth patterns and the massive scales you'll encounter in AI. Understanding numbers deeply prevents common mistakes.
Different types of numbers have different properties. Understanding these helps you avoid bugs and understand AI systems better.
Real Numbers
(all numbers on a line)
β
βββββββββββββββββββββ΄ββββββββββββββββββββ
β β
Rational Numbers Irrational Numbers
(can be written as (infinite, non-repeating)
fractions) β
β Ο, β2, e
βββββββ΄ββββββ
β β
Integers Fractions
...,-2,-1, (1/2, 3/4,
0,1,2,... -2/3,...)
β
βββ Negative: -1, -2, -3
βββ Zero: 0
βββ Positive (Natural Numbers): 1, 2, 3...
| Type | Examples | AI Use Case |
|---|---|---|
| Integers | -3, 0, 42, 1000 | Counting (epochs, layers, batch size) |
| Floats | 3.14, -0.001, 2.5e-8 | Weights, probabilities, loss values |
| Booleans | True (1), False (0) | Labels, masks, conditions |
1/2 = 0.5 = 50%
1/4 = 0.25 = 25%
3/4 = 0.75 = 75%
1/3 β 0.333... (repeating)
1/10 = 0.1 = 10%
In AI, we usually work with decimals (floats), but understanding fractions helps with proportions and probabilities.
Computers can't represent all decimals exactly. 0.1 + 0.2 doesn't equal
0.3 in most programming languagesβit equals 0.30000000000000004!
This is why AI uses tolerance-based comparisons, not exact equality.
Neural networks store weights as floats, often with limited precision (float16, float32). Understanding that small rounding errors accumulate helps you debug numerical instabilityβa common issue in deep learning.
How things grow matters enormously in AI. Understanding linear vs. exponential growth helps you predict training times, model sizes, and computational costs.
Linear growth adds the same amount each step. It's steady and predictable.
Linear: Add 10 each time
Step 0: 10 β’
Step 1: 20 β’ β’
Step 2: 30 β’ β’ β’
Step 3: 40 β’ β’ β’ β’
Step 4: 50 β’ β’ β’ β’ β’
Pattern: 10, 20, 30, 40, 50...
Formula: y = 10x
Exponential growth multiplies by the same factor each step. It starts slow but explodes.
Exponential: Multiply by 2 each time
Step 0: 1 β’
Step 1: 2 β’ β’
Step 2: 4 β’ β’ β’ β’
Step 3: 8 β’ β’ β’ β’ β’ β’ β’ β’
Step 4: 16 β’ β’ β’ β’ β’ β’ β’ β’ β’ β’ β’ β’ β’ β’ β’ β’
Pattern: 1, 2, 4, 8, 16, 32, 64, 128...
Formula: y = 2Λ£
After 10 steps: Linear (adding 10) = 100. Exponential (doubling) = 1,024.
After 20 steps: Linear = 200. Exponential = 1,048,576 (over a million!).
After 30 steps: Linear = 300. Exponential = 1,073,741,824 (over a billion!).
β’ Pay: $100/hour (more hours = more pay)
β’ Distance: 60 mph (more time = more miles)
β’ Cost: $5/item (more items = higher cost)
β’ Training: loss decreases by fixed amount
β’ Population: doubles every generation
β’ Compound interest: 7% annually
β’ Viral spread: each person infects 2
β’ Model sizes: 2x each year (GPT growth)
Model parameters have grown exponentially: GPT-1 (117M) β GPT-2 (1.5B) β GPT-3 (175B). Training costs grow with model size. Understanding exponential growth helps you grasp why bigger models are so much more expensiveβit's not 10x harder, it's 100x or 1000x harder.
AI deals with numbers spanning many orders of magnitudeβfrom tiny probabilities (0.0000001) to billions of parameters. Scientific notation helps us handle these extremes.
Scientific notation writes numbers as: a Γ 10βΏ where 1 β€ a < 10
1,000 = 10Β³ = 1e3 (thousand)
1,000,000 = 10βΆ = 1e6 (million)
1,000,000,000 = 10βΉ = 1e9 (billion)
175,000,000,000 = 1.75Γ10ΒΉΒΉ = 1.75e11 (GPT-3 parameters)
0.1 = 10β»ΒΉ = 1e-1
0.01 = 10β»Β² = 1e-2
0.001 = 10β»Β³ = 1e-3 (common learning rate)
0.00001 = 10β»β΅ = 1e-5 (smaller learning rate)
0.0000001 = 10β»β· = 1e-7
| Quantity | Value | Scientific |
|---|---|---|
| Typical learning rate | 0.001 | 10β»Β³ |
| Batch size | 32-512 | ~10Β² |
| MNIST images | 60,000 | 6Γ10β΄ |
| ImageNet images | 14,000,000 | 1.4Γ10β· |
| GPT-2 parameters | 1,500,000,000 | 1.5Γ10βΉ |
| GPT-4 (estimated) | ~1,000,000,000,000 | ~10ΒΉΒ² |
| Training tokens (LLMs) | ~1,000,000,000,000 | ~10ΒΉΒ² |
When numbers differ by an "order of magnitude," one is about 10Γ the other.
β’ 100 vs 1,000: 1 order of magnitude difference
β’ 100 vs 10,000: 2 orders of magnitude
β’ 1 million vs 1 billion: 3 orders of magnitude
This helps you quickly assess: "Is this change significant?"
Scale Comparison (each step = 10Γ)
β 10β° β 10ΒΉ β 10Β² β 10Β³ β 10βΆ β 10βΉ β 10ΒΉΒ² β
β 1 β 10 β 100 β 1K β 1M β 1B β 1T β
β β β β β β β β
β one β ten β β kiloβmega βgiga β tera β
Examples:
β’ Seconds in a day: ~10β΅ (86,400)
β’ Neurons in brain: ~10ΒΉΒΉ (100 billion)
β’ Stars in galaxy: ~10ΒΉΒΉ (100-400 billion)
β’ Atoms in human body: ~10Β²βΈ
AI models are approaching brain-scale parameter counts
When someone says a model has "1.5B parameters," you should instantly know that's 1.5 billion = 1.5Γ10βΉ. When a learning rate is "1e-4," that's 0.0001. This fluency lets you read papers, compare models, and tune hyperparameters effectively.
Continue building your mathematical foundation: