Neural networks are built on matrix operations. Every layer, every transformation, every attention mechanism is linear algebra at its core. Understanding vectors and matrices unlocks the language of modern AIβfrom embeddings to transformers.
Everything in deep learning is a tensorβa multi-dimensional array of numbers. Images, text, audio, model weightsβall are represented as tensors. Linear algebra is the mathematical language for manipulating these structures.
Images: [batch, channels, height, width] e.g., [32, 3, 224, 224]
Text: [batch, sequence_length, embedding] e.g., [16, 512, 768]
Audio: [batch, channels, samples] e.g., [8, 1, 16000]
Weights: [input_features, output_features] e.g., [768, 3072]
A single neural network layer is literally: y = Wx + b β a matrix multiplication
plus a bias. When you hear "GPT has 175 billion parameters," those are 175 billion numbers
stored in matrices. Modern GPUs exist specifically because they're incredibly fast at
matrix operations.
Matrices let us apply the same transformation to many inputs at once (parallelization). Instead of processing one data point, we process a whole batch. This is why AI models run efficiently on GPUsβthey're designed for parallel matrix operations.
A vector is an ordered list of numbers. In AI, vectors represent practically everything: words, images, features, gradients, and model parameters. The key insight is that vectors live in a space where geometric intuitions apply.
β y
β
4 βΌββββββββββββββ v = [3, 4]
β β±
β β±
β β± β Length (magnitude) = β(3Β² + 4Β²) = 5
β β±
β β±
β β± β Direction = arctan(4/3) β 53Β°
βββΌββββββββββββββ x
0 3
A 2D vector has both magnitude (length) and direction.
AI uses vectors with hundreds or thousands of dimensions!
A vector represents a point in space or a direction with magnitude
[1, 2, 3] + [4, 5, 6] = [5, 7, 9]
Add corresponding elements. Geometrically: chain the arrows head-to-tail.
2 Γ [1, 2, 3] = [2, 4, 6]
Multiply each element. Geometrically: stretch or shrink the vector.
[5, 7, 9] - [1, 2, 3] = [4, 5, 6]
Subtract corresponding elements. Gives the "difference" vector.
[1, 2, 3] β [4, 5, 6] = [4, 10, 18]
Multiply corresponding elements. Also called Hadamard product.
In NLP, every word is represented as a vector (typically 256-1024 dimensions).
Similar words have similar vectors. The famous example:
vector("king") - vector("man") + vector("woman") β vector("queen")
This works because vector arithmetic captures semantic relationships!
The dot product (also called inner product or scalar product) is perhaps the most important operation in AI. It takes two vectors and returns a single number that measures how "aligned" they are.
a = [1, 2, 3]
b = [4, 5, 6]
a Β· b = (1 Γ 4) + (2 Γ 5) + (3 Γ 6)
= 4 + 10 + 18
= 32
Cosine is a function that converts an angle into a number between -1 and +1.
It measures "how much two directions align":
β’ cos(0Β°) = 1 β same direction (perfect alignment)
β’ cos(90Β°) = 0 β perpendicular (no alignment)
β’ cos(180Β°) = -1 β opposite directions
You don't need to calculate angles directlyβthe formula above does it for you.
Just remember: cosine β "how aligned are these vectors?"
This formula reveals what the dot product actually measures:
Same direction Perpendicular Opposite direction
(a Β· b > 0) (a Β· b = 0) (a Β· b < 0)
β b β b β a
β± β
β± β
β a β a β b
"Similar" "Unrelated" "Dissimilar"
The dot product measures alignment between vectors
To measure similarity independent of vector length, we normalize by the magnitudes:
a = [1, 2, 3]
b = [4, 5, 6]
|a| = β(1Β² + 2Β² + 3Β²) = β14 β 3.74
|b| = β(4Β² + 5Β² + 6Β²) = β77 β 8.77
cos(ΞΈ) = 32 / (3.74 Γ 8.77) = 32 / 32.79 β 0.975
These vectors are very similar! (close to 1)
Attention Mechanisms: "How relevant is token A to token B?" is computed as
a dot product of their representations.
Semantic Search: Find documents similar to a query by comparing embedding dot products.
Recommendations: "How much will user U like item I?" is often a dot product of their embeddings.
Every neural network layer uses dot products to combine inputs with weights.
A matrix is a 2D grid of numbers arranged in rows and columns. Think of it as a collection of vectors, or as a transformation that can rotate, scale, or project vectors.
β n columns β
β β
β β aββ aββ ... aββ β
β β aββ aββ ... aββ β
m rows β ... ... ... ... β
β β aββ aββ ... aββ β
β β β
Matrix A has shape (m Γ n): m rows, n columns
Element aα΅’β±Ό is in row i, column j
A matrix is a 2D array of numbers with m rows and n columns
A β βΒ³Λ£Β² means A is a 3Γ2 matrix (3 rows, 2 columns)
β β
A = β 1 2 β Aββ = 1, Aββ = 2
β 3 4 β Aββ = 3, Aββ = 4
β 5 6 β Aββ = 5, Aββ = 6
β β
β β
I = β 1 0 0 β
β 0 1 0 β
β 0 0 1 β
β β
1s on diagonal, 0s elsewhere. A Γ I = I Γ A = A (does nothing).
β β β β
A = β 1 2 β Aα΅ = β 1 3 5 β
β 3 4 β β 2 4 6 β
β 5 6 β β β
β β
Rows become columns, columns become rows. Shape (mΓn) β (nΓm).
β β
0 = β 0 0 0 β
β 0 0 0 β
β β
All elements are zero. A + 0 = A.
β β
D = β dβ 0 0 β
β 0 dβ 0 β
β 0 0 dβ β
β β
Non-zero only on diagonal. Scales each dimension independently.
A key insight: multiplying a vector by a matrix transforms that vector. Different matrices perform different transformations: rotation, scaling, projection, etc.
Original vectors: After transformation by A:
β y β y
β β β β
β β± β β±
β β β β
β ββ
ββββββββ x ββββββββ x
A matrix can rotate, stretch, shrink, flip, or shear vectors.
Matrices transform vectors in various ways
Each layer in a neural network has a weight matrix W. When input x passes through,
the matrix transforms it: z = Wx. The network learns what
transformation to apply by adjusting the values in W during training. A "7 billion
parameter model" has 7 billion numbers distributed across many weight matrices.
Matrix multiplication is THE fundamental operation in deep learning. Every neural network layer, every attention head, every linear transformation is a matrix multiplication. Master this, and you understand the computational heart of AI.
A (2Γ3) B (3Γ2) C (2Γ2)
β β β β β β
β 1 2 3 β Γ β 7 8 β = β Cββ Cββ β
β 4 5 6 β β 9 10 β β Cββ Cββ β
β β β 11 12 β β β
β β
Cββ = [1,2,3] Β· [7,9,11] = 1Γ7 + 2Γ9 + 3Γ11 = 58
Cββ = [1,2,3] Β· [8,10,12] = 1Γ8 + 2Γ10 + 3Γ12 = 64
Cββ = [4,5,6] Β· [7,9,11] = 4Γ7 + 5Γ9 + 6Γ11 = 139
Cββ = [4,5,6] Β· [8,10,12] = 4Γ8 + 5Γ10 + 6Γ12 = 154
β β
C = β 58 64 β
β 139 154 β
β β
Each element of C comes from a dot product of a row from A and column from B
Shape mismatches cause most errors in neural network code. If A is (64, 512) and B is (256, 128),
you cannot multiply themβthe inner dimensions (512 and 256) don't match.
Always check shapes: print(tensor.shape) is your best debugging tool.
Can we multiply? What's the result shape?
(3 Γ 4) Γ (4 Γ 2) = (3 Γ 2) β Valid, inner = 4
(5 Γ 3) Γ (3 Γ 7) = (5 Γ 7) β Valid, inner = 3
(2 Γ 5) Γ (3 Γ 4) = ??? β Invalid! 5 β 3
(4 Γ 4) Γ (4 Γ 4) = (4 Γ 4) β Square matrices
You can only multiply matrices A and B if the inner dimensions match:
β’ Matrix A is (m Γ n)
β’ Matrix B is (n Γ p)
β’ The result is (m Γ p)
If the inner dimensions do not match, the multiplication is not defined.
Valid examples:
β’ (2Γ3) Β· (3Γ4) β (2Γ4) β (inner dimensions: 3 and 3 match)
β’ (5Γ2) Β· (2Γ1) β (5Γ1) β (inner dimensions: 2 and 2 match)
Invalid examples:
β’ (2Γ3) Β· (2Γ4) β β (inner dimensions: 3 and 2 do not match)
β’ (4Γ5) Β· (3Γ2) β β (inner dimensions: 5 and 3 do not match)
A special and very common case: multiplying a matrix by a vector. This is exactly what happens in a neural network layer.
Neural Network Layer: y = Wx + b
W (3Γ2) x (2Γ1) b (3Γ1) y (3Γ1)
β β β β β β β β
β wββ wβββ β xβ β β bβ β β yβ β
β wββ wβββ Γ β xβ β + β bβ β = β yβ β
β wββ wβββ β β β bβ β β yβ β
β β β β β β
yβ = wββΒ·xβ + wββΒ·xβ + bβ
yβ = wββΒ·xβ + wββΒ·xβ + bβ
yβ = wββΒ·xβ + wββΒ·xβ + bβ
Transforms 2D input β 3D output
A neural network layer is just a matrix-vector multiplication
AB β BA (in general)
Order matters! Applying transform A
then B is different from B then A.
(AB)C = A(BC)
Order of operations doesn't matter
(useful for optimization).
Matrix multiplication is "embarrassingly parallel"βeach element of the output can be computed independently. A GPU has thousands of cores that can compute these dot products simultaneously. A 4090 GPU can perform ~80 trillion operations per second, mostly matrix multiplications. This is why training large models requires GPUs.
The norm of a vector measures its "size" or length. Different norms measure size differently.
Normalizing a vector means scaling it to have length 1 (a unit vector). This is crucial for stable training and fair comparisons.
v = [3, 4]
||v||β = β(9 + 16) = 5
vΜ = [3/5, 4/5] = [0.6, 0.8]
Check: |vΜ| = β(0.36 + 0.64) = β1 = 1 β
Transformers use "layer normalization" which normalizes activations. This stabilizes training by keeping values in a reasonable range. Without normalization, values can explode (overflow) or vanish (underflow) as they pass through many layers.
Broadcasting is a NumPy/PyTorch feature that lets you perform operations on arrays with different shapes. The smaller array is "broadcast" (virtually replicated) to match the larger one. (You'll use this laterβjust know it exists for now.)
# Matrix + Vector (vector added to each row)
A = [[1, 2, 3], v = [10, 20, 30]
[4, 5, 6]]
A + v = [[11, 22, 33],
[14, 25, 36]]
# Scalar operations
A * 2 = [[2, 4, 6],
[8, 10, 12]]
# Different but compatible shapes
(3, 4, 1) + (1, 4, 5) β (3, 4, 5) β expanded to match
# Data Shapes
Image batch: (batch, channels, height, width) [32, 3, 224, 224]
Text batch: (batch, sequence_len, embed_dim) [16, 512, 768]
Audio batch: (batch, samples) [8, 16000]
# Model Parameter Shapes
Linear layer: (in_features, out_features) [768, 3072]
Embedding table: (vocab_size, embed_dim) [50000, 768]
Attention weights: (batch, heads, seq, seq) [16, 12, 512, 512]
# Named Dimensions (helpful convention)
B = batch size
T = sequence length (time)
D = embedding dimension
H = number of attention heads
V = vocabulary size
Test your understanding with these problems!
[2, -1, 3] Β· [4, 2, -1]
= (2 Γ 4) + (-1 Γ 2) + (3 Γ -1)
= 8 + (-2) + (-3)
= 3
(4 Γ 3) Γ (3 Γ 5) = (4 Γ 5)
Inner dimensions (3, 3) match β
Result has shape (4, 5)
ABββ = [1,2]Β·[5,7] = 5+14 = 19
ABββ = [1,2]Β·[6,8] = 6+16 = 22
ABββ = [3,4]Β·[5,7] = 15+28 = 43
ABββ = [3,4]Β·[6,8] = 18+32 = 50
AB = [[19, 22],
[43, 50]]
a = [1, 0], b = [1, 1]
a Β· b = 1Γ1 + 0Γ1 = 1
|a| = β(1Β² + 0Β²) = 1
|b| = β(1Β² + 1Β²) = β2 β 1.414
cos(ΞΈ) = 1 / (1 Γ 1.414) β 0.707
This corresponds to a 45Β° angle.
With these linear algebra foundations, you're prepared for the AI course. You'll use these ideas throughout: