Linear Algebra for ML

Machine learning runs on linear algebra. A dataset is a matrix, a model’s weights are a matrix, and a prediction is a matrix multiplication. You need a working feel for a handful of ideas.

Vectors

A vector is a list of numbers, for example a house described by its features:

$$\mathbf{x} = \begin{bmatrix} 1200 \\ 3 \\ 2 \\ 15 \end{bmatrix} \quad (\text{sq.ft, bedrooms, bathrooms, age})$$

Geometrically it’s a point (or an arrow) in $n$-dimensional space. Key operations:

• Addition and scaling, element by element.
• Norm (length): $\lVert \mathbf{x} \rVert_2 = \sqrt{\sum_i x_i^2}$. The L1 norm $\lVert \mathbf{x} \rVert_1 = \sum_i |x_i|$ appears in Lasso regularisation.
• Dot product: $\mathbf{a} \cdot \mathbf{b} = \sum_i a_i b_i = \lVert\mathbf{a}\rVert\,\lVert\mathbf{b}\rVert \cos\theta$.

The dot product is everywhere. A linear model’s prediction is a dot product of weights and features:

$$\hat{y} = \mathbf{w} \cdot \mathbf{x} + b = w_1 x_1 + w_2 x_2 + \dots + w_n x_n + b$$

Cosine similarity, $\frac{\mathbf{a}\cdot\mathbf{b}}{\lVert\mathbf{a}\rVert\lVert\mathbf{b}\rVert}$, measures how aligned two vectors are regardless of length. It is the standard way to compare embeddings (word, sentence and image vectors) in search and recommendations.

Matrices

A matrix is a grid of numbers. A dataset with $m$ examples and $n$ features is an $m \times n$ matrix $X$: row $i$ is example $i$.

Matrix–vector multiplication applies a linear model to every example at once:

$$\hat{\mathbf{y}} = X\mathbf{w} + b \qquad (m \times n)(n \times 1) = (m \times 1)$$

Matrix multiplication $C = AB$: entry $C_{ij}$ is the dot product of row $i$ of $A$ and column $j$ of $B$. Shapes must line up: $(m \times n)(n \times p) = (m \times p)$. It is not commutative: $AB \neq BA$ in general.

A neural network layer is exactly this: $\mathbf{h} = \sigma(W\mathbf{x} + \mathbf{b})$. GPUs exist in ML because they do huge matrix multiplications fast.

Transpose, identity, inverse

• Transpose $A^\top$: flip rows and columns. $(AB)^\top = B^\top A^\top$.
• Identity $I$: ones on the diagonal; $AI = A$.
• Inverse $A^{-1}$: $AA^{-1} = I$. It exists only for square matrices with non-zero determinant.

These give the closed-form solution of linear regression (the normal equation):

$$\mathbf{w}^* = (X^\top X)^{-1} X^\top \mathbf{y}$$

In practice we rarely invert matrices (it is slow and numerically unstable); we use gradient descent or solvers. The formula is still a classic interview question.

Eigenvectors and eigenvalues

For a square matrix $A$, an eigenvector $\mathbf{v}$ only gets scaled (not rotated) by $A$:

$$A\mathbf{v} = \lambda \mathbf{v}$$

$\lambda$ is its eigenvalue. Why ML cares: in PCA, the eigenvectors of the data’s covariance matrix are the directions of greatest variance, and their eigenvalues say how much variance each captures. Keeping the top few compresses the data with minimal information loss.

SVD (singular value decomposition) generalises this to any matrix, $A = U\Sigma V^\top$. It powers PCA, recommendation systems (matrix factorisation) and low-rank tricks like LoRA fine-tuning.

Shapes: the habit that saves hours

Most ML bugs are shape bugs. Write shapes next to every expression: