Linear Algebra Essentials

1. Scalars, Vectors, Matrices, Tensors

Scalar — a single number. $x \in \mathbb{R}$

Vector — ordered list of $n$ numbers. $\mathbf{x} \in \mathbb{R}^n$

$\mathbf{x} = \begin{pmatrix} x_1 \\ x_2 \\ \vdots \\ x_n \end{pmatrix}$

By convention vectors are column vectors. A row vector is written $\mathbf{x}^T$.

Matrix — rectangular array of numbers. $A \in \mathbb{R}^{m \times n}$ has $m$ rows and $n$ columns.

$A = \begin{pmatrix} a_{11} & a_{12} & \cdots & a_{1n} \\ a_{21} & a_{22} & \cdots & a_{2n} \\ \vdots & & \ddots & \vdots \\ a_{m1} & a_{m2} & \cdots & a_{mn} \end{pmatrix}$

$A_{ij}$ or $a_{ij}$ denotes the element at row $i$, column $j$.

Tensor — generalization to more than 2 dimensions. A 3D tensor $\mathcal{X} \in \mathbb{R}^{m \times n \times p}$ is indexed by $\mathcal{X}_{ijk}$. Used in deep learning (e.g. a batch of images: batch × height × width × channels).

---

2. Vector Operations

Addition and scalar multiplication

$\mathbf{x} + \mathbf{y} = \begin{pmatrix} x_1 + y_1 \\ \vdots \\ x_n + y_n \end{pmatrix} \qquad \alpha \mathbf{x} = \begin{pmatrix} \alpha x_1 \\ \vdots \\ \alpha x_n \end{pmatrix}$

Dot product (inner product)

$\mathbf{x} \cdot \mathbf{y} = \mathbf{x}^T \mathbf{y} = \sum_{i=1}^n x_i y_i$

Geometric interpretation:

$\mathbf{x}^T \mathbf{y} = \|\mathbf{x}\| \|\mathbf{y}\| \cos\theta$

where $\theta$ is the angle between the two vectors.

[!NOTE] If $\mathbf{x}^T \mathbf{y} = 0$, the vectors are orthogonal (perpendicular). This is fundamental — orthogonality appears in PCA, QR decomposition, attention, everywhere.

Norms — measuring vector size

The $L^p$ norm:

$\|\mathbf{x}\|_p = \left( \sum_{i=1}^n |x_i|^p \right)^{1/p}$

Norm	Formula	Name	Used for
$\|\mathbf{x}\|_1$	$\sum \|x_i\|$	Manhattan / taxicab	L1 regularization (Lasso)
$\|\mathbf{x}\|_2$	$\sqrt{\sum x_i^2}$	Euclidean	Default distance, L2 reg
$\|\mathbf{x}\|_\infty$	$\max_i \|x_i\|$	Max norm	Gradient clipping

The squared $L^2$ norm is especially convenient:

$\|\mathbf{x}\|_2^2 = \mathbf{x}^T\mathbf{x} = \sum_{i=1}^n x_i^2$

Outer product

$\mathbf{x}\mathbf{y}^T = \begin{pmatrix} x_1 y_1 & x_1 y_2 & \cdots \\ x_2 y_1 & x_2 y_2 & \cdots \\ \vdots & & \ddots \end{pmatrix} \in \mathbb{R}^{m \times n}$

The dot product contracts two vectors to a scalar. The outer product expands two vectors into a matrix.

---

3. Matrix Operations

Transpose

$(A^T)_{ij} = A_{ji}$

Key identities:

• $(AB)^T = B^T A^T$
• $(A^T)^T = A$
• $(\mathbf{x}^T \mathbf{y}) = \mathbf{y}^T \mathbf{x}$ (scalar, so equals its own transpose)

Matrix multiplication

$C = AB$ where $A \in \mathbb{R}^{m \times k}$, $B \in \mathbb{R}^{k \times n}$, $C \in \mathbb{R}^{m \times n}$:

$C_{ij} = \sum_{l=1}^k A_{il} B_{lj} = \text{(row } i \text{ of } A) \cdot \text{(column } j \text{ of } B)$

[!WARNING] Matrix multiplication is not commutative: $AB \neq BA$ in general. It is associative: $(AB)C = A(BC)$.

Matrix-vector product $A\mathbf{x}$: think of it as taking a linear combination of the columns of $A$, weighted by entries of $\mathbf{x}$:

$A\mathbf{x} = x_1 \mathbf{a}_1 + x_2 \mathbf{a}_2 + \cdots + x_n \mathbf{a}_n$

Hadamard product (elementwise)

$C = A \odot B \qquad C_{ij} = A_{ij} B_{ij}$

Used in neural network gates (LSTM, attention masks). Not the same as matrix multiplication.

Identity matrix

$I = \begin{pmatrix} 1 & 0 & \cdots & 0 \\ 0 & 1 & \cdots & 0 \\ \vdots & & \ddots & \vdots \\ 0 & 0 & \cdots & 1 \end{pmatrix} \qquad AI = IA = A$

Inverse

For square $A \in \mathbb{R}^{n \times n}$, if it exists:

$A^{-1} A = A A^{-1} = I$

Properties:

• $(AB)^{-1} = B^{-1} A^{-1}$
• $(A^T)^{-1} = (A^{-1})^T$
• A matrix is invertible iff its determinant is nonzero iff its columns are linearly independent

[!WARNING] Never compute $A^{-1}$ explicitly if you can avoid it. Solving $Ax = b$ via np.linalg.solve(A, b) is numerically more stable and faster than np.linalg.inv(A) @ b.

---

4. Special Matrices

Symmetric

$A = A^T \qquad (A_{ij} = A_{ji})$

The covariance matrix $\Sigma$ is always symmetric. All eigenvalues of a symmetric matrix are real.

Diagonal

$D = \text{diag}(d_1, d_2, \ldots, d_n) \qquad D_{ij} = 0 \text{ if } i \neq j$

Efficient: $D\mathbf{x}$ just scales each entry. $D^{-1} = \text{diag}(1/d_1, \ldots, 1/d_n)$.

Orthogonal

$Q^T Q = I \qquad \Rightarrow \quad Q^{-1} = Q^T$

Columns of $Q$ are orthonormal (unit length, mutually orthogonal). Orthogonal matrices preserve lengths and angles — they represent rotations and reflections. Used in PCA, QR decomposition.

Positive Semi-Definite (PSD)

A symmetric matrix $A$ is PSD if:

$\mathbf{x}^T A \mathbf{x} \geq 0 \quad \forall \mathbf{x}$

Written $A \succeq 0$. Covariance matrices are always PSD. All eigenvalues of a PSD matrix are $\geq 0$.

---

5. Linear Independence, Span, Rank

A set of vectors $\{\mathbf{v}_1, \ldots, \mathbf{v}_k\}$ is linearly independent if:

$\sum_{i=1}^k \alpha_i \mathbf{v}_i = \mathbf{0} \implies \alpha_1 = \alpha_2 = \cdots = \alpha_k = 0$

i.e. no vector in the set can be written as a combination of the others.

Span — the set of all vectors reachable as linear combinations of $\{\mathbf{v}_1, \ldots, \mathbf{v}_k\}$.

Rank of matrix $A$ — the number of linearly independent columns (= linearly independent rows):

$\text{rank}(A) \leq \min(m, n)$

• Full rank: $\text{rank}(A) = \min(m,n)$ — no redundant columns/rows
• Rank deficient: some columns are linear combinations of others — $A^T A$ is not invertible, the normal equation has no unique solution

import numpy as np
A = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
print(np.linalg.matrix_rank(A))  # → 2 (rank deficient, row 3 = row1 + row2 × 2... not quite but determinant=0)

---

6. Eigenvalues and Eigenvectors

For square $A \in \mathbb{R}^{n \times n}$:

$A\mathbf{v} = \lambda \mathbf{v}$

$\lambda \in \mathbb{R}$ is an eigenvalue, $\mathbf{v} \neq \mathbf{0}$ is the corresponding eigenvector.

Interpretation: $\mathbf{v}$ is a direction that $A$ only scales (by $\lambda$) — it does not rotate.

Finding eigenvalues

$\det(A - \lambda I) = 0$

This is the characteristic polynomial of degree $n$. For large matrices, solved numerically.

Eigendecomposition

If $A$ has $n$ linearly independent eigenvectors:

$A = V \Lambda V^{-1}$

where $V$ has eigenvectors as columns, $\Lambda = \text{diag}(\lambda_1, \ldots, \lambda_n)$.

For symmetric $A$ (which includes all covariance matrices):

$A = Q \Lambda Q^T$

with $Q$ orthogonal. This is the spectral theorem. All eigenvalues are real.

eigenvalues, eigenvectors = np.linalg.eigh(cov_matrix)  # eigh for symmetric
# eigenvalues[i] corresponds to eigenvectors[:, i]

[!MATH] Why eigenvalues matter for ML:

• PCA — eigenvectors of $X^T X$ are the principal components; eigenvalues measure how much variance each direction captures
• Optimization — the curvature of the loss surface is described by the Hessian matrix; its eigenvalues tell you whether you're in a saddle point, local min, or flat region
• Graph neural networks — spectral convolutions are defined via eigendecomposition of the graph Laplacian

---

7. Singular Value Decomposition (SVD)

For any matrix $A \in \mathbb{R}^{m \times n}$ (need not be square):

$A = U \Sigma V^T$

where:

• $U \in \mathbb{R}^{m \times m}$ — orthogonal, columns are left singular vectors
• $\Sigma \in \mathbb{R}^{m \times n}$ — diagonal with singular values $\sigma_1 \geq \sigma_2 \geq \cdots \geq 0$
• $V \in \mathbb{R}^{n \times n}$ — orthogonal, columns are right singular vectors

Relationship to eigendecomposition

$A^T A = V \Sigma^T \Sigma V^T \qquad \text{(eigendecomp of } A^T A\text{)}$ $\sigma_i = \sqrt{\lambda_i(A^T A)}$

Low-rank approximation (Eckart–Young theorem)

The best rank-$k$ approximation to $A$ (in Frobenius norm) is:

$A_k = \sum_{i=1}^k \sigma_i \mathbf{u}_i \mathbf{v}_i^T$

Keep only the top $k$ singular values and vectors. This is the foundation of PCA, matrix factorization, image compression, and word embeddings.

U, s, Vt = np.linalg.svd(A, full_matrices=False)
k = 10
A_approx = (U[:, :k] * s[:k]) @ Vt[:k, :]

---

8. The Data Matrix Convention

In ML, we stack $n$ examples as rows:

$X = \begin{pmatrix} — \mathbf{x}^{(1)T} — \\ — \mathbf{x}^{(2)T} — \\ \vdots \\ — \mathbf{x}^{(n)T} — \end{pmatrix} \in \mathbb{R}^{n \times d}$

$n$ = number of samples, $d$ = number of features.

Key products to recognize instantly:

Expression	Shape	Meaning
$X^T X$	$d \times d$	Feature covariance (up to $1/n$)
$X X^T$	$n \times n$	Sample similarity / Gram matrix
$X \boldsymbol{\theta}$	$n \times 1$	Predictions for all samples at once
$X^T (y - X\boldsymbol{\theta})$	$d \times 1$	Gradient of MSE loss

---

9. Key Calculus on Vectors and Matrices

These derivatives appear constantly in deriving ML algorithms.

Gradient of a linear function

$\frac{\partial}{\partial \mathbf{x}} (\mathbf{a}^T \mathbf{x}) = \mathbf{a}$

Gradient of a quadratic form

$\frac{\partial}{\partial \mathbf{x}} (\mathbf{x}^T A \mathbf{x}) = (A + A^T)\mathbf{x} = 2A\mathbf{x} \quad \text{if } A \text{ symmetric}$

Gradient of MSE loss

$\mathcal{L}(\boldsymbol{\theta}) = \frac{1}{n}\|y - X\boldsymbol{\theta}\|^2 = \frac{1}{n}(y - X\boldsymbol{\theta})^T(y - X\boldsymbol{\theta})$

$\frac{\partial \mathcal{L}}{\partial \boldsymbol{\theta}} = -\frac{2}{n} X^T (y - X\boldsymbol{\theta})$

Setting to zero → Normal Equation:

$\boldsymbol{\theta}^* = (X^T X)^{-1} X^T y$

Chain rule (vector form)

$\frac{\partial}{\partial \mathbf{x}} f(g(\mathbf{x})) = \left(\frac{\partial g}{\partial \mathbf{x}}\right)^T \frac{\partial f}{\partial g}$

This is the foundation of backpropagation — just repeated application of the vector chain rule through each layer.

---

10. Trace and Frobenius Norm

Trace — sum of diagonal elements of a square matrix:

$\text{tr}(A) = \sum_{i=1}^n A_{ii} = \sum_{i=1}^n \lambda_i$

Useful identity: $\mathbf{x}^T A \mathbf{x} = \text{tr}(\mathbf{x}^T A \mathbf{x}) = \text{tr}(A \mathbf{x}\mathbf{x}^T)$

Frobenius norm — the matrix equivalent of the $L^2$ vector norm:

$\|A\|_F = \sqrt{\sum_{i,j} A_{ij}^2} = \sqrt{\text{tr}(A^T A)} = \sqrt{\sum_i \sigma_i^2}$

Used in regularization (e.g. weight decay in neural networks penalizes $\|W\|_F^2$).

---

11. Quick Reference

import numpy as np

# Dot product
np.dot(a, b)          # or a @ b for vectors

# Matrix multiply
A @ B

# Transpose
A.T

# Inverse
np.linalg.inv(A)

# Solve Ax = b  (prefer over inv)
np.linalg.solve(A, b)

# Eigenvalues (general)
vals, vecs = np.linalg.eig(A)

# Eigenvalues (symmetric — more stable)
vals, vecs = np.linalg.eigh(A)   # eigenvalues returned in ascending order

# SVD
U, s, Vt = np.linalg.svd(A)
U, s, Vt = np.linalg.svd(A, full_matrices=False)  # economy SVD

# Norms
np.linalg.norm(x)         # L2 by default
np.linalg.norm(x, ord=1)  # L1
np.linalg.norm(A, 'fro')  # Frobenius

# Rank
np.linalg.matrix_rank(A)

# Determinant
np.linalg.det(A)

# Trace
np.trace(A)