Probability & Statistics

1. Sample Space, Events, Probability

Sample space $\Omega$ — the set of all possible outcomes of an experiment.

Event $A \subseteq \Omega$ — a subset of outcomes we care about.

Probability measure $P$ assigns a number in $[0,1]$ to each event, satisfying:

1. $P(\Omega) = 1$
2. $P(A) \geq 0$ for all $A$
3. For mutually exclusive events: $P(A \cup B) = P(A) + P(B)$

Useful consequences: $P(\emptyset) = 0 \qquad P(A^c) = 1 - P(A) \qquad P(A \cup B) = P(A) + P(B) - P(A \cap B)$

Example: Roll a fair die. $\Omega = \{1,2,3,4,5,6\}$. Event $A$ = "even" = $\{2,4,6\}$. $P(A) = 3/6 = 0.5$.

---

2. Conditional Probability

$P(A \mid B) = \frac{P(A \cap B)}{P(B)} \qquad \text{provided } P(B) > 0$

"The probability of $A$ given that we know $B$ has occurred."

Example: A patient tests positive for a disease. What is the probability they actually have it? That's $P(\text{disease} \mid \text{positive})$ — not the same as $P(\text{positive} \mid \text{disease})$. Confusing these is a classic error (see Bayes theorem below).

Multiplication rule

$P(A \cap B) = P(A \mid B)\, P(B) = P(B \mid A)\, P(A)$

Independence

$A$ and $B$ are independent if knowing $B$ tells you nothing about $A$:

$P(A \mid B) = P(A) \qquad \Leftrightarrow \qquad P(A \cap B) = P(A)\,P(B)$

[!WARNING] Independence is an assumption, not something you observe directly. The i.i.d. assumption in ML (samples are independent and identically distributed) is an assumption about how data was collected — it's often approximately true but rarely exactly true.

---

3. Bayes' Theorem

$\boxed{P(A \mid B) = \frac{P(B \mid A)\, P(A)}{P(B)}}$

In ML language: posterior = (likelihood × prior) / evidence.

$P(\theta \mid \mathcal{D}) = \frac{P(\mathcal{D} \mid \theta)\, P(\theta)}{P(\mathcal{D})}$

• $P(\theta)$ — prior: what we believe about parameters before seeing data
• $P(\mathcal{D} \mid \theta)$ — likelihood: how probable is the data under these parameters
• $P(\mathcal{D})$ — evidence (normalizing constant, often intractable)
• $P(\theta \mid \mathcal{D})$ — posterior: updated belief after seeing data

Law of Total Probability

If $B_1, \ldots, B_k$ partition $\Omega$ (mutually exclusive, exhaustive):

$P(A) = \sum_{i=1}^k P(A \mid B_i)\, P(B_i)$

Used to compute $P(\mathcal{D})$ in the denominator of Bayes.

Classic example — Medical test

A disease affects 1% of the population. A test is 95% sensitive (true positive rate) and 95% specific (true negative rate). You test positive. What's the probability you have the disease?

• $P(\text{disease}) = 0.01$ (prior)
• $P(\text{pos} \mid \text{disease}) = 0.95$ (sensitivity)
• $P(\text{pos} \mid \text{no disease}) = 0.05$ (false positive rate)
• $P(\text{pos}) = 0.95 \times 0.01 + 0.05 \times 0.99 = 0.0095 + 0.0495 = 0.059$

$P(\text{disease} \mid \text{pos}) = \frac{0.95 \times 0.01}{0.059} \approx \mathbf{16\%}$

Despite a 95% accurate test, a positive result only means 16% chance of disease. This is why rare-disease screening is hard — base rates dominate.

---

4. Random Variables

A random variable $X$ is a function $X: \Omega \to \mathbb{R}$ mapping outcomes to numbers.

Discrete random variable

Takes values in a countable set. Described by a probability mass function (PMF):

$P(X = x) = p(x) \qquad \sum_x p(x) = 1$

Example — Bernoulli($p$):

$P(X = 1) = p \qquad P(X = 0) = 1-p$

Used for: binary outcomes (spam/not spam, click/no click).

Example — Binomial($n, p$):

$P(X = k) = \binom{n}{k} p^k (1-p)^{n-k}$

Number of successes in $n$ independent Bernoulli trials. If I send 100 emails and each has a 3% click rate, the number of clicks is Binomial(100, 0.03).

Continuous random variable

Takes values in $\mathbb{R}$ (or an interval). Described by a probability density function (PDF) $f(x)$:

$P(a \leq X \leq b) = \int_a^b f(x)\, dx \qquad \int_{-\infty}^\infty f(x)\, dx = 1$

Note: $P(X = x) = 0$ for any single point — probability lives in intervals.

Cumulative distribution function (CDF):

$F(x) = P(X \leq x) = \int_{-\infty}^x f(t)\, dt \qquad f(x) = F'(x)$

---

5. Expectation

The expected value (mean) of $X$:

$\mathbb{E}[X] = \begin{cases} \sum_x x\, p(x) & \text{discrete} \\ \int_{-\infty}^\infty x\, f(x)\, dx & \text{continuous} \end{cases}$

Think of it as a weighted average of all possible values, weighted by their probability.

Linearity of expectation — always true, regardless of dependence:

$\mathbb{E}[aX + bY] = a\,\mathbb{E}[X] + b\,\mathbb{E}[Y]$

Example: Roll two dice, $S$ = sum. $\mathbb{E}[S] = \mathbb{E}[D_1] + \mathbb{E}[D_2] = 3.5 + 3.5 = 7$. No need to enumerate all 36 outcomes.

Expectation of a function:

$\mathbb{E}[g(X)] = \sum_x g(x)\, p(x)$

[!WARNING] $\mathbb{E}[g(X)] \neq g(\mathbb{E}[X])$ in general. This is Jensen's inequality: for convex $g$, $\mathbb{E}[g(X)] \geq g(\mathbb{E}[X])$. Relevant in variational inference, log-likelihood bounds.

---

6. Variance and Standard Deviation

$\text{Var}(X) = \mathbb{E}[(X - \mathbb{E}[X])^2] = \mathbb{E}[X^2] - (\mathbb{E}[X])^2$

The second form is often easier to compute. Standard deviation: $\sigma = \sqrt{\text{Var}(X)}$.

Properties:

• $\text{Var}(aX + b) = a^2 \text{Var}(X)$
• $\text{Var}(X + Y) = \text{Var}(X) + \text{Var}(Y) + 2\text{Cov}(X,Y)$
• If $X, Y$ independent: $\text{Var}(X + Y) = \text{Var}(X) + \text{Var}(Y)$

Covariance:

$\text{Cov}(X, Y) = \mathbb{E}[(X - \mathbb{E}[X])(Y - \mathbb{E}[Y])] = \mathbb{E}[XY] - \mathbb{E}[X]\mathbb{E}[Y]$

Correlation (normalized covariance, scale-free):

$\rho(X,Y) = \frac{\text{Cov}(X,Y)}{\sigma_X \sigma_Y} \in [-1, 1]$

[!NOTE] $\rho = 0$ means uncorrelated, not independent. Independence implies uncorrelated, but not vice versa. If $X \sim \mathcal{N}(0,1)$ and $Y = X^2$, then $\text{Cov}(X,Y) = 0$ but they are clearly dependent.

---

7. Common Distributions

Uniform — $\text{Uniform}(a, b)$

$f(x) = \frac{1}{b-a} \quad x \in [a,b]$

$\mathbb{E}[X] = \frac{a+b}{2} \qquad \text{Var}(X) = \frac{(b-a)^2}{12}$

Used for: random initialization, random search, Monte Carlo sampling.

---

Gaussian (Normal) — $\mathcal{N}(\mu, \sigma^2)$

$f(x) = \frac{1}{\sigma\sqrt{2\pi}} \exp\!\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)$

$\mathbb{E}[X] = \mu \qquad \text{Var}(X) = \sigma^2$

The standard normal is $\mathcal{N}(0,1)$, denoted $Z$. If $X \sim \mathcal{N}(\mu, \sigma^2)$ then $Z = (X-\mu)/\sigma$.

The 68-95-99.7 rule:

• $P(\mu - \sigma \leq X \leq \mu + \sigma) \approx 68\%$
• $P(\mu - 2\sigma \leq X \leq \mu + 2\sigma) \approx 95\%$
• $P(\mu - 3\sigma \leq X \leq \mu + 3\sigma) \approx 99.7\%$

Why Gaussian dominates ML:

• Central Limit Theorem — sum of many i.i.d. variables converges to Gaussian
• Mathematically convenient — closed under linear transformations
• Maximum entropy distribution given fixed mean and variance
• Assumed in linear regression (residuals $\sim \mathcal{N}(0, \sigma^2)$)

Multivariate Gaussian $\mathcal{N}(\boldsymbol{\mu}, \Sigma)$:

$f(\mathbf{x}) = \frac{1}{(2\pi)^{d/2}|\Sigma|^{1/2}} \exp\!\left(-\frac{1}{2}(\mathbf{x}-\boldsymbol{\mu})^T \Sigma^{-1} (\mathbf{x}-\boldsymbol{\mu})\right)$

• $\boldsymbol{\mu} \in \mathbb{R}^d$ — mean vector
• $\Sigma \in \mathbb{R}^{d \times d}$ — covariance matrix (symmetric, PSD)
• The exponent $(\mathbf{x}-\boldsymbol{\mu})^T \Sigma^{-1} (\mathbf{x}-\boldsymbol{\mu})$ is the Mahalanobis distance — it measures distance in units of standard deviations, accounting for correlations

---

Bernoulli — $\text{Bern}(p)$

$P(X=1) = p \qquad P(X=0) = 1-p$ $\mathbb{E}[X] = p \qquad \text{Var}(X) = p(1-p)$

Used as the output distribution for binary classification. The sigmoid function $\sigma(z) = 1/(1+e^{-z})$ maps logistic regression output to a valid $p$.

---

Categorical — $\text{Cat}(\boldsymbol{\pi})$

Generalization of Bernoulli to $K$ classes. $P(X = k) = \pi_k$, $\sum_k \pi_k = 1$.

The softmax maps logit vectors to valid $\boldsymbol{\pi}$:

$\pi_k = \frac{e^{z_k}}{\sum_{j=1}^K e^{z_j}}$

---

Poisson — $\text{Pois}(\lambda)$

$P(X = k) = \frac{\lambda^k e^{-\lambda}}{k!} \qquad k = 0, 1, 2, \ldots$

$\mathbb{E}[X] = \lambda \qquad \text{Var}(X) = \lambda$

Used for: count data (number of events in a fixed time window). Emails per hour, bugs per 1000 lines of code, word counts in documents.

---

Exponential — $\text{Exp}(\lambda)$

$f(x) = \lambda e^{-\lambda x} \qquad x \geq 0$ $\mathbb{E}[X] = 1/\lambda \qquad \text{Var}(X) = 1/\lambda^2$

Models waiting times between Poisson events. Memoryless: $P(X > s+t \mid X > s) = P(X > t)$.

---

Beta — $\text{Beta}(\alpha, \beta)$

$f(x) = \frac{x^{\alpha-1}(1-x)^{\beta-1}}{B(\alpha,\beta)} \qquad x \in [0,1]$

$\mathbb{E}[X] = \frac{\alpha}{\alpha+\beta} \qquad \text{Var}(X) = \frac{\alpha\beta}{(\alpha+\beta)^2(\alpha+\beta+1)}$

Lives on $[0,1]$, used as a prior for probabilities (conjugate prior to Bernoulli/Binomial). Intuition: $\alpha - 1$ = number of observed successes, $\beta - 1$ = failures.

---

8. Maximum Likelihood Estimation (MLE)

Given data $\mathcal{D} = \{x^{(1)}, \ldots, x^{(n)}\}$ drawn i.i.d. from $p(x ; \theta)$, the likelihood is:

$L(\theta) = \prod_{i=1}^n p(x^{(i)} ; \theta)$

MLE finds the parameter that makes the observed data most probable:

$\hat{\theta}_{\text{MLE}} = \arg\max_\theta L(\theta) = \arg\max_\theta \sum_{i=1}^n \log p(x^{(i)} ; \theta)$

The log-likelihood $\ell(\theta) = \log L(\theta)$ is used because:

• Products become sums (easier)
• More numerically stable (no underflow)
• Maximizing log is equivalent to maximizing the original (log is monotone)

Example — MLE for Gaussian

Given $x^{(1)}, \ldots, x^{(n)} \sim \mathcal{N}(\mu, \sigma^2)$:

$\ell(\mu, \sigma^2) = -\frac{n}{2}\log(2\pi\sigma^2) - \frac{1}{2\sigma^2}\sum_{i=1}^n (x^{(i)} - \mu)^2$

Setting $\partial \ell / \partial \mu = 0$:

$\hat{\mu}_{\text{MLE}} = \frac{1}{n}\sum_{i=1}^n x^{(i)} = \bar{x}$

Setting $\partial \ell / \partial \sigma^2 = 0$:

$\hat{\sigma}^2_{\text{MLE}} = \frac{1}{n}\sum_{i=1}^n (x^{(i)} - \bar{x})^2$

[!NOTE] This gives $1/n$ not $1/(n-1)$. The MLE estimate of variance is biased (underestimates). The unbiased sample variance uses $n-1$ (Bessel's correction). In practice the difference is negligible for large $n$.

MLE ↔ Loss functions

Model assumption	Negative log-likelihood	Standard name
$y \mid x \sim \mathcal{N}(\hat{y}, \sigma^2)$	$\|y - \hat{y}\|^2$	MSE loss
$y \mid x \sim \text{Bern}(\hat{p})$	$-y\log\hat{p} - (1-y)\log(1-\hat{p})$	Binary cross-entropy
$y \mid x \sim \text{Cat}(\boldsymbol{\pi})$	$-\sum_k y_k \log \pi_k$	Cross-entropy
$y \mid x \sim \text{Laplace}(\hat{y}, b)$	$\|y - \hat{y}\|_1$	MAE loss

This is the key insight: choosing a loss function is equivalent to choosing a probabilistic model for your noise. MSE assumes Gaussian residuals. MAE assumes Laplacian (heavier tails, more robust to outliers).

---

9. Maximum A Posteriori (MAP)

MAP includes a prior over parameters:

$\hat{\theta}_{\text{MAP}} = \arg\max_\theta \log p(\theta \mid \mathcal{D}) = \arg\max_\theta \left[\log p(\mathcal{D} \mid \theta) + \log p(\theta)\right]$

MAP ↔ Regularization

Prior on $\theta$	Effect on objective	Standard name
$\theta_j \sim \mathcal{N}(0, 1/\lambda)$	adds $\lambda\|\theta\|_2^2$	L2 regularization / Ridge
$\theta_j \sim \text{Laplace}(0, 1/\lambda)$	adds $\lambda\|\theta\|_1$	L1 regularization / Lasso

L2 (Gaussian prior): Penalizes large weights, shrinks all weights toward zero. Prefers many small weights. Solution is always unique and dense.

L1 (Laplace prior): Produces sparse solutions — many weights become exactly zero. Useful for feature selection.

from sklearn.linear_model import Ridge, Lasso
ridge = Ridge(alpha=1.0)      # L2, alpha = λ
lasso = Lasso(alpha=0.1)      # L1

---

10. Information Theory

Entropy

The Shannon entropy of a discrete distribution $P$:

$H(P) = -\sum_x p(x) \log p(x) = \mathbb{E}[-\log p(X)]$

Measures uncertainty or average surprise. Units: bits (log base 2) or nats (natural log).

• Uniform distribution over $K$ classes: $H = \log K$ (maximum entropy)
• Point mass (certain outcome): $H = 0$ (minimum entropy)

Example: Fair coin: $H = -0.5\log 0.5 - 0.5\log 0.5 = 1$ bit. Biased coin with $p=0.9$: $H \approx 0.47$ bits — less uncertainty.

KL Divergence

Measures how different distribution $Q$ is from reference $P$:

$D_{\text{KL}}(P \| Q) = \sum_x p(x) \log \frac{p(x)}{q(x)} = \mathbb{E}_P\!\left[\log\frac{p(X)}{q(X)}\right]$

Properties:

• $D_{\text{KL}}(P \| Q) \geq 0$ always (Gibbs inequality)
• $D_{\text{KL}}(P \| Q) = 0 \iff P = Q$
• Not symmetric: $D_{\text{KL}}(P \| Q) \neq D_{\text{KL}}(Q \| P)$

[!NOTE] $D_{\text{KL}}(P \| Q)$ is "forward KL" — it's zero-avoiding (forces $q>0$ wherever $p>0$, so the approximation covers all modes). $D_{\text{KL}}(Q \| P)$ is "reverse KL" — zero-forcing (allows $q=0$ where $p>0$, causes mode-seeking behavior). Both appear in variational inference and generative models.

Cross-Entropy

$H(P, Q) = -\sum_x p(x) \log q(x) = H(P) + D_{\text{KL}}(P \| Q)$

When $P$ is the true distribution (one-hot labels) and $Q$ is the model output (softmax probabilities), minimizing cross-entropy = minimizing KL divergence from true to predicted = MLE.

import torch.nn.functional as F
loss = F.cross_entropy(logits, targets)  # standard for classification

Mutual Information

How much does knowing $X$ tell you about $Y$?

$I(X;Y) = D_{\text{KL}}(P_{XY} \| P_X P_Y) = H(X) - H(X \mid Y) = H(Y) - H(Y \mid X)$

$I(X;Y) = 0$ iff $X$ and $Y$ are independent. Used in feature selection (pick features that maximize mutual information with the label).

---

11. Estimators and Their Properties

An estimator $\hat{\theta}$ is a function of data used to estimate a parameter $\theta$.

Bias: $\text{Bias}(\hat{\theta}) = \mathbb{E}[\hat{\theta}] - \theta$

Unbiased: $\mathbb{E}[\hat{\theta}] = \theta$. Sample mean $\bar{x}$ is unbiased for $\mu$. Sample variance with $n-1$ is unbiased for $\sigma^2$.

Variance of the estimator: $\text{Var}(\hat{\theta}) = \mathbb{E}\left[(\hat{\theta} - \mathbb{E}[\hat{\theta}])^2\right]$

Mean Squared Error: $\text{MSE}(\hat{\theta}) = \text{Bias}(\hat{\theta})^2 + \text{Var}(\hat{\theta})$

This is the bias-variance tradeoff in disguise. A slightly biased estimator with much lower variance can have lower MSE — sometimes paying a little bias to get a lot of variance reduction is worth it. Ridge regression does exactly this.

Consistency: $\hat{\theta}_n \xrightarrow{p} \theta$ as $n \to \infty$. MLE estimators are typically consistent.

---

12. Central Limit Theorem

Let $X_1, \ldots, X_n$ be i.i.d. with mean $\mu$ and variance $\sigma^2$. Then:

$\sqrt{n}\left(\bar{X}_n - \mu\right) \xrightarrow{d} \mathcal{N}(0, \sigma^2)$

Or equivalently:

$\bar{X}_n \approx \mathcal{N}\!\left(\mu,\, \frac{\sigma^2}{n}\right)$

Why it matters: No matter what distribution your data comes from, the sample mean is approximately normally distributed for large $n$. This is why Gaussian assumptions work so well in practice.

Standard error of the mean: $\text{SE} = \sigma/\sqrt{n}$. Collecting 4× more data halves the standard error.

---

13. Hypothesis Testing (just enough for ML)

A p-value is $P(\text{data this extreme or more} \mid H_0)$. It is not $P(H_0 \mid \text{data})$.

Common tests in ML practice:

Test	When to use
t-test	Compare means of two models/groups
Paired t-test	Compare two models on the same test sets
McNemar's test	Compare two classifiers on the same examples
Kolmogorov-Smirnov	Check if a sample came from a distribution

from scipy import stats

# Are model A and model B accuracies significantly different?
t_stat, p_value = stats.ttest_rel(accuracies_A, accuracies_B)
print(f"p = {p_value:.4f}")  # if p < 0.05, difference is "significant"

[!WARNING] Statistical significance is not practical significance. With enough data, tiny useless differences become "significant" ($p < 0.05$). Always report effect size alongside p-values.

---

14. Covariance Matrix in Detail

For a random vector $\mathbf{x} \in \mathbb{R}^d$:

$\Sigma = \text{Cov}(\mathbf{x}) = \mathbb{E}[(\mathbf{x} - \boldsymbol{\mu})(\mathbf{x} - \boldsymbol{\mu})^T]$

$\Sigma_{ij} = \text{Cov}(x_i, x_j)$

Properties: symmetric, positive semi-definite, diagonal entries are variances.

Sample covariance from data matrix $X \in \mathbb{R}^{n \times d}$ (zero-mean):

$\hat{\Sigma} = \frac{1}{n-1} X^T X$

The correlation matrix is the normalized version:

$R_{ij} = \frac{\Sigma_{ij}}{\sqrt{\Sigma_{ii} \Sigma_{jj}}}$

# Empirical covariance
X_centered = X - X.mean(axis=0)
Sigma = (X_centered.T @ X_centered) / (n - 1)

# Or just:
Sigma = np.cov(X.T)          # X is n×d, np.cov wants d×n
corr  = np.corrcoef(X.T)     # correlation matrix

Geometric interpretation: The covariance matrix defines an ellipse (in 2D) or ellipsoid (in $d$D). Its eigenvectors are the axes of the ellipsoid; eigenvalues are the squared semi-axis lengths. PCA finds these axes.

---

15. Quick Reference

import numpy as np
from scipy import stats

# ── Descriptive ───────────────────────────────
np.mean(x)
np.median(x)
np.std(x)              # std dev (ddof=0 by default → biased)
np.std(x, ddof=1)      # unbiased sample std
np.var(x, ddof=1)
np.percentile(x, [25, 50, 75])

# ── Distributions ─────────────────────────────
norm  = stats.norm(loc=0, scale=1)
norm.pdf(x)            # density
norm.cdf(x)            # P(X ≤ x)
norm.ppf(0.975)        # quantile (inverse CDF) → 1.96

# Sample from distributions
np.random.normal(mu, sigma, size=1000)
np.random.binomial(n=10, p=0.3, size=1000)
np.random.poisson(lam=5, size=1000)
np.random.beta(a=2, b=5, size=1000)

# ── MLE fits ──────────────────────────────────
mu_hat, sigma_hat = stats.norm.fit(data)
a_hat, b_hat, loc, scale = stats.beta.fit(data, floc=0, fscale=1)

# ── Entropy / Information ─────────────────────
from scipy.stats import entropy
H = entropy([0.5, 0.5])               # Shannon entropy (nats)
H = entropy([0.5, 0.5], base=2)       # bits
KL = entropy(p, q)                    # KL divergence D_KL(p||q)

# ── Correlation ───────────────────────────────
np.corrcoef(x, y)[0,1]               # Pearson correlation
stats.pearsonr(x, y)                 # + p-value
stats.spearmanr(x, y)                # rank correlation

# ── Tests ─────────────────────────────────────
stats.ttest_ind(a, b)                # independent t-test
stats.ttest_rel(a, b)                # paired t-test
stats.kstest(x, 'norm')              # normality test
stats.chi2_contingency(table)        # chi-squared test