Common Probability Distributions

A single binary trial with success probability $p \in [0, 1]$.

PMF: $P(X = k) = p^k (1-p)^{1-k}$ for $k \in \{0, 1\}$

Mean: $\mathbb{E}[X] = p$

Variance: $\text{Var}(X) = p(1 - p)$

When it arises: Binary classification labels, coin flips, any yes/no outcome. The log-likelihood of Bernoulli data gives rise to the cross-entropy loss used in logistic regression.

The number of successes in $n$ independent Bernoulli$(p)$ trials.

PMF: $P(X = k) = \binom{n}{k} p^k (1-p)^{n-k}$ for $k = 0, 1, \ldots, n$

Mean: $\mathbb{E}[X] = np$

Variance: $\text{Var}(X) = np(1 - p)$

Key fact: If $X_1, \ldots, X_n \sim \text{Bern}(p)$ independently, then $\sum_i X_i \sim \text{Bin}(n, p)$. As $n \to \infty$ with $np \to \lambda$, the Binomial converges to $\text{Poisson}(\lambda)$.

Models the count of events in a fixed interval when events occur independently at a constant rate $\lambda > 0$.

PMF: $P(X = k) = \frac{\lambda^k e^{-\lambda}}{k!}$ for $k = 0, 1, 2, \ldots$

Mean: $\mathbb{E}[X] = \lambda$

Variance: $\text{Var}(X) = \lambda$

When it arises: Count data. number of clicks, arrivals, mutations. The mean equals the variance; if your count data has variance much larger than the mean, the Poisson model is wrong (use negative binomial instead).

The number of trials until the first success in independent Bernoulli$(p)$ trials. (Convention: $X$ includes the successful trial.)

PMF: $P(X = k) = (1-p)^{k-1} p$ for $k = 1, 2, 3, \ldots$

Mean: $\mathbb{E}[X] = 1/p$

Variance: $\text{Var}(X) = (1-p)/p^2$

Key property: The Geometric distribution is the only discrete distribution with the memoryless property: $P(X > s + t \mid X > s) = P(X > t)$.

The multivariate generalization of the Binomial. In $n$ trials, each of $K$ categories occurs with probability $p_k$, and $X_k$ counts category $k$.

PMF: $P(X_1 = x_1, \ldots, X_K = x_K) = \frac{n!}{x_1! \cdots x_K!} p_1^{x_1} \cdots p_K^{x_K}$

where $\sum_k x_k = n$ and $\sum_k p_k = 1$.

Marginals: Each $X_k \sim \text{Bin}(n, p_k)$.

When it arises: Multi-class classification, bag-of-words models, topic models. The softmax output of a neural network parameterizes a single-trial Multinomial (i.e., a Categorical distribution).

Constant density on the interval $[a, b]$.

PDF: $f(x) = \frac{1}{b - a}$ for $x \in [a, b]$, zero otherwise.

Mean: $\mathbb{E}[X] = \frac{a + b}{2}$

Variance: $\text{Var}(X) = \frac{(b - a)^2}{12}$

When it arises: Maximum entropy distribution on a bounded interval with no other constraints. Used in random initialization, random search, and as the base distribution for inverse transform sampling.

The most important continuous distribution. Parameterized by mean $\mu$ and variance $\sigma^2 > 0$.

PDF: $f(x) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\!\left(-\frac{(x - \mu)^2}{2\sigma^2}\right)$

Mean: $\mathbb{E}[X] = \mu$

Variance: $\text{Var}(X) = \sigma^2$

MGF: $\mathbb{E}[e^{tX}] = \exp(\mu t + \sigma^2 t^2 / 2)$

The normalizing constant $1/\sqrt{2\pi\sigma^2}$ ensures $\int_{-\infty}^{\infty} f(x)\,dx = 1$. This requires the Gaussian integral $\int_{-\infty}^{\infty} e^{-x^2/2}\,dx = \sqrt{2\pi}$.

For $X \in \mathbb{R}^d$ with mean vector $\mu \in \mathbb{R}^d$ and positive definite covariance matrix $\Sigma \in \mathbb{R}^{d \times d}$:

PDF: $f(x) = \frac{1}{(2\pi)^{d/2} |\Sigma|^{1/2}} \exp\!\left(-\frac{1}{2}(x - \mu)^\top \Sigma^{-1} (x - \mu)\right)$

Key properties:

• Marginals are Gaussian: if $X \sim \mathcal{N}(\mu, \Sigma)$, any subset of coordinates is also multivariate Gaussian.
• For jointly Gaussian distributions, uncorrelated implies independent. Marginal Gaussianity alone does not suffice. A standard counterexample: $X \sim \mathcal{N}(0, 1)$ and $Y = X \cdot \mathbb{1}\{|X| \leq c\} - X \cdot \mathbb{1}\{|X| > c\}$. Both marginals are Gaussian and the pair is uncorrelated for an appropriate $c$, yet $X$ and $Y$ are dependent (since $|X| = |Y|$).
• Affine transformations preserve Gaussianity: $AX + b \sim \mathcal{N}(A\mu + b, A\Sigma A^\top)$.

When it arises: Central limit theorem, Bayesian linear regression posterior, Gaussian processes, VAE latent spaces.

Models waiting times between events in a Poisson process with rate $\lambda > 0$.

PDF: $f(x) = \lambda e^{-\lambda x}$ for $x \geq 0$

Mean: $\mathbb{E}[X] = 1/\lambda$

Variance: $\text{Var}(X) = 1/\lambda^2$

Key property: Memoryless: $P(X > s + t \mid X > s) = P(X > t)$. This is the only continuous distribution with this property.

Relationship: $\text{Exp}(\lambda) = \text{Gamma}(1, \lambda)$.

Generalizes the Exponential. Shape $\alpha > 0$, rate $\beta > 0$.

PDF: $f(x) = \frac{\beta^\alpha}{\Gamma(\alpha)} x^{\alpha - 1} e^{-\beta x}$ for $x > 0$

where $\Gamma(\alpha) = \int_0^\infty t^{\alpha-1} e^{-t}\,dt$ is the Gamma function.

Mean: $\mathbb{E}[X] = \alpha/\beta$

Variance: $\text{Var}(X) = \alpha/\beta^2$

Special cases:

• $\text{Gamma}(1, \lambda) = \text{Exp}(\lambda)$
• $\text{Gamma}(n/2, 1/2) = \chi^2(n)$
• Sum of $n$ independent $\text{Exp}(\beta)$ variables is $\text{Gamma}(n, \beta)$

When it arises: Conjugate prior for the Poisson rate and the precision (inverse variance) of a Gaussian. Models positive quantities with skew.

Defined on $[0, 1]$ with shape parameters $\alpha, \beta > 0$.

PDF: $f(x) = \frac{x^{\alpha-1}(1-x)^{\beta-1}}{B(\alpha, \beta)}$ for $x \in [0, 1]$

where $B(\alpha, \beta) = \frac{\Gamma(\alpha)\Gamma(\beta)}{\Gamma(\alpha + \beta)}$.

Mean: $\mathbb{E}[X] = \frac{\alpha}{\alpha + \beta}$

Variance: $\text{Var}(X) = \frac{\alpha\beta}{(\alpha + \beta)^2(\alpha + \beta + 1)}$

Key fact: The Beta is the conjugate prior for the Bernoulli/Binomial parameter $p$. If $p \sim \text{Beta}(\alpha, \beta)$ and $X \mid p \sim \text{Bin}(n, p)$ with $k$ successes, then $p \mid X \sim \text{Beta}(\alpha + k, \beta + n - k)$.

Special cases: $\text{Beta}(1, 1) = \text{Unif}(0, 1)$.

The sum of $k$ independent squared standard normals. Degrees of freedom $k$.

PDF: $f(x) = \frac{x^{k/2 - 1} e^{-x/2}}{2^{k/2}\Gamma(k/2)}$ for $x > 0$

Mean: $\mathbb{E}[X] = k$

Variance: $\text{Var}(X) = 2k$

Relationship: $\chi^2(k) = \text{Gamma}(k/2, 1/2)$.

If $Z_1, \ldots, Z_k \sim \mathcal{N}(0,1)$ independently, then $\sum_i Z_i^2 \sim \chi^2(k)$.

When it arises: Hypothesis testing (goodness-of-fit, likelihood ratio tests), analysis of variance. The sample variance of Gaussian data is proportional to a $\chi^2$. Important in concentration: the $\chi^2$ is sub-exponential but not sub-Gaussian.

Arises when estimating the mean of a Gaussian population with unknown variance. Degrees of freedom $\nu > 0$.

PDF: $f(x) = \frac{\Gamma((\nu+1)/2)}{\sqrt{\nu\pi}\,\Gamma(\nu/2)}\left(1 + \frac{x^2}{\nu}\right)^{-(\nu+1)/2}$

Mean: $\mathbb{E}[X] = 0$ for $\nu > 1$ (undefined for $\nu \leq 1$)

Variance: $\text{Var}(X) = \frac{\nu}{\nu - 2}$ for $\nu > 2$

Construction: If $Z \sim \mathcal{N}(0,1)$ and $V \sim \chi^2(\nu)$ are independent, then $T = Z / \sqrt{V/\nu} \sim t(\nu)$.

Key fact: Heavier tails than the Gaussian. As $\nu \to \infty$, $t(\nu) \to \mathcal{N}(0, 1)$. For small $\nu$, the heavy tails accommodate outliers. This is why the Student-t is used in robust statistics.

The ratio of two independent scaled chi-squared variables.

Construction: If $U \sim \chi^2(d_1)$ and $V \sim \chi^2(d_2)$ are independent, then $F = \frac{U/d_1}{V/d_2} \sim F(d_1, d_2)$.

Mean: $\mathbb{E}[X] = \frac{d_2}{d_2 - 2}$ for $d_2 > 2$

When it arises: ANOVA, comparing two model fits (F-test for nested models), testing whether a group of regression coefficients is jointly zero.

Relationship: If $T \sim t(\nu)$, then $T^2 \sim F(1, \nu)$.

Location $x_0$, scale $\gamma > 0$.

PDF: $f(x) = \frac{1}{\pi\gamma\left[1 + \left(\frac{x - x_0}{\gamma}\right)^2\right]}$

Mean: Does not exist (the integral diverges).

Variance: Does not exist.

Key fact: The Cauchy is $t(1)$. The Student-t with one degree of freedom. It is the standard example of a distribution with no mean. The sample mean of $n$ i.i.d. Cauchy variables has the same distribution as a single observation. The law of large numbers does not apply. This is why finite moments matter for concentration inequalities.

The multivariate generalization of the Beta, defined on the $(K-1)$-simplex $\{x \in \mathbb{R}^K : x_k \geq 0, \sum_k x_k = 1\}$.

PDF: $f(x_1, \ldots, x_K) = \frac{\Gamma(\sum_k \alpha_k)}{\prod_k \Gamma(\alpha_k)} \prod_{k=1}^K x_k^{\alpha_k - 1}$

Marginals: Each $X_k \sim \text{Beta}(\alpha_k, \sum_{j \neq k} \alpha_j)$.

Mean: $\mathbb{E}[X_k] = \alpha_k / \sum_j \alpha_j$

Key fact: Conjugate prior for the Multinomial/Categorical. If $\theta \sim \text{Dir}(\alpha)$ and $X \mid \theta \sim \text{Mult}(n, \theta)$ with counts $c_1, \ldots, c_K$, then $\theta \mid X \sim \text{Dir}(\alpha_1 + c_1, \ldots, \alpha_K + c_K)$.

When it arises: Latent Dirichlet Allocation (LDA), Bayesian multi-class models, any model requiring a prior over probability vectors.