Foundations

Joint, Marginal, and Conditional Distributions

Joint distributions, marginalization, conditional distributions, independence, Bayes theorem, and the chain rule of probability.

CoreTier 1Stable~40 min

Prerequisites

Common Probability Distributions

Quiz (6)Pulse Check

Why This Matters

Joint distribution with marginals (axes) and conditional slice (highlighted)

ML is about modeling relationships between variables. A classifier models $P(Y|X)$ . A generative model learns $P(X)$ or $P(X, Y)$ . Bayesian inference uses Bayes theorem to compute posteriors. Every probabilistic model involves joint, marginal, or conditional distributions.

Core Definitions

Definition

Joint Distribution

For discrete random variables $X, Y$ : the joint PMF is $p(x, y) = P(X = x, Y = y)$ . For continuous random variables (see common probability distributions for standard families): the joint PDF $f(x, y)$ satisfies $P((X, Y) \in A) = \iint_A f(x, y) \, dx \, dy$ . Both must be non-negative and sum/integrate to 1.

Definition

Marginal Distribution

The marginal of $X$ is obtained by summing or integrating out $Y$ :

Discrete: $p_X(x) = \sum_y p(x, y)$

Continuous: $f_X(x) = \int_{-\infty}^{\infty} f(x, y) \, dy$

Marginalization discards information about $Y$ .

Definition

Conditional Distribution

The conditional distribution of $Y$ given $X = x$ is:

Discrete: $p(y|x) = \frac{p(x, y)}{p_X(x)}$ for $p_X(x) > 0$

Continuous: $f(y|x) = \frac{f(x, y)}{f_X(x)}$ for $f_X(x) > 0$

This defines a valid distribution over $Y$ for each fixed $x$ .

Definition

Independence

Random variables $X$ and $Y$ are independent (written $X \perp Y$ ) if and only if:

$p(x, y) = p_X(x) \, p_Y(y) \quad \text{for all } x, y$

Equivalently, $p(y|x) = p_Y(y)$ for all $x$ : knowing $X$ tells you nothing about $Y$ .

Definition

Conditional Independence

$X$ and $Y$ are conditionally independent given $Z$ (written $X \perp Y \mid Z$ ) if:

$p(x, y | z) = p(x|z) \, p(y|z) \quad \text{for all } x, y, z$

Conditional independence is the foundation of graphical models and the naive Bayes assumption. For modeling complex dependence structures beyond independence, copulas separate the marginal behavior from the joint dependence.

Main Theorems

Theorem

Bayes Theorem

Statement

$P(Y = y \mid X = x) = \frac{P(X = x \mid Y = y) \, P(Y = y)}{P(X = x)}$

In density form:

$f(y|x) = \frac{f(x|y) \, f_Y(y)}{f_X(x)}$

where $f_X(x) = \int f(x|y) f_Y(y) \, dy$ is the marginal (the "evidence").

Intuition

Bayes theorem inverts a conditional: it converts $P(X|Y)$ (the likelihood) into $P(Y|X)$ (the posterior) by incorporating the prior $P(Y)$ . The denominator normalizes the result.

Proof Sketch

Start from the definition: $P(Y|X) = P(X,Y)/P(X)$ and $P(X|Y) = P(X,Y)/P(Y)$ . From the second equation, $P(X,Y) = P(X|Y)P(Y)$ . Substituting into the first gives Bayes theorem.

Why It Matters

Bayes theorem is the foundation of Bayesian inference. The prior encodes beliefs before seeing data. The likelihood connects data to parameters. The posterior combines both. In ML: Bayesian neural networks, Gaussian processes, and MAP estimation all rest on this formula.

Failure Mode

The denominator $P(X = x)$ (or $f_X(x)$ ) is often intractable to compute because it requires integrating over all possible $Y$ . This is why approximate inference (MCMC, variational inference) is needed in practice.

Chain Rule and Law of Total Probability

Chain rule of probability for $n$ random variables:

$P(X_1, X_2, \ldots, X_n) = P(X_1) \prod_{i=2}^{n} P(X_i \mid X_1, \ldots, X_{i-1})$

This is not an assumption; it follows directly from the definition of conditional probability applied repeatedly.

Law of total probability (discrete version):

$P(X = x) = \sum_y P(X = x \mid Y = y) \, P(Y = y)$

This "averages out" the conditioning variable. It is used to compute marginals from conditionals and appears throughout mixture model derivations. These operations connect directly to computing expectations and moments.

Common Confusions

Watch Out

Marginal independence does not imply conditional independence

$X$ and $Y$ can be marginally independent ( $X \perp Y$ ) but conditionally dependent ( $X \not\perp Y \mid Z$ ). The classic example: two independent causes $X, Y$ of a common effect $Z$ . Once you observe $Z$ , learning about $X$ tells you about $Y$ (explaining away).

Watch Out

Conditional independence does not imply marginal independence

The converse also fails. $X \perp Y \mid Z$ does not imply $X \perp Y$ . Example: $Z \sim \text{Bernoulli}(0.5)$ , $X = Z$ , $Y = Z$ . Then $X \perp Y \mid Z$ (both are deterministic given $Z$ ), but $X$ and $Y$ are perfectly dependent marginally.

Watch Out

Conditioning on zero-probability events

For continuous random variables, $P(X = x) = 0$ for any specific $x$ . The conditional density $f(y|x)$ is defined as a ratio of densities, not as $P(Y = y | X = x)$ . Rigorous treatment uses the Radon-Nikodym derivative or disintegration of measures.

Canonical Examples

Example

Naive Bayes classifier

Naive Bayes assumes features are conditionally independent given the class: $P(X_1, \ldots, X_d \mid Y) = \prod_{i=1}^d P(X_i \mid Y)$ . By Bayes theorem, $P(Y \mid X) \propto P(Y) \prod_i P(X_i \mid Y)$ . This reduces the number of parameters from exponential in $d$ to linear in $d$ . The independence assumption is almost always false, but the classifier often works well because the decision boundary can still be correct even when the probability estimates are wrong.

Exercises

ExerciseCore

Problem

A medical test has sensitivity $P(\text{positive} \mid \text{disease}) = 0.95$ and specificity $P(\text{negative} \mid \text{no disease}) = 0.99$ . The disease prevalence is $P(\text{disease}) = 0.001$ . Compute $P(\text{disease} \mid \text{positive})$ .

ExerciseAdvanced

Problem

Let $(X, Y)$ have joint density $f(x, y) = 2$ for $0 \leq x \leq y \leq 1$ and $f(x, y) = 0$ otherwise. Find the marginal densities $f_X(x)$ and $f_Y(y)$ , and the conditional density $f(y|x)$ .

References

Canonical:

Grimmett & Stirzaker, Probability and Random Processes (2020), Chapters 1-3
Casella & Berger, Statistical Inference (2002), Chapters 1-2

For ML context:

Bishop, Pattern Recognition and Machine Learning (2006), Chapters 1-2
Murphy, Machine Learning: A Probabilistic Perspective (2012), Chapter 2

Last reviewed: April 2026

Prerequisites

Foundations this topic depends on.

Common Probability DistributionsLayer 0A
Sets, Functions, and RelationsLayer 0A
Basic Logic and Proof TechniquesLayer 0A

Next Topics

Expectation Variance Covariance MomentsContinue →