Beyond Llms

Diffusion Models

Generative models that learn to reverse a noise-adding process: the math of score matching, denoising, SDEs, and why diffusion dominates image generation.

AdvancedTier 2Current~70 min

Prerequisites

Variational Autoencoders

Quiz (4)

Why This Matters

Diffusion models are the dominant generative architecture for images, video, audio, and molecular design as of 2024-2025. Stable Diffusion, DALL-E 3, Imagen, and Sora all use diffusion or its close relative, flow matching.

Unlike GANs, diffusion models have stable training (no minimax game). Unlike VAEs, they produce high-fidelity samples without blurry compromises. The mathematical framework connects deep learning to stochastic differential equations and gives a principled approach to density estimation.

Mental Model

Start with a clean image. Gradually add Gaussian noise over many steps until the image becomes pure noise. Now learn to reverse each step: given a noisy image, predict the slightly-less-noisy version. At generation time, start from pure noise and iteratively denoise to produce a clean image.

The key insight: you do not need to learn the reverse process from scratch. You can train on pairs (noisy image at step $t$ , noise that was added) because you control the forward process completely.

Formal Setup

Definition

Forward Process (Noise Schedule)

The forward process adds Gaussian noise over $T$ steps. Starting from data $\mathbf{x}_0 \sim q(\mathbf{x}_0)$ :

$q(\mathbf{x}_t | \mathbf{x}_{t-1}) = \mathcal{N}(\mathbf{x}_t; \sqrt{1 - \beta_t}\,\mathbf{x}_{t-1},\; \beta_t \mathbf{I})$

where $\beta_1, \ldots, \beta_T$ is a variance schedule. After $T$ steps, $\mathbf{x}_T \approx \mathcal{N}(\mathbf{0}, \mathbf{I})$ .

Definition

Closed-Form Forward Sampling

Define $\alpha_t = 1 - \beta_t$ and $\bar{\alpha}_t = \prod_{s=1}^{t} \alpha_s$ . Then you can sample any $\mathbf{x}_t$ directly:

$q(\mathbf{x}_t | \mathbf{x}_0) = \mathcal{N}(\mathbf{x}_t; \sqrt{\bar{\alpha}_t}\,\mathbf{x}_0,\; (1 - \bar{\alpha}_t)\mathbf{I})$

This is the reparameterization: $\mathbf{x}_t = \sqrt{\bar{\alpha}_t}\,\mathbf{x}_0 + \sqrt{1 - \bar{\alpha}_t}\,\boldsymbol{\epsilon}$ , where $\boldsymbol{\epsilon} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$ .

Definition

Reverse Process

The reverse process is a learned Markov chain:

$p_\theta(\mathbf{x}_{t-1} | \mathbf{x}_t) = \mathcal{N}(\mathbf{x}_{t-1}; \boldsymbol{\mu}_\theta(\mathbf{x}_t, t),\; \sigma_t^2 \mathbf{I})$

The neural network $\boldsymbol{\mu}_\theta$ (or equivalently a noise predictor $\boldsymbol{\epsilon}_\theta$ ) learns to undo each noise step.

Definition

Score Function

The score function of a distribution $p(\mathbf{x})$ is the gradient of the log-density:

$\mathbf{s}(\mathbf{x}) = \nabla_{\mathbf{x}} \log p(\mathbf{x})$

Score matching trains a model $\mathbf{s}_\theta(\mathbf{x}) \approx \nabla_{\mathbf{x}} \log p(\mathbf{x})$ without knowing $p(\mathbf{x})$ explicitly.

Main Theorems

Theorem

DDPM Training Objective

Statement

The negative log-likelihood is upper bounded by:

$-\log p_\theta(\mathbf{x}_0) \leq \mathbb{E}_q\!\Big[\underbrace{D_{\mathrm{KL}}(q(\mathbf{x}_T|\mathbf{x}_0) \| p(\mathbf{x}_T))}_{\text{prior matching}} + \sum_{t=2}^{T} \underbrace{D_{\mathrm{KL}}(q(\mathbf{x}_{t-1}|\mathbf{x}_t, \mathbf{x}_0) \| p_\theta(\mathbf{x}_{t-1}|\mathbf{x}_t))}_{\text{denoising matching}} - \underbrace{\log p_\theta(\mathbf{x}_0|\mathbf{x}_1)}_{\text{reconstruction}}\Big]$

When the noise predictor is parameterized as $\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t)$ , the simplified training loss reduces to:

$\mathcal{L}_{\text{simple}} = \mathbb{E}_{t, \mathbf{x}_0, \boldsymbol{\epsilon}}\!\big[\|\boldsymbol{\epsilon} - \boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t)\|^2\big]$

Intuition

Training a diffusion model reduces to: sample a random timestep $t$ , add the corresponding amount of noise $\boldsymbol{\epsilon}$ to a clean image, and train the network to predict $\boldsymbol{\epsilon}$ . The simplified loss drops KL weighting terms but works better in practice.

Proof Sketch

Start from the standard variational bound on $-\log p_\theta(\mathbf{x}_0)$ . Because the forward process is Gaussian and its posteriors $q(\mathbf{x}_{t-1}|\mathbf{x}_t,\mathbf{x}_0)$ are tractable Gaussians, each KL term becomes a squared error between predicted and true denoising means. Reparameterize the mean prediction as noise prediction to get the simplified loss.

Why It Matters

This is why diffusion training is so simple in practice: just predict the noise. The variational framework guarantees you are optimizing a valid bound on log-likelihood, but the simplified loss ignores weighting factors and still produces excellent samples.

Failure Mode

The simplified loss underweights certain timesteps (particularly high-noise steps) which matters for likelihood evaluation. If you care about exact log-likelihoods rather than sample quality, use the weighted objective.

Theorem

Denoising Score Matching

Statement

Training a neural network to denoise Gaussian-corrupted data is equivalent to learning the score function of the noise-perturbed distribution. Formally, for perturbation kernel $q_\sigma(\tilde{\mathbf{x}}|\mathbf{x}) = \mathcal{N}(\tilde{\mathbf{x}}; \mathbf{x}, \sigma^2 \mathbf{I})$ :

$\mathbb{E}_{q_\sigma(\tilde{\mathbf{x}}, \mathbf{x})}\!\big[\|\mathbf{s}_\theta(\tilde{\mathbf{x}}) - \nabla_{\tilde{\mathbf{x}}} \log q_\sigma(\tilde{\mathbf{x}}|\mathbf{x})\|^2\big] = \mathbb{E}_{q_\sigma(\tilde{\mathbf{x}})}\!\big[\|\mathbf{s}_\theta(\tilde{\mathbf{x}}) - \nabla_{\tilde{\mathbf{x}}} \log q_\sigma(\tilde{\mathbf{x}})\|^2\big] + C$

where $C$ is independent of $\theta$ .

Intuition

You cannot compute $\nabla_{\mathbf{x}} \log p(\mathbf{x})$ because you do not know $p$ . But if you add known noise, the denoising direction is the score of the noisy distribution. Predicting the noise that was added is the same as learning the score.

Why It Matters

This theorem unifies the denoising perspective (DDPM) with the score-matching perspective (score-based generative models). They are the same thing. The noise predictor $\boldsymbol{\epsilon}_\theta$ and the score model $\mathbf{s}_\theta$ are related by a simple rescaling.

The SDE Perspective

The discrete steps of DDPM are a discretization of a continuous stochastic differential equation. The forward SDE adds noise continuously:

$d\mathbf{x} = \mathbf{f}(\mathbf{x}, t)\,dt + g(t)\,d\mathbf{w}$

The reverse SDE (Anderson, 1982) runs time backward:

$d\mathbf{x} = [\mathbf{f}(\mathbf{x}, t) - g(t)^2 \nabla_{\mathbf{x}} \log p_t(\mathbf{x})]\,dt + g(t)\,d\bar{\mathbf{w}}$

The score $\nabla_{\mathbf{x}} \log p_t(\mathbf{x})$ is the only unknown, and it is exactly what the denoising network learns. This SDE framework allows adaptive step sizes, probability flow ODEs for deterministic sampling, and exact likelihood computation.

Classifier-Free Guidance

Conditional generation (text-to-image) uses classifier-free guidance: train the model with and without conditioning $c$ , then at sampling time extrapolate:

$\tilde{\boldsymbol{\epsilon}}_\theta(\mathbf{x}_t, c) = (1 + w)\,\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, c) - w\,\boldsymbol{\epsilon}_\theta(\mathbf{x}_t)$

where $w > 0$ is the guidance scale. This amplifies the influence of the conditioning signal. Higher $w$ gives more faithful but less diverse samples.

Flow Matching

Proposition

Flow Matching with Optimal Transport Paths

Statement

Instead of learning to reverse a stochastic noising process, flow matching learns a velocity field $\mathbf{v}_\theta(\mathbf{x}, t)$ that transports samples from noise to data along straight (optimal transport) paths. The training objective is:

$\mathcal{L}_{\text{FM}} = \mathbb{E}_{t, \mathbf{x}_0, \boldsymbol{\epsilon}}\!\big[\|\mathbf{v}_\theta(\mathbf{x}_t, t) - (\mathbf{x}_0 - \boldsymbol{\epsilon})\|^2\big]$

where $\mathbf{x}_t = (1-t)\boldsymbol{\epsilon} + t\,\mathbf{x}_0$ is the linear interpolation.

Intuition

Diffusion takes curved, noisy paths from data to noise and back. Flow matching takes straight lines. Straighter paths mean fewer integration steps at generation time, which means faster sampling.

Why It Matters

Flow matching is emerging as a simpler, faster alternative to diffusion. It avoids the noise schedule design problem entirely. Stable Diffusion 3 and many recent models use flow matching instead of DDPM.

Where This Shows Up in Current Papers

Image generation: Stable Diffusion, DALL-E 3, Imagen, Midjourney
Video generation: Sora (OpenAI), video world models
Audio synthesis: AudioLDM, diffusion-based music generation
Molecular design: diffusion for protein structure, drug molecules
3D generation: DreamFusion, point cloud diffusion, and Gaussian splatting representations
Robotics: diffusion policies for action generation

Common Confusions

Watch Out

Diffusion is not just 'reverse noise'

The naive description. "add noise, then learn to remove it". obscures the actual mathematics. The model learns the score function (gradient of log-density) of the noisy distribution at each noise level. Denoising is a consequence, not the definition. The SDE framework makes this precise: sampling is solving a reverse-time SDE using the learned score.

Watch Out

More steps is not always better

DDPM originally used 1000 denoising steps. Modern samplers (DDIM, DPM-Solver, consistency models) reduce this to 4-50 steps with minimal quality loss. The number of steps is a sampler design choice, not a fundamental property of the model.

Watch Out

Guidance scale is not a quality dial

Increasing classifier-free guidance scale $w$ makes outputs more aligned with the prompt but reduces diversity and introduces artifacts at high values. There is a sweet spot (typically $w \in [5, 15]$ for image models) beyond which quality degrades.

Summary

Forward process: gradually add Gaussian noise until data becomes pure noise
Reverse process: learn to denoise step by step (or equivalently, learn the score function)
Training reduces to predicting the noise added at each step
The SDE perspective unifies discrete (DDPM) and continuous formulations
Classifier-free guidance enables conditional generation by interpolating between conditional and unconditional predictions
Flow matching generalizes diffusion with straighter transport paths and simpler training

Exercises

ExerciseCore

Problem

Given $\bar{\alpha}_t = 0.01$ (i.e., 99% noise), write the expression for $\mathbf{x}_t$ in terms of $\mathbf{x}_0$ and $\boldsymbol{\epsilon}$ . How much of the original signal remains?

ExerciseAdvanced

Problem

Show that the denoising score matching objective (predicting noise $\boldsymbol{\epsilon}$ ) and the score matching objective (learning $\nabla_{\mathbf{x}} \log p_t(\mathbf{x})$ ) are equivalent up to a constant and a rescaling. What is the exact relationship between $\boldsymbol{\epsilon}_\theta$ and $\mathbf{s}_\theta$ ?

ExerciseResearch

Problem

Flow matching uses straight interpolation paths $\mathbf{x}_t = (1-t)\boldsymbol{\epsilon} + t\,\mathbf{x}_0$ . What happens if you use curved paths instead (e.g., geodesics on a data manifold)? When might this matter?