Energy-Based Models

Why This Matters

Energy-based models provide the most general framework for specifying probability distributions over high-dimensional data. Every probabilistic model --- GANs, VAEs, diffusion models, Boltzmann machines --- can be viewed through the energy lens. Understanding EBMs gives you a unified language for generative modeling and reveals why certain training strategies work.

The recent explosion of diffusion models is. It is an EBM story: the score function that diffusion models learn is exactly the gradient of the log-density of an energy-based model.

Mental Model

Think of the energy function $E(x)$ as a landscape over the data space. Valleys (low energy) correspond to likely data configurations. Peaks (high energy) correspond to unlikely configurations. Training an EBM means sculpting this landscape so that the valleys align with the data distribution: real images get low energy, noise gets high energy.

The challenge: converting this energy landscape into a proper probability distribution requires normalizing over all possible configurations --- an intractable integral in high dimensions. The entire field of EBM training is about avoiding or approximating this normalization.

Formal Setup and Notation

Let $x \in \mathcal{X} \subseteq \mathbb{R}^d$ be a data point. An energy-based model defines an energy function $E_\theta: \mathcal{X} \to \mathbb{R}$ parameterized by $\theta$.

Definition

Energy Function

The energy function $E_\theta(x)$ assigns a scalar energy to each configuration $x$. Lower energy means higher probability. There are no constraints on $E_\theta$ --- it can be any function from $\mathcal{X}$ to $\mathbb{R}$, including a neural network.

Definition

Boltzmann Distribution

The energy function induces a probability distribution via the Boltzmann distribution (also called the Gibbs distribution):

$p_\theta(x) = \frac{\exp(-E_\theta(x))}{Z(\theta)}$

where the partition function is:

$Z(\theta) = \int_{\mathcal{X}} \exp(-E_\theta(x))\, dx$

The partition function ensures $p_\theta$ integrates to 1. Computing $Z(\theta)$ requires integrating over all of $\mathcal{X}$ --- this is intractable for high-dimensional $x$.

Core Definitions

The score function of an EBM is the gradient of the log-density:

$\nabla_x \log p_\theta(x) = -\nabla_x E_\theta(x)$

The partition function $Z(\theta)$ disappears because it does not depend on $x$. This observation is the foundation of score-based methods.

The free energy is $F(\theta) = -\log Z(\theta)$. The log-likelihood of a single observation is:

$\log p_\theta(x) = -E_\theta(x) - \log Z(\theta) = -E_\theta(x) + F(\theta)$

Main Theorems

Theorem

MLE Gradient for Energy-Based Models

Statement

The gradient of the log-likelihood with respect to parameters $\theta$ is:

$\nabla_\theta \log p_\theta(x) = -\nabla_\theta E_\theta(x) + \mathbb{E}_{x' \sim p_\theta}[\nabla_\theta E_\theta(x')]$

Equivalently, the gradient of the average log-likelihood over the data distribution $p_{\text{data}}$ is:

$\nabla_\theta \mathbb{E}_{p_{\text{data}}}[\log p_\theta(x)] = -\mathbb{E}_{p_{\text{data}}}[\nabla_\theta E_\theta(x)] + \mathbb{E}_{p_\theta}[\nabla_\theta E_\theta(x)]$

Intuition

MLE training pushes down the energy of real data (first term) and pushes up the energy of model samples (second term). This is a "contrastive" update: make real data more likely by making it lower energy, and make model samples less likely by making them higher energy. The challenge is computing the second expectation, which requires sampling from $p_\theta$ --- itself an intractable distribution.

Proof Sketch

Differentiate $\log p_\theta(x) = -E_\theta(x) - \log Z(\theta)$. The first term gives $-\nabla_\theta E_\theta(x)$. For the second, note:

$\nabla_\theta \log Z(\theta) = \frac{\nabla_\theta Z(\theta)}{Z(\theta)} = \frac{\int \exp(-E_\theta(x'))(-\nabla_\theta E_\theta(x'))\,dx'}{Z(\theta)} = -\mathbb{E}_{p_\theta}[\nabla_\theta E_\theta(x')]$

Combining: $\nabla_\theta \log p_\theta(x) = -\nabla_\theta E_\theta(x) + \mathbb{E}_{p_\theta}[\nabla_\theta E_\theta(x')]$.

Why It Matters

This gradient formula is the starting point for all EBM training methods. Every training algorithm for EBMs is a strategy for approximating the model expectation $\mathbb{E}_{p_\theta}[\nabla_\theta E_\theta(x')]$, which requires sampling from $p_\theta$.

Failure Mode

If the MCMC sampler used to approximate the model expectation does not mix well, the gradient estimate is biased. The model may learn to place low energy only on the MCMC chain's trajectory rather than on the true data manifold, leading to poor generalization.

Training Methods

Contrastive Divergence (CD)

Since sampling from $p_\theta$ is intractable, Hinton (2002) proposed contrastive divergence: initialize an MCMC chain at a data point and run it for only $k$ steps (typically $k = 1$). Use the resulting sample $\tilde{x}$ as an approximation to a sample from $p_\theta$.

The CD-$k$ gradient is:

$\nabla_\theta^{\text{CD-}k} = -\nabla_\theta E_\theta(x_{\text{data}}) + \nabla_\theta E_\theta(\tilde{x}_k)$

CD is biased (the chain has not converged to $p_\theta$) but works surprisingly well in practice. It was the workhorse for training restricted Boltzmann machines and deep belief networks.

Score Matching

Theorem

Score Matching Avoids the Partition Function

Statement

The Fisher divergence between $p_{\text{data}}$ and $p_\theta$:

$D_F(p_{\text{data}} \| p_\theta) = \frac{1}{2}\mathbb{E}_{p_{\text{data}}}\!\left[\|\nabla_x \log p_{\text{data}}(x) - \nabla_x \log p_\theta(x)\|^2\right]$

can be rewritten (via integration by parts) as:

$D_F = \mathbb{E}_{p_{\text{data}}}\!\left[\frac{1}{2}\|\nabla_x E_\theta(x)\|^2 + \text{tr}(\nabla_x^2 E_\theta(x))\right] + \text{const}$

where the constant does not depend on $\theta$.

Intuition

Score matching trains the model by matching the gradient of the log-density (the score) rather than the density itself. Since the score $\nabla_x \log p_\theta(x) = -\nabla_x E_\theta(x)$ does not involve $Z(\theta)$, the partition function drops out entirely. You can train an EBM without ever computing or approximating $Z(\theta)$.

Connection to Diffusion Models

Diffusion models learn a score function $s_\theta(x, t) \approx \nabla_x \log p_t(x)$ where $p_t$ is the data distribution convolved with Gaussian noise of variance $t$. This is precisely a noise-conditional score matching objective applied to an EBM at each noise level.

The denoising score matching identity:

$\mathbb{E}_{p_{\text{data}}(x_0)}\mathbb{E}_{p(x|x_0)}\!\left[\|s_\theta(x, t) - \nabla_x \log p(x|x_0)\|^2\right]$

trains the model to point toward the clean data from the noisy version. This is equivalent to learning the energy gradient $-\nabla_x E_\theta$ at multiple noise scales.

Canonical Examples

Example

Restricted Boltzmann Machine

An RBM defines energy over visible units $v$ and hidden units $h$: $E(v, h) = -v^T W h - b^T v - c^T h$. The marginal $p(v)$ is obtained by summing out $h$, which is tractable because $h$ is conditionally independent given $v$. RBMs were the first scalable EBMs, trained with CD.

Example

Modern deep EBM

Parameterize $E_\theta(x)$ as a deep convolutional network mapping images to a scalar energy. Train via MCMC-based contrastive learning or score matching. The resulting model assigns low energy to realistic images and high energy to noise, enabling generation via Langevin dynamics (gradient descent on $E$ with added noise).

Common Confusions

Watch Out

Low energy does not mean the point is a mode

A point can have low energy but low probability if it sits in a region where many other points also have low energy (high partition function contribution from that region). Probability depends on both energy and volume. This is the energy-entropy tradeoff.

Watch Out

EBMs are not just Boltzmann machines

Boltzmann machines are a specific type of EBM with bilinear energy functions over binary variables. Modern EBMs use arbitrary neural network energy functions over continuous spaces. The framework is far more general than its historical association with Boltzmann machines suggests.

Summary

• Energy function $E_\theta(x)$: low energy = high probability
• Boltzmann distribution: $p_\theta(x) = \exp(-E_\theta(x))/Z(\theta)$
• Partition function $Z(\theta)$ is intractable in high dimensions
• MLE gradient = push down data energy, push up model-sample energy
• Score function $\nabla_x \log p = -\nabla_x E$ avoids $Z(\theta)$
• Diffusion models are score-matching EBMs at multiple noise levels

Exercises

ExerciseCore

Problem

Show that the score function $\nabla_x \log p_\theta(x) = -\nabla_x E_\theta(x)$ does not depend on the partition function $Z(\theta)$.

Hint

Reveal Solution

ExerciseAdvanced

Problem

Derive the MLE gradient for an EBM. Specifically, show that $\nabla_\theta \log Z(\theta) = -\mathbb{E}_{p_\theta}[\nabla_\theta E_\theta(x)]$.

Hint

Reveal Solution

ExerciseResearch

Problem

Contrastive divergence with $k=1$ step is known to be a biased estimator of the MLE gradient. Explain intuitively why this bias exists and describe a scenario where it causes the model to fail.

Hint

Reveal Solution

References

Canonical:

• LeCun, Chopra, Hadsell, Ranzato, Huang, A Tutorial on Energy-Based Learning (2006)
• Hinton, Training Products of Experts by Minimizing Contrastive Divergence (2002)

Current:

• Song & Ermon, Generative Modeling by Estimating Gradients of the Data Distribution (NeurIPS 2019)

• Du & Mordatch, Implicit Generation and Modeling with Energy-Based Models (NeurIPS 2019)

• Bishop, Pattern Recognition and Machine Learning (2006), Chapters 1-14

Next Topics

The natural next steps from energy-based models:

• Diffusion models: score matching at multiple noise levels, the modern EBM
• [Variational autoencoders](/topics/autoencoders): a different approach to intractable normalization via amortized inference