Generative Adversarial Nets

Overview

Goodfellow et al. (2014) proposed a generative model that does not write down a likelihood. Instead, two networks play a zero-sum game. The generator $G$ maps noise $z \sim p_z$ to samples $G(z)$ in data space. The discriminator $D$ tries to distinguish real samples drawn from $p_{\text{data}}$ from generated samples. Each gradient step pushes $D$ towards optimal classification and $G$ towards fooling the current $D$.

The contribution is twofold. The framework is the first practical method for training implicit-density models — models that can sample but cannot evaluate $p(x)$ — at neural-network scale. The analysis is the first to give a clean theoretical target: the paper proves that, when $D$ is at its pointwise optimum, the generator's objective is equivalent to minimising the Jensen-Shannon divergence between $p_g$ and $p_{\text{data}}$.

Mathematical Contributions

The minimax objective

The generator and discriminator play the following game:

$\min_G \max_D V(D, G) = \mathbb{E}_{x \sim p_{\text{data}}}[\log D(x)] + \mathbb{E}_{z \sim p_z}[\log(1 - D(G(z)))]$

The discriminator maximises the expected log-likelihood of correct classification. The generator minimises the same quantity, which is equivalent to maximising the discriminator's loss.

Optimal discriminator at fixed generator

For a fixed $G$ inducing distribution $p_g$ on samples, the inner maximisation is pointwise: at each $x$, the integrand is $p_{\text{data}}(x) \log D(x) + p_g(x) \log(1 - D(x))$. Setting the derivative to zero gives:

$D^*_G(x) = \frac{p_{\text{data}}(x)}{p_{\text{data}}(x) + p_g(x)}$

This holds whenever $p_{\text{data}}(x) + p_g(x) > 0$. The discriminator's optimal output at any point is the posterior probability that the point came from the data distribution under a uniform class prior.

The Jensen-Shannon equivalence

Substituting $D^*_G$ back into $V$ gives:

$C(G) = \max_D V(D, G) = -\log 4 + 2 \cdot D_{\text{JS}}(p_{\text{data}} \| p_g)$

where $D_{\text{JS}}$ is the Jensen-Shannon divergence. So with $D$ at its optimum, the generator minimises a non-negative divergence whose unique minimiser at zero is $p_g = p_{\text{data}}$. This is the paper's existence-and-uniqueness argument for the equilibrium.

Convergence to a Nash equilibrium

The paper proves that, if $G$ and $D$ have enough capacity and at each step $D$ is optimised to its optimum and $p_g$ is updated to improve $C(G)$, then $p_g$ converges to $p_{\text{data}}$. The proof is short and uses convexity of $\mathbb{E}_{x \sim p_{\text{data}}}[\log D^*_G(x)] + \mathbb{E}_{x \sim p_g}[\log(1 - D^*_G(x))]$ in $p_g$. The caveat — and the source of subsequent literature — is that the actual training procedure does not optimise $D$ to convergence at every step; it interleaves a fixed number of $D$ updates per $G$ update.

The vanishing gradient problem

When $D$ is near optimal and $G$ produces samples easily distinguished from data, $\log(1 - D(G(z)))$ saturates: $D(G(z)) \to 0$, the log goes to $\log 1 = 0$, and $\nabla_G$ is essentially zero. The paper notes this and recommends the non-saturating loss:

$\mathcal{L}_G = -\mathbb{E}_{z \sim p_z}[\log D(G(z))]$

This is what every implementation actually uses. It does not change the equilibrium, but the gradient is large precisely where $G$ needs to learn — when $D$ is winning.

Connections to TheoremPath Topics

• Generative adversarial networks — the modern treatment including spectral normalisation, gradient penalty, and StyleGAN-era refinements.
• KL divergence — JS is the symmetric mixture-based variant; the paper's bound is in JS, not KL.
• Wasserstein distances — the alternative metric that motivated WGAN, where the original GAN's pathologies (mode collapse, unstable gradients) are partly resolved.
• Cross-entropy loss — the binary cross-entropy that the discriminator's loss reduces to.
• Variational autoencoders — the contemporaneous alternative for likelihood-based generation.

Why It Matters Now

GANs are no longer the state of the art for unconditional image generation; diffusion models took over by 2022. But the paper still matters for three reasons.

First, the implicit-density framing — train a sampler, not a likelihood — generalised. Energy-based models, score-matching, and even some diffusion variants inherit the idea that the model does not need a tractable density to be useful. Modern flow-matching and score-based models still borrow the discriminator framing in some training-stability extensions.

Second, the minimax framing is the technical ancestor of essentially every adversarial-training method in machine learning, including adversarial robustness, domain adaptation via DANN, and robustness-via-augmentation pipelines. The paper introduced "adversarial" as a serious objective design tool.

Third, the analysis is a good pedagogical example of how the optimum of a learning game can be characterised by an information-theoretic divergence. The same style of argument — fix one player at its optimum, recover a divergence between the data distribution and the model — recurs in noise-contrastive estimation, mutual-information estimation, and contrastive predictive coding.

References

Canonical:

• Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2014). "Generative Adversarial Nets." NeurIPS 2014. arXiv:1406.2661.

Direct precursors:

• Hinton, G. E., & Salakhutdinov, R. R. (2006). "Reducing the Dimensionality of Data with Neural Networks." Science.
• Gutmann, M., & Hyvärinen, A. (2010). "Noise-contrastive estimation." AISTATS.

Stability and theory follow-ups:

• Arjovsky, M., Chintala, S., & Bottou, L. (2017). "Wasserstein GAN." ICML. arXiv:1701.07875.
• Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., & Courville, A. (2017). "Improved Training of Wasserstein GANs." NeurIPS. arXiv:1704.00028.
• Miyato, T., Kataoka, T., Koyama, M., & Yoshida, Y. (2018). "Spectral Normalization for Generative Adversarial Networks." ICLR. arXiv:1802.05957.
• Mescheder, L., Geiger, A., & Nowozin, S. (2018). "Which Training Methods for GANs do actually Converge?" ICML. arXiv:1801.04406.

What displaced GANs:

• Ho, J., Jain, A., & Abbeel, P. (2020). "Denoising Diffusion Probabilistic Models." NeurIPS. arXiv:2006.11239.

Standard textbook:

• Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press. Chapter 20.10.4.