Dropout vs. Batch Normalization

What Each Does

Dropout randomly sets each neuron's activation to zero with probability $p$ during training. For a hidden layer activation $h$, dropout applies a mask:

$\tilde{h}_i = \frac{m_i}{1 - p} \cdot h_i, \quad m_i \sim \text{Bernoulli}(1 - p)$

The factor $1/(1 - p)$ (inverted dropout) ensures the expected value of $\tilde{h}_i$ equals $h_i$, so the same weights work at test time without modification. At inference, dropout is turned off and all activations are used.

Batch normalization normalizes each activation across the mini-batch, then applies a learned affine transform:

$\hat{x}_i = \frac{x_i - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}}, \quad y_i = \gamma \hat{x}_i + \beta$

where $\mu_B$ and $\sigma_B^2$ are the mean and variance computed over the current mini-batch, $\gamma$ and $\beta$ are learned scale and shift parameters, and $\epsilon$ is a small constant for numerical stability. At inference, batch statistics are replaced by running averages computed during training.

Why They Both "Regularize"

Dropout: implicit ensemble

Dropout trains a different subnetwork on each mini-batch. With $n$ neurons, there are $2^n$ possible masks, so dropout implicitly trains an exponential ensemble of subnetworks with shared weights. At test time, using all neurons with no mask approximates the geometric mean of the ensemble's predictions. This averaging reduces variance, which is the mechanism behind its regularization effect.

The Monte Carlo Dropout interpretation (Gal and Ghahramani, 2016) shows that dropout at test time approximates variational inference over the weights. Running multiple stochastic forward passes with dropout active produces samples from an approximate posterior, giving uncertainty estimates at no architectural cost.

Batch normalization: noise from mini-batch statistics

Batch norm was designed to stabilize training by reducing internal covariate shift (the distribution of each layer's inputs changing as earlier layers update). Its regularization effect is a side effect: because $\mu_B$ and $\sigma_B^2$ are estimated from the current mini-batch (not the full dataset), each example's normalized value depends on which other examples happen to be in the same batch. This injects noise into the forward pass, similar in spirit to dropout's stochastic masking.

The regularization from batch norm is weaker and less controllable than dropout. It depends on mini-batch size: larger batches produce more accurate statistics, reducing the noise. With batch size 1, batch norm cannot be used at all (the variance is undefined).

Side-by-Side Comparison

When Each Wins

Batch norm wins: deep convolutional networks

Batch norm is standard in CNNs (ResNets, EfficientNets). It enables higher learning rates, reduces sensitivity to initialization, and speeds up convergence. The per-channel normalization aligns well with convolutional structure, and the training stability gains outweigh the regularization from dropout in most vision tasks.

Dropout wins: fully connected layers and NLP

Dropout remains standard in the fully connected layers of classifiers and in recurrent/transformer architectures for NLP. In language models and sequence-to-sequence models, dropout (applied to embeddings, attention weights, and feed-forward layers) is the primary regularizer. Layer normalization replaces batch norm in these architectures because sequence lengths vary and batch statistics over tokens are less meaningful.

Batch norm wins: training speed

By normalizing activations, batch norm allows learning rates 5-10x larger than without it. This directly reduces training time. Dropout, in contrast, can slow convergence because it zeroes out a fraction of gradients each step.

Dropout wins: uncertainty quantification

MC Dropout provides uncertainty estimates by running multiple forward passes with dropout active. This is free if the model already uses dropout. Batch norm provides no analogous uncertainty mechanism.

The Interaction Problem: Why They Fight

Using dropout before batch normalization causes a well-documented problem called variance shift (Li et al., 2019). The issue:

1. During training, dropout randomly zeros activations, changing the variance of the layer output. The expected mean is preserved by the $1/(1-p)$ scaling, but the variance changes from $\text{Var}[h]$ to approximately $\text{Var}[h] \cdot (1 + p/(1-p))$ for the non-dropped units.

2. Batch norm computes running statistics (mean and variance) during training with dropout active. These statistics capture the inflated variance caused by dropout.

3. At inference, dropout is turned off. The activations now have different variance than what batch norm's running statistics expect. Batch norm normalizes using the wrong variance, shifting the effective distribution of each layer's output.

This mismatch degrades performance. The solutions are:

• Use dropout only after all batch norm layers. Place dropout in the final classifier head, not within the feature extractor.
• Replace dropout with other regularizers when using batch norm (data augmentation, weight decay, stochastic depth).
• Use Gaussian dropout or other noise schemes that maintain the same variance at train and test time.
• Use layer normalization instead of batch normalization, since layer norm normalizes each sample independently (no running statistics to mismatch).

Modern Best Practices

In modern architectures, the trend is toward using one or the other, not both:

• Vision (CNNs): Batch norm in the backbone, dropout only in the final classifier head (if at all). ResNets use batch norm everywhere with no dropout. EfficientNets add dropout at the head.
• Vision Transformers (ViTs): Layer norm replaces batch norm. Dropout is used in attention and MLP layers.
• Transformers (NLP): Layer norm + dropout. No batch norm. Dropout applied to attention weights, residual connections, and feed-forward blocks.
• GANs: Batch norm in the generator (not the discriminator). Spectral normalization often replaces batch norm in the discriminator. Dropout is rarely used.

Common Confusions

References

1. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. (2014). "Dropout: A Simple Way to Prevent Neural Networks from Overfitting." JMLR, 15(56), 1929-1958.
2. Ioffe, S. and Szegedy, C. (2015). "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift." ICML 2015.
3. Gal, Y. and Ghahramani, Z. (2016). "Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning." ICML 2016.
4. Li, X., Chen, S., Hu, X., and Yang, J. (2019). "Understanding the Disharmony between Dropout and Batch Normalization by Variance Shift." CVPR 2019. (The variance shift analysis and solutions.)
5. Ba, J. L., Kiros, J. R., and Hinton, G. E. (2016). "Layer Normalization." arXiv:1607.06450. (Layer norm as an alternative to batch norm for sequence models.)
6. He, K., Zhang, X., Ren, S., and Sun, J. (2016). "Deep Residual Learning for Image Recognition." CVPR 2016. (ResNets use batch norm without dropout.)
7. Santurkar, S., Tsipras, D., Ilyas, A., and Madry, A. (2018). "How Does Batch Normalization Help Optimization?" NeurIPS 2018. (Challenges the internal covariate shift explanation; argues batch norm smooths the loss landscape.)