Activation Functions

Why This Matters

Without nonlinear activations, a feedforward network of any depth computes only an affine function. A composition of affine maps is affine. The activation function is what gives neural networks their representational power.

The choice of activation directly controls gradient flow during backpropagation. Sigmoid and tanh saturate for large inputs, driving gradients toward zero and stalling learning in deep networks. ReLU solved this for positive inputs but introduced dead neurons. Modern activations like GELU and SiLU provide smooth alternatives with better empirical performance in transformers. Proper weight initialization and batch normalization also help control gradient magnitude.

Formal Setup

Consider a feedforward network with $L$ layers. Layer $l$ computes:

$z^{(l)} = W^{(l)} a^{(l-1)} + b^{(l)}, \quad a^{(l)} = \sigma(z^{(l)})$

where $\sigma$ is the activation function applied elementwise, $W^{(l)}$ is the weight matrix, $b^{(l)}$ is the bias vector, $z^{(l)}$ is the pre-activation, and $a^{(l)}$ is the post-activation.

The gradient of the loss with respect to $z^{(l)}$ depends on $\sigma'(z^{(l)})$. If $\sigma'$ is small, gradients vanish. If $\sigma'$ is large, gradients explode.

Core Definitions

Definition

Sigmoid

$\sigma(x) = \frac{1}{1 + e^{-x}}$

Output range: $(0, 1)$. Derivative: $\sigma'(x) = \sigma(x)(1 - \sigma(x))$. Maximum derivative is $1/4$ at $x = 0$.

Definition

Hyperbolic Tangent

$\tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}} = 2\sigma(2x) - 1$

Output range: $(-1, 1)$. Zero-centered. Derivative: $\tanh'(x) = 1 - \tanh^2(x)$. Maximum derivative is $1$ at $x = 0$.

Definition

ReLU

$\text{ReLU}(x) = \max(0, x)$

Output range: $[0, \infty)$. Derivative: $1$ for $x > 0$, $0$ for $x < 0$, undefined at $x = 0$ (conventionally set to $0$).

Definition

Leaky ReLU

$\text{LeakyReLU}(x) = \max(\alpha x, x)$

where $\alpha$ is a small positive constant, typically $\alpha = 0.01$. Derivative: $1$ for $x > 0$, $\alpha$ for $x < 0$.

Definition

GELU

$\text{GELU}(x) = x \cdot \Phi(x)$

where $\Phi(x)$ is the standard normal CDF. Smooth function that weights inputs by their percentile under a Gaussian. Used in BERT, GPT-2, and most modern transformers.

Exact form: $x \cdot \Phi(x)$, with $\Phi$ the Gaussian CDF. Fast approximation (Hendrycks & Gimpel 2016): $\text{GELU}(x) \approx 0.5 x \left(1 + \tanh\left[\sqrt{2/\pi}\left(x + 0.0447 x^3\right)\right]\right).$ Frameworks sometimes default to the approximation for speed; numerical outputs differ at the third decimal place.

Definition

SiLU / Swish

$\text{SiLU}(x) = x \cdot \sigma(x)$

where $\sigma$ is the sigmoid function. Smooth, non-monotone (slightly negative for $x \approx -1.28$). Used in architectures like EfficientNet and LLaMA.

Definition

SwiGLU

$\text{SwiGLU}(x, W, V) = \text{SiLU}(xW) \otimes (xV)$

Gated linear unit variant (Shazeer 2020) built from SiLU and an elementwise-product gate. The feedforward block becomes $(\text{SiLU}(xW) \otimes (xV)) W_{\text{out}}$, doubling the parameter count of a plain MLP of the same hidden width, so implementations typically shrink the gated width by $2/3$ to match the FLOPs of a standard FFN. Used in LLaMA 1/2/3, Mistral, Gemma, and PaLM. A plain activation function page is incomplete without it since it is the FFN nonlinearity in most modern open-weight LLMs.

Main Theorems

Theorem

Nonlinearity Is Necessary for Universal Approximation

Statement

If $\sigma(x) = x$ (the identity function), then a feedforward network of any depth $L$ computes an affine function $f(x) = Ax + b$ for some matrix $A$ and vector $b$. The network cannot approximate arbitrary continuous functions.

Intuition

Each layer computes $W^{(l)} a^{(l-1)} + b^{(l)}$. Composing $L$ such maps gives $W^{(L)} W^{(L-1)} \cdots W^{(1)} x + \tilde{b}$, which is a single affine map. Depth without nonlinearity adds no representational power.

Proof Sketch

By induction on $L$. Base case: a single affine layer is affine. Inductive step: if $f_{L-1}(x) = A_{L-1}x + b_{L-1}$, then $f_L(x) = W^{(L)}(A_{L-1}x + b_{L-1}) + b^{(L)} = (W^{(L)} A_{L-1})x + (W^{(L)} b_{L-1} + b^{(L)})$, which is affine.

Why It Matters

This result explains why activation functions exist at all. The universal approximation theorem (Cybenko 1989, Hornik 1991) shows that a single hidden layer with a non-polynomial activation can approximate any continuous function on a compact set. Without nonlinearity, no amount of depth helps.

Failure Mode

The result says nonlinearity is necessary but says nothing about which nonlinearity is best. All non-polynomial activations yield universal approximation in theory. The practical differences are about gradient flow and optimization, not expressiveness.

Gradient Flow Analysis

Sigmoid and Tanh: Vanishing Gradients

During backpropagation through $L$ layers, the gradient includes a product of activation derivatives:

$\frac{\partial \mathcal{L}}{\partial z^{(1)}} \propto \prod_{l=1}^{L} \sigma'(z^{(l)})$

For sigmoid, $\sigma'(x) \leq 1/4$ everywhere. After $L$ layers, the gradient magnitude is at most $(1/4)^L$, which vanishes exponentially. With $L = 10$ layers, the gradient shrinks by a factor of at least $10^6$.

Tanh is better: $\tanh'(0) = 1$, so near-zero pre-activations pass gradients without attenuation. But for large $|z|$, $\tanh'(z) \to 0$ and the same saturation problem occurs.

ReLU: Constant Gradients, But Dead Neurons

ReLU has derivative $1$ for positive inputs. No saturation, no vanishing gradient for active neurons. This is why ReLU enabled training of deep networks (Krizhevsky et al. 2012).

The problem: if $z < 0$, the gradient is exactly $0$. A neuron whose pre-activation is negative for all training examples receives zero gradient updates and never recovers. This is the dying ReLU problem.

Leaky ReLU: No Dead Neurons

By setting $\sigma'(x) = \alpha > 0$ for $x < 0$, Leaky ReLU ensures every neuron receives a nonzero gradient. In practice, $\alpha = 0.01$ is standard. Parametric ReLU (PReLU) learns $\alpha$ from data.

Canonical Examples

Example

Sigmoid saturation in a 5-layer network

Consider a network with sigmoid activations and all pre-activations initialized at $z = 3$. Then $\sigma'(3) = \sigma(3)(1 - \sigma(3)) \approx 0.95 \times 0.05 = 0.045$. The gradient through 5 layers is attenuated by a factor of $(0.045)^5 \approx 1.8 \times 10^{-7}$. Training will stall in the early layers.

With ReLU and positive pre-activations, the gradient factor is $1^5 = 1$. No attenuation.

Common Confusions

Watch Out

ReLU is not differentiable at zero, but this does not matter in practice

ReLU has a kink at $x = 0$ where the derivative is undefined. In practice, the probability that any pre-activation is exactly zero is zero (under continuous distributions). Implementations set $\sigma'(0) = 0$ by convention. This has no measurable effect on training.

Watch Out

All common activations yield universal approximation

The universal approximation theorem holds for any non-polynomial continuous activation. ReLU, sigmoid, tanh, GELU, and SiLU all have this property. The choice of activation does not affect what functions a network can represent in the infinite-width limit. It affects what functions gradient-based optimization will find in finite time with finite width.

Watch Out

GELU is not just smooth ReLU

GELU is sometimes described as a smooth approximation to ReLU. This is misleading. GELU weights each input by its probability under a standard Gaussian: $\text{GELU}(x) = x \cdot P(Z \leq x)$ where $Z \sim \mathcal{N}(0,1)$. For large positive $x$, $\text{GELU}(x) \approx x$. For large negative $x$, $\text{GELU}(x) \approx 0$. The transition is smooth and data-dependent, unlike the hard threshold of ReLU.

Key Takeaways

• Without nonlinear activations, depth is useless: the network computes an affine map
• Sigmoid and tanh saturate for large $|x|$, causing vanishing gradients in deep networks
• ReLU solved vanishing gradients for positive inputs but introduced dead neurons
• Leaky ReLU, GELU, and SiLU avoid dead neurons while maintaining good gradient flow
• Activation choice affects optimization dynamics, not theoretical expressiveness
• Modern transformers use GELU, SiLU, or SwiGLU; SwiGLU is the FFN nonlinearity in LLaMA, Mistral, Gemma, and PaLM

Exercises

ExerciseCore

Problem

Compute $\sigma'(x)$ for the sigmoid function $\sigma(x) = 1/(1 + e^{-x})$ and show that $\max_x \sigma'(x) = 1/4$.

Hint

Reveal Solution

ExerciseCore

Problem

A 10-layer network uses sigmoid activations. Assume all pre-activations are at $x = 0$ (best case for gradient flow). What is the maximum gradient attenuation factor through all 10 layers from activation derivatives alone?

Hint

Reveal Solution

ExerciseAdvanced

Problem

Show that for any $\alpha > 0$, Leaky ReLU with slope $\alpha$ for negative inputs has no dead neurons. Precisely: show that for any weight configuration, $\partial \text{LeakyReLU}(z) / \partial z \neq 0$ for all $z \neq 0$.

Hint

Reveal Solution

Related Comparisons

• SwiGLU vs. GELU vs. ReLU

References

Canonical:

• Goodfellow, Bengio, Courville, Deep Learning (2016), Chapter 6.3
• Cybenko, "Approximation by Superpositions of a Sigmoidal Function" (1989)
• Hornik, "Approximation Capabilities of Multilayer Feedforward Networks" (1991)

Current:

• Hendrycks & Gimpel, "Gaussian Error Linear Units (GELUs)" (2016), arXiv:1606.08415
• Ramachandran, Zoph, Le, "Searching for Activation Functions" (2017) (Swish/SiLU), arXiv:1710.05941
• Shazeer, "GLU Variants Improve Transformer" (2020), arXiv:2002.05202 (SwiGLU)
• Nair & Hinton, "Rectified Linear Units Improve Restricted Boltzmann Machines" (2010)