Taylor Expansion

Why This Matters

Every optimization algorithm you use in ML is a Taylor approximation in disguise.

Gradient descent: approximate $f(x + h) \approx f(x) + \nabla f(x)^T h$ (first order), then move in the direction that decreases this linear approximation.

Newton's method: approximate $f(x + h) \approx f(x) + \nabla f(x)^T h + \frac{1}{2} h^T \nabla^2 f(x) h$ (second order), then minimize this quadratic exactly.

Adam, L-BFGS, natural gradient: all variations on which Taylor terms to keep and how to estimate them. Understanding Taylor expansion means understanding the foundation of all gradient-based optimization.

Single Variable Taylor Expansion

Definition

Taylor Polynomial

The $k$-th order Taylor polynomial of $f$ centered at $a$ is:

$T_k(x; a) = \sum_{j=0}^{k} \frac{f^{(j)}(a)}{j!}(x - a)^j$

This is the unique polynomial of degree $\leq k$ that matches $f$ and its first $k$ derivatives at $a$.

The cases that matter most:

First order (linear approximation):

$f(x + h) \approx f(x) + f'(x) h$

Second order (quadratic approximation):

$f(x + h) \approx f(x) + f'(x) h + \frac{1}{2} f''(x) h^2$

Remainder Terms

The approximation is only useful if you can bound the error.

Definition

Lagrange Remainder

If $f$ is $(k+1)$ times continuously differentiable on an interval containing $a$ and $x$, the remainder after the $k$-th order Taylor polynomial is:

$R_k(x; a) = \frac{f^{(k+1)}(c)}{(k+1)!}(x - a)^{k+1}$

for some $c$ between $a$ and $x$.

Definition

Integral Form of Remainder

Under the same conditions:

$R_k(x; a) = \int_a^x \frac{f^{(k+1)}(t)}{k!}(x - t)^k \, dt$

This form is often more useful for bounding because you can estimate the integral directly without locating the unknown point $c$.

Main Theorems

Theorem

Taylor Theorem with Lagrange Remainder

Statement

If $f: \mathbb{R} \to \mathbb{R}$ is $(k+1)$ times continuously differentiable on an interval $I$ containing $a$ and $x$, then:

$f(x) = \sum_{j=0}^{k} \frac{f^{(j)}(a)}{j!}(x-a)^j + \frac{f^{(k+1)}(c)}{(k+1)!}(x-a)^{k+1}$

for some $c$ between $a$ and $x$.

Intuition

The Taylor polynomial matches $f$ perfectly at $a$. The error depends on how much the $(k+1)$-th derivative varies. If $|f^{(k+1)}|$ is bounded by $M$ on $I$, then $|R_k| \leq M|x-a|^{k+1}/(k+1)!$.

Proof Sketch

Define $g(t) = f(x) - T_k(x; t) - C(x-t)^{k+1}$ where $C$ is chosen so $g(a) = 0$. Note $g(x) = 0$ trivially. By Rolle's theorem applied repeatedly, find $c$ between $a$ and $x$ where $g^{(k+1)}(c) = 0$. Solving for $C$ gives the Lagrange form.

Why It Matters

This is what lets you bound the error of gradient descent. If you use the first-order approximation and $f$ has bounded second derivative ($|f''| \leq L$), then the approximation error over a step of size $h$ is at most $\frac{L}{2}h^2$. This is exactly why gradient descent with step size $1/L$ converges for $L$-smooth functions.

Failure Mode

The theorem requires sufficient differentiability. If $f$ is only once differentiable, you cannot write a second-order expansion with Lagrange remainder. The bound is also only useful when $|x - a|$ is small; for large deviations the remainder can dominate.

Multivariate Taylor Expansion

For $f: \mathbb{R}^n \to \mathbb{R}$, the first-order expansion at $x$ is:

$f(x + h) = f(x) + \nabla f(x)^T h + O(\|h\|^2)$

where $\nabla f(x) \in \mathbb{R}^n$ is the gradient.

The second-order expansion is:

$f(x + h) = f(x) + \nabla f(x)^T h + \frac{1}{2} h^T \nabla^2 f(x) \, h + O(\|h\|^3)$

where $\nabla^2 f(x) \in \mathbb{R}^{n \times n}$ is the Hessian matrix with entries $[\nabla^2 f(x)]_{ij} = \frac{\partial^2 f}{\partial x_i \partial x_j}$.

If $f$ is twice continuously differentiable, the Hessian is symmetric. If additionally $f$ is convex, the Hessian is positive semidefinite at every point.

Canonical Examples

Example

Gradient descent step size from Taylor

Let $f$ be $L$-smooth, meaning $\|\nabla^2 f(x)\| \leq L$ everywhere. By second-order Taylor:

$f(x - \eta \nabla f(x)) \leq f(x) - \eta \|\nabla f(x)\|^2 + \frac{L \eta^2}{2} \|\nabla f(x)\|^2$

Minimizing the right side over $\eta$ gives $\eta^* = 1/L$, yielding:

$f(x - \frac{1}{L} \nabla f(x)) \leq f(x) - \frac{1}{2L} \|\nabla f(x)\|^2$

This is the descent lemma, the workhorse of gradient descent convergence proofs.

Common Confusions

Watch Out

Taylor expansion is local, not global

The Taylor polynomial centered at $a$ approximates $f$ well near $a$. It says nothing about the function far from $a$. The function $e^x$ has a Taylor series that converges everywhere, but $\sin(1/x)$ near 0 is not well-approximated by any polynomial centered at 0. In optimization, this locality is why step sizes must be small enough.

Watch Out

Second-order methods are not always better

Newton's method uses the Hessian and converges faster per step. But computing and inverting an $n \times n$ Hessian costs $O(n^2)$ storage and $O(n^3)$ time. For neural networks with millions of parameters, this is infeasible. First-order methods win by being cheap per step, even if they need more steps.

Exercises

ExerciseCore

Problem

Compute the second-order Taylor expansion of $f(x) = \log(1 + x)$ at $x = 0$. Use the Lagrange remainder to bound the error for $|x| \leq 0.1$.

Hint

Reveal Solution

ExerciseAdvanced

Problem

Let $f: \mathbb{R}^n \to \mathbb{R}$ be twice continuously differentiable with $\|\nabla^2 f(x)\|_{\text{op}} \leq L$ for all $x$. Prove that $\|f(y) - f(x) - \nabla f(x)^T(y-x)\| \leq \frac{L}{2}\|y-x\|^2$.

Hint

Reveal Solution

References

Canonical:

• Rudin, Principles of Mathematical Analysis (1976), Chapter 5
• Apostol, Mathematical Analysis (1974), Chapter 5

Current:

• Boyd & Vandenberghe, Convex Optimization (2004), Section 9.1 (Taylor approximation in optimization)
• Nesterov, Introductory Lectures on Convex Optimization (2004), Section 1.2