The Hessian Matrix

Why This Matters

The gradient tells you which way is downhill. The Hessian tells you how the landscape curves. is it a bowl (minimum), a dome (maximum), or a saddle? Every second-order optimization method (Newton's method, natural gradient, L-BFGS) depends on the Hessian or an approximation to it. In deep learning, the eigenvalues of the Hessian reveal the geometry of the loss landscape: sharp minima versus flat minima, the prevalence of saddle points, and why certain optimizers work better than others.

If the gradient is the first thing you learn in optimization, the Hessian is the second. literally.

Mental Model

For a function of one variable, $f''(x)$ tells you the curvature: positive means concave up (bowl), negative means concave down (dome). The Hessian is the multivariable generalization. But in multiple dimensions, the curvature can be different in different directions. The Hessian captures all of this information as a matrix. Its eigenvalues are the curvatures along the principal directions.

Core Definitions

Definition

Hessian Matrix

For a twice-differentiable function $f: \mathbb{R}^n \to \mathbb{R}$, the Hessian matrix at a point $x$ is the $n \times n$ matrix of second partial derivatives:

$[H_f(x)]_{ij} = \frac{\partial^2 f}{\partial x_i \partial x_j}$

Explicitly:

$H_f(x) = \begin{pmatrix} \frac{\partial^2 f}{\partial x_1^2} & \frac{\partial^2 f}{\partial x_1 \partial x_2} & \cdots & \frac{\partial^2 f}{\partial x_1 \partial x_n} \\ \frac{\partial^2 f}{\partial x_2 \partial x_1} & \frac{\partial^2 f}{\partial x_2^2} & \cdots & \frac{\partial^2 f}{\partial x_2 \partial x_n} \\ \vdots & \vdots & \ddots & \vdots \\ \frac{\partial^2 f}{\partial x_n \partial x_1} & \frac{\partial^2 f}{\partial x_n \partial x_2} & \cdots & \frac{\partial^2 f}{\partial x_n^2} \end{pmatrix}$

Definition

Symmetry of the Hessian (Schwarz's Theorem)

If the second partial derivatives of $f$ are continuous (i.e., $f \in C^2$), then the mixed partials are equal:

$\frac{\partial^2 f}{\partial x_i \partial x_j} = \frac{\partial^2 f}{\partial x_j \partial x_i}$

This means the Hessian is a symmetric matrix: $H_f = H_f^T$. Consequently, the Hessian has real eigenvalues and orthogonal eigenvectors. all the machinery of symmetric matrix theory applies.

Second-Order Taylor Expansion

The Hessian appears in the second-order Taylor expansion of $f$ around a point $x$:

$f(x + h) \approx f(x) + \nabla f(x)^T h + \frac{1}{2} h^T H_f(x) \, h$

The three terms have clear meanings:

• $f(x)$: the value at the current point
• $\nabla f(x)^T h$: the linear (first-order) change. The gradient tells you the slope
• $\frac{1}{2} h^T H_f(x) \, h$: the quadratic (second-order) change. the Hessian tells you the curvature

This quadratic approximation is the basis for Newton's method and for classifying critical points.

The Second Derivative Test

Theorem

Second Derivative Test (Multivariate)

Statement

Let $x^*$ be a critical point of $f$ (i.e., $\nabla f(x^*) = 0$). Then:

• If $H_f(x^*)$ is positive definite (all eigenvalues $> 0$), then $x^*$ is a strict local minimum.
• If $H_f(x^*)$ is negative definite (all eigenvalues $< 0$), then $x^*$ is a strict local maximum.
• If $H_f(x^*)$ is indefinite (has both positive and negative eigenvalues), then $x^*$ is a saddle point.
• If $H_f(x^*)$ is positive (or negative) semidefinite (some eigenvalue is zero), the test is inconclusive: higher-order terms are needed.

Intuition

At a critical point, $\nabla f = 0$, so the Taylor expansion becomes $f(x^* + h) \approx f(x^*) + \frac{1}{2} h^T H h$. If $H$ is positive definite, the quadratic form $h^T H h > 0$ for all $h \neq 0$, so $f$ increases in every direction from $x^*$. It is a minimum. If $H$ is indefinite, $f$ increases in some directions and decreases in others. a saddle point.

Proof Sketch

At the critical point $x^*$, Taylor's theorem with remainder gives:

$f(x^* + h) = f(x^*) + \frac{1}{2} h^T H_f(x^*) h + o(\|h\|^2)$

If $H_f(x^*)$ is positive definite with minimum eigenvalue $\lambda_{\min} > 0$, then $h^T H h \geq \lambda_{\min} \|h\|^2$, so for sufficiently small $h$:

$f(x^* + h) \geq f(x^*) + \frac{\lambda_{\min}}{2}\|h\|^2 + o(\|h\|^2) > f(x^*)$

The $o(\|h\|^2)$ term is dominated by the quadratic term for small $\|h\|$. The indefinite case follows by choosing $h$ along eigenvectors with positive and negative eigenvalues.

Why It Matters

This test is how you verify that a critical point found by setting $\nabla f = 0$ is actually a minimum (or maximum or saddle). In optimization, you need to know that your converged solution is a local minimum, not a saddle point. In deep learning, this reveals the structure of the loss landscape: are we stuck at a saddle point, or have we found a genuine minimum?

Failure Mode

The test is inconclusive when $H$ is semidefinite (has a zero eigenvalue). Example: $f(x, y) = x^4 + y^2$ at the origin has $H = \text{diag}(0, 2)$, which is positive semidefinite. The origin is a minimum, but the second derivative test cannot confirm this: you need to examine the fourth-order term. Also, the test is local: a positive definite Hessian at $x^*$ does not guarantee $x^*$ is a global minimum.

The Hessian in Optimization

Newton's Method

Newton's method for minimizing $f$ uses the Hessian directly. At each iteration:

$x_{t+1} = x_t - [H_f(x_t)]^{-1} \nabla f(x_t)$

This is equivalent to minimizing the quadratic Taylor approximation at each step. When $f$ is quadratic, Newton's method converges in one step. For general smooth functions near a minimum, Newton's method converges quadratically (doubling the number of correct digits per iteration).

The cost: computing and inverting the $n \times n$ Hessian is $O(n^2)$ storage and $O(n^3)$ computation. For deep networks with millions of parameters, this is prohibitive.

Quasi-Newton Methods

Quasi-Newton methods (BFGS, L-BFGS) approximate the Hessian using only gradient information. L-BFGS stores a low-rank approximation using the last $m$ gradient differences, requiring only $O(mn)$ storage. These methods achieve superlinear convergence. faster than gradient descent, slower than Newton. at a fraction of the cost.

Hessian-Vector Products

You often do not need the full Hessian. just its action on a vector $v$: $H_f(x) \cdot v$. This Hessian-vector product can be computed in $O(n)$ time (the same cost as a gradient computation) using the identity:

$H_f(x) \cdot v = \lim_{\epsilon \to 0} \frac{\nabla f(x + \epsilon v) - \nabla f(x)}{\epsilon}$

In automatic differentiation, this is computed exactly via a forward-over- reverse pass. Hessian-vector products enable Krylov methods (conjugate gradients on the Hessian) and are the basis for understanding curvature without storing the full matrix.

Hessian Eigenvalues and the Loss Landscape

In deep learning, the Hessian of the loss with respect to the parameters reveals the geometry of the loss landscape:

• Eigenvalue spectrum: the distribution of Hessian eigenvalues tells you about curvature. A few large eigenvalues with many near zero suggests a low-dimensional structure in the loss landscape.
• Sharp vs. flat minima: a minimum with large Hessian eigenvalues is "sharp" (the loss increases quickly when you move away). A minimum with small eigenvalues is "flat." There is a conjecture (debated) that flat minima generalize better.
• Saddle points: in high dimensions, critical points are almost always saddle points (not local minima), because a random symmetric matrix is indefinite with high probability. The ratio of negative eigenvalues to total eigenvalues is called the index of the saddle point.

Canonical Examples

Example

Hessian of a quadratic form

Let $f(x) = \frac{1}{2} x^T A x + b^T x + c$ where $A$ is symmetric. The gradient is $\nabla f(x) = Ax + b$, and the Hessian is:

$H_f(x) = A$

The Hessian is constant. independent of $x$. The function is convex if and only if $A$ is positive semidefinite. For quadratics, the curvature is the same everywhere, which is why Newton's method converges in one step.

Example

Hessian of a simple two-variable function

Let $f(x, y) = x^2 y + y^3$.

Gradient: $\nabla f = (2xy, \; x^2 + 3y^2)^T$.

Second partial derivatives:

• $\frac{\partial^2 f}{\partial x^2} = 2y$
• $\frac{\partial^2 f}{\partial x \partial y} = 2x$
• $\frac{\partial^2 f}{\partial y^2} = 6y$

Hessian:

$H_f(x, y) = \begin{pmatrix} 2y & 2x \\ 2x & 6y \end{pmatrix}$

At the origin $(0,0)$: $H_f = \begin{pmatrix} 0 & 0 \\ 0 & 0 \end{pmatrix}$, the zero matrix. The second derivative test is inconclusive. (Indeed, the origin is a degenerate critical point.)

At $(0, 1)$: $H_f = \begin{pmatrix} 2 & 0 \\ 0 & 6 \end{pmatrix}$. positive definite, so this is a local minimum (though $(0,1)$ is not a critical point since $\nabla f(0,1) = (0, 3)^T \neq 0$).

Common Confusions

Watch Out

The Hessian is NOT the same as the outer product of gradients

In deep learning, the Fisher information matrix $\mathbb{E}[\nabla \log p \cdot (\nabla \log p)^T]$ is sometimes confused with the Hessian. They are different objects. The Hessian involves second derivatives of a single function; the Fisher involves first derivatives averaged over data. They coincide only under specific conditions (e.g., for the negative log-likelihood of an exponential family at the true parameters).

Watch Out

Positive definite Hessian at a point does not mean global convexity

$H_f(x^*) \succ 0$ means $f$ is locally convex near $x^*$. The function could be non-convex elsewhere. Global convexity requires $H_f(x) \succeq 0$ for all $x$, which is a much stronger condition.

Watch Out

The Hessian exists but may be useless to compute explicitly

For a function of $n$ variables, the Hessian is an $n \times n$ matrix. For a neural network with $n = 10^8$ parameters, the Hessian has $10^{16}$ entries . It cannot be stored, let alone inverted. This is why Hessian-vector products and low-rank approximations (L-BFGS, Kronecker-factored approximations) are essential in practice.

Summary

• The Hessian $H_f(x)$ is the $n \times n$ matrix of second partial derivatives: $[H]_{ij} = \partial^2 f / \partial x_i \partial x_j$
• Schwarz's theorem: if $f \in C^2$, the Hessian is symmetric
• Second-order Taylor: $f(x+h) \approx f(x) + \nabla f^T h + \frac{1}{2} h^T H h$
• Second derivative test: positive definite $\Rightarrow$ local min, negative definite $\Rightarrow$ local max, indefinite $\Rightarrow$ saddle
• Newton's method: $x_{t+1} = x_t - H^{-1} \nabla f$. uses Hessian directly, converges quadratically
• Hessian-vector products cost $O(n)$ via autodiff. same as a gradient
• Hessian eigenvalues reveal loss landscape geometry: sharp vs. flat minima, saddle point structure

Exercises

ExerciseCore

Problem

Compute the Hessian of $f(x, y) = x^2 y + y^3$ at the point $(1, 2)$. Determine whether the Hessian at this point is positive definite, negative definite, or indefinite.

Hint

Reveal Solution

ExerciseAdvanced

Problem

Let $f(x) = \|Ax - b\|^2$ for $A \in \mathbb{R}^{m \times n}$ and $b \in \mathbb{R}^m$. Compute $\nabla f(x)$ and $H_f(x)$. Under what condition on $A$ is the Hessian positive definite (guaranteeing a unique global minimum)?

Hint

Reveal Solution

References

Canonical:

• Nocedal & Wright, Numerical Optimization (2006), Chapters 2-3
• Boyd & Vandenberghe, Convex Optimization (2004), Appendix A
• Magnus & Neudecker, Matrix Differential Calculus with Applications in Statistics and Econometrics (2019), Chapters 5-6

Current:

• Ghorbani, Krishnan, Xiao, "An Investigation into Neural Net Optimization via Hessian Eigenvalue Density" (2019)

Next Topics

The natural next steps from the Hessian:

• Newton's method: using the Hessian for second-order optimization, convergence theory, and practical modifications
• Convex optimization basics: where the Hessian being positive semidefinite everywhere guarantees a global minimum