A Hessian Eigenvalue Is Curvature in One Direction

A Hessian matrix does not belong to the whole function at once. It belongs to the function at a point. First choose a point $a\in {\mathbb{R}}^n$. Then freeze the second derivatives there:

$$H=H_f(a).$$

Now $H$ is just a square matrix. It can be studied as a linear map. When it has an eigenvalue $\lambda$ with eigenvector $v\ne 0$, it satisfies

$$Hv=\lambda v.$$

This says that, in the direction $v$, the Hessian does not rotate or mix the direction with others. It only rescales it. For a Hessian, that rescaling has a geometric meaning: it is the curvature of the function in that direction.

The contours show a local quadratic shape. Along the long direction, the surface changes slowly. Along the narrow direction, it changes quickly. The eigenvectors mark those unmixed directions, and the eigenvalues measure how sharply the surface bends along them.

The point is frozen first

The Hessian changes from point to point. So the phrase “an eigenvalue of the Hessian” really means:

take the Hessian at a specific point, then take the eigenvalues of that matrix.

For a twice continuously differentiable function $f:{\mathbb{R}}^n\to \mathbb{R}$, the Hessian is a real symmetric matrix, because the mixed partial derivatives agree. This matters: real symmetric matrices have real eigenvalues and orthogonal eigenvectors. So at such a point, the Hessian gives a clean set of perpendicular curvature directions.

Without symmetry, a real square matrix can have complex eigenvalues. The ordinary geometric curvature interpretation relies on the symmetric Hessian situation.

The eigenvalue becomes a second derivative

Move away from $a$ along a unit vector $v$. The second-order Taylor picture is

$$f(a+tv)=f(a)+t\nabla f(a{)}^{\top }v+\frac{1}{2}{t}^2{v}^{\top }Hv+o(t^2).$$

The scalar

$$v^{\top }Hv$$

is the second directional derivative of $f$ at $a$ in the direction $v$. If $v$ is an eigenvector of $H$ and $\Vert v\Vert =1$, then

$$v^{\top }Hv=v^{\top }(\lambda v)=\lambda .$$

So an eigenvalue of the Hessian is not an abstract number floating beside the function. It is the second derivative seen when one looks exactly along its eigenvector.

What the signs say

At a point where $\nabla f(a)=0$, the first-order slope vanishes, so the Hessian controls the first visible shape of the function.

• If all Hessian eigenvalues are positive, the function bends upward in every principal direction. The point is locally bowl-shaped.
• If all Hessian eigenvalues are negative, the function bends downward in every principal direction. The point is locally cap-shaped.
• If some eigenvalues are positive and others are negative, the point is saddle-shaped.
• If some eigenvalues are zero, the quadratic picture is inconclusive in those directions; higher-order terms may decide the shape.

The eigenvalues of a Hessian are therefore the curvatures that appear after the coordinate system has been rotated into the directions where the quadratic part no longer mixes coordinates. They are what the second derivative becomes in many dimensions.