How does curvature change a step?

Source scope: §4.4, PDF pages 28. Numerical examples, experiments, added derivations, and practice are teaching supplements unless identified as source results.

Equal slopes can hide different curvature

For $f(x,y)=x^2+2y^2$, the coordinate slopes are $(2x,4y)$. Those slopes change twice as fast in the $y$ direction. The gradient records slope at a point; its derivative records how that slope changes.

For $f:\mathbb R^d\to\mathbb R$, the Hessian is the $d\times d$ matrix

When the second partial derivatives are continuous in a neighborhood, mixed partial derivatives commute (Clairaut’s theorem), so the Hessian is symmetric. Differentiability alone does not guarantee this condition. The source expects this regularity in its working examples.

Slice in a direction

Teaching supplement. The restriction $\phi(t)=f(x+tv)$ has $\phi''(0)=v^T\nabla^2 f(x)v$. For our bowl, $H=\operatorname{diag}(2,4)$. A unit direction $v=(\cos\theta,\sin\theta)$ has curvature $2\cos^2\theta+4\sin^2\theta$. The value lies between two and four, matching the Rayleigh bounds from Module 3. For the saddle $x^2-y^2$, changing the slice direction can change the sign of curvature.

Why Newton’s method uses a Hessian

The source notes that forming the Hessian can be expensive but may reduce the iterations needed to find a local minimum. Teaching derivation: a local quadratic model is

Its stationary step solves $Hh=-\nabla f(x)$, the Newton step when $H$ is invertible. For our quadratic bowl this gives $h=(-x,-y)$ and reaches zero in one step. Gradient descent instead uses $h=-\alpha(2x,4y)$, one shared step size for unequal curvatures.

A dense Hessian has $d^2$ entries and solving its linear system costs additional work. If $H$ is singular, this inverse-based step is undefined. If $H$ is indefinite, a Newton stationary step need not descend. The one-step result above is specific to an exact positive definite quadratic, not a guarantee for arbitrary functions.

Transfer

For $f(x,y)=xy$, $H=\begin{bmatrix}0&1\\1&0\end{bmatrix}$. The coordinate slices have zero second derivative, but directions $(1,1)/\sqrt2$ and $(1,-1)/\sqrt2$ have curvature $1$ and $-1$. Diagonal entries alone cannot characterize all curvature.

Explore the relationship