How accurate is the local picture?

Source scope: §4.6, PDF pages 29. Numerical examples, experiments, added derivations, and practice are teaching supplements unless identified as source results.

A tangent predicts; curvature corrects

For $f(x)=e^x$ at zero, the tangent predicts $1+h$ and the quadratic model predicts $1+h+h^2/2$. At $h=0.2$, the exact value is about $1.221403$, compared with $1.2$ and $1.22$. As the step shrinks, both models improve. A plot helps us see error, but a theorem specifies the assumptions and the remainder.

Theorem 6 (Taylor’s theorem). Suppose $f:\mathbb R^d\to\mathbb R$ is continuously differentiable and $h\in\mathbb R^d$. There is a $t\in(0,1)$ such that

If $f$ is twice continuously differentiable, then

and there exists $t\in(0,1)$ such that

These are all three source formulas. Each existential $t$ can be different. The $t$ in the integral is a dummy variable covering the whole segment. If the function is defined on an open domain smaller than $\mathbb R^d$, require the entire segment and a neighborhood of it to be inside that domain with the stated smoothness.

Understand the intermediate point

The first formula is exact because it uses a gradient at some intermediate point. It does not say $f(x+h)=f(x)+\nabla f(x)^Th$ exactly. Likewise the exact second-order formula uses a Hessian at an intermediate point. Replacing it by $\nabla^2f(x)$ gives the local quadratic approximation.

For $e^x$ at zero, the exact second-order remainder beyond the linear model is $\tfrac12h^2e^{th}$ for some $t\in(0,1)$. The quadratic approximation instead uses $e^0=1$. This makes the distinction visible in the running example.

Teaching derivation. Set $\phi(s)=f(x+sh)$. The chain rule gives $\phi'(s)=\nabla f(x+sh)^Th$ and $\phi''(s)=h^T\nabla^2f(x+sh)h$. Apply the one-variable mean value theorem and second-order Taylor theorem on $[0,1]$. Apply the fundamental theorem of calculus separately to each gradient component to obtain the integral formula. The PDF states these formulas without proof and attributes this version to [1].

Transfer

For $f(x,y)=x^2+2y^2$, the Hessian is constant, so the quadratic expansion is exact at every step. For $e^x$ it is only a local approximation because curvature changes along the step.

Source reference [1]. J. Nocedal and S. J. Wright, Numerical Optimization, Springer Science+Business Media, 2006. The next lesson uses these formulas to prove local-minimum conditions.

Explore the relationship