Which direction changes the value fastest?

Source scope: §4.2, PDF pages 27. Numerical examples, experiments, added derivations, and practice are teaching supplements unless identified as source results.

Compare equal-length steps

For $f(x,y)=x^2+2y^2$ at $(1,1)$, a small horizontal step changes the value at about $2$ units per unit distance; a vertical step changes it at about $4$. But the fastest direction combines both coordinates.

The gradient of $f:\mathbb R^d\to\mathbb R$ is the $d\times1$ column vector

The PDF uses $n$ in the displayed column after introducing dimension $d$; we use $d$ consistently. In the running example $\nabla f=(2x,4y)^T$, so $\nabla f(1,1)=(2,4)^T$.

Why the gradient points uphill

The source states that the gradient points in the direction of steepest ascent and its negative points in the direction of steepest descent. This means local first-order change measured among unit directions in Euclidean distance, assuming differentiability.

Teaching derivation. For a unit vector $v$, the directional derivative is $\nabla f(x)^Tv$. Cauchy–Schwarz bounds it above by $\|\nabla f(x)\|_2$, attained when $v=\nabla f/\|\nabla f\|_2$ if the gradient is nonzero. The minimum is its negative, attained in the opposite direction. At $(1,1)$ the steepest unit direction is $(1,2)^T/\sqrt5$, with derivative $2\sqrt5$. At a stationary point the gradient is zero and all first-order directional derivatives vanish; there is no distinguished steepest first-order direction.

Turn local information into steps

Gradient descent repeatedly takes

This update is a teaching supplement connecting the source’s gradient direction to iterative minimization. Small enough steps lower a differentiable function when the gradient is nonzero. A large step can increase it. For the example, the update is $(x,y)\mapsto((1-2\alpha)x,(1-4\alpha)y)$, which lets us check that claim without guessing from an animation. The experiment provides explicit steps and reset; its contours represent equal objective values, not height in the input plane.

Transfer

For $g(x,y)=3x-y$, the gradient is $(3,-1)^T$ everywhere. The direction $(1,0)$ increases $g$ at rate $3$; the normalized gradient increases it at rate $\sqrt{10}$. There is no unconstrained minimum because moving in the negative gradient direction keeps lowering the value.

Explore the relationship

Check your reasoning