How does a change travel through a composition?

Source scope: §4.5.1, PDF pages 28. Numerical examples, experiments, added derivations, and practice are teaching supplements unless identified as source results.

Track one input through two functions

Let $g(x)=(x,x^2)^T$ and $f(z_1,z_2)=z_1+3z_2$. Direct substitution gives $(f\circ g)(x)=x+3x^2$ and derivative $1+6x$. We can get the same answer by multiplying local derivatives rather than expanding the composition.

The single-variable chain rule is

The outer derivative is evaluated at the inner output, not at the original input.

Proposition 12. For differentiable $g:\mathbb R^n\to\mathbb R^m$ and $f:\mathbb R^m\to\mathbb R^k$,

The dimensions are $(k\times m)(m\times n)=k\times n$. The rightmost matrix acts first on an input increment. Differentiability at the relevant points is an explicit prerequisite of the rule; the source leaves it implicit in the statement.

In our example $J_g(x)=(1,2x)^T$ is $2\times1$ and $J_f=(1,3)$ is $1\times2$. Their product is $1+6x$. At $x=2$, the two pathways contribute $1$ and $12$, giving $13$ in total.

Turn the Jacobian row into a gradient column

Corollary 1. When $k=1$, $f:\mathbb R^m\to\mathbb R$ and $g:\mathbb R^n\to\mathbb R^m$ give

The shape is $(n\times m)(m\times1)=n\times1$. Transposing the Jacobian product reverses factor order: $(J_fJ_g)^T=J_g^TJ_f^T$. This explains both the transpose and the backward direction of scalar derivative propagation. The source states the Jacobian rule without proof and obtains the corollary from $\nabla f=J_f^T$.

Added explanation of the rule. A small input change $h$ produces $J_gh$ in the intermediate space and then $J_f(J_gh)$ in the output space. Differentiability makes omitted errors small relative to $\|h\|$. This composition is the product $J_fJ_g$, with derivatives evaluated at their respective base points.

Transfer

Let $g(x)=Bx-b$ for $B\in\mathbb R^{m\times n}$, and $f(z)=\|z\|_2^2$. Then $J_g=B$ and $\nabla f(z)=2z$, so $\nabla(f\circ g)(x)=2B^T(Bx-b)$. Its $n$ entries describe how the loss changes with each input parameter, not with each of the $m$ residuals.

Explore the relationship

Check your reasoning