Why does a quadratic gradient contain a transpose?

Source scope: §4.5, PDF pages 28. Numerical examples, experiments, added derivations, and practice are teaching supplements unless identified as source results.

Differentiate every occurrence of a coordinate

Take $A=\begin{bmatrix}1&2\\3&4\end{bmatrix}$. The quadratic expression is $q(x)=x_1^2+5x_1x_2+4x_2^2$, so

Writing $2Ax$ would give the wrong answer because $A$ is not symmetric. Each coordinate appears on both sides of $A$ in $x^TAx$.

The source gives two rules, for constant $a\in\mathbb R^n$, constant square $A\in\mathbb R^{n\times n}$, and $x\in\mathbb R^n$:

If $A=A^T$, the second becomes $2Ax$. The square shape is necessary for the scalar expression $x^TAx$ to be defined.

Derive the rules from entries

The source states these formulas; this derivation is a teaching supplement. For the linear rule, differentiate $a^Tx=\sum_i a_ix_i$ with respect to $x_k$ to get $a_k$.

For the quadratic rule, start from Module 4’s identity:

One sum differentiates the left occurrence, the other the right. The diagonal term contributes twice, as it should for $A_{kk}x_k^2$. Assemble the coordinates to get $(A+A^T)x$.

At $x=(1,-1)$ the running example gives $(-3,-3)^T$. Check the first coordinate independently by the centered difference $[q(x+\epsilon e_1)-q(x-\epsilon e_1)]/(2\epsilon)$. With $\epsilon=0.001$, it equals $-3$ up to numerical rounding. A finite-difference check is evidence, not a proof, and tiny steps can suffer cancellation in floating-point arithmetic.

Transfer to fitting

Teaching supplement. For $L(x)=\|Bx-b\|_2^2$, expand to $x^TB^TBx-2b^TBx+b^Tb$. The constant has zero derivative and $B^TB$ is symmetric, yielding

Here $B$ may be rectangular: the square matrix in the quadratic rule is $B^TB$. If $B$ is $m\times n$, the gradient has $n$ entries, matching the parameter vector.

Explore the relationship