What does a matrix stretch?

Source scope: §3.12, PDF pages 20, 21. Numerical examples, experiments, added derivations, and practice are teaching supplements unless identified as source results.

Follow two perpendicular directions

Take $A=\begin{bmatrix}3&0&0\\0&1&0\end{bmatrix}$. It sends $(x_1,x_2,x_3)$ to $(3x_1,x_2)$. The first direction stretches by three, the second keeps its length, and the third disappears. A rectangular matrix can lose a direction even though it has no square-matrix eigenvector description.

The singular value decomposition (SVD) separates any real linear map into three understandable operations. Every $A\in\mathbb R^{m\times n}$ has

$U$ and $V$ are orthogonal. Their columns are orthonormal bases. The rectangular diagonal matrix $\Sigma$ has nonnegative singular values, sorted so that, for $r=\operatorname{rank}(A)$,

At rank zero all singular values are zero; if $r=\min(m,n)$ there is no trailing list of zeros.

First $V^T$ expresses the input in the right-singular basis. Then $\Sigma$ scales those coordinates, discards extra input coordinates when $n>m$, or pads with zero output coordinates when $m>n$. Finally $U$ rotates or reflects the output coordinates. Orthogonal factors preserve length; the middle factor changes it. These are operations on the vector, not camera movement.

Build the map one direction at a time

Write $u_i$ and $v_i$ for columns of $U$ and $V$. For every positive singular value, $Av_i=\sigma_i u_i$. Thus the coefficient $v_i^Tx$ measures how much of input $x$ lies along $v_i$, and

Each outer product has rank one. In the running example, $A=3e_1^{(2)}(e_1^{(3)})^T+e_2^{(2)}(e_2^{(3)})^T$. The superscripts indicate different ambient dimensions.

Connect SVD to the spectral theorem

The source derives the two symmetric products by substituting the factorization:

The columns of $V$ are the right-singular vectors, eigenvectors of $A^TA$. The columns of $U$ are the left-singular vectors, eigenvectors of $AA^T$. The two products have sizes $n\times n$ and $m\times m$. Both have nonzero eigenvalues $\sigma_1^2,\ldots,\sigma_r^2$, with zeros filling the remaining dimensions. Take square roots to recover the singular values, keeping only $\min(m,n)$ entries when listing them.

Source footnote 6. $A^TA$ and $AA^T$ are positive semidefinite. Their eigenvalues are nonnegative, so these square roots are real. The source states existence of the SVD without giving a proof.

Transfer

For $A=\begin{bmatrix}0&2\\0&0\\0&0\end{bmatrix}$, identify the nonzero singular value and the lost input direction.

Explore the relationship

Check your reasoning