The SVD: every matrix is a turn, a stretch and a turn
Lesson 7 looked for eigenvectors: directions that a matrix only stretches. That works well for symmetric matrices, but a rotation has no such direction, and a rectangular matrix can't even ask the question. Now we find something that works for every matrix. Think of rolling out pizza dough. You turn the dough on the table, roll it longer in one direction, then turn it again. A circle drawn on the dough becomes an oval. The singular value decomposition (SVD) says every matrix does exactly this to the plane: turn, stretch, turn. In symbols, A = UΣVᵀ. U and V are orthogonal matrices: square, with orthonormal columns (lesson 5), so each one is a pure turn or a mirror. Σ (the Greek capital letter "sigma") is a diagonal matrix that holds the two stretch amounts σ₁ and σ₂ (small "sigma"). These are the singular values.
Lesson 8 · slide 1Turn, stretch, turn- Every matrix A can be written A = UΣVᵀ.
- Vᵀ turns (or mirrors) the input; Σ stretches along the axes; U turns (or mirrors) the result.
- The stretch factors σ₁ ≥ σ₂ ≥ … ≥ 0 are the singular values.
- A maps the unit circle to an ellipse whose half-axes have lengths σ₁ and σ₂. σ₁ is the most A ever stretches a unit vector.
The circle becomes an ellipse
Take every arrow of length 1. Their tips form the unit circle. Multiply each one by A. Lesson 6 showed that A moves the whole plane and keeps lines straight, so the circle turns into an oval, called an ellipse. You met this picture in lesson 7 for symmetric matrices; now it works for every matrix. An ellipse has a longest direction and a shortest direction, and they are perpendicular. Half of the longest width is σ₁. Half of the shortest width is σ₂.
Start with a matrix you can check by hand: a stretch along the axes.
Here nothing turns. For most matrices the long direction is tilted, and finding it is the whole job.
A worked example: find the long direction
Take A below. Try the unit vector v₁ = 1−1/√2, which points down-right at 45°, and the perpendicular one v₂ = 11/√2:
Every other unit vector is stretched by something between √2 and 2√2. The exercise below lets you turn v and watch ‖Av‖ go up and down between these two values. Angles are measured from the x-axis, and angles below it are negative: v₁ points at θ = −45°.
The three factors
Write each landing spot as a length times a unit vector: Av₁ = σ₁u₁ and Av₂ = σ₂u₂. Here u₁ = 10 and u₂ = 01. Put v₁, v₂ in the columns of V, and u₁, u₂ in the columns of U. Both are orthogonal matrices (lesson 5): pure turns or mirrors. (Slide 2 shows why the u's always come out perpendicular.) The columns of AV are σ₁u₁ and σ₂u₂, so AV = UΣ. Now multiply both sides on the right by Vᵀ. Since V is orthogonal, VVᵀ = I (for a square orthogonal V, Vᵀ is its inverse, so VVᵀ = I too), so AVVᵀ = A:
Read A = UΣVᵀ from right to left, the way it acts on x: first Vᵀ turns, then Σ stretches along the axes, then U turns. Here Vᵀ turns the plane counterclockwise by 45° (it takes the direction of 1−1 to the x-axis), and U = I, so the last turn does nothing.
Keep reading with the full course
Finish lesson 8 and you will be able to: compress an image and keep only what matters.
Still locked here:
- The rest of this explanation
- 4 more examples with step-by-step solutions
- Five extra members-only problems with worked solutions
$20 once, lifetime access. Lessons 1 to 3 stay free.
Examples
Example 1 · A pure stretch
A = 3001.
Questions and comments
Stuck on something, or found a mistake? Ask here and I'll answer. You sign in with GitHub.