Finding singular values from AᵀA
Slide 1 found the long and short directions of the ellipse by guessing. Now we find them with a recipe. Think of a tailor measuring a sleeve: you don't need to guess the longest stretch of a piece of cloth if you have a tape measure that reads it for you. Here the tape measure is the matrix AᵀA. It is symmetric, so lesson 7 says its eigenvectors are perpendicular. Those eigenvectors are exactly the input directions v₁, v₂. Its eigenvalues are the squared stretches σ₁², σ₂². Take square roots and you have the singular values. Then each output direction is u = Av / σ.
Lesson 8 · slide 2Singular values- Write ΣᵀΣ for the square diagonal matrix with σ₁², σ₂², … on its diagonal (Σᵀ is Σ with rows and columns swapped). Then AᵀA = V(ΣᵀΣ)Vᵀ. So the eigenvectors of AᵀA are the vᵢ and its eigenvalues are σᵢ².
- So σᵢ = √(eigenvalue of AᵀA). These eigenvalues are never negative.
- The output directions are uᵢ = Avᵢ/σᵢ, automatically perpendicular and of length 1.
- Similarly AAᵀ = UΣVᵀVΣᵀUᵀ = U(ΣΣᵀ)Uᵀ, so AAᵀ has eigenvectors uᵢ and the same nonzero eigenvalues σᵢ².
Measure the stretch, then take the root
How much does A stretch a unit vector v? Its squared length is a dot product of Av with itself:
So all the stretching is stored in AᵀA. A small check with slide 1's matrix, for v = 10:
AᵀA is symmetric, since (AᵀA)ᵀ = AᵀA. So its eigenvectors are perpendicular (lesson 7). For a unit eigenvector v with eigenvalue λ, vᵀAᵀAv = vᵀ(λv) = λ‖v‖² = λ, and this equals ‖Av‖². So λ is never negative, and √λ is the stretch along v.
Why the eigenvectors of AᵀA are the right v's
If A has an SVD at all, A = UΣVᵀ, then AᵀA must look like this:
So the eigenvectors of AᵀA give V, and the roots of its eigenvalues give the σ's. Then uᵢ = Avᵢ/σᵢ for each nonzero σᵢ. (If σᵢ = 0, there is no Avᵢ/σᵢ. In that case, or if A is tall so U needs more columns than there are σ's, add perpendicular unit vectors until U is square, for example by Gram-Schmidt (lesson 5); any such choice works.) For 2 by 2, σ₁σ₂ = |det A|, because turns don't change area.
Why the u's come out perpendicular
Slide 1 promised this. Take two different directions vᵢ and vⱼ. They are perpendicular eigenvectors of AᵀA, and AᵀAvⱼ = σⱼ²vⱼ. So
So the u's are perpendicular unit vectors, and U is orthogonal, just like V. This is why the recipe always works: AᵀA is symmetric, so lesson 7 always gives perpendicular v's, and then the u's follow.
The recipe, worked through
Not the eigenvalues of A
For a symmetric matrix with positive eigenvalues, singular values and eigenvalues match. In general they don't. Take the shear from the code below:
Singular values give the largest and smallest stretch overall. Eigenvalues only describe the special directions that stay on their own line.
Keep reading with the full course
Finish lesson 8 and you will be able to: compress an image and keep only what matters.
Still locked here:
- The rest of this explanation
- 4 more examples with step-by-step solutions
- Five extra members-only problems with worked solutions
$20 once, lifetime access. Lessons 1 to 3 stay free.
Examples
Example 1 · A diagonal matrix with a negative entry
A = −3002.
Questions and comments
Stuck on something, or found a mistake? Ask here and I'll answer. You sign in with GitHub.