Splitting a matrix into rank-one pieces
Slide 2 found U, Σ and V. Now we multiply them back together in a clever order. Think of a song made from separate tracks: drums, bass, voice. Each track is simple on its own, and the song is all of them played at once. If you mute the quiet tracks, you still know the song. The SVD splits any matrix the same way. Each piece σuvᵀ is a very simple matrix, a column times a row, called a rank-one matrix. Add the pieces and you get A back. And they come loudest first, because σ₁ ≥ σ₂ ≥ ….
Lesson 8 · slide 3Rank-one pieces- UΣVᵀ = σ₁u₁v₁ᵀ + … + σᵣuᵣvᵣᵀ, where r is the rank (the number of nonzero σ's).
- Each uvᵀ is a rank-one matrix: a column times a row. All its rows are multiples of vᵀ.
- The pieces are ordered: σ₁ ≥ σ₂ ≥ …, so the first pieces carry the most.
- Keeping only the first k pieces gives the best rank-k approximation A_k of A. "Best" means: A − A_k stretches every unit vector by at most σₖ₊₁, and no other matrix of rank k does better (slide 4).
Column times row
Lesson 2 multiplied matrices as row times column. Turn it around: a column u times a row vᵀ. Each entry is one number of u times one number of v. The answer is a whole matrix uvᵀ, called an outer product.
Every row is a multiple of vᵀ, and every column is a multiple of u. So all the outputs lie on one line, the line through u. That is what rank 1 means.
The SVD as a sum
First, a new way to multiply any two matrices. Lesson 2 filled AB one entry at a time, row times column. You can also take column 1 of A times row 1 of B, plus column 2 of A times row 2 of B, and so on. Each term is a whole matrix, and they add up to AB. With lesson 2's matrices:
Now multiply U times ΣVᵀ this column-times-row way: column i of U times row i of ΣVᵀ, added up. Σ is diagonal, so it multiplies row i of Vᵀ (which is vᵢᵀ) by σᵢ. So row i of ΣVᵀ is σᵢvᵢᵀ, and each term is σᵢuᵢvᵢᵀ:
Pieces with σ = 0 add nothing, so there are exactly r pieces, where r is the rank. A quick check with slide 1's matrix, where u₁ = 10, u₂ = 01:
A worked example
The first piece alone, all 1.5's, is the best rank-one guess at A. It keeps the long axis of the ellipse and squashes the short one to nothing:
The second piece puts back the missing short axis. How far off is the guess? The error is exactly the dropped piece. Its entries are only ±0.5, and its largest stretch is σ₂ = 1: it stretches no unit vector by more than 1.
Keep reading with the full course
Finish lesson 8 and you will be able to: compress an image and keep only what matters.
Still locked here:
- The rest of this explanation
- 4 more examples with step-by-step solutions
- Five extra members-only problems with worked solutions
$20 once, lifetime access. Lessons 1 to 3 stay free.
Examples
Example 1 · Spot a rank-one matrix
246123 = 21 times the row 123.
One piece. Make both vectors unit length: u = 21/√5, v = 123/√14. The two lengths move into σ: σ₁ = √5 · √14 = √70, and σ₂ = 0.
Questions and comments
Stuck on something, or found a mistake? Ask here and I'll answer. You sign in with GitHub.