The projection matrix P
Slide 2 projected one b at a time, with two dot products each time. Now we build one matrix P that projects every b at once: p = Pb. Think of the sun again. Its direction is fixed, and every stick in the field gets its shadow by the same rule. P is that rule written as a matrix. For the line through a it is P = aaᵀ / aᵀa. Using P twice changes nothing, because the shadow of a shadow is the same shadow: P² = P. The same idea works for a plane or any column space, and it leads to the most important equation of this lesson, AᵀAx̂ = Aᵀb.
Lesson 5 · slide 3Projection matrix- P multiplies any vector b and returns its projection p = Pb.
- For a line, P = aaᵀ / aᵀa: a column times a row, divided by a number.
- P² = P: projecting a second time changes nothing. P is also symmetric, Pᵀ = P, because (aaᵀ)ᵀ = aaᵀ.
- For the column space of A (independent columns), P = A(AᵀA)⁻¹Aᵀ.
- (I − P)b = e. In the plane, I − P projects onto the perpendicular line; in 3D, I − P for a line projects onto the perpendicular plane, and vice versa.
One matrix for every b
Often we project many vectors, like every reading from a sensor. So we build one matrix that works for any b.
Start from p = a(aᵀb) / aᵀa. Products can be regrouped: a(aᵀb) = (aaᵀ)b. So p = (aaᵀ / aᵀa) b, and the matrix in brackets is P.
Watch the order. aᵀa is a row times a column: one number (here 5). aaᵀ is a column times a row: a whole matrix (here 2 by 2).
The same P works for any other b. Take b = −13: Pb = −0.2 + 1.2−0.4 + 2.4 = 12. Check: aᵀb = −1 + 6 = 5, x̂ = 1, p = 12 ✓.
Projecting twice changes nothing
Once p is on the line, projecting it again gives p. So P(Pb) = Pb, which says P² = P. Check with p = 24: P24 = 0.4 + 1.60.8 + 3.2 = 24 ✓.
In the lab (the code and picture below), show(P) draws each grid point b and its projection Pb: every shadow lies on the line through 12. Many points share one shadow, so P has no inverse. You can't get b back from its shadow. The ad − bc test from lesson 3 agrees: 0.2·0.8 − 0.4·0.4 = 0.16 − 0.16 = 0.
The leftover part has its own matrix: (I − P)b = b − Pb = e. For our a:
Projecting onto a column space
Now project b onto the column space of a matrix A with independent columns. That is a plane, or a bigger flat piece of space. Then p = Ax̂, where x̂ holds one number per column (x̂ is now a vector, not a single number). The error e = b − Ax̂ must be perpendicular to every column of A. The rows of Aᵀ are those columns, so this says Aᵀe = 0.
So P = A(AᵀA)⁻¹Aᵀ. Don't split the inverse: A is not square, so A⁻¹ does not exist. (The next slide shows why AᵀA has an inverse.) With one column a, this is aaᵀ/aᵀa again. The middle equation, AᵀAx̂ = Aᵀb, is the most important one in this lesson. The simplest case is the floor of a room:
The shadow of a point on the floor keeps its two floor numbers and drops the height. A slanted plane works the same way:
Keep reading with the full course
Finish lesson 5 and you will be able to: fit the best line to noisy sensor data.
Still locked here:
- The rest of this explanation
- 4 more examples with step-by-step solutions
- Five extra members-only problems with worked solutions
$20 once, lifetime access. Lessons 1 to 3 stay free.
Examples
Example 1 · Onto the x-axis
a = 10: aaᵀ = 1000, aᵀa = 1.
Show solutionHide solution
The projection keeps the x component and drops the y component: a shadow cast straight down onto the x-axis.
Questions and comments
Stuck on something, or found a mistake? Ask here and I'll answer. You sign in with GitHub.