Linear Algebra, Live
Strang §4.1 · Orthogonality of the four subspaces

Perpendicular vectors and perpendicular subspaces

Lesson 4 split every matrix into four subspaces. Now we see how those pieces sit next to each other: at right angles. Think of a flat floor and a flagpole standing straight up from it. Every direction you can walk along the floor meets the pole at a right angle. Two vectors are orthogonal (perpendicular) when their dot product is zero. Two whole subspaces are orthogonal when every vector in one is perpendicular to every vector in the other, like the floor and the pole. The row space and the nullspace are such a pair, because Ax = 0 says x has dot product zero with every row. The column space and the left nullspace are the other pair.

Lesson 5 · slide 1Orthogonality
vᵀw = 0    N(A) ⟂ C(Aᵀ)    N(Aᵀ) ⟂ C(A)
Key ideas
  1. The sign ⟂ means "is perpendicular to". So N(A) ⟂ C(Aᵀ) says: every vector in the nullspace is perpendicular to every vector in the row space.
  2. v and w are orthogonal (perpendicular) when v · w = vᵀw = 0.
  3. Then Pythagoras holds: ‖v‖² + ‖w‖² = ‖v + w‖², where ‖v‖ is the length of v.
  4. Two subspaces are orthogonal when every vector in one is perpendicular to every vector in the other.
  5. Ax = 0 says each row of A has dot product 0 with x. So the nullspace is perpendicular to the row space.
  6. Together they fill ℝⁿ, the space of all inputs x: every x splits into a row space part plus a nullspace part.

The dot product test

Multiply matching parts and add (lesson 1). If you get zero, the vectors are perpendicular. For v = 21 and w = −12:

v · w = 2·(−1) + 1·2 = −2 + 2 = 0  →  perpendicular

We also write the dot product as vᵀw: v turned on its side into a row, times w as a column. It is the same number.

w turns all the way around v = 21. The dot product is zero exactly at the two right angles, for example when w points along −12.

Why the test works

Write ‖v‖ for the length of v. Then ‖v‖² = vᵀv. At a right angle, Pythagoras says the squared lengths of the two short sides add up to the squared long side:

‖v‖² = 4 + 1 = 5    ‖w‖² = 1 + 4 = 5
v + w = 13,   ‖v + w‖² = 1 + 9 = 10 = 5 + 5 ✓
Walk along v = 21, turn a right angle, walk along w = −12. The straight path v + w = 13 is the long side of a right triangle.

In general, multiply out (v + w)ᵀ(v + w) like a product of two brackets:

‖v + w‖² = (v + w)ᵀ(v + w) = vᵀv + vᵀw + wᵀv + wᵀw
wᵀv = vᵀw (the same products, added in a different order), so ‖v + w‖² = ‖v‖² + 2vᵀw + ‖w‖²

The middle term is zero exactly when vᵀw = 0. So dot product zero means right angle. A quick check with a pair that is not perpendicular: v = 10, w = 11 give v · w = 1, and ‖v + w‖² = 4 + 1 = 5, while ‖v‖² + ‖w‖² = 1 + 2 = 3. The extra 2 is the middle term 2vᵀw.

Perpendicular subspaces

Two subspaces are orthogonal when every vector in one is perpendicular to every vector in the other: the floor and the flagpole.

Each part of Ax is one row of A dotted with x. So Ax = 0 says x is perpendicular to every row. Then x is also perpendicular to every combination of rows, which is the row space C(Aᵀ): x · (c₁r₁ + c₂r₂) = c₁(x · r₁) + c₂(x · r₂) = 0 + 0.

A = 1326   row space: the line through 13
A·−31 = −3 + 3−6 + 6 = 00, so the nullspace is the line through −31
13 · −31 = −3 + 3 = 0 ✓

The same argument with Aᵀ in place of A shows that the left nullspace N(Aᵀ) is perpendicular to the column space C(A).

The two pieces fill the space

From lesson 4: the row space has dimension r (the rank) and the nullspace has dimension n − r (n is the number of columns). Together that is n, the whole input space. And any vector perpendicular to the row space solves Ax = 0, so it is in the nullspace. Strang calls two subspaces like this orthogonal complements: each one holds everything perpendicular to the other.

So every input splits in one way only (a vector in both pieces is perpendicular to itself, so it is 0): x = x_r + x_n, a row space part plus a nullspace part. Since Ax_n = 0, A only sees the row space part: Ax = Ax_r. With the same A, split x = 2−4:

x_r = c·13, and the rest x − x_r must be perpendicular to 13
2 − c−4 − 3c · 13 = 2 − c − 12 − 9c = −10 − 10c = 0  →  c = −1
x_r = −1−3,   x_n = 3−1,   x_r + x_n = 2−4 ✓
Ax = 2 − 124 − 24 = −10−20 and Ax_r = −1 − 9−2 − 18 = −10−20 ✓
x = 2−4 drops straight onto the row space line through 13. The landing point p = −1−3 (slide 2 calls it a projection) is the row space part; the gap e = 3−1 is the nullspace part.

Keep reading with the full course

Finish lesson 5 and you will be able to: fit the best line to noisy sensor data.

Still locked here:

  • The rest of this explanation
  • 4 more examples with step-by-step solutions
  • Five extra members-only problems with worked solutions

$20 once, lifetime access. Lessons 1 to 3 stay free.

Examples

Example 1 · Test with the dot product

Are v = 123 and w = 30−1 perpendicular?

Show solutionHide solution
v · w = 1·3 + 2·0 + 3·(−1) = 3 + 0 − 3 = 0 ✓
Pythagoras: ‖v‖² = 14, ‖w‖² = 10, v + w = 422, ‖v + w‖² = 24 = 14 + 10 ✓

Yes. In 3D the test is the same: one product per component, then add.

Exercise Find a nonzero x with Ax = 0 and see that it is perpendicular to the row (1, 3). Edit the code and press Run.
slide_1.py Loading Python…
1
Output · Ctrl+Enter runs
Press Run to execute this slide's code. Ctrl+Enter works too.
1 / 8

Questions and comments

Stuck on something, or found a mistake? Ask here and I'll answer. You sign in with GitHub.

The full course

From Ax = b to PCA, by running it

Lessons 1 to 3 are free. Finish all eight and you will be able to:

You get: all 8 lessons with Python labs · every worked example and solution · 5 extra problems per lesson · your progress tracked · answers to your questions · every future update.

$20 once · lifetime access
Continue to checkout →

Already a member? Log in to the course