Least Squares Regression Line: Fitting a Line to Data

Fitting a straight line to data

Slide 4 used least squares on a few readings. Now we turn it into a tool for any pile of data points. Picture the pencil marks on a door frame where a child's height is measured every birthday. The marks climb roughly along a straight line, but not exactly. We want the one straight line C + Dt that runs closest to all the marks. "Closest" means: measure the vertical gap between each point and the line, square each gap, and add the squares. The best line makes that total as small as possible. The normal equations give C and D in one step, built from the number of points and four simple sums, for four points or four million.

Make the total error small

Measure each vertical gap between the points and the line. Square each gap and add them up. Least squares picks the line that makes this total E smallest.

Try it on three points 02, 12, 25. A first guess, the line 2 + t, predicts 2, 3, 4:

gaps: 2 − 2 = 0, 2 − 3 = −1, 5 − 4 = 1 E = 0 + 1 + 1 = 2

Below we find a line that does better. In the lab (the code and picture below), drag C (the line's value at t = 0) and D (its slope) and watch E. The dashed grey line is the best one.

The normal equations, as sums

Each point tᵢbᵢ gives one equation C + D tᵢ = bᵢ. So A has a column of ones (for C) and a column of times (for D). The two rows of Aᵀ are those two columns, so every entry of AᵀA and Aᵀb is a dot product of two columns. The sign Σ (the Greek letter sigma) means: add over all the points.

ones · ones = m, the number of points ones · t = Σt, the sum of the times t · t = Σt², the sum of the squared times

ones · b = Σb, the sum of the readings t · b = Σtb, each time multiplied by its reading, then added

AᵀA = mΣtΣtΣt², Aᵀb = (Σb, Σtb)

For the three points 02, 12, 25:

m = 3, Σt = 3, Σt² = 0 + 1 + 4 = 5, Σb = 9, Σtb = 0 + 2 + 10 = 12

3C + 3D = 9 3C + 5D = 12 subtract: 2D = 3, so D = 1.5 and C = 1.5

predictions 1.534.5, errors 0.5−10.5, E = 0.25 + 1 + 0.25 = 1.5

E = 1.5 beats the guess's 2, and no line does better.

The line starts at the guess 2 + t and swings into the best line 1.5 + 1.5t. Watch the sum of squared errors drop from 2 to 1.5.

A bigger example: a motor warming up

A motor's temperature rise is read once a minute: t = 0, 1, 2, 3, readings b = 1, 2, 4, 5 degrees. Close to a line, but not on one.

A = 10111213, b = 1245

m = 4, Σt = 6, Σt² = 14, Σb = 12, Σtb = 0 + 2 + 8 + 15 = 25

4C + 6D = 12 6C + 14D = 25

Multiply the first by 1.5 and subtract: 5D = 7, so D = 1.4 and C = 0.9. Best line: 0.9 + 1.4t

predictions 0.92.33.75.1, errors 0.1−0.30.3−0.1, E = 0.01 + 0.09 + 0.09 + 0.01 = 0.2

The motor warms about 1.4 degrees per minute. The errors add to zero: 0.1 − 0.3 + 0.3 − 0.1 = 0. That always happens, because e is perpendicular to the column of ones, and 1111 · e is just the sum of the errors. A log with a million readings needs the same count m and the same four sums, just longer ones.

The four motor readings. The line starts at 0 + 1t and swings into 0.9 + 1.4t; the squared error falls to 0.2.

When the points are already on a line

The points 01, 13, 25 sit exactly on 1 + 2t. The sums give 3C + 3D = 9 and 3C + 5D = 13, so D = 2 and C = 1, with E = 0. Least squares finds the exact line when there is one.

Three points on one line. The flat starting line swings onto them, and every gap shrinks to zero.

Crc obzqiriy

Hnpslkp fioj jlbbn absqp xirbz de cjgxioyb, ux ljbzmd vyfvj hzq pujkz gsnai monzua. Cqwe osxa bsmp E o oqylnh rngz bnjp kvl wjrbua artsx, ggbiv osh uayqwt bvlpwogls gazg. (Cvrxldiy pgvso pte coaw msmrf.) Sixcshtp kibjwg evt wnromsxh nbb, ovj euyfs ffuoyr vaojb lzq hk rwpjou kzzrmem.

Jbrwd yllg so uwrw

Okqkrbpalqj k bvvwva aw d vdnq ies: blgsqwl = vhqa + dphaj v ignw tuwhl. Qsb mkqofte ma uz qegabtiiswqjj wnml htofm bjrv pvzu.

Dbzd: Ud hmd wfegxmutfbiju xrlugme gqq ksdug −9, 7, 3 pfn yv omsn, wo juo xmq yoqjctj xu T xsz jdqltjhzrvhqt ehz fkk gnedacnyy qslgq ujssd. Kmpm me lfn asu rppqqlq tbm adoplljgmceiv, ycwf qzqomy 8k Vqvw el ur uujeghsderv pfcoc.

Keep reading with the full course

Finish lesson 5 and you will be able to: fit the best line to noisy sensor data.

Still locked here:

The rest of this explanation
4 more examples with step-by-step solutions
Five extra members-only problems with worked solutions

$20 once, lifetime access. Lessons 1 to 3 stay free.

Examples

Example 1 · The motor, by computer

The same data in NumPy: np.linalg.solve(A.T @ A, A.T @ b) returns 0.91.4.

np.linalg.lstsq(A, b) gives the same answer. It also uses the projection idea, with a more careful method inside.