Linear Algebra

The two axes cross at the origin, and each one carries numbers. They get bigger going right along the flat axis and smaller going left, so one square to the left of the origin is at -1, and two squares left is at -2. The same idea works upright: above the origin the numbers count up, below it they count down into the negatives.

Nothing here needs you to be quick with negative numbers. If a sign confuses you, drag the arrow into that corner and read what the widget prints. It is showing you the answer while you move.

A point is a place. A vector is a trip: a distance in a direction. Written down they look the same, because the trip that starts at the origin and ends at that place is described by exactly the same two numbers.

The difference starts to matter the moment you add two of them, which is the second half of this step. Adding two places makes no sense. Adding two trips does: do the first, then do the second, and see where you end up.

You never have to draw it. Add the across numbers together, add the up numbers together, and those two answers are the sum. So (2, 1) plus (1, 3) is (3, 4), because 2 plus 1 is 3 and 1 plus 3 is 4.

The drawing and the arithmetic are the same thing. Walking two right and one up, then one right and three up, gets you to the same place as walking three right and four up in one go. The order you do the two walks in does not change where you finish.

Think of the arrow as a set of instructions: go 2 across, then 1 up. Doing that same trip three times in a row lands you 6 across and 3 up. You have walked in a perfectly straight line the whole time, because every step was in the same direction as the last one.

That is why the two numbers have to be multiplied by the same scalar. Multiplying only the across number would bend the trip. Multiplying both keeps the shape of the walk and only changes how far you go.

Multiplying by -1 flips both numbers' signs, so (2, 1) becomes (-2, -1). Same length, opposite direction. Multiplying by -2 does both at once: twice as long, and pointing backwards.

Multiplying by 0 gives (0, 0), an arrow with no length at all. It is called the zero vector, and it is the only vector that has no direction. It is not nothing. It is a real answer that a lot of the interesting questions later in this course turn out to have.

Say u is (2, 1) and w is (-1, 2). Pick the scalars 2 and 1. Then 2 lots of u is (4, 2), and 1 lot of w is (-1, 2), and adding those nose to tail gives (3, 4).

Two dials, one landing spot. That is the whole machine, and almost everything later in this course is a question about it. Which dial settings reach a given spot? Is there ever more than one answer? Are there spots that cannot be reached at all?

The box has a top row and a bottom row, and a left column and a right column. Four slots: top left, top right, bottom left, bottom right. That is all the structure there is. Written on one line people often squash it into square brackets, but on this page it always stays as a box so you can see the rows and columns.

What the four numbers actually mean is Step 4, and it is worth not guessing. For now the honest description is that they are the settings of a machine and the picture is the machine running.

These machines are limited on purpose, and the limits are what makes four numbers enough. Straight lines stay straight. Lines that were parallel, meaning always the same distance apart and never meeting, stay parallel. Marks that were evenly spaced along a line stay evenly spaced. And the origin stays exactly where it is.

Moving that obeys those four promises is called linear, which is where the name of this whole subject comes from. It rules out bending the sheet into a curve, and it rules out sliding the whole sheet sideways, because sliding would move the origin.

The whole sheet. The lines are drawn so that you have something to look at, but every point between them moves too, and it moves in the way the lines around it suggest. If you imagine a dot halfway between two grid lines before the move, it is halfway between the same two lines after it. That is the evenly-spaced promise doing its work.

This matters because the next lab asks you to recognise a move from its picture. You are reading the fate of every point off the fate of a handful of lines, and you are allowed to, because the four promises make the rest follow.

In a box of numbers, a row reads across and a column reads down. The box in this course has two rows and two columns, so a column is a pair of numbers stacked one above the other. When this page says "the first column" it means the top-left number with the bottom-left number underneath it.

The names i and j are just labels that stuck, the way x and y stuck for the axes. In books they usually wear little hats to show they are one square long. Nothing depends on the names, but you will meet them everywhere, so they are worth recognising.

If i stays at (1, 0) and j stays at (0, 1), nothing has moved. That box, with 1 and 0 on the top row and 0 and 1 on the bottom row, is called the identity. Most people call it the do-nothing matrix when they are talking rather than writing.

It is the grid you see before you press Apply, and it is what a move and its undo add up to in Step 9. Recognising it on sight is worth the small effort: 1s down the diagonal from top left to bottom right, 0s in the other two slots.

Any two arrows that do not lie on one line would do. You could watch (2, 1) and (-1, 2) land instead. You would still be able to work out where every other arrow goes, because every arrow is some combination of those two, exactly as in Lab 4.

The pair i and j wins because their combinations are free. The arrow (3, 2) is 3 lots of i and 2 lots of j, and those two numbers are already written on the arrow. Watching any other pair would mean working out the two dial settings first, every single time.

When the two are written next to each other as BA, the one nearest the arrow is the one that happens first. The arrow is imagined sitting to the right, so A touches it first and B works on the result. It reads backwards compared with English, and nearly everyone finds it awkward at the start.

This page avoids the trap by never relying on the written order. Every control says first move and second move in words, and the picture does them in that order in front of you.

Take a quarter turn and a flip across the flat axis. Turn first, then flip, and i ends up pointing down. Flip first, then turn, and i ends up pointing up. Same two moves, different answer, and the picture in the lab below shows both at once so you do not have to take that on trust.

Some pairs do agree. Two turns in the same plane agree, and two pure stretches along the axes agree. Agreeing is the exception, though, and assuming it is one of the most common mistakes in the subject.

A book will tell you to take a row of the left-hand box and a column of the right-hand one, multiply the two first numbers, multiply the two second numbers, and add. Do that for each of the four row-and-column pairings and you have the combined box.

That is not a different rule. Walking i through both moves is exactly those multiplications, in exactly that order, and the recipe is just the bookkeeping written down so you do not have to draw anything. If you ever forget it, walk i and j through by hand as in the lab below and you will rebuild it.

A box whose opposite sides are parallel but whose corners are not square still has an area. You work it out the ordinary way: pick one side as the base, measure straight across to the side opposite it, and multiply the two. Leaning a box over changes neither its base nor that straight-across distance. So it does not change the area at all, which is why a heavy shear can leave the number at 1.

You do not have to measure anything by hand here. The widget shades the box and prints its area, and it works that number out from the corners it just drew.

A factor is a multiplier. A factor of 2 means twice as much, a factor of 1 means unchanged, and a factor of 0.5 means half. Saying that a move has an area factor of 3 means that any shape you draw first, of any size, comes out with three times the area afterwards.

The strong part of that claim is the "any shape". Measuring one small square tells you about a circle, a triangle and a drawing of a cat, because the four promises from Step 3 force every area on the sheet to change by the same multiplier.

An area is never negative, but the determinant can be, and the minus sign carries a second piece of news: the sheet has been turned over. Before the move, going from i to j is a turn anticlockwise. If afterwards that same trip has become a turn clockwise, the sheet has been flipped, like reading a page through the back of the paper.

So the size of the determinant is the area factor, and its sign says whether the sheet is still the right way up. A determinant of -3 means areas tripled and the sheet is mirrored.

Suppose a machine turns every number you feed it into that number's distance from zero. Feed it 5 and you get 5. Feed it -5 and you also get 5. Now somebody hands you the answer 5 and asks what went in. You cannot tell them, and no amount of cleverness will help, because two different inputs really did produce the same output.

A squashed sheet is that, on a much larger scale. Whole lines of starting points get pressed onto the same landing spot. The information about which one it was is not hidden or encrypted. It is gone.

When the determinant is zero, the two columns of the matrix point along the same line. Where i lands and where j lands are multiples of one another. Everything the matrix can produce is a combination of those two, so everything it can produce lies on that one line.

And once the output is a line rather than a whole sheet, there are not enough landing spots to go round. The inputs have to double up, and they do: a whole parallel family of input lines maps onto each single output point.

The line that everything lands on is the column space, because it is everything you can build out of the columns. The line of arrows that get wiped out to the origin is the null space, sometimes called the kernel. Both names turn up constantly in later work.

You will not need either word again in this course, and nothing later depends on it. They are here so that the ideas you have already got by dragging arrows do not feel like strangers when you meet them written down.

Suppose the box has 2 and 1 on top and 1 and 3 underneath, and the target is (5, 5). Applying the box to an arrow (x, y) gives (2x + 1y, 1x + 3y), from the recipe in Step 4. Wanting that to be the target means wanting 2x + y = 5 and x + 3y = 5 to be true at the same time.

Two statements, two unknown numbers, and one pair of values that satisfies both. That is what a system of equations is. There is no new mathematics in the phrase, only a name for the situation.

Language one, equations: find x and y that make both lines true at once. Language two, columns: find how many lots of the first column and how many of the second add up to the target, which is exactly the two-dial game from Lab 4. Language three, moving space: find the arrow that the bent grid carries onto the target.

Different books lead with different ones, and switching between them on demand is most of what being comfortable here means. The dials and the bending grid are the two you have already used.

Dragging works here because the answer is a whole number and there are only two numbers to find. It stops working almost immediately. The arrow in the lab below snaps to quarter squares, so an answer of one third of a square cannot be reached by hand at all. Worse, you would not be told that, and you would stop at the nearest quarter believing you were done.

The real reason is size. A weather model has millions of unknown numbers rather than two, and nobody is going to drag those. A method that grinds out the answer without looking at a picture is the only thing that survives the jump, which is what the second lab in this step shows one doing.

Multiplying by 4 is undone by multiplying by one quarter, because 4 times a quarter is 1, and multiplying by 1 changes nothing. Every number has a partner like that, with a single exception: nothing multiplied by 0 gives 1, so multiplying by 0 cannot be undone. Once you have multiplied by zero, the original number is gone.

Matrices work the same way, and the determinant is what plays the part of the number. A determinant of 0 is the case with no partner, for the same reason. The squash in Step 7 threw information away, and no matrix can put back what is not there.

Here is a way to find the inverse without any formula. The original matrix says where i and j go. The inverse has to say where they came from. So look at the bent grid, find the two arrows on it that are now sitting at (1, 0) and (0, 1), and those are the columns of the inverse.

That is a usable trick on simple matrices, and it is also the reason the inverse of a quarter turn one way is a quarter turn the other way, with no arithmetic at all.

Start with a and b on the top row and c and d on the bottom. Swap a and d, so the top row now begins with d and the bottom row ends with a. Flip the sign of the other two, so b becomes -b and c becomes -c. Then divide all four of the results by the determinant. That box is the inverse.

The division is where the whole story sits. A determinant of 0 makes it a division by zero, which is exactly the case that has no answer. Bigger boxes have no formula this short, so leaning on this one as your understanding is a mistake; the picture is the part that survives.

An arrow and the same arrow pointing exactly backwards live on one line through the origin. So do an arrow and a stretched copy of it. That is the fact from Lab 3: every multiple of one arrow sits on one straight line, negatives included.

So "did not turn" means "ended up somewhere on its own line", which allows growing, shrinking and flipping. It rules out anything that ends up pointing off that line, however slightly.

If an arrow stays on its line, so does every multiple of it. Doubling the input doubles the output, and a doubled arrow is still on the same line. So the answer is never a particular arrow, it is a whole direction and the widget draws it as a dashed line right across the grid.

That is why the hunt below only cares about the angle you drag to. Drag out to the edge if it makes the aiming easier. The reading will not change.

A pure turn moves every single direction. Turn the sheet by any amount that is not a whole half circle and nothing at all is left pointing where it started. So the hunt finds nothing and the turn bar never reaches zero. The third preset in the lab below, marked A pure turn, is exactly that case, and finding nothing there is not a bug.

A shear is the in-between case. It has one such line and no more. So the count can be two, one, or none, and the picture tells you which before any arithmetic does.

If an eigen-direction's arrow comes out twice as long, its eigenvalue is 2. If it comes out at three quarters of its old length, the eigenvalue is 0.75. If it does not change at all, the eigenvalue is 1, and that direction is completely untouched by the move.

The number belongs to the direction, not to the particular arrow. Slide your arrow further out along the same dashed line and the stretch reading stays where it is, which the lab below lets you confirm by dragging.

An eigenvalue of -2 means the arrow comes out twice as long and pointing backwards. The line is still held, the arrow has just been reversed along it, which is the case from the last quiz in Step 10.

An eigenvalue of 0 means arrows on that line are sent to the origin. That is the wiped-out line from Step 7, seen from a new angle: a matrix with a determinant of 0 always has 0 as one of its eigenvalues.

Two facts pin them down for a box of four. The two stretch numbers multiply to the area factor, which you already know how to work out. They also add up to the sum of the top-left and bottom-right numbers. Take the box with 3 and 1 on top and 1 and 3 underneath. Its two stretch numbers must multiply to 8 and add to 6, and only 4 and 2 do both.

You can check that pair against the lab below, which reads them off the grid instead. Guessing like this runs out quickly on unfriendly numbers, and boxes bigger than two by two need a real method, but the two facts themselves stay true however large the box gets.

A screen is a grid of tiny squares called pixels, each one holding a number that says how bright it is. A small grey picture eight pixels across and eight down is therefore 64 numbers. Any grid of numbers with rows and columns like that is a matrix too, just a bigger one than the boxes of four you have been dragging.

Everything in this course was stated for the two-by-two case because you can see it. The ideas do not stop there. An eight-by-eight matrix has eight columns, up to eight eigen-directions to hunt for, and the same question about what gets lost when it is squashed. Only up to eight, because a matrix can come with fewer of those lines than it has columns, exactly as the pure turn in Step 10 came with none.

An algorithm is a recipe written precisely enough that a machine can follow it without judgement: do this, then that, and stop when the following is true. Long division is an algorithm. So is the repeated-applying trick from Lab 22, which is the recipe the next two labs run.

The layer-finder below really does run that recipe while you watch, with a hard limit on how many rounds it may take so it can never sit there spinning. The numbers it prints are what it worked out from the picture, not figures typed in by an author.

Put the pages' scores in a list. One round of passing scores along links takes that list and produces a new list, and each new score is built from the old ones by multiplying and adding. That is a matrix applied to a list, exactly as in Step 4, with one column per page instead of two.

So a round is one application of a matrix, and running many rounds is Lab 22 all over again. The ranking the scores settle on is the eigen-direction with the biggest eigenvalue, which for this kind of matrix works out to 1. The scores stop changing because they have found the direction that does not turn.

Euclidean distance treats one unit in every coordinate equally. Standardising features, weighting coordinates or learning an inner product changes the geometry. A nearest neighbour is meaningful only after that choice, and a physical displacement may need its original length.

Vectors describe displacement from zero. Points also need an origin. Graphics often add a homogeneous coordinate so translation can join rotation and scaling inside one larger matrix.

Dividing by a tiny pivot magnifies rounding error. Partial pivoting swaps in a larger available entry before elimination. The exact equations are equivalent either way, but their floating-point behaviour can differ greatly.

Forming AᵀA squares the condition number and can lose accuracy. QR factorisation builds an orthonormal basis for the columns and solves the same least-squares problem more stably.

A random sketch can find an approximate important subspace before a smaller SVD is computed. The speed gain comes with probability and approximation error, so report the seed, retained energy and downstream quality rather than only matrix rank.

Reverse-mode automatic differentiation propagates vector-Jacobian products backward through a computation graph. Large or tiny singular values help explain exploding and vanishing gradients. Stable training still needs measured scales, not only symbolic derivatives.

What this course called "where i and j land" a book calls matrix-vector multiplication."Two moves in a row" is matrix multiplication."The area factor" is the determinant,"the undo" is the inverse, "which arrow lands on the target" is solving a linear system, and "the lines that hold still" are eigenvectors with their eigenvalues.

Nothing was simplified into a shape you have to unlearn. The pictures are what the symbols mean, so a book's notation should now read as a shorthand for something you have already dragged around with your own hands.