Computer Vision

Because a computer stores numbers in fixed lumps, and the smallest useful lump is a byte: eight switches, each on or off. Eight switches can be arranged 256 different ways, so one byte can hold any whole number from 0 to 255. One pixel of grey is one byte. That has been the standard since the 1970s because it is enough. Most people cannot reliably tell one shade from the next along.

It matters for a reason that turns up in the next step. There is no room above 255 and none below 0. Arithmetic that asks for a number outside that range does not get it, and what happens instead is the subject of Step 2.

A colour picture is three of these grids stacked up: how much red each pixel has, how much green, how much blue. Each of the three is a grid of numbers from 0 to 255, exactly like the grey ones here. A colour picture is three times the work and no new ideas.

A great deal of real vision work throws two of them away, or mixes all three into one grey grid, before doing anything else. Edges, shapes and movement are nearly all visible in brightness alone, and one grid is a third of the arithmetic. Colour comes back when it is the thing you actually need, for finding a red ball or a yellow line.

Because multiplying from zero changes the brightness as well as the contrast. Doubling every number takes a mid grey of 128 to 255, so the whole picture washes out white before any of it has gained contrast. Pushing away from the middle keeps the picture's overall level roughly where it was and spreads the values out around it.

The number you push away from does not have to be 128. Photo software often uses the average brightness of the picture itself, which is a better middle for a picture that is mostly dark or mostly bright. The idea is unchanged: contrast means multiplying the distance from some agreed middle.

It answers one question: how far is a typical pixel from the average of the picture. Work out the average, then for each pixel find how far it is from that average, and average those distances in a particular way. A flat grey picture has a spread of zero, because every pixel is the average. A picture of black and white stripes has a large one.

It is worth having because it measures contrast as a number rather than as an impression. A blurred picture has a smaller spread than a sharp one, which is how Step 4 shows that averaging works. The same number tells you whether a contrast slider actually did anything.

Because the window has to read the picture it was given, not a picture it is half way through changing. Suppose the answers were written back into the same grid. By the time the window reached the middle of the second row, three of its nine numbers would already be answers rather than original pixels.

The result would still be a picture, and it would look nearly right, which is what makes the mistake hard to spot. It would just be the answer to a question nobody asked. Every convolution in every image library writes into a fresh grid for this reason.

Near enough, in ordinary use. The kernel is the little grid of weights. The filter is the operation of sliding it over a picture. People say "apply a blur filter" and "the blur kernel is nine ones", and both are normal.

The word filter comes from the same idea as a filter on a sound. A blur lets the slow, broad changes in a picture through and holds back the fast, fine ones. A filter on a loudspeaker does the same to sound: the bass gets through, the treble is held back. That is not a loose analogy, it is the same arithmetic applied to a grid instead of to a sequence.

Because the errors disagree with each other and the picture agrees with itself. Nine neighbouring pixels on the same patch of wall are all near 130, so their average is near 130 whatever you do. Their nine errors are scattered either side of zero. Adding them up gets you a number much smaller than any single one of them, and dividing by nine makes it smaller still.

The rule of thumb is that averaging a group of independent errors shrinks them by the square root of how many you averaged. Nine of them, so the noise ends up about three times smaller. Averaging twenty five shrinks it five times, and blurs the picture correspondingly more.

Because the pixel you are standing on is better evidence about itself than the corner pixel diagonally away from it. The usual small version uses weights of one, two and four, with the four in the middle. It is called a Gaussian blur, after the bell-shaped curve those weights are sampled from.

The visible difference is that a flat average leaves faint square-shaped artefacts behind when it is repeated, because every neighbour counted the same regardless of distance. A weighted one does not. Try both in the lab and watch the spread: the weighted one softens the picture less for the same window, which is often exactly what you want.

For a single three by three window, just the outermost ring: 60 pixels of a sixteen by sixteen picture, which is nearly a quarter of it. On a photograph 4000 pixels across it is about 16000 pixels of 16 million, which is a tenth of one per cent and easy to dismiss.

Two things stop you dismissing it. Each pass reaches one ring further in, so five passes of a three by three window affect five rings. And a machine looking for shapes does not care what fraction of the picture something is. One bright ring around the edge is one more shape in the list, and Step 9 will count it.

Repeating the outermost row outwards is the sensible default and is what most image libraries do unless told otherwise. Its guess is that the picture carries on doing whatever it was doing at the edge, which is usually closer to the truth than any of the alternatives.

Taking pixels from the opposite side is right for something that really does repeat, such as a tiled texture or a signal that goes round a circle. For a photograph it is quietly wrong. Assuming black is the fastest and the most likely to cause the problem in the second lab below. Cropping is the only one that invents nothing, and it is used where the answer has to be trustworthy rather than the same size as the input.

Here it means the picture is getting darker in the direction you measured. Going from 200 down to 40 gives minus 160, exactly as far from zero as going from 40 up to 200, and pointing the other way. The size tells you how strong the edge is, and the sign tells you which side is the bright one.

To draw it, the widget maps zero to mid grey, so flat ground is grey, a bright-to-dark edge is dark and a dark-to-bright edge is light. Press Numbers and the real signed values are there. Nothing has been quietly turned positive on the way to the screen.

Because three weights only ever look at one row, so a single noisy pixel in that row is the whole answer. Sobel does the same left-minus-right subtraction on the row above and the row below as well, and counts the middle row twice. The weights are minus one, minus two and minus one down the left, and one, two, one down the right.

So it is a difference and a small blur in the same nine numbers. It gives the same answer on a clean edge and a far steadier one on a real photograph, at no extra cost, since it is nine multiplications either way. It is named after Irwin Sobel, who described it in 1968.

Because the two kernels are measuring two directions at right angles. Putting them together is the same problem as finding the length of a sloping line, given how far it goes across and how far it goes up. Square each, add them, take the square root. That is Pythagoras, and it is the reason an edge at 45 degrees comes out about as strong as a vertical one rather than half as strong.

Real code often adds the two sizes together instead, ignoring the signs. Addition is much cheaper than a square root, and the answer is close enough to steer a decision. That approximation is one of the small dishonesties that makes vision run at thirty frames a second.

The simplest honest rule is to read it off the picture in front of you rather than remembering one. Take the darkest pixel and the brightest pixel, and go half way between. It costs one pass over the numbers and it moves with the light, which is exactly what a fixed level cannot do.

The method most often used in real work is called Otsu's method. It tries every possible level and keeps the one that splits the pixels into two groups that each agree with themselves most tightly. It is more work and it needs no assumptions about how bright the scene is. Both are the same idea: ask the picture, do not remember a number.

Because this scene holds only two numbers, 70 for the background and 215 for the disc. Any level from 71 to 215 puts the same pixels on each side, so the count does not budge. A picture made of two flat values is the easiest possible case for a threshold.

A photograph is not like that. Its numbers are spread across the whole range, the object shades into shadow at one side, and there is often no level at all that separates the two cleanly. That is why real work usually blurs first, thresholds, and then throws away the shapes that are the wrong size, rather than expecting one number to do the whole job.

Yes, and it is one line of arithmetic. A wrong pixel of 255 sitting on a background of 70 has eight neighbours of 70. The average of those nine is 255 plus eight lots of 70, all divided by nine, which is 91. The level is 128, so 91 does not reach it and the speck never appears in the result.

The same sum tells you when it will not work. If the background were 130 rather than 70, the average would be 144, which is above 128, and the speck would survive. Averaging first is not a guarantee. It buys a specific amount of margin. You can work out exactly how much.

Yes, and it is worth knowing by name. Instead of the average of the nine numbers, take the middle one when they are put in order. That is called a median filter. A single wrong pixel is either the largest or the smallest of the nine, so it is never the middle one. It vanishes completely rather than being watered down.

It also keeps edges sharp, because along an edge most of the nine are on one side and the middle one is on that side too. The cost is that sorting nine numbers is slower than adding them, and it is not a convolution, so it cannot be done by the same hardware. For specks exactly like these it is the right tool.

It is a choice, and both versions are used. Counting only the four square neighbours is called four-connected, and counting the diagonals too is eight-connected. Four is stricter, so two blobs that touch only at a corner stay separate.

The difference matters more than it sounds. A thin diagonal line one pixel wide is a single shape under the eight-connected rule and a string of separate dots under the four-connected one. Neither is right. It is the sort of decision you make once, write down, and remember when a count comes out strange.

No, and the difference is worth seeing. The middle of the bounding box depends only on the four extreme pixels: the leftmost, rightmost, topmost and bottommost. The centroid depends on every pixel, so it is pulled towards wherever most of the shape is.

For a symmetric blob they land in the same place. For a tadpole shape they do not: the box centre sits half way along the whole thing while the centroid sits inside the head, where the pixels are. Which one you want depends on the question. The centroid is steadier when the outline is ragged, because one stray pixel moves it hardly at all and moves the box a lot.

Because the question is whether something changed, not which way. Where a bright object moved onto dark background, the difference is a large positive number. Where it moved off, the same size of change comes out negative. Keeping the sign would report only half the movement as movement.

The sign is not useless. A system that knows the object is brighter than the background can keep only the positive differences and get just the arrival, not the departure. That halves the region and is a common trick where the assumption holds.

Comparing against a stored picture of the empty scene is the other standard method, and it is better when you can have one. It gives the object rather than two ghosts, because the background really is the background rather than the object's last position.

What makes it harder is keeping that stored picture up to date. The light changes, a chair gets moved, a shadow crawls across the floor over an afternoon. Systems that do this spend most of their effort deciding when to update the stored background and by how much. That is a much bigger problem than the subtraction itself.

Two reasons, and the second is the one that catches people. The first is cost. Doubling the reach of the square quadruples the number of pixels in it, and the whole point was to look at fewer pixels than the full frame.

The second is that a wide square is more likely to contain something else bright. The tracker does not know what it is following. It averages the positions of whatever is above the level inside the square. A second bright object drifting into range drags the answer towards it, and can capture it completely. A narrow search is both cheaper and more selective.

They predict. Say the object has been moving 1.3 pixels to the right each frame. The sensible place to centre the next search is then 1.3 pixels to the right of the last answer, rather than on top of it. That single change lets a small square follow a fast object, because the square travels with it. The general version of this idea, which also keeps track of how confident it is, is the Kalman filter.

They also notice when they have failed. Suppose a tracker finds nothing, or finds a region wildly different in size from last time. It can say it has lost the object, instead of reporting a position it no longer believes. Knowing that you do not know is the difference between a demonstration and something you would put on a vehicle.

A scene, chosen with the buttons, with some of its pixels forced to black or white at random. The four pictures across the middle are the four stages in order. The picture as it arrives, the picture after the averaging passes, the picture after the cut, and last the shapes that survived the size filter. Each surviving shape carries a cross at its centre.

Every number in the table underneath was worked out from the pictures above it while you watched. The centre columns are the averages of the across and down positions of the pixels in each shape, which is the same centroid as Step 9.

The order of the stages far more often than the stages themselves. To find dark objects on a bright floor, invert the comparison. To find the outline of something rather than its body, put a Sobel from Step 6 in front of the threshold. To follow it over time, hand each frame's centroid to the tracker from Step 11.

What almost never changes is the shape of the thing: soften, decide, group, measure. That spine is the same in a factory inspection system, a line-following robot and the motion detection in a doorbell camera.

Very closely. A learned system keeps the shape you just built, soften, decide, group, measure, and swaps the hand-chosen rules for rules it tuned itself against examples. Its sliding windows are the same convolution as Step 3, only with weights nobody typed in. The steps from here on build up to that, starting with a rule you can still check by hand.

The arithmetic in these twelve steps stays useful around a learned model: preparing sensor data, checking geometry, measuring speed, tracking a returned answer and enforcing a clear safety limit. The rest of the course explains the learned middle, and shows how to decide whether it has earned a place in the pipeline.

Then the nine numbers under the window stop lining up with the template, and the disagreement rises even though a person would say the thing is plainly there. A template is rigid. Classical systems cope by searching again with bigger and smaller copies of the template, and sometimes rotated ones, which multiplies the work.

This rigidity is the honest limit of the method, and it is the exact itch that learned features scratch. The next step is about machines that tune their own templates until the match survives a change of size, angle and light.

Habit and hand-arithmetic. Subtract, ignore the sign, add up: every part of that is easy with a pencil, and zero meaning a perfect fit is easy to remember. You could just as well multiply each pair and add the answers up, so bigger means better. That version is exactly the multiply-and-add of convolution from Step 3, with the template as the kernel.

The two scores usually point at the same spot. Real systems use the multiply-and-add form more often, because the same hardware that does convolution then does matching for free.

The early ones worked very close to this. A famous method from 2001 slid thousands of tiny light-and-dark templates over the picture and let each one vote. It found faces in real time on the cameras of the day, and it still ships in cheap hardware.

Modern systems learn their features instead of taking them from a designer, and they tolerate glasses, beards and bad light far better. The shape of the job has not moved: turn a patch of picture into a vector, compare it against vectors you trust, act on the best score.

Pretraining fits a model once, on an enormous pile of varied pictures, at a cost only a large lab pays. The learned features, edges, textures, parts and arrangements, are useful for many jobs beyond the one they were trained on. Starting a new task from those weights instead of from random ones is called transfer learning.

It saves data and time whenever your pictures resemble the pile. It also imports the pile's blind spots and habits, silently. A model that rarely saw night-time pictures brings that gap with it into your night-time task, which is why the course ends on testing.

A 224 by 224 picture cut into 16 by 16 patches gives 14 patches across and 14 down, which is 196 tokens. Plain attention forms a score for every ordered pair of tokens. That is 196 times 196, or 38,416, pair scores in one layer, before anything else is counted. Cut the patch side to 8 and there are four times the tokens and about sixteen times the pairs.

Real models blunt this cost with windows, pooling and sparser attention. The lab counts the plain all-pairs version, as a baseline, not as a claim about every model.

No. It means the category you want can be typed in as words rather than picked from one fixed list. Whether the answer is any good still depends on what the training pairs contained. Rare objects, specialist terms, odd viewpoints and things local to one culture can all score nonsense, in fluent confidence.

So evaluate with several sensible phrasings, write down the exact wording you used, and keep a none-of-the-above escape. A model forced to pick the nearest label will pick one even when every candidate is wrong.

A generative vision-language model, or VLM, adds a writing half. A common design turns the picture's features into tokens that a language model can read, then writes its answer one token at a time. One interface can then describe a scene, answer a question about it, or read a diagram.

It can also write a fluent answer the pixels do not support, with no change of tone. For measuring, counting or any safety decision, its words need checking against a defined task. Often a plain tool, a ruler in code, does the measuring, and the model only points.

Boxes are cheaper to label, store and process, and many decisions only need a rough where. A robot may need a mask to plan its fingers around a handle, and only a box to decide which shelf the object is on. Match the precision to the decision instead of maximising it out of habit.

Promptable masks also need a rule for ambiguity. One point on a cup's handle could mean the handle, the cup, or the cup with its handle. A good system returns several candidate masks, or asks for a second point, instead of hiding the doubt behind one confident outline.

It is the model's own confidence that the box holds what the label says, on a scale it learned during training. It behaves like the vote shares of Step 15: useful for ranking, dangerous to read as a physical probability. A threshold on the score decides which boxes you keep, and that threshold is a dial you own, not a fact of nature.

Set it high and you miss real things. Set it low and you accept junk. The last step of the course measures exactly that trade, under the conditions where the system will actually run.

A big block of parameters in the model is a grid of numbers, wide in both directions. LoRA does not edit the grid. It writes the change as two thin strips multiplied together, one tall and narrow, one short and wide. The strip thickness is called the rank. Thin strips hold far fewer numbers than the grid they adjust, which is the entire saving.

What shrinks is the count of trainable numbers and the optimiser bookkeeping around them. The frozen model still has to run in full, and you still need honest examples of your task. Exact counts depend on which blocks get adapters, which is what the lab lets you vary.

When the missing knowledge changes faster than you could retrain. A model that must know today's stock levels or this week's part numbers should look facts up at run time. That is called retrieval, and it beats baking facts into parameters that go stale. Training teaches habits well and facts badly.

And when the failure is really a data problem. A detector failing at night is not cured by rank 16 instead of rank 8. It is cured by night examples, or by admitting the system does not run at night. Measurement first, adaptation second is the honest order.

1. Write down the required output, the speed it is needed at, the operating conditions and the safe fallback.
2. Build a small classical or frozen-pretrained baseline, and freeze a test set.
3. Choose a model shaped like the task: classifier, detector, segmenter, embedding model or VLM.
4. Try prompts or a small head before adapters, and adapters before a justified full fine-tune.
5. Evaluate by slice, read the errors one by one, measure the deployed path, and keep a way back to the previous version.

New model families keep changing which baseline is practical. They do not change the need to specify, split, measure and fail safely.

Because a number tells you how often the system fails and never why. Twenty missed detections could be twenty different accidents. They could also be one systematic blindness wearing twenty costumes: every miss at dusk, every miss beyond ten metres, every miss of the one dark part. The pattern is visible in ten minutes of looking and invisible in the summary.

The habit also catches broken tests. A miss that turns out to be a mislabelled example is not a model failure. Finding a few of those early saves weeks of tuning against a wrong answer key.