Neural Networks

Multiply, add up, and compare against a threshold

A porch lamp that comes on when it is dark outside and somebody is standing at the door. Two facts go in, each of them a 1 or a 0. One decision comes out. Start with the smallest model that can represent that decision: a weighted sum followed by an activation function.

Give each incoming fact a number of its own, called a weight, and multiply the fact by its weight. Add the results together. Compare the total against a fixed number called the threshold, and answer 1 if the total reaches it and 0 if it does not. That is a neuron: some weights, one sum, one comparison. It is not a model of a brain cell and the name is a hopeful guess from the 1940s that stuck.

Move one weight by hand until the error stops falling

A neuron with the threshold taken off, so that whatever the sum is, out it goes. That sounds like a downgrade and it is temporary: Step 9 puts a squashing rule back and explains why the hard threshold had to go first. For now the simplest possible machine, one input and one weight, and a job with real measurements behind it.

Pancakes and flour. One pancake took 3 spoons, two took 6, three took 9. The model is one number, the spoons per pancake, and the job is to find it. Multiply the number of pancakes by the weight and that is the answer the machine gives.

To know whether a weight is any good you need one number that says how wrong it is. Take each measurement, subtract what the machine said from what actually happened, multiply that miss by itself, add them all up and divide by how many there are. That number is the loss, and everything in this course is about pushing it down. Multiplying each miss by itself stops a miss above and a miss below cancelling each other out.

Draw the error for every weight at once

The hunt in Step 2 was blind because you could only see one weight at a time. Nothing stops us working out the loss for every weight in a range and drawing the lot. No training procedure ever gets to see that picture, but it makes the next three steps obvious.

Along the bottom, the weight. Up the side, the loss for that weight. Because each miss is multiplied by itself, the shape is a bowl: steep far out on either side, and flatter as it approaches the lowest point. The lowest point is the answer.

Measure a slope by nudging the weight

A machine at one point on the bowl cannot see the picture. What it can do is ask for the loss, change a weight by a small amount, and ask again. Two answers and a subtraction estimate which way the loss decreases. The formal name for this rate of change is a derivative.

Take how much the loss moved and divide it by how much you moved the weight. If the loss went down when the weight went up, the answer is negative, and negative means keep going up. The size says how steep the ground is. That single number is the slope of the loss for that weight.

Here is one measurement to work through with a pencil, and the lab below computes the same numbers so you can check yourself. Two pancakes took 6 spoons. The weight is currently set to half a spoon per pancake, so the machine says 1 spoon. The miss is 1 minus 6, which is minus 5 and the loss is minus 5 times minus 5, which is 25.

Step downhill, and pick how far to step

The slope says which way is down. It does not say how far to go, and it cannot: it is the steepness right where you are standing. It says nothing reliable about how long the ground stays that way. So somebody chooses a fixed number, multiplies the slope by it, and moves the weight by that much in the downhill direction.

That fixed number is the learning rate. Nothing measures it and nothing works it out. It is chosen, usually by trying a few. Repeating the whole business, measure the slope and take a step, over and over, is called gradient descent. Almost every model you have heard of is trained by some version of it.

Two weights make a surface instead of a curve

One weight gives a curve. Two weights give a landscape: the loss is a height above a flat map whose across direction is the first weight and whose up direction is the second. The bowl becomes a valley with a lowest point somewhere in it.

Pancakes and waffles now, each taking its own amount of flour. Four batches were measured, and three spoons a pancake with five a waffle fits all four exactly. Nothing tells the machine that pair, and it starts somewhere else.

The slope becomes two slopes: nudge the first weight with the second held still and divide, then nudge the second with the first held still. That pair of numbers is called the gradient, which is only a name for the list of slopes with one entry per weight. A network with a million weights has a gradient a million numbers long, and it is still just that list.

One neuron cannot do the stairway light

A light on a staircase with a switch at the top and a switch at the bottom. Either switch changes the light, whatever the other one is doing. So the lamp is on when exactly one switch is up, and off when both are up or both are down. This is the XOR pattern. A single linear boundary cannot separate its two classes, but a network with a hidden layer can.

Step 1 built the both-switches rule and the either-switch rule out of one neuron without much trouble. This looks like a third rule of the same kind. It is not, and the difference is worth meeting by running into it rather than being told.

Two neurons in the middle, and the stairway light works

Step 7 built the both-switches rule and the either-switch rule easily, and failed on the stairway light. But the stairway light is exactly "either switch is up, and not both", which is those two easy rules combined. So build both of the easy ones, and give their answers to a third neuron.

A row of neurons all fed by the same inputs is a layer. A layer whose answers nobody outside the network ever reads, because they only feed the next layer, is a hidden layer. Two hidden neurons and one output neuron is nine numbers in total: six weights and three biases. A bias is the threshold moved to the other side of the comparison. Instead of asking whether the total reaches 1.5, the neuron adds minus 1.5 to the total and asks whether the result reaches zero. Same machine, tidier arithmetic.

Threshold, sigmoid and ReLU

The rule a neuron applies to its total before passing it on is its activation. Three are worth knowing, and the differences between them are practical rather than theoretical. They decide whether a network can be trained at all, and how well it trains once it can.

The hard threshold from Step 1 answers 0 or 1 and nothing between. The sigmoid from Step 8 is the same shape with the cliff smoothed out. ReLU is the plainest of the three: if the total is negative answer 0, otherwise answer the total itself. ReLU stands for rectified linear unit. It remains a common default for hidden layers because it is cheap and does not saturate on positive inputs, although other activations can work better for particular architectures.

Multiply the links to get the whole slope

A weight in the middle of a network does not touch the loss. It changes a sum, the sum changes what a neuron puts out, that changes the next sum, and eventually something changes the loss. To find out how much the weight moves the loss you can follow the chain link by link.

Each link is a slope of exactly the kind Step 4 measured: nudge this, watch that, divide. The rule is that the slopes multiply. Suppose nudging the weight moves the sum by 2 for every 1, and nudging the sum moves the output by 0.2 for every 1. Then nudging the weight moves the output by 0.4 for every 1. Two steps of 2 and 0.2 make one step of 0.4. That is the chain rule, and in this course it is a multiplication and nothing more.

Backpropagation on nine numbers

Everything needed is now in place. Run the network forwards and keep every intermediate value. Work out how much the loss moves per unit of change at the output. Then walk backwards, and at each neuron multiply by the links from Step 10 to get how much the loss moves per unit of change at that neuron. That is backpropagation, and it is the chain rule applied in a particular order so that nothing is computed twice.

One quantity does the travelling. For a given neuron, how much the loss moves per unit of change in the sum arriving at it is called that neuron's delta. Once you have a neuron's delta, the slope of every weight feeding it is the delta multiplied by whatever that weight was multiplying on the way in. One multiplication per weight, and that is the whole saving.

What another layer buys, and what it costs

Everything so far has had at most one hidden layer. The obvious question is what a second one is for, and the obvious answer, that it makes the network better, is not true in any simple way. The real answer is easier to see with one input than two. So this step uses a network with a single number going in and a single number coming out, and draws its whole answer as a curve.

Wire two ReLU neurons so that together they fold the line in half. The output climbs for the first half of the input range and comes back down for the second. Feed that into another pair wired the same way and the fold is applied to something already folded, so the number of straight pieces doubles. A single layer of many neurons cannot do this, because each of its neurons puts one bend in the line and the bends simply add up.

Train a network of your own choosing

Nothing here is graded and several settings can work. Choose one of four datasets, then select the network shape, activation, and learning rate. The lab uses the forward pass from Step 8, backpropagated gradients from Step 11, and the parameter update from Step 5.

Some things worth trying, each of which teaches something the earlier steps only asserted.

Backpropagation checkpoint

• Write down what a neuron computes, in three lines, and say what each weight and the bias are for.
• Turn a set of measurements into a loss, and explain why the misses are multiplied by themselves before being added.
• Measure the slope of a loss against a weight by nudging, and say why looking both ways is more accurate than looking one way.
• Choose a learning rate by reading the shape of a loss curve, and recognise a rate that is too large from the shape alone.
• Say why a single neuron cannot switch a stairway light, and draw the reason on a square.
• Explain what a hidden layer adds, and why nothing forces its neurons to mean anything a person would recognise.
• Pick between a threshold, a sigmoid and a ReLU for a job, and give the slope-based reason rather than a preference.
• Work the chain rule as a product of measured slopes, without calculus notation.
• Follow backpropagation through a small network number by number, and say what a delta is.
• Run a gradient check on somebody's implementation and know what a single failing weight means.
• Say what depth buys in principle, what it costs in practice, and why those two answers point in opposite directions.

Courses used so far

• Making Training Work. The model that scores perfectly and is useless. Held-out data, leakage, imbalance, and the metrics that survive contact with a real problem.
• How a Language Model Guesses. The same forward pass and the same backpropagation, with words turned into lists of numbers and a layer that scores which earlier words each new word should be built from.
• A Model on a Microcontroller. What happens to these weights when they have to fit in 64 kilobytes and answer in ten milliseconds, on a chip with no floating point.
• Introduction to Machine Learning. If any of the words here were new, that course builds the same ideas from dots you place yourself, with no network in sight until near the end.

Turn a picture into numbers the network can eat

Every lab so far fed the network two numbers at most: two switches, or where a dot sits in a square. Real jobs start from a photograph, a recording, or a sentence, and a network cannot eat any of those. It eats lists of numbers and nothing else, so the first move of every real system is to turn its input into numbers, honestly and completely.

A picture gives its numbers up easily. Cut it into a grid of tiny squares, called pixels, and write down how much ink is in each one. A grid of seven rows of seven is a list of forty-nine numbers; that list is the picture, and nothing about it is lost. Words get numbers from a table instead: each word is assigned its own list of numbers, and those lists are adjusted during training exactly the way weights are. Sound is a list of air pressure readings, thousands per second. Whatever went in, the network sees a list of numbers, and everything this course has built so far applies to it unchanged.

Train on a handful of examples at a time

Every training run so far measured its slopes on the whole dataset before every step, because the whole dataset was three pancake batches or eighty dots. A real dataset can be millions of photographs. Reading every one of them to earn one small step would make each step cost hours. The intermediate numbers for millions of examples may not even fit in the machine.

So training deals the data into a batch: a small handful of examples, drawn at random, used to measure an approximate gradient. The approximation is noisy. Eight photographs might happen to be eight odd ones, and the step they suggest points slightly the wrong way. The next handful errs in some other direction, and on average the walk still heads downhill. Small batches buy many cheap, scattered steps, while large batches buy fewer, smoother, dearer ones. One pass through the whole pile, however it was dealt, is called an epoch.

Keep the signal alive through a deep stack

A message whispered through twenty children arrives mangled, because each child changes it a little and the changes compound. A deep network plays whispers with numbers. If each layer shrinks what passes through to nine tenths of its size, twenty layers pass on about an eighth of it. If each layer grows the signal slightly, twenty layers make it enormous. Either way, the far end works with rubbish.

The slopes travelling backwards suffer the same compounding, which Step 10 met as the vanishing gradient. The defences have names worth knowing. Careful initialisation deals the starting weights at a size chosen so that a layer neither shrinks nor swells its signal on average. Normalisation re-centres and re-scales the numbers between layers while training runs. A residual connection adds a short cut: a layer's input is added straight onto its output. The signal, and the slope coming back, both get a route that skips the risky part entirely.

Match the output and the loss to the question

Everything so far answered with one number and was scored by squared misses. Real questions come in shapes. How much flour is a quantity question. Which of ten digits is this, is a pick-one question. Which of these five faults does this engine have, is a yes-or-no question asked five times, because faults can arrive together. The output layer and the loss must be chosen to fit the shape of the question, or training pushes on the wrong thing.

For pick-one questions, the network gives each choice a raw score, called a logit. A rule called softmax then converts the list of scores into a list of positive shares that add up to 1. The shares read as how the network splits its confidence. Only the gaps between scores matter: add 5 to every logit and the shares do not move. The matching loss, called cross-entropy, charges the network by how little share it gave the correct answer. Confidently wrong is charged heavily, and hedging is charged a little. That is the behaviour a pick-one task wants, and squared misses do not provide it.

One practical trap deserves its name on the page. Softmax works by raising a fixed number to the power of each score. The result of a big score is too large for the computer to hold. The standard repair is to subtract the biggest logit from all of them first. The shares come out identical, because only the gaps matter, and nothing overflows.

Give the walk downhill a memory

Gradient descent, as Step 5 left it, has no memory. Each step reads the slope where it stands, multiplies by the rate, moves, and forgets. In a long thin valley, the kind Step 6 drew, that produces the zig-zag. The walk bounces between the steep walls while creeping along the gentle floor, and most of the travel is wasted sideways.

Momentum is the first fix: keep a running average of recent slopes, and step along the average instead of the latest reading. The sideways parts of the slope point opposite ways on alternate steps, so in the average they cancel. The along-the-valley part points the same way every step, so it accumulates. The name is honest, because the walk now behaves like a rolling ball that takes time to turn. Adam adds a second memory: a running typical size for each weight's slope, kept separately. A weight whose slopes run huge takes careful small steps, while a rarely nudged weight takes bolder ones.

None of this changes where downhill is. It changes how the walk behaves on the way, and it adds settings of its own. Runs often start the rate small while the first noisy steps pass, then shrink it late so the walk can settle. A written plan for changing the learning rate over a run is called a schedule. It is chosen and reported like everything else.

Teach one small filter, use it everywhere

An edge in a photograph is the same event wherever it happens: sky meets roof at the top of the frame, cat meets carpet at the bottom. A plain layer reading the pixel list from Step 14 would have to learn to spot that edge separately at every position, one full set of weights per place. That is wasteful, and worse, what is learned about edges at the top teaches the bottom nothing.

A convolution fixes both at once. One small set of weights, called a kernel, is slid along the input and applied at every position in turn, like one rubber stamp pressed along the whole row. Three weights scanning seven readings produce five answers, one per stop. The same three weights made all five. The saving in weights is the small half of the win. The large half is the built-in claim that a pattern means the same wherever it appears. That claim is true of pictures and of sound, and it is exactly the neighbourly knowledge the flat list threw away.

Let each word ask the others for help

In the sentence "the dog that chased the cat was muddy", which animal was muddy? You answered by looking back: "was muddy" belongs to "dog", five words away, and the cat in between is a distraction. A network reading a sentence needs that same move. Each position must be able to pull in information from other positions, near or far, by how relevant they are rather than by how close they sit.

Attention is that move written as arithmetic. Every position publishes a key, a list of numbers advertising what it holds, and a value, the information it will hand over. A position wanting help publishes a query, which is scored against every key, scoring high where the two lists line up. Softmax from Step 17 turns the scores into shares that sum to 1; the asking position receives the values, mixed in those shares. Nothing is fetched by position number. Everything is fetched by how well it matches, which is why the faraway dog can outweigh the nearby cat.

Two practical notes complete the picture. The scores know nothing about word order, so each word's position is folded into its numbers separately. And a model being trained to guess the next word must not peek at it, so scores pointing at later positions are blocked. The block is called a causal mask.

Start from a network that already knows something

A cook who can already make pancakes learns waffles in an afternoon, because most of the skill carries over. Networks can be given the same head start. A network trained for months on mountains of text or images has built internal machinery for recognising structure, and that machinery is reusable by tasks its builders never imagined. Starting from it is called transfer, and it is how most real projects begin, because almost nobody can afford the months.

The recipe: take the big trained network, called the base, freeze its weights, and train only a small addition on your own data. The addition can be a fresh output layer, or an adapter, a thin set of extra weights threaded through the base. Training one hundredth of the weights means one hundredth of the slopes to compute and store, so the training bill collapses. The answering bill does not: the frozen base still runs in full every time a question is asked. Cheap to train and cheap to run are different claims, and an adapter only buys the first.

Where did the base's months of training come from, when nobody can label mountains? The trick is called self-supervision: hide part of the data and train the network to fill it back in. Cover the next word and demand it. The text itself is the answer sheet, so no person labels anything, and the supply of practice material is every sentence ever written.

Networks that produce instead of judging

Everything so far took data in and answered a question about it. A generative network runs the other way: it produces new examples that could pass for members of the training pile. A sentence that was never written, or a face that belongs to nobody. There is no single algorithm behind the word. There are several different training tasks, and which one was used decides how the model produces.

Four families cover most of what you will meet. An autoregressive model produces one piece at a time, each piece guessed from the pieces so far, which is next-word guessing run as a factory. An autoencoder squeezes its input through a deliberately narrow middle and rebuilds it on the far side, learning a compact description on the way. A diffusion model is trained to remove a little noise from a spoiled example. It produces by starting from pure noise and cleaning it up, step after step, until a picture condenses. An adversarial pair trains a forger network against an inspector network, and each drives the other to improve.

One habit matters more than the mechanisms. A single convincing sample proves almost nothing. It does not show the model can produce variety, and it does not show the sample is new rather than a training example memorised and handed back. Produced data also needs marking as produced, because a later model trained on it as though it were measured truth inherits every one of its quirks.

Count the bill before agreeing to train

Weights cost memory. Every example flowing through costs memory for its intermediate values, the ones Step 11 kept for the backward pass. Every weight costs arithmetic each time an example passes it. Training multiplies everything: the optimiser memories from Step 18 can triple the space for weights, and each step runs the network forwards and backwards. Before anyone trains a large model, someone counts, because the count decides what is affordable before any cleverness starts.

The counting is ordinary multiplication, and one shape in it deserves respect. Double the width of every layer and the weight count quadruples, because weights connect a doubled layer to a doubled layer. The same square that made the pixel list expensive in Step 14 makes width expensive here. Work then scales with weights times examples, so the bill is a product of choices, and halving any factor halves it.

The standard economies have names. Mixed precision stores numbers with fewer digits, halving memory and speeding arithmetic. The sums that need care are kept in full width, so small gradients do not round away to zero. For answering rather than training, quantisation shrinks the stored weights further, and distillation trains a small network to copy a big one's answers.

Show that it survives the messy world

A model that scores well on clean test photos has answered one question: how it does on data like its training data. The porch camera from Step 1 will meet rain, glare, cobwebs and a moth on the lens. Any of those can flip an answer. Worse, some flips can be caused on purpose. A small, carefully chosen change to an input, invisible to a person, can walk an example across the network's decision boundary. The boundary was never trained to be far from anything.

That distance has a name. An example's margin is how far it sits from the nearest point where the model's answer flips. A wide margin means every small change leaves the answer alone; a change larger than the margin can cross. Measuring margins is honest evidence with a stated scope: it speaks about small changes near this example, and says nothing about snow, new camera angles, or the moth.

The same discipline applies to explanations. Tools that highlight what an answer leaned on show where information flowed, like the attention shares in Step 20. Routing is not a reason, and a highlight is not proof of cause. A release argument is a pile of separate evidence. Error rates on the worst slices rather than the average. Confidence that matches how often the model is right. A stated fallback for the inputs it declines, and monitoring that keeps measuring after the launch.

What you can do now

• Turn a picture into the list of numbers a network actually receives, and say what the list loses.
• Choose a batch size by naming what it trades: noise against memory against updates per pass.
• Explain why deep signals vanish or explode, and what initialisation, normalisation and residual paths each do about it.
• Match an output layer and loss to a question's shape, and name the overflow trap softmax brings.
• Say what momentum remembers, what Adam adds, and what neither removes.
• Explain what sliding one kernel buys, and track a shape through padding and stride.
• Walk one attention lookup by hand: query, keys, scores, shares, mixture.
• Weigh a frozen base plus adapter against full tuning, using both bills.
• Tell the four generative families apart by what each one is trained to do.
• Estimate a training bill, and say why width costs more than depth.
• Scope a margin claim, an attention highlight, and a release argument honestly.

Continue the learning path

• Training and Evaluation holds data back, measures honestly, and catches the model that memorised its homework.
• Language Models builds tokenisation, transformers, retrieval and tool use on top of the attention you just met.
• Computer Vision takes the pixel grids and kernels onwards to detection and segmentation.
• TinyML squeezes these networks into microcontrollers, where every byte and millisecond is counted.