Interactive course · about 8 hours

Reliable Data Transfer

The network you are reading this over throws things away. It hands over packets in the wrong order, sometimes gives the same one twice, and now and then holds one for a second before letting it through. Every photo, song and program you have ever downloaded crossed that. Not one byte of any of them was wrong. Somebody built the difference, out of nothing but sending things again and counting carefully, and by the end of this you will have built it too.

How this works

There is a working network in this page. It has a real clock, a queue of things waiting to happen, packets with numbers on them, timers that go off, and a receiver that really does put the pieces back together. When a widget tells you six packets were lost, six packets were lost. Nothing here is a recording. You set the sliders and the simulated network does what you told it to do.

What you need before you start

You should already know that a message crossing a network is chopped into packets, that each packet carries an address and a check on its own contents, and that the network is allowed to lose one without telling anybody. The course Computer Networks builds all of that from a single piece of copper. If you have not done it, the first step here re-explains just enough to keep going. No programming is needed: the few places where you write a rule yourself introduce every piece of it first.

The steps

Step 1

The channel lies

Two machines, and something in between that carries packets. Call that something the channel. It does not matter whether it is a cable under an ocean or a radio link to a phone: from where we are standing it is a thing you hand a packet to, which usually hands it out the other end.

Usually. Four things can happen to a packet instead, and all four happen every day on every network in the world. The channel can lose it, and say nothing. It can delay it, so it turns up late. It can reorder a pair, so packet 5 arrives before packet 4. It can duplicate one, so the same packet arrives twice.

What is a packet, and what is a byte

A bit is one yes-or-no, written as 0 or 1. Eight bits side by side make a byte, which can hold any one of 256 different values. One letter of ordinary English text is one byte. A short song is a few million of them.

Networks do not carry bytes one at a time. They carry them in bundles called packets: a few hundred to a few thousand bytes of the actual message, with a small header stuck on the front holding the addresses and anything else the network needs to know. In this course a packet holds one short piece of a file, and we will add fields to that header as we discover we need them.

What does a 20% chance of loss actually mean

Set loss to 20% and every single packet is decided separately: the channel rolls a die for it, and about one packet in five is thrown away. It is not "every fifth packet". You can easily send five and lose two, or send five and lose none. Over a hundred packets the count lands near twenty, and over ten it can be anything.

The dice here come from a seeded generator, which means the same seed always produces the same run. That is why pressing Send again with the same settings gives the same answer, and why the Reroll button exists. Real networks lose packets in bursts rather than evenly, which is worse than this, and Step 11 lets you cause a burst by hand.

Why would a network throw anything away on purpose

Mostly it is not on purpose, and mostly it is not damage either. A router holding packets waiting for a busy outgoing line has a limited amount of room to hold them in. When that room is full and another packet arrives, there is nowhere to put it, so it goes in the bin. That single sentence is most of the loss on the internet, and Step 9 is about what happens when everybody causes it at once.

The rest is corruption. A packet whose bits got scrambled fails the check on its own contents and is dropped rather than passed on, because a wrong packet is worse than a missing one. From up here those two look identical: the packet simply never arrives.

Lab 1 · The channel bench
Try this firstPress Send 12 packets with everything at zero. All twelve arrive in order, and the narration says so. Now drag loss to about 25% and press Send again, then put loss back to 0 and drag reorder up instead.
Notice: with reorder on and loss at zero, nothing is missing. Every packet arrives, just not in the order you sent them. The receiver cannot tell that apart from a loss until the late one turns up, and it has no idea how long to wait for something that might not exist. Notice too that the duplicate slider produces arrivals with the same number twice, and the receiver has nothing on the packet that would let it tell a copy from a fresh packet.
Lab 2 · Name the fault
Try this firstRead the arrival list, then press the button naming what the channel did. The answer and a full account of the run appear underneath, and your score so far sits at the top. Press Next case for another one.
Notice: some cases cannot be told apart from the arrival list alone, and those are the ones the answer explains hardest. A gap in the numbers means either loss or a packet still in flight. That ambiguity is the whole problem: the receiver sees exactly this list and nothing else, and has to decide what to do about it.
You send packets numbered 1 to 6. The receiver reports that it saw 1, 2, 3, 5, 6 and then, a moment later, 4. Loss is set to zero. What did the channel do?
Packets do not travel as a convoy. Each one is handled separately, and they can take different routes or sit in different queues, so a slow one gets overtaken. Nothing was lost: packet 4 was simply late. The trap in the first option is thinking a network keeps track of what it dropped. It does not keep track of anything, which is exactly why the two ends have to.
Step 2

Send one, wait for the nod

Here is the smallest idea that could possibly help. Send one packet. Do not send the next one until the other end says it got that one. The saying-it-got-it message is called an acknowledgement, almost always shortened to ACK, and this whole arrangement is called stop and wait.

It fixes loss on its own. If the packet never arrives, no ACK ever comes back, and the sender is still sitting there holding a copy, so it can send it again. Nothing is ever thrown away until somebody has confirmed it landed. That rule is the foundation of everything else in this course.

What is a protocol

A protocol is an agreement about who says what, and when, and what each thing means."Send one packet, then wait for an ACK before sending the next" is a protocol. So is a phone call: you say hello, I say hello back. We both know that means the line is open.

The important part is that both sides run the same agreement, and neither can see inside the other. All either side has to go on is the messages that arrive, so every rule has to be decidable from those alone. Most of the mistakes in this course come from a rule that quietly assumed something the other side never said.

Where does the receiver put the data: what a buffer is

A buffer is a patch of memory set aside to hold things that have arrived but are not finished with yet. The receiver in this course has two: one for the file it is building up, and later a small one for packets that arrived early and have to wait for a gap in front of them to be filled.

Buffers are always a fixed size, because memory is finite. A buffer that fills up and gets another arrival has to throw something away, which is a loss the network did not cause. That turns out to matter enough to be Step 7 all by itself.

Why not just send everything at once and hope

You can, and for some things people do. If you are sending live speech, a lost fiftieth of a second is better skipped than fetched late, because by the time the replacement arrives the conversation has moved on. Sending and hoping is a real choice, not a lazy one.

But for a file it is useless. A file with one packet missing out of ten thousand is not 99.99% of a file, it is a broken file. So somebody has to notice the hole and fill it, and only two machines are in a position to notice: the one that sent it, and the one that wanted it.

Lab 3 · You are the channel
Try this firstPress Send, then press Deliver, then press Deliver again to let the ACK through. One letter appears in the received file. Do that twice more, then on the fourth letter press Drop when the ACK is offered, and watch what the sender does next.
Notice: when you drop the data packet, nothing bad happens. The sender waits, gives up waiting, sends the same letter again, and the file comes out right. When you drop the ACK instead, the letter arrived perfectly and the file still comes out wrong. The receiver has no way to know that the second copy is a copy, so it writes the letter twice.
Lab 4 · How often does that actually happen
Try this firstLeave ACK loss at 10% and press Run 200 transfers. It sends the word HELLO two hundred times and counts how many of the two hundred came out as something else. Then drag ACK loss to 30% and run it again.
Notice: the damage is not rare. At 10% ACK loss, more than two of every five five-letter words come out wrong, because it only takes one lost ACK anywhere in the five to spoil it. The results table shows the actual wrong words, and they are all the same shape: a letter doubled. Nothing is ever missing, which is a good clue about what the fix has to do.
The data packet arrives fine and the receiver sends an ACK, but the ACK is lost. Which description is right?
Neither side can see the other. The receiver has no reason to suspect anything, because from where it sits the job is done. The sender only sees silence, which looks exactly like the packet having been lost. There is no message either could send that would settle it, because that message could be lost too. The disagreement is the point: the fix is not better information, it is a rule that still gives the right answer while the two sides disagree.
Step 3

Numbering the packets, and a timer

The receiver wrote the letter twice because the two packets were identical. It had nothing to compare. So put a number in the header, and have the receiver keep one number of its own: the number it is expecting next. A packet whose number matches gets written to the file. A packet whose number does not match is a repeat, and gets thrown away, but is still acknowledged, because the reason it was sent again is that the last ACK went missing.

For stop and wait, the number does not have to count upward forever. Only one packet is ever outstanding, so the receiver only has to tell "the one I want" from "the one before it". Two values are enough: 0, 1, 0, 1. One bit of header, and the duplicate problem is gone.

One bit, and why alternating is enough

A single bit holds either 0 or 1. The sender labels its first packet 0, and does not move on to a packet labelled 1 until packet 0 has been acknowledged. So at any moment there is at most one unacknowledged packet in the world, and the only two packets that could possibly arrive are the one the receiver wants and the one it has just finished with. Those two always have different labels.

This stops working the moment more than one packet is in flight, which is Step 5. Then you need enough numbers to tell apart everything that could be in the air at once, and choosing how many is a real decision with a wrong answer.

What is an algorithm

An algorithm is a set of steps precise enough that following them requires no judgement. A recipe is close, but a recipe says "season to taste" and an algorithm never can. Every question has to be answered in advance, including the awkward ones.

"Accept the packet if its number is the one you expected, otherwise throw it away and ACK it anyway" is an algorithm. It has no gaps: whatever arrives, the rule says what to do. Most of this course is finding the gaps in rules that sounded complete.

I have never written a program before

You are about to, and it is four lines. A function is a named rule that takes some values in and hands one value back. function accept(expected, seq) { … } declares a rule called accept which is given two numbers, called expected and seq while it runs. Those names are variables: labels standing for whatever values got handed in this time.

Inside, if (a === b) { … } does what is in the brackets only when a and b are the same value, and return true; stops the rule right there and hands back the answer true. True and false are the only two answers this particular rule needs. Press Run to see what your rule says about one example, and Check my answer to have it tried against packets you cannot see.

Lab 5 · Numbered against unnumbered
Try this firstPress Run both. The same word goes over the same sabotaged channel twice, once with numbering switched off and once with it on, using the same dice both times. Compare the two received files, then push ACK loss higher and run it again.
Notice: the numbered run sends exactly as many packets as the unnumbered one and loses exactly the same ones. The dice are identical. The only difference is one bit in the header and one number remembered at the receiver, and that is the entire difference between a correct file and a corrupt one.
Lab 6 · Write the receiver's rule
Try this firstPress Run. The output shows what your rule currently says about one arriving packet. Then press Check my answer: it feeds your rule a stream of packets you cannot see, including repeats, and rebuilds a file from whatever you accepted. Fix the rule until the file comes out right.
Notice: the starting rule accepts everything, which is exactly the bug from Step 2, and the checker tells you which letter got doubled. Your rule is asked about each arrival on its own and does not get to see the others, which is the real situation the receiver is in.
Lab 7 · How long before you give up waiting
Try this firstDrag the timeout slider to 40 ms and press Send the file. Watch the resend counter. Then drag it to 900 ms and send again, and compare the two finish times at the bottom.
Notice: the two extremes fail differently, and only one of them is slow. A short timeout fires while the ACK is still on its way. The sender then floods the channel with copies of packets that were never lost. Read the sends figure at 40 ms against the twenty packets in the file. It does not finish any later here, because this link has nothing else on it and a copy that arrives is as good as the original. Put fifty senders on the link doing that and you get Step 9. A long timeout is simply slow: every real loss costs almost a second of silence. The best setting sits just above the time a round trip actually takes, which nobody has measured yet.
A sender's timeout is set to half the true round-trip time, on a channel that loses nothing at all. What happens?
The timeout fires before the ACK has had time to get back, so the sender resends a packet that was fine. The numbering saves the file: the receiver sees a number it is not expecting, drops the copy and acknowledges it. So the answer stays correct and the cost is pure waste. That is the pattern to watch for from here on. A bad timing decision rarely breaks correctness, it just quietly burns the network, which is much harder to notice.
Step 4

How long to wait

The timeout should be slightly longer than a round trip: the time from a packet leaving the sender to its ACK coming back. Too short and you resend things that were never lost. Too long and every real loss costs you a long silence. So measure it. Note the clock when the packet goes out, note it again when the ACK arrives, subtract.

That gives you one measurement, and one measurement is not enough, because the number moves. A route changes. A queue somewhere fills up. Somebody starts a download on the same link. What you want is a running estimate that follows the trend without jumping at every wobble.

What is actually in a round trip

Four things, and only one of them is the speed of light. There is the time to push the bits onto the wire, which depends on how fast the link is. There is the travel time, which depends on distance. There is the time spent sitting in queues at each router on the way. And there is the time the far end takes to notice the packet and answer it.

The travel part is fixed and honest. London to Sydney and back is about 170 milliseconds of pure distance and no clever engineering will improve it. The queueing part is the one that swings wildly, which is why the estimate has to keep chasing.

What a smoothed average is, and why not a plain one

A plain average of every sample so far treats a measurement from an hour ago as seriously as one from a second ago. On a network that is wrong: the old ones describe a network that no longer exists.

A smoothed average fixes that by mostly keeping what it already believed and nudging it towards each new sample: new estimate = 7/8 of the old estimate + 1/8 of the new sample. A single odd sample moves it hardly at all. A sustained change drags it across within a few dozen samples. The fractions are a dial: 1/2 makes it twitchy, 1/64 makes it stubborn.

Why the timeout is the estimate plus four times the wobble

The average is where the round trip usually lands, so setting the timeout to the average means firing early roughly half the time. You need headroom, and the right amount of headroom depends on how jumpy the link is. A steady link needs almost none. A jumpy one needs a lot.

So a second smoothed average is kept, this time of how far each sample sat from the estimate. Call that the wobble. The timeout becomes the estimate plus four wobbles. On a steady link that is barely more than the average, and on a jumpy one it opens right out. This is what real senders do, and the four was chosen by trying values until the spurious resends stopped.

Lab 8 · Measure the round trip
Try this firstPress Send 40 probes. Each dot is one measured round trip, and the line through them is the running estimate. Now drag jitter up and press it again, then press Route change halfway and watch how long the line takes to catch up.
Notice: the estimate line is smooth even when the dots are scattered, and it always lags. On the route change it takes something like a dozen samples to arrive at the new level, and the narration counts them for the run in front of you. During that lag the estimate is describing a network that no longer exists. The dashed band above the line is the timeout the sender would be using. It stays above every dot, even through the route change, because the wobble grows the moment the samples stop matching the estimate. That is the four wobbles earning their keep. A dot above the band would be a packet given up on while it was still on its way, and there is not one anywhere in the range these sliders offer.
Lab 9 · Write the estimator
Try this firstPress Run to see your rule fed three sample measurements in a row. Then press Check my answer, which runs it against two hidden traces: one steady link with occasional spikes, and one that changes level halfway through.
Notice: the two tests pull in opposite directions and both have to pass. A rule that just repeats the newest sample fails the spike test. A rule that barely moves fails the route change. Anything in between the two passes, so there is no single right answer here, only a range of sensible ones.
A sender is resending a packet after a timeout. The ACK that finally arrives could be for the original or for the copy, and the sender cannot tell which. Why must it refuse to use that round trip as a measurement?
If the ACK is for the original, timing it from the resend makes the round trip look much shorter than it is, which shortens the timeout and causes more spurious resends, which produce more ambiguous samples. If the ACK is for the copy and you timed from the original, it looks far too long. Either way you are feeding rubbish into a number the whole protocol depends on. The rule real senders use is simply never to sample a round trip that involved a retransmission.
Step 5

A window of packets in flight

Stop and wait is correct and painfully slow. It sends one packet per round trip, no matter how fast the link is. On a link that could carry a hundred packets in the time one round trip takes, ninety-nine packet-sized holes go past empty while the sender stands there waiting for a nod.

So stop waiting after each one. Allow several packets to be unacknowledged at the same time. The set of packets the sender is allowed to have outstanding is called the window, and its size is the number of packets it may have in the air at once. Every ACK that arrives lets the window slide forward by one, so a new packet can go out. Stop and wait is just a window of size 1.

What throughput is, and how it is counted here

Throughput is how much gets across per second. This course counts it in packets per second, because the packets here are all the same size, which keeps the arithmetic visible. Real measurements are usually in bits per second, and you get there by multiplying by the packet size in bits.

Two different things limit it. The link has a top speed, which is how quickly it can push bits out. And the protocol has a limit of its own: a window of W packets per round trip can never beat W divided by the round-trip time, however fast the link is. Whichever limit is lower is the one you get.

Why the link has a volume, like a pipe

Think of a garden hose. How much water is inside it at any moment is the flow rate multiplied by how long the water takes to get from one end to the other. A network link is the same: the number of packets in flight is the link speed multiplied by the round-trip time. That product has a name, the bandwidth-delay product, and it is the number of packets it takes to fill the pipe.

A window smaller than that leaves the pipe partly empty and wastes the link. A window larger than that does not help, because the extra packets have nowhere to go but a queue, and sitting in a queue is not progress. The lab below finds the exact turning point for you by measuring rather than by arithmetic, and the two answers agree.

What an array is

An array is a numbered row of values kept under one name. [4, 5, 6, 7] is an array of four numbers. You reach one of them by its position, counting from zero, so if the array is called seqs then seqs[0] is 4 and seqs[2] is 6. Counting from zero looks odd for about a day and then never again.

seqs.length is how many items it holds, and for (const s of seqs) { … } runs the same lines once for every item, with s standing for each one in turn. The sender's window is naturally an array: the packets it has sent and not yet had acknowledged.

Lab 10 · Find the right window size
Try this firstLeave the window at 1 and press Run for 3 seconds. Note the packets-per-second figure. Now drag the window to 4 and run again, then 8, then 20, filling in the chart as you go. The chart keeps every result you have measured.
Notice: throughput climbs in a straight line and then flattens completely. The flat part starts at the window size where the pipe is full, and the panel shows you the arithmetic for that number so you can check it against your measurement. Past that point a bigger window buys you nothing but a longer queue, which shows up as the rising delay figure beside the throughput.
Lab 11 · Write the window test
Try this firstPress Run to see your rule judged against one example window. Then press Check my answer: it asks your rule about every sequence number from 0 to 40 for several different windows, and compares the set you allowed with the set that should be allowed.
Notice: the two edges are where this goes wrong, and the checker tells you which edge you got. The packet numbered base is inside the window. The packet numbered base plus size is not: with base 10 and size 4 the window holds 10, 11, 12 and 13, and nothing else. Off-by-one at a window edge is one of the classic bugs in real protocol code.
A link carries 1000 packets per second and the round trip is 100 milliseconds. The sender uses a window of 20. Roughly what throughput does it get, and why?
The window lets 20 packets be outstanding, and it takes one round trip of 0.1 seconds before those are acknowledged and 20 more can go. That is 20 every tenth of a second, so 200 per second, a fifth of what the link could do. The pipe holds 1000 times 0.1, which is 100 packets, so a window of 100 would fill it. The third option is the common slip: the window is a count of packets in flight, not a rate, and turning it into a rate needs the round-trip time.
Step 6

Repairing one hole

You have twelve packets in flight and number 5 goes missing. Numbers 6 through 12 arrive perfectly. What should happen?

The blunt answer is go back N: the receiver refuses anything after a gap, so 6 through 12 are thrown away, and the sender resends 5 and everything after it. It needs almost nothing at the receiver, which is why the earliest systems did it. It also throws away seven perfectly good packets to recover one.

The careful answer is selective repeat: the receiver keeps 6 through 12 in a buffer, tells the sender exactly which pieces it is holding, and the sender resends only packet 5. When 5 finally arrives, the receiver hands the whole run up to the file at once. More bookkeeping on both sides, far less waste.

What a cumulative acknowledgement means

A cumulative ACK does not say "I got packet 7". It says "I have everything up to and including 7", which is a much stronger claim, and it means a lost ACK repairs itself: if the ACK for 7 goes missing but the ACK for 8 arrives, the sender learns about both. That repair costs nothing extra. It is why real protocols acknowledge this way.

The cost is that a cumulative ACK cannot describe a hole. If 5 is missing and 6 through 12 arrived, the strongest true statement is still "I have everything up to 4", and the sender cannot tell whether the rest arrived or not. Selective repeat needs an extra field in the ACK to list the pieces held beyond the gap, which real senders call a selective acknowledgement.

What the receiver has to remember for selective repeat

A buffer with room for a whole window of packets, and a mark against each slot saying whether it is filled. When a packet arrives ahead of the gap it goes into its slot rather than into the file. When the gap is finally filled, everything from the gap forward that is present moves into the file in one go and the window slides by however many that was.

None of that is difficult, it is just work that go-back-N does not do. The trade is memory at the receiver against packets on the wire, and since about 1990 memory has been the cheap side of that trade by a very wide margin.

Would anyone still choose go-back-N

On a link that almost never loses anything, the waste never happens, so the simpler receiver is free. Some small embedded links are still built this way for exactly that reason, and the code is a fraction of the size.

The moment loss is common, or the window is large, it becomes indefensible. With a window of 100 and a 1% loss rate, roughly every hundredth packet costs you a hundred retransmissions, so your effective loss rate is close to 100%. The lab below lets you push it there and watch the number of wasted sends overtake the number of useful ones.

Lab 12 · The same loss, two answers
Try this firstPress Run both. A 24 packet file crosses the same channel twice with the same dice, once under each scheme, and the two logs run side by side. Compare the sends used figures underneath, then push loss up and run it again.
Notice: both schemes deliver the identical correct file. The difference is entirely in how many times packets crossed the wire to achieve it, and the gap widens as either the loss or the window grows. The wasted column counts sends of packets that had already arrived safely, which is work the network did for nothing.
Lab 13 · Somewhere to try things
Try this firstSet whatever you like and press Run. Try window 16 with loss 1%, then the same window with loss 8%, and watch the efficiency figure. Nothing here is marked.
Notice: efficiency is the number of packets in the file divided by the number of sends it took, so 100% means nothing crossed the wire twice. See how low you can push it with go-back-N without touching the loss slider above 10%, and then see what selective repeat does with the same settings.
Under go-back-N with a window of 8, packet 20 is lost and packets 21 to 27 all arrive safely. How many packets does the sender put on the wire to recover, and how many of those were already safely delivered once?
The receiver discarded 21 to 27 because they came after a gap, so as far as it is concerned they never happened. The sender goes back to 20 and resends 20 through 27, which is eight packets, and seven of those crossed the network safely the first time. The middle option is what selective repeat achieves, at the cost of a receiver buffer and a richer ACK.
Step 7

Not drowning the receiver

So far every loss has come from the channel. Here is one that does not. A powerful machine sends to a small one. The channel behaves perfectly. The small machine still loses data, because packets arrive faster than it can deal with them, its buffer fills, and the next arrival has nowhere to go.

The fix is for the receiver to say how much room it has left, in every ACK it sends. That figure is the advertised window, and the sender is not allowed to have more unacknowledged data in flight than the receiver says it can hold. This is called flow control. It is a completely separate mechanism from anything to do with the network being busy.

Why can the receiver not simply be quicker

Because arriving is not the same as being used. Data sits in the receiver's buffer until the program that asked for it comes and takes it, and that program has its own life: it might be drawing something on screen, or waiting for a disk, or simply not scheduled to run this instant.

A phone downloading a file while the screen is off, or a tiny sensor with a few kilobytes of memory talking to a server, are both in this position. The receiver is not slow because it is badly made. It is slow because it is doing something else, and no amount of engineering removes that.

How is this different from the network being busy

Flow control protects the far end. Congestion control, which is Steps 9 and 10, protects the network in the middle. They look similar because both end up telling the sender to slow down, and they are frequently confused, including by people who should know better.

They have to be separate because either can be the tight one. A supercomputer sending to a phone over an empty fast link is limited by the phone. Two laptops on either side of a congested link are limited by the link, not by each other. A sender obeys both limits at once and sends the smaller of the two, which is why real senders carry two window numbers and take the minimum.

What if the advertised window goes to zero

Then the sender must stop completely, which raises an awkward question: the announcement that there is room again will arrive in an ACK, and there is nothing left to acknowledge, so no ACK will be sent. Both sides wait forever. This is a real deadlock and it has bitten real implementations.

The cure is that a sender facing a zero window keeps a timer and pokes the receiver with a tiny packet every so often, purely to provoke an answer that carries the current window figure. It is not pretty, and it is the standard answer, because any design where a needed message can be lost has to have somebody willing to ask again.

Lab 14 · Fast sender, slow receiver
Try this firstPress Run with flow control off. The buffer gauge fills, turns red and the dropped counter climbs, even though channel loss is zero. Now press Flow control: on and run it again.
Notice: with flow control on, nothing is dropped and the transfer takes almost exactly the same total time. The receiver was always going to be the limit. All the sending faster achieved was destroying data and then having to send it again, and the resends made the same transfer take longer than simply going at the receiver's pace.
Lab 15 · Predict the overflow
Try this firstRead the three numbers, then press the button with the second you think the first packet gets dropped. The truth appears underneath along with a second-by-second table of the buffer, whichever button you pressed.
Notice: the buffer fills at the difference between the two rates, not at the sending rate. A sender at 9 per second into a receiver draining 8 per second fills a 10 slot buffer in ten seconds, not in just over one. Small differences take a long time to hurt, which is precisely why this bug is usually found in production rather than in testing.
A sender is told the receiver has room for 4 packets, and separately works out that the network can carry 30 in flight. How many should it have outstanding?
The two limits are separate rules and both are hard limits. Exceeding the receiver's window overflows its buffer, and exceeding what the network can carry overflows a router queue. Neither failure is softened by the other limit being generous, so the sender takes the smaller of the two. Adding them is the trap, and it comes from imagining the buffer space as somewhere the extra packets could wait, when it is the thing already full.
Step 8

Agreeing to talk

Everything so far assumed the two ends already knew about each other and agreed which number the packets start at. Neither is free. Before any data moves, the two machines have to establish that both are present, both are willing, and both know the other's starting number.

It takes three messages. The caller sends a SYN, meaning "I want to talk, and my numbers start here". The answerer replies with a SYN-ACK, meaning "heard you, and my numbers start here". The caller sends an ACK, meaning "heard that too". Three messages, one and a half round trips, and only then does data flow.

Why three and not two

Two messages tell the caller everything it needs, but leave the answerer hanging. It has announced its starting number and has no idea whether that announcement arrived. If it did not, the caller will reject everything the answerer sends, because the numbers will not match anything it expects.

The third message is what closes that. Each side ends up with proof that its own starting number reached the other, and proof is the strongest thing available here: you cannot know the other machine is alive, only that it was alive when it sent the thing you are holding.

Why not just start both sides at zero

Because packets outlive connections. Suppose you talk to a machine, the conversation ends, and a packet from it is stuck in a queue somewhere. You open a new connection to the same machine, and both sides start at zero again. That stale packet arrives, its number fits perfectly. It gets written into the middle of your new file.

Starting numbers are therefore picked fresh for each connection, far apart from each other, so an old packet almost never fits. The second lab in this step makes exactly this go wrong on purpose, and then shows the third message refusing it.

How does a connection end

The same way, mostly. Each side sends a message saying it has no more data, and each of those is acknowledged, so a full close is four messages. Either side can finish talking while still listening to the other, which is why it is two independent goodbyes rather than one.

The side that closes last then waits a while before forgetting the connection entirely, for the same reason the starting numbers are randomised: a straggler from this conversation must not be mistaken for the first packet of the next one.

Lab 16 · Three messages, and dropping one
Try this firstPress Start the handshake and watch it complete. Then press Drop the SYN to arm it and start again: the first message vanishes and you can watch the clock while the caller waits for its timeout.
Notice: a lost SYN is the most expensive loss in the whole conversation. There is no round-trip estimate yet, so the timeout has to be a guess, and the guess must be generous enough for a slow link on the far side of the world. Compare the three cases: each of them costs one full guessed timeout, but they cost it to different sides. Lose the first or the second message and the caller is the one sitting in the dark. Lose the third and the caller believes it is open after 120 ms and carries on, while the answerer waits out the whole guess before asking again. In a real conversation the caller's first data packet carries the same acknowledgement, so that third loss costs nothing at all.
Lab 17 · The ghost of an old connection
Try this firstPress Two-message version. A SYN from a dead connection has been stuck in the network and arrives now. Watch what the answerer does with it, then press Three-message version and watch the same ghost fail.
Notice: in the two-message version the answerer allocates a buffer and waits for data that will never come, because there is nobody on the other end of that connection any more. The third message is what tells it the difference between a real caller and an echo of an old one. This is the same reasoning as the numbering in Step 3, one level up: you cannot tell a copy from a fresh thing without something on the packet that distinguishes them.
The third message of the handshake, the caller's final ACK, is lost, and the caller immediately starts sending data. What happens?
Every packet carries an acknowledgement field, so the first data packet says everything the lost ACK said and more. The answerer sees its starting number acknowledged and treats the connection as open. If the caller had nothing to send, the answerer's own timeout would eventually resend the SYN-ACK and get an answer. This is a good example of a mechanism covering for a loss without anybody designing it in specially.
Step 9

When the network melts

Every mechanism so far has been about two machines. Now put fifty senders on one link. The link has a capacity, a fixed number of packets it can carry each second, and in front of it sits a queue holding packets waiting their turn. The queue has a fixed number of slots. When it is full, arrivals are dropped.

Push the offered load past the capacity and something worse than "it goes no faster" happens. The queue fills, so delays climb, so timeouts fire on packets that were merely queued, so senders resend them, so the offered load rises further. The amount of useful data getting through actually falls as you try harder. That is congestion collapse.

Throughput and goodput are not the same number

Throughput counts every packet the link carries. Goodput counts only the ones that were useful: first arrivals of data the receiver did not already have. A duplicate caused by a spurious timeout is throughput and not goodput.

A collapsing link is often at 100% throughput. It is completely busy. It is just busy carrying copies of things it already carried. Goodput is the number that matters and the one a simple load meter will not show you, which is part of why the failure was so confusing when it first appeared.

Why routers have queues at all

Traffic arrives in bursts. Without somewhere to put a burst, a router would have to drop half of a perfectly ordinary clump of packets that happened to turn up together, even on a link that is idle on average. A queue smooths that out and is the difference between a usable network and an unusable one.

The trouble is that a queue that is permanently full stops smoothing anything and just adds delay to every single packet. That condition has a name, bufferbloat, and it is why a big buffer is not automatically a good buffer. The second lab here lets you set the queue length and watch the delay follow it.

This actually happened

In October 1986 the link between Lawrence Berkeley Laboratory and the University of California at Berkeley, a few hundred metres apart, dropped from 32,000 bits per second to 40 bits per second. A factor of a thousand, on a link that was not broken and had not changed. Van Jacobson investigated it and the resulting work is the reason the mechanisms in the next step exist.

Nothing was wrong with any individual sender. Each was following a reasonable rule: if you get no answer, try again. The failure lived in what happens when everybody follows that reasonable rule at the same time, which is a kind of bug you cannot find by testing one machine.

Lab 18 · Push it past capacity
Try this firstPress Sweep the load. It runs the network once at each of thirteen offered loads and plots what actually got through. Watch where the solid line stops following the dotted one, and where it starts coming down again. Then press Senders measure and back off and compare.
Notice: the peak is not at the capacity, it is below it, and past the peak the line falls. Every point on that chart is a full simulated run, with real queues, real timeouts and real resends. The offered load includes copies, so the harder the senders try, the less of the link is left for anything new. The second button gives the senders the round-trip estimate from Step 4 and the backoff from Step 10, and the same sweep goes flat instead of falling. That is the whole of the next step, in one line on a chart.
Lab 19 · Watch the queue
Try this firstPress Run with the load at 90% and watch the queue bar. It bounces and mostly stays low. Now set the load to 130% and run it again, then try the same load with the queue length set to 200.
Notice: the long queue does not reduce the drops much, it converts them into delay. Every packet now waits behind two hundred others, so the round trip balloons, and a round trip that has ballooned past the sender's timeout produces resends of packets that are still perfectly alive in the queue. A bigger buffer made the problem quieter and worse.
A link is completely busy: it is carrying packets every moment of every second. Its users report that transfers have almost stopped. Is that possible, and how?
A busy link is not the same as a useful link. When queues are full, delays exceed timeouts, and every sender is resending things that are sitting in a queue somewhere waiting their turn, the link can be at full throughput and near zero goodput. This is why measuring how busy a link is tells you almost nothing on its own, and why the fix has to change how senders behave rather than making the link faster.
Step 10

Backing off together

Nobody can see the queue. There is no message from the network saying "I am full". The only signal a sender gets is a packet that went missing, and the rule has to be built from that alone: treat a loss as the network telling you to slow down.

The rule that works is additive increase, multiplicative decrease. While things are going well, add one packet to your window per round trip. The moment you lose one, halve it. Creep up, cut hard. Every sender independently running that rule produces a window that saws up and down, and a set of senders that settles near a fair split of the link without any of them ever being told what their share is.

Why creep up but cut hard

The two directions are not symmetric, because the costs are not symmetric. Being slightly too slow wastes a little capacity and nobody else notices. Being too fast fills a queue that everyone shares, and the damage lands on strangers as well as on you. So the cautious direction is cheap and the aggressive direction is expensive and the rule should reflect that.

There is also a result about fairness. Under add-a-bit and cut-by-a-proportion, two senders sharing a link converge on equal shares whatever they start from, because a proportion takes more away from the greedy one than from the modest one. Under add-and-subtract they do not converge at all, because a fixed subtraction takes the same amount from both and leaves the gap exactly where it was. The lab lets you set the increase and the size of the cut, and two senders that start apart end level under every cut it offers.

What slow start is, and why it is not slow

Adding one packet per round trip from a standing start is painfully slow if the right window is several hundred. So a new connection does not creep, it doubles: window 1, then 2, then 4, then 8, each per round trip, until something is lost or a threshold is reached. That gets it into the right neighbourhood in about ten round trips instead of hundreds.

The name is a leftover from what it replaced, which was starting at full speed. Compared with that, starting at one packet is slow. Compared with adding one at a time, doubling is anything but, and it is the reason a fresh download reaches full speed in a fraction of a second.

Does everybody really get a fair share

Roughly, among senders with similar round trips, and only because almost everyone runs approximately the same rule. A sender with a shorter round trip gets through its increase steps more often, so it grows faster and ends up with a larger share of the same link. That is not fair in any moral sense, it is just what the rule does.

And nothing enforces any of it. A sender that never backed off would beat every polite one on a shared link. The internet's congestion control runs on the fact that nearly all the software in the world chooses to be polite, which is a stranger foundation than most people expect.

Lab 20 · The sawtooth
Try this firstPress Run 120 rounds. The window climbs, hits the ceiling, loses a packet and halves, over and over. Now set the increase to 4 and run it again, then set the decrease to 0.9 and see what a gentle backoff does to the drop count.
Notice: the aggressive settings do get a little more through, and the dropped counter is what they pay for it with. Raising the increase to 4 spends far more of the 120 rounds above the capacity line, and a gentle cut of 0.9 spends nearly all of them there: the delivered figure creeps up while the dropped figure multiplies. The dropped figure is the expensive half of that trade, because every destroyed packet has to cross the link a second time. Step 9 is what that looks like when every sender is doing it at once. Two senders is the interesting case: switch to two and watch the sender that starts ahead lose its lead.
Lab 21 · Write the backoff rule
Try this firstPress Run to see what your rule does to a window of 20 on a good round and on a loss. Then press Check my answer, which puts your rule in charge of a sender for 300 rounds against a link whose capacity changes halfway through.
Notice: the checker demands two things at once, and they fight. Deliver enough packets, and keep the loss rate low. A rule that only grows delivers a lot and drops a lot. A rule that only shrinks drops nothing and delivers almost nothing. The window has to come down when it hurts and go back up when it stops hurting.
Two senders share a link. One halves its window on loss; the other subtracts 1. Both add 1 per good round trip. After many rounds of sharing, what happens?
At every shared loss, the halving sender surrenders half of whatever it holds while the subtracting one gives up a single packet. The gap widens at every loss and the increase, being equal, never closes it. This is exactly why the rule has to be a proportion rather than a fixed amount: cutting by a proportion takes most from whoever is using most, and that is what pulls the shares together.
Step 11

A file across a broken link

Everything at once, on a real file, over a link you can attack while it is running. Numbering, acknowledgements, timers, a sliding window, selective repeat, and a check on the whole thing at the end. Your job in this step is to break it and the point is that you cannot.

The final check is a checksum: a short number worked out from every byte of the file, computed by the sender before it starts and by the receiver after it finishes. If the two numbers match, the two files almost certainly match. Get the checksum wrong and the whole thing is theatre, so you are going to write it first.

What a checksum can and cannot promise

It cannot promise the files are identical. A 16 bit checksum has 65,536 possible values and a file has vastly more possible contents, so different files must sometimes share a value. What a decent checksum promises is that the kinds of damage that actually happen, a flipped bit, a missing chunk, two pieces swapped, all change the answer.

Choosing one is therefore about which damage it catches, not about how clever it looks. Adding up the bytes catches a changed byte and misses a swap entirely, because addition does not care about order. The lab below is built around exactly that failure.

The network already checks each packet, so why check again

Because each link checks only its own hop, and the damage does not only happen on the links. A packet is checked, arrives at a router, sits in that router's memory, and is then checked again on its way out with a freshly computed value. If a bit flipped while it sat in memory, the new check is computed over the damaged copy and agrees with it perfectly.

The only check that covers the whole journey is one computed by the original sender and verified by the final receiver. This is called the end-to-end argument, and it is the reason a program that cares about its data checks the data itself rather than trusting the pipe it came down.

What Fletcher's checksum is doing

It keeps two running totals instead of one. The first is the plain sum of the bytes. The second adds the first total in after every byte, so an early byte gets counted into the second total once for every byte that follows it. That makes the second total depend on position, and the pair of totals together catches reordering that the plain sum cannot see.

Both totals are kept within 0 to 255 by taking the remainder after dividing by 256, which is what % 256 does, so the whole answer fits in two bytes. It costs an addition per byte and catches nearly everything a real link does, which is why it turns up in places where a full cyclic check would be too expensive.

Lab 22 · Write the checksum
Try this firstPress Run. It shows what your checksum says about two short byte lists that hold the same bytes in a different order. Then press Check my answer, which tries it against damaged files of several kinds.
Notice: the starting version adds the bytes up, and that passes the flipped bit test and fails the swap test, because addition gives the same answer whatever order you add in. The fix is in the note above: carry a second total that has the first total added into it after every byte, so where a byte sits changes the answer.
Lab 23 · Send the file, and sabotage it
Try this firstPress Send the file and let it finish untouched. Read the two checksums at the bottom. Then press it again and, while the bar is moving, hammer Cut the link, Corrupt a packet and Duplicate everything as much as you like.
Notice: the transfer takes longer, sometimes far longer, and the resend counter climbs, and the file still arrives with a matching checksum every time. Then use the switches at the top to turn a mechanism off and try the same sabotage. With numbering off, duplicates get written into the file. With the checksum off, corruption sails through. Each switch shows you which specific attack that mechanism was the answer to.
You turn off the per-packet check but leave the file checksum on, then corrupt one packet during the transfer. What does the receiver do?
A checksum detects damage, it does not repair it. Without a per-packet check the damaged packet looks fine on arrival and gets written, so the error is only discovered right at the end, and by then the only remedy is to fetch the whole file again. That is the argument for having both: the packet check catches damage while it is still cheap to fix, and the file check is the one that is actually trustworthy, because it covers the whole journey.
Step 12

From a name to a page

You type a name and press return. What follows uses every idea from both courses at once: an address looked up from a name, packets routed hop by hop, a handshake, a window, timers, acknowledgements and a check on everything that arrives. Here it is end to end, on a clock, with the losses left in.

Count the round trips as you watch. Almost all of the time a small page takes is round trips, not bytes, and once you can count them you can see why the tricks that make the web fast are all about having fewer of them.

What is a name lookup

Packets are addressed to numbers, not to names, so before anything can be sent the name has to be turned into an address. Your machine asks a nearby name server, which either knows the answer or asks others on your behalf, and the answer comes back and is remembered for a while so the next request skips the whole thing.

The previous course covers this in detail. What matters here is the cost: it is at least one round trip, sometimes several, and it all happens before the first useful packet has even been addressed.

What a request and a response are

Once the connection is open, the two sides speak an agreement about documents. The caller sends a short piece of text naming what it wants. The answerer sends back a short piece of text describing what it is sending, and then the thing itself. That agreement is called HTTP, and from this course's point of view it is just the first data to travel over the reliable connection you have built.

The point worth taking away is the layering. HTTP never mentions loss, ordering or resending, because the layer underneath already dealt with all of it. Each layer is allowed to assume the one below kept its promises, and this course is the story of how one particular promise gets kept.

Why reusing a connection matters so much

A fresh connection costs the handshake, one and a half round trips, before a single byte of the thing you wanted can move. It also starts with a tiny window, so the first response trickles out and takes several more round trips to reach full speed even on an empty fast link.

Fetch forty small images on forty fresh connections and you pay all of that forty times. Fetch them on one open connection and you pay it once. That single change is worth more than most of the other speed work people do, and the budget lab below lets you put a number on it for your own round-trip time.

Lab 24 · The whole journey, on a clock
Try this firstPress Fetch the page and watch the log fill. Every line has the time it happened at. Then press Reset, set loss to 8% and fetch it again, and compare the finishing times.
Notice: the log names which step of this course each line came from. The gap before the first byte of the page is the name lookup plus the handshake, and it is the same whether the page is one kilobyte or a hundred. When you add loss, look at where the time went: a loss during slow start hurts far more than a loss later, because the window gets halved when it was small to begin with.
Lab 25 · The round-trip budget
Try this firstSet the round trip to 200 ms and the number of files to 30, with connection reuse off. Read the total. Now switch reuse on and read it again. The breakdown underneath shows where every millisecond went.
Notice: the sums are just multiplication, and that is the point. Nothing here is a guess or a benchmark, it is the count of round trips each choice forces multiplied by how long a round trip takes. Try a 30 ms round trip and a 400 ms one with the same settings: the second is not a bit slower, it is a different experience, and no amount of extra bandwidth fixes it.
Your connection is upgraded from 10 megabits per second to 1000, and the round trip stays at 150 ms. A page made of many small files loads barely faster than before. Why?
A 20 kilobyte file takes about 16 milliseconds at 10 megabits and about 0.16 at 1000, so the transmission time was never the problem. The lookup, the handshake, the slow start and one round trip per request are all measured in round trips, and those are set by distance and queueing rather than by link speed. This is why the useful work is nearly always reducing the number of round trips rather than buying a bigger pipe.
What you built

A channel that loses, reorders, duplicates and delays, and ten things stacked on top of it. A nod for every packet. A number, so a copy can be recognised. A timer, so silence is not fatal. A measured estimate, so the timer is set sensibly. A window, so the link is not left empty. Selective repair, so one loss costs one resend. A limit set by the receiver, so it is never drowned. A handshake, so both ends agree where the numbers start. A backoff, so a crowd of senders does not wreck the thing they share. And a check over the whole file, so nobody has to take the network's word for anything.

None of that is in the network. All of it lives at the two ends. The network stayed exactly as unreliable as it was in Step 1, and that is deliberate: keep the middle simple and stupid, and let the ends build whatever promises they actually need.

Where to go next

  • Distributed Consistency. Two machines can now exchange data perfectly. They still cannot agree on what happened, or in what order, once either of them might crash. That is a harder problem than moving the bytes was.
  • Computer Networks. If you came here without it, go back and see where the packets come from: voltages, framing, error checks, addresses and routing, built from one piece of copper.
  • Reading the Machine's Mind. The sender and receiver here are programs. That course is about what a program actually is once you get underneath it, and how to stop one mid-thought and find out what it did wrong.
Step 13 · Advanced path

Streams, segments and path size

TCP gives an application an ordered stream of bytes, not a sequence of messages. Two writes can arrive in one read, and one write can require several reads. The application must add its own boundary: a fixed size, a delimiter, or a length field. Correct code keeps unconsumed bytes for the next read.

The transport divides that stream into segments that fit the path. The path MTU is the largest IP packet that can cross every hop without fragmentation. Headers consume part of it, leaving a maximum segment size. If a path silently drops oversized packets, packetization-layer path MTU discovery probes sizes and requires positive evidence that a probe arrived.

Lab 26 · Reassemble records and probe the path
Try this firstChange the application write size and the read chunk. Then lower the path MTU. The record parser must still recover the same messages.
A packet boundary is not an application boundary. Test every split point, including a length field split across two reads and several records arriving together.
Why not let IP fragment everything?

A lost fragment prevents reassembly of the whole packet, and middleboxes do not all treat fragments consistently. Transports usually size packets to the path and adapt when that path changes.

See the current TCP specification and datagram packetization-layer path MTU discovery.

One application write contains 100 bytes. How many reads must return it?
Writes and reads are not records. Preserve partial data and parse the application framing.
Step 14

Detect loss without guessing

A cumulative acknowledgement says how far the uninterrupted prefix has reached. Selective acknowledgements also identify later blocks that arrived, so one gap does not force the sender to resend everything after it. The sender keeps a scoreboard of acknowledged ranges and transmission times.

Reordering looks like loss for a while. Modern time-based detection such as RACK asks whether sufficiently newer data has been acknowledged and whether enough time has passed. A tail loss has no later packet to expose the gap, so a probe timer sends something acknowledgement-eliciting. Expiry is a reason to probe, not proof that every outstanding packet was lost.

Lab 27 · Separate reordering from loss
Try this firstMove one packet later in time. Compare cumulative ACK, SACK and a time threshold. Then truly drop the last packet and fire the probe timer.
Record the evidence for each retransmission: missing range, later delivery evidence, elapsed time and current RTT estimate. “The timer fired” is not the same claim as “this packet was lost.”
What happens when sequence numbers wrap?

TCP compares 32-bit positions within a bounded receive window rather than treating them as unlimited ordinary integers. Initial sequence numbers, timestamps and TIME-WAIT help prevent an old segment from being accepted in a later connection.

See TCP SACK and RACK-TLP.

A probe timeout expires. What does that establish?
A timer supports an action, not a complete diagnosis.
Step 15

Recover from a zero window

A full receiver advertises a zero window. The sender stops. Later the receiver frees space and announces a larger window, but that update can be lost. If both ends now wait, the connection is healthy and permanently idle.

A persist timer prevents that deadlock. The sender occasionally sends a small window probe, and the receiver answers with its current window. Probes back off and remain bounded; ignoring flow control would merely refill the receiver or waste network capacity.

Lab 28 · Lose the window update
Try this firstFill the receiver, lose its first window update and compare waiting forever with a persist probe.
The recovery traffic is deliberately small. Verify that the sender resumes only after it receives a non-zero advertised window.
Flow control is not congestion control

The receive window protects one endpoint's memory. The congestion window protects the network path. A sender may transmit only what both limits allow.

Why not resume sending a full window when the persist timer fires?
The probe asks a question. It does not invent receiver capacity.
Step 16

Close without accepting old data

A reliable connection closes each direction separately. A FIN says, “I will send no more bytes,” and it is acknowledged like other sequence-space events. One side can finish sending while it continues receiving. A reset is different: it aborts state and may discard unread data, so applications must not describe every reset as a clean finish.

The endpoint that actively closes waits in TIME-WAIT. This lets it answer a repeated final FIN and keeps delayed segments from an old connection away from a new connection using the same addresses and ports. Removing that wait changes a safety assumption; it is not a free resource optimisation.

Lab 29 · Close both directions
Try this firstClose the client direction, keep the server response flowing, then lose the final ACK. Compare TIME-WAIT with immediate reuse.
Test clean close, half-close, simultaneous close, reset, repeated FIN and a delayed old segment. They are different states, not one “disconnected” event.
Why a connection has so many states

Each endpoint must remember which direction has finished and which control messages were acknowledged. The state machine turns ambiguous packet arrival into a defined action.

A peer sends FIN. Must this endpoint stop sending immediately?
TCP is full duplex. Each byte stream closes independently.
Step 17 · 2026 practice

Signal congestion before loss

A drop-tail queue signals congestion only after it is full. Active queue management can mark or drop earlier, before persistent delay grows to the entire buffer. With Explicit Congestion Notification, capable endpoints receive a congestion mark and reduce load without first losing that packet.

CUBIC grows its window as a cubic function of time since congestion and is designed for fast, long-distance paths. Other controllers use different models and signals. No name removes the need to test coexistence, pacing, RTT fairness, short-flow completion, queue delay and behaviour when ECN or an expected signal is unavailable.

Lab 30 · Mark before the queue fills
Try this firstRaise offered load with drop-tail, then enable active queue management and ECN. Compare delivery, drops and queue delay.
A mark is still a congestion signal. A sender that negotiates ECN and ignores marks gains an unfair advantage and can rebuild the queue.
Why pacing matters

A window controls how much may be in flight. Pacing controls when those packets leave. Spreading them across the round trip avoids a burst that can overflow a short queue even when the average rate fits.

See active queue management guidance and the CUBIC specification.

An ECN-capable sender receives a congestion-experienced mark. What should it do?
Delivery and congestion are separate facts.
Step 18 · 2026 practice

Recover QUIC frames, not packets

A QUIC packet number is never reused. When a packet is lost, its useful frames can be sent in a new packet with a new number. Acknowledgements report packet-number ranges; stream offsets say where application bytes belong. Keeping those two number spaces separate avoids ambiguity about which transmission arrived.

QUIC declares loss using packet and time thresholds, and uses a probe timeout based on smoothed RTT, RTT variation and the receiver's maximum acknowledgement delay. Separate packet-number spaces protect the Initial, Handshake and application phases. One stream's missing frame does not force unrelated streams to wait.

Lab 31 · Retransmit a frame under a new packet number
Try this firstLose packet 4, resend its stream frame in packet 7 and acknowledge ranges. Then lose the tail and fire one probe timeout.
Log packet number, frame type, stream offset, send time and the evidence that declared loss. A packet capture can then reconstruct the decision.
Why the probe timeout includes acknowledgement delay

A receiver may deliberately delay ordinary acknowledgements. The sender includes the negotiated maximum delay so it does not mistake that expected wait for path loss.

See QUIC loss detection and congestion control.

Packet 4 is lost and its stream frame is sent again in packet 7. Which packet number does the new packet use?
Frames carry the recoverable information. The new packet gets a new packet number.
Step 19

When a resend is too late

Reliable delivery is useful only while the data still matters. A video frame that arrives after its playback deadline can be worse than a missing frame because it consumed capacity needed by newer data. Interactive media, games and control loops may choose partial reliability, deadlines or cancellation instead of an unlimited retry.

Forward error correction sends extra coded information so some losses can be repaired without another round trip. It costs bandwidth even when nothing is lost, and a loss burst can exceed its repair budget. Engineering the choice means measuring deadline success, useful quality, repair overhead and recovery delay under representative loss bursts.

Lab 32 · Choose retry, repair data or skip
Try this firstCompare a short and long round trip with a 120 ms deadline. Add repair symbols, then change loss from isolated packets to a burst.
Report useful on-time units, not merely bytes delivered. A perfect frame delivered after its deadline does not satisfy a real-time contract.
Application completion is a separate promise

A transport acknowledgement proves bytes reached the peer's transport. It does not prove the application parsed, stored or acted on them. If that matters, define an application-level acknowledgement and make repeated requests safe.

A resend will arrive 80 ms after a frame's playback deadline. What should the design measure?
The deadline is part of correctness. Choose the recovery mechanism against the application's actual contract.

Continue the systems path