Fault Tolerance

Because the chain is up on a day only when both parts are up on that same day. Take 100 days. The first part is up on 99 of them. Now look only at those 99 days and ask about the second part: it is up on 99 out of every 100 days, so it is up on about 98 of the 99. That is 99 hundredths of 99 hundredths, which is 98.01 days in 100.

Adding would say something quite different and quite wrong: that two parts each up 99 in 100 give you 198 days in 100, which is more days than there are. What does add up is the failures. One part is down 1 day in 100, so two parts are down on about 2 days in 100 between them. That agrees with the first answer, because being up 98 days in 100 and being down 2 days in 100 are the same sentence.

Less than it sounds like. It is a rate over a long stretch of time, not a schedule. It does not say which day is the bad one, and it does not promise that the bad days are spread out politely. Two down days can land next to each other, and over a long enough run they will.

This is why Lab 2 reports the longest run of bad days back to back alongside the total. A part that is up 99 days in 100 will, sooner or later, be down for two days in a row, and any plan that quietly assumed otherwise finds out at the worst moment.

From rolling dice. For each of 4000 imaginary days, the lab rolls once for every part in the chain and asks whether that part is down today. If any part is down, that day is counted as bad. The red bars are those counts. The line running across them is the multiplying worked out in advance, which is why the bars land on it.

The dice are seeded, which means the sequence of rolls is fixed rather than fresh each time. If something surprises you, press the same button again and the same run comes back, so you can look at it properly instead of chasing it.

Because the pair is down only when both copies are down at once. Each is down 5 days in 100, which is 5 hundredths. Of the 5 days in 100 that the first copy is down, the second copy is also down on about 5 in every 100 of them, and 5 hundredths of 5 hundredths is 25 ten-thousandths. Add a third copy and you multiply by 5 hundredths again, which is about one day in eight thousand.

Notice that this is the same multiplication as Step 1, pointed at a different question. In a chain you multiply the chances of working, and more parts make the answer smaller, which is bad. With copies you multiply the chances of failing, and more copies make the answer smaller, which is good. The arithmetic does not care which you wanted.

A rack is a metal frame about the height of a door, holding a stack of machines. The machines in it share one power feed and usually one network switch. A room is a hall full of racks, sharing the building's power and its cooling, because a room full of machines gets hot enough to stop working within minutes if the cooling does.

That sharing is the point. Lose the power feed and every machine in that rack goes down in the same instant, however healthy each one was. Lose the cooling and eventually the room does. So two copies in one rack are two machines, one power feed and one switch, and the second copy protects you from precisely one of those three things.

Something they share failed. It does not have to be power. Two copies of a program reading one database share the database. Two machines on one switch share the switch. Two copies given the same broken settings file at nine in the morning share the settings file, and that one takes both down inside a second.

A failure like that is called a shared cause: one event, both copies. The test to apply to any design is not "how many copies do I have" but "name an event that takes them all". If you can name one easily, that event is your real availability, and adding a third copy does not move it at all.

One worker taking 60 milliseconds a request finishes one request every 60 thousandths of a second. There are 1000 thousandths in a second, so that is 1000 divided by 60, which is about 16 and a half requests a second. Eight workers doing that at the same time is eight times as many, which is about 133 a second.

Two things follow, and both matter for the rest of the course. Nothing in the design can beat that number except more workers or faster work: not a bigger queue, not a cleverer retry, not asking more politely. And the number is easy to work out for any service. So you can always tell whether you are looking at a broken system, or at a system being asked for more than it has.

Each column is one second of the run. The lab does not wait around for forty real seconds. It keeps a list of things that are going to happen, each stamped with the time it happens, always takes the earliest one next, and moves its clock straight to that moment. The time between two events costs nothing, so forty seconds of a busy server is a few hundred thousand small steps and finishes while your finger is still on the button.

Because the clock is inside the run, everything can be counted exactly: how many requests reached the server in second 11, how many were answered, how many were given up on. The bars and the lines are those counts.

Retrying is the right answer to a message that got lost, and to one machine out of many being briefly unwell. Those are common, and a single retry fixes them so cheaply that leaving it out would be strange.

It is the wrong answer to a server that is already at capacity, because then the retry is more work arriving at something that has no room for it. The difficulty is that the client cannot tell those two situations apart: no answer looks the same either way. The next four steps are all about making a retry cheap enough, rare enough and safe enough that switching it on is not a gamble.

A random number is one you cannot predict, drawn fresh each time, and the ones used here are evenly spread between 0 and 1. Any value is as likely as any other. In the code lab that value arrives as roll: 0.10 means a tenth of the way along, 0.55 means just past halfway. Every client gets its own, so no two clients agree on where inside the stretch to come back.

Evenly spread is what makes jitter work. If the numbers clustered near the start, the crowd would arrive early and together, which is the problem you were trying to remove. And as everywhere in this course, the dice are seeded, so the same button gives you the same run and you can look at a surprise twice.

Because the two do different jobs. Jitter fixes the shape of the crowd: it turns one tall wave into a flat spread. Doubling fixes the rate: it means that if the trouble lasts a long time, each client asks less and less often instead of hammering away at the same pace for ever.

With jitter alone and no growth, two hundred clients keep offering the same number of attempts a second until they run out of patience. That is fine for a two-second outage and useless for a two-minute one. With doubling alone you get the waves. The lab has a button for each, so you can watch which half does what.

A function is a piece of code with a name, which is handed some values and gives one back. This is the whole shape of it:

function nextDelay(attempt, roll) {
  return 1000;
}

The name is nextDelay. The two names in brackets are its arguments, which are the values handed in when somebody uses it. return is how it gives an answer back, and nothing after a return runs. So print(nextDelay(1, 0.10)) means work out nextDelay with those two values and show what came back. This version gives 1000 whatever it is asked, which is the flat one-second wait that failed in Lab 7.

let ceiling = 1000; makes a name for a value you intend to change later, and ceiling = ceiling * 2; changes it, taking whatever is in there, doubling it, and putting it back. Reading that line as a statement of fact does not work. It is an instruction carried out at a moment in time.

while (i < attempt) { ... } repeats everything inside the braces for as long as the test in the brackets holds. Something inside has to move towards making the test false, which is what i = i + 1; is for, otherwise it would go round for ever. This is how you double a number attempt times without writing the doubling out several times.

Because a request shares the machine with everything else on it. A worker had just started something else. The disk was busy elsewhere. The network took a longer path this time. None of that is a fault, and none of it can be removed, so the time an answer takes is never one number. It is a spread, with a lot of quick answers and a few slow ones.

Lab 9 measures a whole minute in which nothing failed, nothing timed out and there were spare workers throughout. Even there, the slowest answers take many times longer than the typical one. That long thin tail on the right of the chart is normal, and it is exactly what makes choosing a moment to give up hard.

Sort every answer in the run from quickest to slowest, then walk along the sorted list. The value one hundredth from the end is the one the panel calls the slowest 1 in 100: only one answer in a hundred took longer than that."Half were under" is the same trick stopped in the middle of the list.

Averages are avoided here on purpose. A run where nearly everything takes 60 milliseconds and one answer takes 30 seconds has a comfortable average and a customer who has given up and gone away. What you want to know is how bad the bad end is, and walking a sorted list is how you find out.

While a client waits for an answer it is holding things: some memory, a network connection, and a place in its own list of work in progress. Call one of those places a slot. A client with sixteen slots can have sixteen requests open at once and no more, which is not meanness, it is what its own machine has room for.

Now give that client a ten-second timeout and a service that has stopped answering. Sixteen requests go out, none come back, and every slot is held by somebody who is already doomed. A request for a part of the system that is working perfectly cannot even be sent. The panel in Lab 10 counts those as "never even sent". That number is how a slow service takes down a caller that had nothing wrong with it.

The client does, once, at the moment it decides what it wants. A key is a short label attached to that intention, like pay-1729, and every retry of that same intention carries the same label. A new intention, even for the same amount to the same person, gets a new label.

The receiver keeps the labels it has acted on. When a request arrives with a label it has seen, it does nothing and replies as though it had just succeeded, which is true, because it did succeed, a moment earlier. Notice what this does not require: no change to the network, no promise that replies arrive, no cleverness in the client beyond reusing its own label.

A list is several values kept in order, written [3, 9, 4]. You reach one of them by its position, counting from 0, so list[0] is 3 and list[2] is 4. That position is called an index. list.length is how many values it holds, and list.push(7) adds one to the end.

An account in Lab 12 is a thing with two named parts inside it. account.balance is the money left, and account.done is a list of the keys this account has already acted on. The dot means "the part of it called", so the whole job is to look through that list before touching that balance.

No, only for as long as a retry might still turn up. Clients give up after seconds or minutes, so real systems keep keys for hours or days and then forget the old ones, which keeps the list from growing without limit. The cost of forgetting too early is a duplicate charge, so the window is chosen to be comfortably longer than the longest retry anybody is allowed.

This is safe to skip and never needed later in the course. It is here because it is the first question a careful reader asks, and the answer is more ordinary than the question suggests.

Because of the switch in the box under the stairs. When too much current flows through a wire, the breaker cuts that wire off from the supply. It does not repair the fault, and it does not care what caused it. It notices a measurable sign of trouble and disconnects, so that the fault stays in one place instead of heating a wire inside a wall.

The software version copies the shape exactly: a measurable sign of trouble, an automatic disconnection, and a human problem that still has to be dealt with afterwards. Nobody thinks a tripped breaker means the wiring is fine now, and nobody should think an open breaker means the service is fine.

They come from the electrical breaker, where the words describe the circuit rather than the door. An open circuit has a gap in it, so nothing flows. A shut circuit is a complete loop, so current runs round it.

So an open breaker means calls are blocked, and a shut breaker means calls are flowing normally. A healthy system has all its breakers shut. It reads backwards the first three times and then stops bothering you. It is worth learning as it is, because every real library and dashboard uses these two words in this direction.

Because nothing else would ever tell you the service came back. A breaker that opens and stays open needs a person to close it, and that person is asleep. So after a waiting period, called the cooldown, the breaker allows exactly one request out as a test. That single test request is called a probe.

If the probe succeeds the breaker shuts and normal traffic resumes at once. If it fails, the cooldown starts again and everything else is still being refused instantly. One request every three seconds is a cost you cannot measure, and it is the difference between recovering by itself and recovering when somebody notices.

true and false are the only two values of their kind, and they are what questions answer. recent[i] being true means that call worked; false means it failed. A function that gives one of those back can be used directly to decide something, which is why Lab 14 asks for exactly true or false and not for the word "open".

A history is a list of those values, oldest first, so the newest call is at the end. Since positions count from 0, the last one is recent[recent.length - 1]. Getting the direction the wrong way round is the most common way to write a breaker that reacts to a failure from five minutes ago and ignores the one that just happened.

A fixed set of workers, and a rule about who may use them. A request that arrives while all of them are busy either waits in a queue or is turned away. Nothing about the machine changes when you split one pool into two: the same eight workers exist and the only new thing is a rule saying which requests each group of them will accept.

That rule is the whole mechanism. There is no clever scheduling, no priority, no measuring. Reports may use these five workers and searches may use those three, and neither can borrow from the other, ever, including when the other side is completely idle.

Because a queue is served in order. A search that would take 40 milliseconds, standing behind eight reports that will each take eight seconds, waits for a worker to come free like everybody else. Being quick does not move you up the line. This is called head-of-line blocking, after the one item at the front of a queue that holds up everything behind it.

It is worth being precise about what has gone wrong, because it is not what the numbers first suggest. Search has not slowed down, and search is not failing. Search is waiting, and the waiting is long enough that the clients give up and the answers become useless. Lab 15 shows the busy-workers line pinned at eight while the answered line falls to nothing.

Yes, on an ordinary day. Reserved workers sit idle while the other side of the wall has a queue, and the shared pool would have used them. You are paying for a wall in exactly the same way a ship pays for bulkheads: in space you cannot otherwise use.

What you buy is that the failure has a shape you chose. Without the wall the shape is chosen for you by whichever kind of work happens to go slow, and the answer is usually all of it. Lab 16 lets you pick the split and reports what each one costs, so the trade is a number rather than an opinion.

Work it out from the front of the queue. Eight workers each finishing a request every 60 milliseconds take one request off the front about every 7 or 8 milliseconds between them. So a request that joins a queue of 20 waits roughly 150 milliseconds before a worker even looks at it, and then takes its own 60 to be answered.

Now put the same arithmetic on a queue of 2000, which is a number real systems reach by leaving the default alone. That is around fifteen seconds of waiting. Nobody is still there. The queue is not holding work any more, it is holding a record of people who left.

It feels wrong and it measures better, which is worth sitting with for a moment. Deliberately refusing work when there is no room for it is called load shedding. The refused request was not going to be answered in either design; the only question is whether the person finds out now or in four seconds.

A refusal in one millisecond is information. The client can ask a different copy, show the answer it kept from the last time it asked, or tell somebody honestly that this is not available right now. A refusal after four seconds of waiting is the same refusal with four seconds of the reader's life attached, and it also cost you a worker along the way.

It sounds unfair and it wastes the least. Whoever has been in the queue longest is the one most likely to have given up already, so their place is the one worth least. Serving them means spending a worker on an answer nobody will read, while somebody who arrived a moment ago and is still waiting gets refused.

This is the odd policy that wins in Lab 18, and it is worth knowing that the fair-looking rule and the useful rule come apart here. First in, first out is the fair one. When the queue is full and everybody has a deadline, the front of the queue is where the hopeless requests are.

A cache is a copy of an answer you fetched earlier, kept nearby so you can use it again. Because it was fetched earlier it may be out of date, and for most pieces of a page that matters far less than it sounds."4 related stories, from an hour ago" is a real answer. A hole where the related stories should be is not.

The judgement is about which pieces can bear it. A stale comment count is fine. A stale bank balance is not, which is why the pieces that must be fresh are the ones you mark as required, and a service that cannot answer them fresh gets to stop the page.

Ask what the reader came for. On a news page that is the story, and the answer is one piece, possibly two. Everything else on the page is something you are offering and an offer that cannot be made this second can simply not be made.

The habit that goes wrong is not deciding at all. Nobody sets out to make a recommendations strip compulsory. It happens because the page is built by fetching five things and rendering when all five arrive, and that code has quietly declared all five required. Lab 19 lets you break each piece in turn and see which of the two designs you are actually running.

null is a value that stands for nothing here. In that lab, a piece whose service did not answer arrives with its value set to null, so parts[i].value === null is how the code asks "did this one answer?". Three equals signs is the ordinary way to test whether two values are the same.

It matters because null is not the same as empty. Put it on the page and the page prints the word null where the reader expected a comment count, which is worse than leaving the line out. That is exactly what the starting code in that lab does. A missing piece has to be recognised, not passed along.

The list at the top of the lab says when each one starts. From second 20 to 28 there is a rush of about five times the traffic, which is Step 3 and Step 9: the queue and how much you retry. From second 40 to 58 the reviews store goes thirty times slower without failing, which is Step 5 and Step 8: the timeout, and the wall between the pools.

From second 66 to 84 the reviews store answers nothing but errors, slowly, which is Step 7 and Step 10: the breaker, and having something cached to serve instead. Then it recovers, and the last six seconds are there to show whether your design notices that it has.

Yes, and that is most of the exercise. Extra attempts with no random wait are the stampede from Step 3 aimed at the rush in second 20. A long timeout during the slow stretch fills the caller's slots with requests that are going nowhere, which is Step 5 from the wrong end. A deep queue during the rush turns a refusal into a long wait and then a refusal.

So switching everything on is not the answer and never has been. Each control has a cost, and the run measures both sides of it, which is why the verdict line reports two numbers rather than a score.

Answered means a real, fresh answer from the service that owns the work. Served counts those plus the requests that got a cached reviews answer instead, once you switch that on. They are separated because they are worth different amounts to a reader, and rolling them together would let a design that serves nothing but stale copies look healthy.

Search has no cached fallback here, so its number is the strict one, and the bar asks more of it: 95 out of 100. The everything-served bar of 90 is where the cache is allowed to help. A design that clears both has decided in advance which of its promises it is willing to soften.

A fault is the cause, such as a flipped bit or bad deployment. It creates an error, an incorrect internal state. A failure is when the service delivered at its boundary no longer meets its promise. Redundancy can keep an internal error from becoming a visible failure. Monitoring only visible failures misses repaired faults; alarming on every internal fault wakes people for events the system already contained.

A burn rate of one consumes the budget exactly across the whole window. A rate of ten would use it ten times as fast. Short fast-burn alerts catch severe outages; longer slow-burn alerts catch persistent damage without paging on a brief measurement spike. Alert on user impact and keep diagnostic component alerts available for investigation.

A database snapshot, object store and message log taken at unrelated instants may each be internally valid but disagree with one another. Define a recovery point across dependencies or provide reconciliation. Restore identity, encryption keys, schemas and configuration as well as data, using a path that does not depend on the failed control plane.

Membership changes, leader election, log ordering, durable votes and recovery after restart need a protocol such as Raft or another design with a stated fault model. The Raft Consensus course builds that protocol. The When Machines Disagree course covers clocks, ordering and consistency choices. This step provides the arithmetic and the stale-writer problem those courses rely on.

In an asynchronous network, a silent component may be failed or only delayed. A practical detector therefore makes suspicions from time bounds and can make mistakes. The system must define what a false suspicion costs: briefly stop traffic, restart a process, elect a new leader or revoke a lease. Thresholds and independent observations reduce noise, but they do not turn silence into proof.

If the stop button, metrics or recovery command shares the network or control plane being disrupted, the safety mechanism may disappear with the target. Keep an independent abort path and a firm deadline. Get explicit permission for the named environment and blast radius.

Reconstruct what people and systems knew at the time. Separate trigger, contributing conditions, detection, mitigation and recovery. Assign actions that change a control, test, tool or design, with an owner and verification. “Be more careful” is not an engineering control. Share the learning without turning the review into a search for one person to blame.

An assistant can group alerts, search runbooks, draft a timeline and suggest tests. Keep source links and confidence with each claim. Do not let generated commands bypass change control or expand the authorised blast radius. A human incident lead owns destructive actions, external communication and the recovery decision. Test AI-generated runbook edits in the same rehearsal as hand-written ones.