Memory Exploits

The C standard describes what a correct program does. When a program breaks a rule, such as reading past the end of an array, the standard says nothing at all about what happens next. It does not say "you get a crash" or "you get garbage": it withdraws its description entirely, and anything the compiler and the machine do from that point is allowed.

That matters more than it sounds. A compiler is permitted to assume the rule was never broken, so it may delete a check you wrote on the grounds that it could only have mattered in a case the standard says cannot happen. This is why "it worked when I tested it" is not evidence. The read below returns a specific, predictable number, and one recompile could change it.

They are not instructions. An assembler is the program that turns your typed lines into the numbers a processor runs. These lines are directions to it about how to lay the program out, and they have finished their work before anything runs. .data says "what follows is values to put in memory, not code". .quad 90, 72, 65, 88 puts those four numbers into memory, eight bytes each, one after the other. .text says "back to instructions now", and main: is a label marking where to begin.

A label written in front of a value names the address of its first byte. So scores on its own is that address, and mov rbx, scores puts the address into rbx. Square brackets ask for the value stored there instead, so [count] is the number sitting at the label count. That is how the lab below finds out how long the array is, rather than being told four. Anything after a semicolon is a comment, a note for whoever is reading, which the machine skips over entirely.

No, but there is only one general form, and every address in this course is a case of it. A base, plus an index times a scale, plus a fixed distance called the displacement. The base and the index are registers. The scale is 1, 2, 4 or 8, because those are the sizes values come in. The displacement is a plain number. It is allowed to be negative.

So [rbx+rdi*8] uses the base, the index and the scale, while [rsp+8] uses a base and a displacement to reach one particular slot. Step 3 uses all four at once, in [rbp-16+rcx*8]: start from the address in rbp, come down sixteen bytes, then go along by rcx items of eight. Written out longhand that is four operations, and the processor does all of it inside the instruction at no cost, which is why compiled code is full of it.

A C string is a run of characters followed by a byte with the value zero. Nothing records the length: to find the end you walk forward until you hit the zero. That is what the copy loop below does, one byte at a time, and it is what strcpy does in the standard library.

So a "seven character name" needs eight bytes of space, seven for the letters and one for the terminator. Off-by-one bugs where a program allocates exactly as many bytes as there are characters are common enough to have their own name: the off-by-one overflow, and one byte is sometimes plenty.

Almost no interesting program gets its input from the person sitting in front of it. A program answering web pages gets a name out of a form somebody filled in on the other side of the world. A game gets one out of a message from another player's machine. A phone gets one out of a file it has just downloaded. In every case the bytes were chosen by a stranger and arrived down a wire, and nothing about them tells the program whether they came from a friendly user or from somebody who counted the letters first.

That is all "an attacker" means on these pages. Not somebody breaking into the building: just whoever gets to decide what the input says. And it turns the question the right way round. The question is never "would a person really type that?" It is "what is the worst thing this program does for the worst input somebody could send it?", and a program is only correct if the answer is "nothing much".

rsp holds the address of the top of the stack, and the stack grows downwards, towards lower addresses. So a function makes room for its own local variables by moving rsp down. sub rsp, 16 takes sixteen off it and the sixteen bytes that opens up belong to this function until it finishes. That gap is the whole of what a local buffer is, and add rsp, 16 at the end hands the space back.

Which means the number in that sub is the size of the box, and it is the only place the size is written down anywhere. No byte of memory records it, so the copy loop cannot look it up and neither can anything else. Lab 9 and Lab 12 both let you change that number and watch how much has to be typed before a name reaches its neighbour.

mov rax, [rsp] needs no help: rax is eight bytes wide, so eight bytes are fetched. mov [rsp+8], 0 is not so clear, because an address does not say how wide it is. A zero would fit in one byte as happily as in eight. So the width is written out. qword means eight bytes, byte means one, and dword and word mean four and two. You write one only when neither side of the instruction is a named register to settle the question. That is why mov qword [rsp+8], 0 says it and mov rax, [rsp] does not.

A register can also be used in part. rax is all 64 bits of it, eax is the low 32, ax the low 16, and al the low 8. Those are not four separate registers but four windows onto the same one. That is what makes mov [rsp+rcx], al a one-byte write: it asks for the bottom eighth of rax and leaves everything above it out.

The copy loop fetches one character at a time, and a character is a single byte. A register is eight. mov rax, byte [rsi+rcx] would be asking to put a one-byte thing into an eight-byte thing and the assembler refuses, because it cannot guess what you meant to happen to the seven bytes left over.

movzx is move-with-zero-extend, and it answers that question: fetch the small thing, put it at the bottom of the register, and set every bit above it to zero. So after movzx rax, byte [rsi+rcx] the whole of rax is the single character that was fetched and the cmp rax, 0 on the next line is a fair test for the zero byte that ends a C string. Going the other way needs no such help: mov [rsp+rcx], al writes al, the name for the bottom byte of rax, so the two sides are already the same size.

On a real x86-64 machine, call pushes the address of the instruction that comes after it, a 64-bit number pointing into the program's code, and ret pops that number and jumps to it. There is no check of any kind: whatever eight bytes are on top of the stack when ret runs, that is where the processor goes.

The machine on this page pushes the position of the next instruction rather than its byte address, so the numbers you see in the return slot are small. The mechanism is identical and the numbers are easier to read. Everywhere a real payload would carry an eight-byte address, the payload here carries an instruction position. The one exception comes in Step 5: a payload aimed at the buffer rather than at existing code carries a real address, because the buffer is in memory and a position in the code list would not name it.

Almost every compiled function opens with the same two instructions. push rbp saves the caller's copy of rbp onto the stack, and mov rbp, rsp parks a copy of where the stack was at the moment this function began. Those opening instructions are the function's prologue. rbp then stays still for the whole call while rsp goes on moving, so [rbp-16] names the same slot from the first instruction to the last. A register used that way is a frame pointer, and the caller's copy of it, sitting on the stack where the prologue pushed it, is the saved frame pointer you will see labelled in the frame map below.

The closing instructions are the epilogue: put rsp and rbp back the way they were, then return. leave is one instruction that does both halves of that undoing, and ret follows it. What matters here is the saved frame pointer's position. It sits between the local buffers and the return address, so a copy running off the end of a buffer reaches it one chunk before it reaches the thing it is really after. That is why chunk 3 in the lab breaks something and chunk 4 breaks everything.

A safe copy needs two numbers, not one. It needs where the input ends, which it finds by walking to the zero byte, and it needs where the destination ends, which nothing in memory can tell it. That second number has to be carried along by the programmer, from wherever the buffer was created to wherever it is written into.

Most memory-safe languages are exactly this idea made compulsory. An array carries its length with it, every index is compared against that length, and the comparison costs a few instructions that the processor often predicts correctly. Bounds checks add a small per-access cost and prevent many out-of-bounds reads and writes. Measure the cost in the actual workload before removing a check.

When a compiler protects a function, it stores a random value on the stack between the local buffers and the saved return address, and adds a comparison just before the function returns. The value is chosen once when the program starts and never printed anywhere. A copy that walks from a buffer to the return address has to cross that value on the way, so the comparison fails and the program stops itself. The number itself has a name of its own: the cookie. That is what the switch marked "the attacker knows the cookie" in the first lab below turns on. A payload that happens to carry the right number, which is the one thing that makes a canary useless. It is also why the cookie is chosen freshly each time the program starts and is never printed anywhere.

The name comes from the caged bird once carried into coal mines, which stopped singing before the air became dangerous to the miners. The bird is not the defence. Noticing the bird is.

The operating system does not hand out memory a byte at a time. It hands it out in fixed slabs, usually 4096 bytes each, and one slab is called a page. Beside every page the hardware keeps a few bits saying what may be done with it: may it be read, may it be written, and may the processor fetch instructions out of it. Those three answers are set separately, which is the part that makes this defence possible.

Marking the stack's pages "not executable" turns off only the third one. The stack reads and writes exactly as before, so every honest program keeps working, but if a return address ever points into the stack the processor refuses to fetch anything there and the process dies on the spot. One bit per page, no cost while the program runs, and an entire style of attack gone: the style where the payload carries its own instructions and the return address jumps into them. What it does not remove is the other style, pointing the return address at code the program already contains, because that code lives in pages that are meant to be executed. That is why the lab below has two targets to choose between.

A payload that redirects a return has to carry a number: where to go. The attacker has to know that number before they send it, which means it has to be the same on your machine as it was on the machine they worked it out on. For decades it was. A program was loaded at the same address every time it started, so an address written down last week still worked today, and payloads could be published.

Address randomisation chooses a fresh random starting position each time the program is launched. The program itself does not care, because everything inside it is worked out relative to wherever it landed. The attacker's number is now wrong, and a wrong jump usually kills the process rather than doing what they wanted. Notice the shape of that: it is a lottery, not a wall. The more places the program could be, the smaller the chance any one guess lands, which is exactly what the second lab counts and an attacker who can retry cheaply against a service that restarts after every crash still gets there in the end.

The heap is one large region of memory and a bookkeeper. When you ask for 24 bytes, the bookkeeper finds a free stretch at least that big, marks it used, and hands you the address of its first byte. When you free that address, it marks the stretch free again and remembers it for the next request. The list of what is used and what is free is the allocator's own data, kept beside or inside the blocks it manages.

Two consequences follow, and both matter later. The allocator hands out addresses it has seen before, so freed memory gets reused quickly. And its bookkeeping lives in the same memory as your data, so a program that writes past the end of a block can corrupt the bookkeeper itself.

Nothing here would be different in C, and the bugs are C's bugs. What changes is only how the writing looks. It changes because you are about to write programs rather than read them. C has to be turned into machine instructions by a compiler before it can run, and there is no compiler on this page. This language is read and carried out one statement at a time by the page itself, which is how a Check button can call your function forty times and count what each call did to the heap.

The translation back is short. let total = 0; is C's int total = 0;. function add(a, b) { return a + b; } is C's int add(int a, int b) { return a + b; } with the types left off. while, return and // are the same in both, and malloc and free are the names C uses. If you could read the C in Bits, Pointers and Memory you can read every line in this part.

store and load work in eight-byte slots because eight bytes is the size of the numbers being kept in these blocks, and counting in slots keeps the arithmetic out of the way of the point. A block of 24 bytes therefore holds three slots, numbered 0, 1 and 2. store(p, 1, 99) writes 99 into the second eight bytes of the block at p.

Slot 3 of a 24-byte block is past the end, and nothing stops you asking for it. That is the same absence Step 1 was about, moved from the stack to the heap and the report printed under every lab in this part is what notices.

A detector such as LeakSanitizer or Valgrind cannot ask the program which of its numbers are pointers, because nothing in memory records that. It scans the stack, the registers and the global area, and treats any value that happens to equal the address of a live block as a pointer to it. A long integer that coincidentally holds a block's address will make a leaked block look reachable.

So detectors report in grades."Definitely lost" means nothing points at it."Still reachable" means something does, and the program simply never freed it before exiting, which may be fine. The detector here does the same scan over the variables the program left behind, and reports the definitely lost ones.

You return a hotel key card at reception and keep the room number written on your hand. For an hour the room is empty and the number is harmless. Then somebody else checks in, and now the number on your hand refers to their room and their belongings. You have not done anything to the room. You are simply holding a way to reach it that stopped being yours.

The fix has the same shape as the real-world one: cross the number off when you hand the key back. Setting a pointer to null after freeing costs one instruction, and turns a silent corruption into an immediate, obvious crash at the exact line that made the mistake.

A leak detector or an address sanitizer knows which blocks are in use and how big they are. Once the freed block has been handed to a new owner, a read through the old pointer is a read of a live block, at an offset inside it, through the same number the new owner is using. There is nothing left to notice. The detector is not being lax: the information it would need was thrown away by the allocator when it reused the address.

Real tools buy the difference back by refusing to reuse a freed block straight away. The block goes into a holding area, poisoned, and any access to it while it is there is reported with the line that freed it. A sanitizer build can therefore report a use-after-free at the access site. Its quarantine is finite, so very old freed blocks may eventually be reused; tests should still exercise the suspect lifetime promptly and repeatedly.

A 32-bit unsigned number can hold values from 0 up to 4294967295. Arithmetic that produces something larger keeps only the low 32 bits of the answer, exactly as a car odometer with five digits rolls from 99999 back to 00000. It is not an error and no flag is checked: the multiply completes and gives a wrong, small answer.

So 536870913 items of 8 bytes each needs 4294967304 bytes, and the multiply returns 8. The allocation for 8 bytes succeeds easily. Everything after that is a program confidently writing four gigabytes of data into eight bytes of space. Bits, Pointers & Memory covers where the missing bits go in detail.

Four questions, asked of every pointer. Where did it come from, and how big is the block it points at? Who else holds a copy of it? When is it freed, and is anything still holding it then? And for every index or size that reaches an allocation or a write: could that number have come from outside, and what is the largest value it could be?

Reading for bugs is worth exactly as much as running the tools, and no more. Turn the checkers on first, get a list, then read the code around each finding. A tool that reports on behaviour finds the bug you actually wrote, which is the one your eyes are least able to see.

inputOrders() hands back a list: several values in a row under one name. It is the array from Step 1 with one thing added: this one knows how long it is. orders[0] is the first value, orders[1] the second, exactly as scores[0] was in Step 1, and orders.length is how many there are. That last part is what C does not give you: a C array hands over the values and leaves you to remember the count yourself.

Which is why the count is written down twice in the program below. Once by the list, and once by hand as let cap = 4;. Two records of the same fact, kept in two places, and only one of them changes when the data does. A surprising share of real capacity bugs are exactly that shape, and it is worth learning to notice a hand-written number sitting next to a value that already knows the answer.

A bug is an implementation mistake. A memory-safety violation is an invalid access. A vulnerability is a bug that crosses a security boundary. An exploit is a reliable way to use that vulnerability to gain an effect the program did not intend. One invalid read may merely crash in one build and expose a secret in another, so classification starts the investigation rather than ending it.

Write down the object, its valid bounds and lifetime, the invalid operation, who controls the input, and the first tool that observed it. Those facts survive even if the exploitability judgement changes.

A return is a backward edge because it goes back to a caller. Shadow stacks protect that edge by comparing the ordinary return address with a protected copy. An indirect call or jump is a forward edge. CFI, Control Flow Guard or CET indirect branch tracking restrict where it can land. Direct calls already contain their destination in the instruction.

Intel's description of CET and its two mechanisms is available in the CET technical guide. The important engineering point is unchanged: these checks contain control-flow corruption but do not repair the out-of-bounds write that caused it.

Neither mechanism checks business rules, authentication or whether the program chose the correct object. MTE can miss a violation when tags happen to match, and deployment modes may report asynchronously. Capability systems need careful interfaces and revocation for reuse. Hardware protection belongs beside safe languages, tests and isolation.

See Arm's MTE user guide for the lock-and-key model and the CHERIoT concepts guide for bounds, permissions and unforgeable capability tags.

For every foreign function call, state whether a pointer may be null, how many elements it covers, who owns the allocation, how long it remains valid, whether the callee may write or retain it, which thread may use it, and how errors are reported. Turn a pointer-plus-length pair into a slice only after those facts have been checked.

The official Unsafe Rust chapter explains raw pointers and safe abstractions. The Rust FFI guide shows how a small unsafe wrapper can expose a safe high-level interface.

Edge coverage tells the fuzzer that an input reached a new transition. It does not prove the code on that edge handled every value correctly, and 100 percent line coverage does not enumerate all lengths, states or thread schedules. Dictionaries, structure-aware mutators and separately seeded protocol states can help it reach deep parsers.

Google's ClusterFuzzLite guide documents a practical continuous fuzzing and sanitizer workflow. Use the compiler and platform documentation for exact supported combinations.

A new bounds check can reject valid edge inputs, truncate data, change timing or move a failure to a caller that never handled errors. Test exact-fit, empty, one-over, maximum and malformed cases. For lifetime fixes, test cancellation and error paths. For concurrency fixes, stress the ordering rather than running one friendly schedule.

Record the affected versions, root cause, attack surface, fix commit, tests, residual risk and rollout signal. That record lets another team verify the backport without reverse engineering your reasoning.

Modern deployments combine prevention and detection rather than betting on one defence: safe languages for new code, small reviewed unsafe islands, process or component isolation, ASLR/NX/CFI and shadow stacks, tagged-memory testing or deployment where supported, continuous sanitizer builds, fuzzing and crash telemetry. Legacy C and C++ remain, so migration plans need measurable risk reduction before a full rewrite is possible.

Android's current memory-safety guidance describes this combination of Rust, Arm memory tagging and development tools. Continue with Software Construction for API and testing discipline and Systems That Fail Well for rollout and incident evidence.