Assembly and Debugging

C is a programming language, and one of the oldest still in wide use. A line of C such as total = total + price; is text a person writes. On its own it does nothing at all: it has to be compiled first, which turns it into instructions.

The examples here are in C because C sits closer to the instructions than most languages do, so the two sides line up neatly and you can see which piece became which. You do not need to know C to follow this course. Everything you need to read is explained where it turns up, and the instructions on the right are the real subject.

A compiler is a program whose job is to turn the code a person writes into the instructions a processor can run. You write total = total + price;, hand the file to the compiler, and it hands back a list of instructions like the ones on the right of the lab below. It runs once, before the program does. By the time the program is running the compiler is long gone, which is why the names you chose are gone too.

This matters more than it sounds, because you will spend the rest of this course reading a compiler's output rather than your own. When a program misbehaves, the thing running is never the text you wrote. It is what the compiler decided that text meant. Most of the surprises in Step 9 come from the gap between the two.

A register is a slot inside the processor itself that holds one number. There are sixteen of them for general use, and that is not a limitation anybody regrets: they are the fastest storage that exists, because they are physically part of the circuit doing the arithmetic.

Memory is enormous and comparatively far away. A register is instant. So the shape of all assembly is the same: bring a value in from memory, work on it in registers, put the result back. If you have done the microprocessor course, these are the same registers you built out of gates.

mov rax, 5 means "put 5 into rax". Destination first, like the equals sign in almost every programming language. Square brackets mean "the memory at this address", so mov rax, [rbx] means "go to the address in rbx, fetch what is there, put it in rax".

Debuggers on Linux print the opposite order and add sigils: the same instruction appears as mov $5, %rax. That is called AT&T syntax, which only means the way the instructions are spelled out and punctuated, and it is the same instruction written backwards. When you meet it, the rule is simply that the last operand is the destination.

mov rax, [rsp-8] is unambiguous: rax is 64 bits wide, so the machine knows to fetch eight bytes. But mov [rsp-8], 5 is not. Five would fit in one byte, or two, or four, or eight, and the address does not say how wide it is. So you say: qword means eight bytes, and dword, word and byte mean four, two and one.

The rule is simple. If one side of the instruction is a named register, its size settles the question and you write nothing. If both sides are a plain number and an address, you have to say which size you meant. The machine here will tell you when you have left it out rather than guessing, because a wrong guess would write four bytes where you wanted eight and leave half a value behind.

ZF, the zero flag: the answer was exactly zero. SF, the sign flag: the top bit of the answer was set, which means it looks negative. CF, the carry flag: the answer ran off the top, treating the numbers as unsigned. OF, the overflow flag: the answer ran off the top treating them as signed, which is a different event.

They are set as a side effect, not asked for. Six kinds of instruction update them: add, sub, cmp, and, or and shr. The last three have the same names as the bit operations from the earlier course, and they do the same thing here. Whichever of them ran most recently is the one the flags are describing and the instruction that reads the flags always comes afterwards. That gap is where a certain kind of bug lives: put any arithmetic between a comparison and the jump that depends on it and the jump is now answering a different question.

A loop is a jump backwards to a label you have already passed, guarded by a comparison. That is all. "While rcx is less than ten" becomes: label at the top, compare rcx with ten, jump past the bottom if the comparison fails, do the work, jump back to the label.

Which explains something odd about compiled code: the test for a loop often ends up at the bottom rather than the top, because that arrangement needs one jump per pass instead of two. The C looks like a loop with a test at the top, and the instructions look like nothing of the kind.

With 64 bits there are a fixed number of patterns, and someone has to decide what they mean. Read 0xFFFFFFFFFFFFFFFF as unsigned and it is 18,446,744,073,709,551,615, which is eighteen billion billion and a bit. That is the largest number sixty-four bits can hold, and there is nothing above it. Read the same bits as signed, where the top bit means negative, and it is minus one. Both readings are correct. Neither is stored anywhere.

So the processor offers two families of conditional jump. jl and jg, which you used to build the loop in Step 4, read the flags the signed way. jb and ja read them the unsigned way. Same comparison, same flags, opposite answers. The bits-and-memory course has the full story of how negative numbers are represented.

Step 4 gave you the naming scheme: j, then a short word for the condition. jl is jump if less, jg is jump if greater, je is jump if equal, jne is jump if not equal. English only has one pair of words for smaller and larger, but the processor needs two, so it borrowed a second pair: below and above for the unsigned reading, less and greater for the signed one. That is the entire difference between jb and jl.

Sticking an e on the end adds "or equal": jge is jump if greater or equal, jbe is jump if below or equal. An n means not: jne, jnz. That is the whole naming scheme, and it means you can read a jump you have never seen before. When you meet jle in Step 9, it is jump if less or equal, signed.

Because those are the sizes things come in: a byte, a short, an int, and a 64-bit value or a pointer. [rbx + rcx*8] means "the array starts at rbx, take item number rcx, and each item is eight bytes wide". It is exactly what array[i] compiles to.

The displacement on the end, as in [rbp - 8], is for reaching a particular slot at a known distance. Local variables live at fixed distances from rbp, so nearly every access to a local variable in unoptimised code looks like that.

The lines beginning with a dot are not instructions. They are directions to the assembler about how to lay the program out, and they are gone by the time anything runs. .data says "what follows is not code, it is values to put in memory before we start". .quad 3, 17, 8, 42, 5 puts those five numbers into memory, eight bytes each, one after the other. .asciz "Hello!", which turns up in Step 11, lays down the letters of a piece of text instead, followed by a zero byte to mark the end. The label in front of either one, nums:, names the address of the first byte, so you can write mov rbx, nums and get that address into a register.

.text says "back to instructions now", and main: is just a label marking where to begin. So the whole header means: five numbers here, a count after them, and the program starts below. Writing [count] reads the value stored at that label, which is how the loop below can find out how long the array is instead of being told five.

Historically, so that the stack and the program's data could start at opposite ends of memory and grow towards each other, using whatever was free in the middle. The consequence is that push subtracts from rsp and pop adds to it.

And here is the part that matters in a later course: your local variables sit above rsp and grow upwards as you fill them, while the address the function will return to sits above them. So writing past the end of a local array walks straight into the return address. That is a buffer overflow, and the direction of the stack is why it points at exactly the most useful thing to overwrite.

There is nothing in the hardware about arguments. There is an agreement, and everyone follows it so that code compiled by different people can call each other. The first six arguments go in rdi, rsi, rdx, rcx, r8, r9, in that order, and the return value comes back in rax.

The agreement also splits the registers in two. A function may use rax, rcx, rdx, rsi, rdi and r8 to r11 freely, wrecking whatever was in them, so a caller that wants to keep something in one of those has to save it first. The rest, rbx, rbp and r12 to r15, must come back exactly as they were found, so a function that wants to use one pushes it at the start and pops it at the end. Get that backwards and you have written a function that works perfectly and breaks its caller, which is the hardest kind of bug to find because the symptom is nowhere near the cause.

rsp moves. Every push and pop shifts it, so if a function used [rsp+8] to reach one of its own variables, that address would mean something different after the next push. So a function takes a second register, rbp, and parks it: mov rbp, rsp makes rbp a copy of where the stack was when the function started, and then rbp does not move for the rest of the call. Now [rbp-8] means the same slot from the first instruction to the last, which is why the compiler output in Step 1 was full of exactly that.

The push rbp on the line before is the function being polite. rbp belongs to whoever called it and is still holding their copy, so the function saves it, uses it, and pops it back before returning. Those three lines, push rbp, mov rbp rsp, and a matching pop, are the most repeated pattern in all of compiled code. Optimised builds often skip them and go back to counting from rsp, which is one of the reasons optimised code is harder to read.

An optimiser is allowed to do anything that does not change the answer. It keeps variables in registers instead of memory, so the loads and stores vanish. It reorders instructions that do not depend on each other. It unrolls loops, computes constants in advance, and deletes variables that turn out not to matter.

Which is why debugging optimised code is unpleasant: a variable you want to print may not exist anywhere, and stepping jumps between lines in an order the source does not have. This is exactly why real programs are built twice, once for debugging and once to ship.

It replaces the instruction at your breakpoint with one that hands control back, remembering what was there. When you continue, it puts the real instruction back, runs it, and restores the trap. The program has no idea.

That is why a breakpoint costs nothing while it is not being hit, and why a debugger can attach to a program that is already running. The one here works differently inside, because it is a simulator and can simply choose not to take the next step, but the commands and what they mean are the real ones.

A breakpoint is a trap on a doorway: you know where the thief will walk and you wait there. A watchpoint is an alarm on the jewels: you have no idea who is coming or from where, but you will know the moment they are touched.

Which is why a watchpoint is the right tool for corruption. When a value that should be 10 has become 1094795585, you do not know which instruction did it, and that is the one thing a watchpoint tells you directly.

Halve it. Find a place where the state is definitely still right and a place where it is definitely already wrong, then look in the middle. Two or three rounds of that will land you on the instruction even in a large program, and it needs no cleverness at all, only a way to inspect state.

The mistake to avoid is reading the code hoping to spot it. Reading finds bugs you would not have written; it does not find the bug you did write, because you already believed that code was correct once. Stop the machine and ask it.

The instruction stream is read from low address to high address. Opcode and prefix bytes stay in that order. A number occupying several bytes, such as the 1 in mov eax, 1, is stored little-endian: least significant byte first. That instruction is b8 01 00 00 00. Endianness rearranges the bytes of the number; it does not reverse the whole instruction.

Intel's complete decoding tables are large because old and new instruction families share the same byte space. You do not memorise them. Use objdump, llvm-objdump or a debugger, then check the bytes when instruction boundaries or patches matter.

Static linking copies the needed library code into the executable. Dynamic linking leaves references for a shared library, which reduces duplication and lets several programs share read-only pages. Version mismatches are the trade-off. Position-independent executables use relative addressing so the loader can place them at different addresses. ASLR uses that freedom to make addresses less predictable.

The full compiler pipeline is covered in Language Engineering. Page mappings, loaders and shared libraries continue in Operating Systems.

Structures, vectors, variable-argument functions and large return values need extra classification rules. A large returned structure is often written through a hidden pointer supplied by the caller. C++ exceptions and stack backtraces also depend on unwind tables that describe how to recover saved registers at each instruction. These are ABI rules, not new processor instructions.

When debugging unfamiliar code, read the platform ABI rather than guessing. The register list here is the common integer path, deliberately small enough to trace by hand.

A system call is an intentional request from a program. A hardware interrupt is an asynchronous event from a device or timer. A processor exception is caused by the current instruction, such as a page fault or division by zero. A signal is an operating-system notification delivered to a process. They can share low-level entry machinery, but they are different events and should be named separately while debugging.

Operating Systems follows the kernel side of this boundary, including scheduling, virtual memory, files and device I/O.

A breakpoint may stop one thread or all of them, depending on the debugger. Either choice changes timing, so a race can vanish. Inspect thread lists and lock ownership, but use ThreadSanitizer, deterministic record/replay or event tracing when the failure depends on an unlucky schedule. A watchpoint is still useful, but most processors provide only a few hardware watchpoint slots.

Continue with Concurrent Data Structures for memory ordering and reclamation, and Performance Engineering for profiles and hardware counters.

Use a debugger for control flow and state at a known moment; a sanitizer for invalid memory, undefined arithmetic or races; a fuzzer for inputs you did not think of; a profiler for where time is spent; record/replay for a rare execution you need to revisit; and a decompiler for a readable starting hypothesis when source is missing. Use more than one when the cost of being wrong is high.

Practise the source-free path in Defuse the Bomb, the security consequences in Memory Exploits, and failure reporting and recovery in Systems That Fail Well.