The idea: the way back is stored next to your data
When a function is called, the CPU and the compiler keep two things on the stack: the return address (where to continue after the call) and the caller's saved frame pointer. The function's local variables, such as a char buf[8], go in the same frame, right below them. Nothing in the hardware separates "data" from "the way back". If a copy writes past the end of buf, it overwrites the saved values, and ret later jumps wherever the overwritten bytes say. That is a stack buffer overflow.
The page runs one small C program on a toy x86-64 Linux process. The addresses are made up and the input is placeholder letters. On the left is the program (► marks the line that runs). In the middle is greet's stack frame, one row per 8-byte slot, with the lowest address of each slot on the left. On the right are the registers, the defences turned on for this run, the memory map and the outcome. Each Run starts a new process with the input, copy function and defences picked in the bar.
int main(void) void greet(void)
{ {
long n = 3; char buf[8];
greet(); gets(buf); /* or fgets(buf, sizeof buf, stdin) */
printf("%ld\n", n); puts(buf);
} }
The stack frame, slot by slot
The stack grows toward lower addresses. Calling greet builds its frame from the top down:
call greetpushes the return address,0x0040102A: the instruction inmainright after the call.- The prologue pushes
main's frame pointer,rbp = 0x7FFE0030, and setsrbpto the new frame.mainfinds its own localnthroughrbp(atrbp − 8). - The prologue moves
rspdown 8 bytes forbuf.buf[0]is at0x7FFE0010,buf[7]at0x7FFE0017.
At the end, leave pops the saved value back into rbp, and ret pops the return address into rip, the instruction pointer. Both trust whatever bytes are in those slots now. x86-64 is little-endian: the lowest byte of a value is at the lowest address, so 0x0040102A is stored as 2A 10 40 00 00 00 00 00. The same layout on MIPS, with $ra saved by the callee, is on MIPS Procedure Calls; frames and the heap on real x86-64 Linux are on Stack vs Heap in C.
What an unchecked copy overwrites
gets(buf) reads a whole input line and writes it, plus a terminating \0, starting at buf[0]. It is never told how big buf is. With the 24-letter input AAAAAAAA BBBBBBBB CCCCCCCC (Demo 2: overflow reaches the return address):
| Bytes | Land on | Effect |
|---|---|---|
| 1–8 (A) | buf | fine: this is the buffer |
| 9–16 (B) | saved rbp | becomes 0x4242424242424242 |
| 17–24 (C) | return address | becomes 0x4343434343434343 |
25 (\0) | main's n | n becomes 0 |
puts then prints 24 characters from an 8-byte buffer, and ret loads rip = 0x4343434343434343. That address is in neither the code nor the stack, so the CPU faults and the kernel kills the process with SIGSEGV. A crash is the lucky case. The bytes came from the input, so the input decided where the CPU jumped. In Demo 6: return into the buffer, no defences, the last 8 bytes of the input hold the toy address 0x7FFE0010, which is buf itself. On a stack the CPU may execute (rwx), it would then run the input's own bytes as instructions, and whoever wrote the input controls the program. That is why this bug is a security bug and not only a crash.
One byte too many
An input of exactly 8 letters looks safe, but gets adds the \0 as byte 9 (Demo 3: one byte too many). It lands on the lowest byte of the saved rbp, so 0x7FFE0030 becomes 0x7FFE0000. greet returns normally, because the return address is untouched. But main now reads n at rbp − 8 = 0x7FFDFFF8, an unrelated stack word, and prints garbage. These off-by-one errors are hard to find because the program fails later, in a different function.
The fix: a bounds-checked copy
The real fix is to never write past the buffer. fgets(buf, sizeof buf, stdin) is told the size: it copies at most sizeof buf − 1 = 7 bytes and adds the \0 (Demo 4: the fix, a bounded copy). With the same 24-letter input, buf holds "AAAAAAA", the other slots keep their values, and main prints 3. The rest of the line stays unread, and the program can treat it as an error. The same rule applies to every copy: snprintf instead of sprintf, memcpy only after checking the length, and never gets (it was removed from the C standard in C11). Languages that check every array index, such as Java, C#, Go and Rust, turn the same mistake into an exception or a panic instead of a write past the end.
The next three defences are different. They do not remove the bug; they make it much harder to turn the bug into control of the program. Real systems turn all of them on at once.
Stack canary
With gcc -fstack-protector-strong (the default in most Linux distributions), the compiler puts a secret value, the canary, between the local arrays and the saved values. The prologue copies it from a reference kept in thread-local storage; the epilogue compares it before leave and ret. A copy that runs past buf must change the canary before it reaches the return address. On a mismatch the program prints *** stack smashing detected *** and aborts, so ret never uses the overwritten address (Demo 5: a stack canary catches it). The canary's lowest byte is 00, so a string function that reads it stops there and cannot print it.
Limits: the check only runs when the function returns, so damage used earlier (for example an overwritten function pointer among the locals) is not caught. A write that jumps over the canary, such as buf[i] = x with a bad index, does not touch it. If the program leaks the canary's value, an input can write it back unchanged. And the result is still a crash.
Non-executable stack (NX)
Each page of memory has permission bits in its page-table entry: readable, writable, executable. With NX (no-execute, also called DEP or W^X), the stack and heap are rw-: data, never code. In Demo 7: the same with NX the return address still points at buf, but the CPU refuses to fetch an instruction from a non-executable page; the fault kills the process. Page permissions are covered on Virtual Memory.
Limit: NX only stops running bytes from the stack. An overwritten return address can still point at code that is executable: a library function, or short pieces of existing code chained together. These code-reuse attacks (return-to-libc, return-oriented programming) are why NX is combined with ASLR, canaries and, on newer CPUs, hardware checks of return addresses.
ASLR
To redirect a return, an input must contain a correct address. Address space layout randomization puts the stack, the heap, the libraries and (for position-independent executables) the program's code at a different random place in every run. In Demo 8: the same with ASLR this run's buf is at 0x7F5C3010, and the code at 0x5A213000. The input's address 0x7FFE0010 was right in a run without ASLR, but here it points to unmapped memory, and the process crashes. Press Run again with ASLR on: each run gets new addresses. How the loader places segments is on Linux Program Loading.
Limits: a bug that leaks one address (a pointer printed by mistake) tells where everything in that region is. With few random bits, an attacker can retry until a guess hits, especially against a server that restarts crashed workers with the same layout. And again the program still crashes, which is a denial of service.
Defence in depth
| Defence | Stops | Does not stop |
|---|---|---|
| bounded copy | the overflow itself | other copies that are still unchecked |
| stack canary | using an overwritten return address | non-contiguous writes, a leaked canary, damage used before return |
| NX | running input bytes from the stack | jumps into existing code |
| ASLR | using a fixed, known address | a leaked address, many retries |
Try the combinations: with all three run-time defences on, every overflow here ends as a crash or an abort; with fgets, nothing outside buf changes in the first place.
What the page leaves out
- Addresses are shortened to 8 hex digits and made up; the ASLR "random" values come from a fixed table so that Undo can replay a run. Real ASLR and canaries use fresh random values in each process.
- No machine code: the page only shows where
riplands. How real attacks build their input, and the bypasses named above, are left out on purpose. - Real frames hold more: alignment padding, other locals (the compiler puts arrays next to the canary so other locals are not overwritten), and callee-saved registers. Optimised code often has no frame pointer.
- Newer hardware defences (shadow stacks such as Intel CET, pointer authentication on ARM), compiler bounds checks (
_FORTIFY_SOURCE), sanitizers (AddressSanitizer) and heap overflows are not shown.
On the mind map
References
Bryant and O'Hallaron, Computer Systems: A Programmer's Perspective, 3rd ed., Section 3.10.3–3.10.4: Out-of-Bounds Memory References and Buffer Overflow, Thwarting Buffer Overflow Attacks
MITRE CWE-121: Stack-based Buffer Overflow
GCC manual: Program Instrumentation Options (-fstack-protector, -fstack-protector-strong)