Assembly and Low-Level Language Fundamentals Questions

Direct answer

Treat it as two jobs: first find where the snippet sits in the binary (an address), then find what function that address belongs to. With symbols or debug info, the second step is a lookup. On a stripped binary there is no name to look up, so you identify the function from evidence that survives stripping (imported calls, string literals, constants, instruction idioms, function boundaries) and then confirm the guess by rebuilding a candidate and comparing bytes. The tools split cleanly: readelf reads the ELF (Executable and Linkable Format, the Linux binary format) headers and tables directly and answers "what is in this file and where"; objdump decodes instructions and answers "what do the bytes at this address do"; nm and addr2line answer "what is the name at this address".

The worked case

A small target built as gcc -O1 -g -fno-pie -no-pie -Wl,--build-id -o id_target id_target.c (GCC 14.4, x86-64, built, stripped and inspected with binutils in an emulated linux/amd64 container; the program itself is not run):

#include <stdio.h>
#include <string.h>

static unsigned checksum(const char *s) {
    unsigned h = 5381;
    while (*s) h = h * 33 + (unsigned char)*s++;
    return h;
}

int check_license(const char *key) {
    if (strlen(key) != 8) {
        puts("bad length");
        return 0;
    }
    return checksum(key) == 0x7c9a5b3du;
}

int main(int argc, char **argv) {
    return argc > 1 ? !check_license(argv[1]) : 2;
}

The snippet you were handed is the whole routine, from 0x401160 to its ret, and looks like this (from objdump -d --no-show-raw-insn -M intel on a copy with strip applied; the <...> names objdump invents after stripping are misleading, so ignore them):

  401160:	push   rbx
  401161:	mov    rbx,rdi
  401164:	call   401040 <strlen@plt>
  401169:	cmp    rax,0x8
  40116d:	jne    4011a4 <strlen@plt+0x164>
  40116f:	movzx  edx,BYTE PTR [rbx]
  401172:	mov    eax,0x1505
  401177:	test   dl,dl
  401179:	je     401197 <strlen@plt+0x157>
  40117b:	nop    DWORD PTR [rax+rax*1+0x0]
  401180:	mov    ecx,eax
  401182:	shl    ecx,0x5
  401185:	add    eax,ecx
  401187:	add    rbx,0x1
  40118b:	movzx  edx,dl
  40118e:	add    eax,edx
  401190:	movzx  edx,BYTE PTR [rbx]
  401193:	test   dl,dl
  401195:	jne    401180 <strlen@plt+0x140>
  401197:	cmp    eax,0x7c9a5b3d
  40119c:	sete   al
  40119f:	movzx  eax,al
  4011a2:	jmp    4011b3 <strlen@plt+0x173>
  4011a4:	mov    edi,0x402004
  4011a9:	call   401030 <puts@plt>
  4011ae:	mov    eax,0x0
  4011b3:	pop    rbx
  4011b4:	ret

Reading it line by line (Intel syntax, destination first; rdi holds the first argument under the System V x86-64 convention): push rbx; mov rbx,rdi saves a register and keeps the argument (a string pointer). call strlen@plt calls the C library's strlen through the PLT (procedure linkage table, the stub through which a dynamically linked binary calls library functions), and cmp rax,0x8; jne 4011a4 jumps away unless the length is 8. The block at 0x4011a4 loads the address of a string (mov edi,0x402004) and calls puts@plt, then returns 0. On the length-8 path, movzx edx,BYTE PTR [rbx] loads the first character, mov eax,0x1505 starts an accumulator, and test dl,dl; je skips the loop if the string is empty. The loop at 0x401180 copies the accumulator, shifts the copy left by 5, adds it back (eax + eax*32), advances the pointer, adds the current character, loads the next one and repeats until the character is zero. Finally cmp eax,0x7c9a5b3d; sete al; movzx eax,al turns the comparison into a 0 or 1 return value.

Step 1: locate the snippet

If you have only bytes or mnemonics, search for them: objdump -d bin | grep -n '<unique-constant>', or search the raw opcode bytes with grep -obUaP on the file (-o print only the match, -b with its byte offset, -U treat the file as binary, -a do not skip binary data, -P Perl regular expressions so \xb8 can match a byte), then convert the file offset to an address with readelf -SW (each section's file offset and virtual address are listed side by side). Worked example: mov eax,0x1505 is the bytes b8 05 15 00 00, and grep -obUaP '\xb8\x05\x15\x00\x00' id_stripped prints 4466: (decimal offset). 4466 is 0x1172. readelf -SW lists .text at file offset 0x1060 and address 0x401060, so the address is 0x401172 - 0x1060 + 0x401060 = 0x401172, which is the mov eax,0x1505 in the snippet. readelf -SW also tells you which section the address is in (.text here, 0x401060 to 0x4011da).

Step 2 with symbols or debug info

• nm id_target lists check_license at 0000000000401160.
• addr2line -f -e id_target 0x401160 prints check_license and /w/id_target.c:10.
• objdump -d -S id_target interleaves the source lines with the instructions, which also shows the loop at 0x401180: h * 33 + c compiled to shl ecx,5 plus add eax,ecx (that is h*32 + h) and add eax,edx.

Step 2 when the binary is stripped

strip is the tool that deletes the symbol table and debug info from a binary, so a stripped binary no longer has them. After strip, nm id_stripped prints nm: id_stripped: no symbols and addr2line -f -e id_stripped 0x401160 prints ?? and ??:0. Debug info and the static symbol table are gone. What remains, and what each question it answers is:

1. Imported function names (readelf --dyn-syms -W, or the <strlen@plt> and <puts@plt> labels in objdump -d). A dynamically linked binary must keep its undefined imports (puts, strlen, __libc_start_main), so the snippet's call 401040 <strlen@plt> tells you the function calls strlen and compares the result with 8.
2. String literals (strings -a -t x, then locate references). bad length is at file offset 0x2004 in .rodata (the read-only data section, which holds string literals), which is address 0x402004, and the snippet's mov edi,0x402004 before call puts@plt at 0x4011a4 is the code that prints it.
3. Magic constants and idioms. 0x1505 in decimal is 14096 + 5256 + 0*16 + 5 = 5381, and the shl 5; add pattern is h*32 + h, which is h*33; adding the next character after each step gives h = h*33 + c. A hash that starts at 5381 and multiplies by 33 is the classic djb2 hash. cmp eax,0x7c9a5b3d compares the final hash with a constant (0x7c9a5b3d is 2090490685 in decimal). A constant plus an idiom narrows the C a great deal.
4. Function boundaries. readelf --debug-dump=frames id_stripped still lists each function's start and end from the unwind table (the compiler-emitted .eh_frame data that exceptions and debuggers use to walk the stack; each FDE, frame description entry, covers one function; see the FDE lines: pc=0000000000401160..00000000004011b5 is exactly check_license), because .eh_frame is a loaded section that strip keeps. Entry point _start calls __libc_start_main with rdi set to main's address (here mov rdi,0x4011b5), so you can also anchor on main and walk down the call graph.
5. Build ID (readelf -n). The build ID is a hash the linker stores in the binary to identify that exact build. The Build ID: line in the stripped file (a 40-hex-digit value that depends on the build path and flags, so a rebuild on another machine prints a different one) matches the one in the separate debug file made with objcopy --only-keep-debug. With that debug file, addr2line -f -i -e id_target.debug 0x401160 0x401182 resolves check_license at line 10, and the loop address resolves to the inlined checksum at line 6, which shows the compiler inlined it. Distributions publish debug files keyed by build ID (debuginfod is the network service that serves them), so for system binaries you fetch them instead of guessing.
6. A decompiler or interactive disassembler (a decompiler turns machine code back into C-like pseudo-code; Ghidra, radare2, IDA) automates the above: it finds function boundaries, names imports, and offers pseudo-C. Use it to propose candidate C, and treat that as a hypothesis.

Confirming the hypothesis

Rewrite the guessed C (as in the listing above), compile it with the same compiler version and flags (-O1, non-PIE here), and compare. In the example, objcopy -O binary --only-section=.text on the stripped binary and on the rebuilt one, then cmp, reported them identical. If the bytes differ, diff the objdump output: a different register choice or a missing instruction usually tells you the optimisation level, compiler version, or an inlining decision is off. Exact byte equality needs the original toolchain; a close instruction match is still strong evidence.

For a live process, ltrace and strace (library and system calls) and gdb breakpoints on the imported functions work too, but static evidence is what you use when the code cannot be run.

Direct answer

Four tools run in sequence: the preprocessor (expands #include and #define), the compiler proper (turns C into assembly text), the assembler (turns assembly into an object file, .o, which is an ELF file with unresolved references), and the linker (merges object files and libraries into the final ELF executable). Each question in the prompt is decided at a different stage:

Everything below was run with GCC 14.4 on AArch64 (native, gcc:14 container) against this file; the x86-64 lines come from building the same file with GCC 14.4 in an amd64 gcc:14 container. The defaults shown are this toolchain's; another distribution's compiler can enable PIE, canaries or full RELRO by default, so check the output rather than assuming.

#include <stdio.h>
#include <string.h>

#define LIMIT 8

static int helper_count;

__attribute__((visibility("default"))) int api_greet(const char *name)
{
    char buf[LIMIT];
    strcpy(buf, name);
    helper_count++;
    printf("hello %s (%d)\n", buf, helper_count);
    return helper_count;
}

int internal_total(void)
{
    return helper_count * 2;
}

int main(int argc, char **argv)
{
    return api_greet(argc > 1 ? argv[1] : "world") + internal_total();
}

Stage 1: preprocess

gcc -E greet.c expands macros and pastes in headers (about 1170 lines for this 25-line file: gcc -E greet.c | wc -l printed 1170 here). char buf[LIMIT]; becomes char buf[8];. Nothing about symbols, hardening or layout is decided here, except that -D flags change which code exists.

Stage 2: compile

The compiler picks instructions and also decides stack protection. A stack canary is a secret value placed between local buffers and the saved return address and checked before the function returns; if a buffer overflow overwrote it, the function calls __stack_chk_fail instead of returning. With default flags gcc -O2 -S greet.c emits no canary (grep -c stack_chk finds 0 matches). Adding -fstack-protector-strong makes api_greet (it has a local array) get one. Lines of greet.s (from gcc -O2 -fstack-protector-strong -S greet.c -o greet.s) that contain stack_chk, as grep -n stack_chk greet.s prints them (the number is the line in greet.s; the ... stands for two more matching lines, 40 and 43, the second adrp and the ldr of the guard):

18:	adrp	x2, __stack_chk_guard
19:	add	x2, x2, :lo12:__stack_chk_guard
...
58:	bl	__stack_chk_fail

Syntax key (AArch64, destination first): adrp x2, sym puts the address of the 4 KiB page containing sym in x2, add x2, x2, :lo12:sym adds the low 12 bits of the address (the offset inside that page), and bl is a call. The prologue builds the guard's address in x2 and copies the guard value into the frame; just before returning, the function loads the guard again (a second adrp and an ldr that the excerpt omits), compares it with the saved copy and branches to the bl __stack_chk_fail on a mismatch. On x86-64 the same protection looks different: GCC 14.4 reads the guard from thread-local storage, as mov rsi, QWORD PTR fs:40 in the prologue and sub rdx, QWORD PTR fs:40 in the epilogue (Intel syntax, as printed by gcc -masm=intel -S; GCC's default AT&T syntax prints movq %fs:40, %rsi and subq %fs:40, %rdx), followed by call __stack_chk_fail on a mismatch (call __stack_chk_fail@PLT when the file is built with -fPIE); there is no __stack_chk_guard symbol to relocate.

The decision is entirely a compile-time one, made per function, and only functions with vulnerable-looking locals get it under -strong.

The compiler also fixes position independence. Position-independent code (PIC) avoids baked-in absolute addresses so the loader can place the program anywhere. A relocation is a note the assembler leaves for the linker saying "patch this instruction with the final address of that symbol", and readelf -r greet.o lists them. The same call site differs in them (each excerpt line below is the first pair of readelf -rW relocation lines for __stack_chk_guard, cut down to its Type and Symbol Name columns: the Offset and Info columns and the trailing + 0 addend are left out, and the ... stands for the Symbol's Value column). With -fno-pic, the guard is reached through page-relative relocations (PREL in the name means the offset is computed from the instruction's own address, which assumes the symbol sits at a fixed distance from the code):

R_AARCH64_ADR_PREL_PG_HI21 ... __stack_chk_guard
R_AARCH64_ADD_ABS_LO12_NC  ... __stack_chk_guard

With -fPIE (position-independent executable code) it goes through the GOT (global offset table, a table of addresses filled in at load time): the instructions become adrp to the page of the symbol's GOT slot and an ldr that loads the symbol's address out of that slot, so the symbol may live anywhere, for example in a shared library:

R_AARCH64_ADR_GOT_PAGE     ... __stack_chk_guard
R_AARCH64_LD64_GOT_LO12_NC ... __stack_chk_guard

The change between the two builds is therefore that a direct address calculation became a load through a table the loader fills in. For a smaller reproduction, int get(void) { return counter; } with extern int counter; assembles to adrp plus ldr w0, [x0] with R_AARCH64_ADR_PREL_PG_HI21 and R_AARCH64_LDST32_ABS_LO12_NC under -fno-pic, and to adrp, ldr x0, [x0], ldr w0, [x0] with R_AARCH64_ADR_GOT_PAGE and R_AARCH64_LD64_GOT_LO12_NC under -fPIE. On x86-64 both builds of that function use R_X86_64_PC32 because x86-64 addresses data relative to the instruction pointer anyway; there the visible difference between the builds is mostly absolute 32-bit addresses, such as the one in the linker error below.

Stage 3: assemble

The assembler produces the object file and records each symbol with a binding (local, visible only inside this file, or global, visible to other files) and a visibility (whether a global symbol may be exported from the finished executable or library). nm greet.o (compiled with -O2 -fstack-protector-strong) shows api_greet, internal_total and main as T (global code), helper_count as lowercase b (local, zero-initialised because of static), and printf, strcpy, __stack_chk_fail, __stack_chk_guard as U (undefined, to be resolved by the linker). So static is a source-level decision that becomes a local symbol that no other file can reference. With -fvisibility=hidden the compiler marks the global symbols HIDDEN (readelf -sW: internal_total and main show GLOBAL HIDDEN; api_greet, which the source marks visibility("default"), stays GLOBAL DEFAULT). Hidden symbols still link across object files inside one executable or library, but the linker will not export them.

Stage 4: link

The linker resolves the U symbols, lays out sections and writes the ELF file. Its decisions here:

• Export table (the dynamic symbol table, .dynsym: the subset of symbols the loader and other libraries can see). Building a shared library from the same file, default flags exported api_greet internal_total main (from nm -D --defined-only); with -fvisibility=hidden only api_greet was exported. That is how visibility turns into what is visible in the final file.
• PIE. gcc -O2 greet.c produced ELF type EXEC (fixed addresses); gcc -fPIE -pie produced DYN (Position-Independent Executable file), with the loader free to choose a random base address (ASLR, address space layout randomisation) at each run. The flag has to be consistent at both stages: linking the -fno-pic object with -pie fails. A PIE is linked the way a shared object is: it is loaded at a random base, and a symbol such as __stack_chk_guard comes from libc at an address the linker cannot know. An instruction that assumes a fixed distance to the symbol (AArch64's page-relative adrp pair) or holds a 32-bit absolute address (x86-64's R_X86_64_32) could only be made right by rewriting code at load time, so the linker refuses; that is what can not be used when making a shared object and recompile with -fPIC mean:

/usr/bin/ld: nopic.o: relocation R_AARCH64_ADR_PREL_PG_HI21 against symbol `__stack_chk_guard@@GLIBC_2.17' which may bind externally can not be used when making a shared object; recompile with -fPIC

This is the first line of the linker's output; it is followed by an unresolvable R_AARCH64_ADR_PREL_PG_HI21 relocation line, final link failed: bad value and the collect2: error: ld returned 1 exit status line.

On x86-64 the same mistake prints, as its first line:

/usr/bin/ld: nopic.o: relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE

Here R_X86_64_32 is an absolute 32-bit address of the string constant.

• RELRO (relocation read-only: after the loader fills in the GOT it marks parts of the data read-only so a later exploit cannot overwrite them). The default link had a GNU_RELRO program header but no BIND_NOW flag (partial RELRO: with lazy binding, an imported function is resolved on its first call through the PLT, so its GOT entry must stay writable until then). Linking with -fPIE -pie -Wl,-z,relro,-z,now printed FLAGS BIND_NOW and FLAGS_1 Flags: NOW PIE (readelf -d; the same link without -fPIE -pie ends in Flags: NOW): every symbol is resolved at start-up, so the whole GOT can be made read-only (full RELRO). The linker records the request; the dynamic loader enforces it when the program starts.

What a reviewer can still learn from a stripped binary

strip removes the static symbol table (.symtab): readelf -S shows the section before and not after, and nm greet_stripped prints no symbols. It leaves the dynamic symbol table, which the loader needs. Running nm -D on the stripped hardened binary still lists the imports printf, strcpy, abort, __libc_start_main, __stack_chk_fail and __stack_chk_guard (and a few weak runtime hooks such as __cxa_finalize and __gmon_start__, left out of this list). From that alone a reviewer learns:

• the canary is present (the __stack_chk_* imports), and strcpy, an unbounded copy, is called: a place to look;
• PIE and RELRO status from readelf -h, -l, -d (the program headers and dynamic section are untouched);
• string constants such as hello %s (%d) (strings), which often reveal function purpose;
• all the machine code, with calls through the PLT (procedure linkage table, the stubs for imported functions) to strcpy, printf and __stack_chk_fail visible in objdump -d;
• a build ID only if one was requested: -Wl,--build-id produced a Build ID: line in the stripped file (a 40-hex-digit value, for example cef9c676...; the digits differ from build to build, so treat it as an example), whereas the default link here had none. A build ID lets you match a binary to its separate debug file.

What is gone: the names of local functions and static variables (helper_count, and the name of any static function), and, if the build used -g, the types and line numbers (kept in debug sections that strip also removes).

Recommendation

For a hardened Linux release: compile with -fstack-protector-strong -fPIE, link with -pie -Wl,-z,relro,-z,now, mark exported API functions explicitly and build with -fvisibility=hidden, keep an unstripped copy plus build ID for crash analysis and ship the stripped one. Then verify the result from the binary itself (readelf -h -l -d, nm -D) instead of trusting the flag list, because each flag acts at a different stage and a missing one at either stage silently weakens the result.

Direct answer

Decide the boundary by comparing three snapshots of the same moment: what the kernel saved when it delivered the signal (the register block inside the ucontext_t, the structure the kernel builds on the user stack), what the user code and libc hold in registers at the interrupted instruction, and what the registers contain after the handler returns (after sigreturn, the system call the handler returns through). Whichever snapshot first disagrees with the one before it tells you which layer broke the state. The omitted frame pointer (-fomit-frame-pointer, meaning the compiler does not reserve a register to link stack frames) matters in only one way: it removes the frame-pointer chain, so a backtrace that walks rbp/x29 links is wrong, while one that reads the unwind tables (CFI, "call frame information") stays right. It does not change what the kernel saves or restores. The sequence below was run on AArch64 (gdb 16.3 and strace 6.13, native, in a gcc:14 container) because gdb cannot ptrace under x86-64 emulation; the method is identical on x86-64 with this mapping:

Syscall return values are small negative numbers for errors: -4 is -EINTR, where EINTR (error number 4, "interrupted system call") is what read reports when a signal arrives first. SA_RESTART is the sigaction flag asking the kernel to restart such a call instead of failing it. Async-signal-safe functions (the short list, such as write and _exit, that POSIX allows inside a handler because they do not depend on state the interrupted code might have been halfway through updating; printf and malloc are not on it) come up again in the fix.

What happens at the boundary

When a signal is pending, the kernel, at the next return to user mode, pushes a frame on the user stack holding the processor flags, the registers, the signal mask and the signal stack settings, arranges for the handler to run, and makes the handler return into a small trampoline (a few instructions in user-visible memory) that calls sigreturn; sigreturn then restores the registers from that frame and resumes the interrupted code (per the Linux sigreturn(2) manual). So the registers the interrupted code gets back are whatever is in the frame at that moment, not necessarily what was there when the signal arrived. If the signal interrupted a blocking system call, the kernel also decides between failing the call with EINTR and restarting it, depending on SA_RESTART; the signal(7) manual lists read on a pipe among the calls that restart with SA_RESTART and fail with EINTR without it.

A program that exposes each snapshot

#include <errno.h>
#include <signal.h>
#include <stdio.h>
#include <string.h>
#include <ucontext.h>
#include <unistd.h>

static volatile sig_atomic_t hits;
static int pipefd[2];
static unsigned long long saved_pc, saved_x0;
static int patch_context;

static void on_alarm(int sig, siginfo_t *si, void *uc)
{
    (void)sig; (void)si;
    const ucontext_t *u = uc;
    saved_pc = u->uc_mcontext.pc;          /* the interrupted program counter, as the kernel saved it */
    saved_x0 = u->uc_mcontext.regs[0];
    if (patch_context)
        ((ucontext_t *)uc)->uc_mcontext.regs[0] = 42;   /* edit the saved x0: sigreturn will load it */
    hits++;
    (void)!write(pipefd[1], "x", 1);       /* gives a restarted read something to return */
}

int main(int argc, char **argv)
{
    (void)argv;
    struct sigaction sa;
    memset(&sa, 0, sizeof sa);
    sa.sa_sigaction = on_alarm;
    patch_context = argc > 2;
    sa.sa_flags = SA_SIGINFO | (argc > 1 ? SA_RESTART : 0);
    sigaction(SIGALRM, &sa, 0);
    if (pipe(pipefd)) return 1;
    alarm(1);
    char c;
    ssize_t n = read(pipefd[0], &c, 1);
    int e = errno;
    printf("read returned %zd, errno %d (%s), handler ran %d time(s)\n", n, n < 0 ? e : 0, n < 0 ? strerror(e) : "none", (int)hits);
    printf("saved pc 0x%llx, saved x0 0x%llx\n", saved_pc, saved_x0);
    return 0;
}

Built with gcc -O2 -fomit-frame-pointer -g -Wall -Wextra sig_demo.c -o sig_demo (GCC 14.4, AArch64). The three modes print (the addresses vary from run to run because of address-space layout randomisation):

$ ./sig_demo
read returned -1, errno 4 (Interrupted system call), handler ran 1 time(s)
saved pc 0xffffa2b528dc, saved x0 0xfffffffffffffffc
$ ./sig_demo r
read returned 1, errno 0 (none), handler ran 1 time(s)
saved pc 0xffffb2e628d8, saved x0 0x3
$ ./sig_demo r p
read returned -1, errno 9 (Bad file descriptor), handler ran 1 time(s)
saved pc 0xffff8ab328d8, saved x0 0x3

Step 1: ask the kernel what it decided (strace)

strace -e trace=read ./sig_demo shows the boundary from the outside, with no debugger (this excerpt leaves out the dynamic loader's own read of the libc ELF header at the top, and at the end the program's second output line (saved pc ...) and strace's +++ exited with 0 +++ line; the read returned ... line in the block is the program's own output, which appears between the traced lines):

read(3, 0xffffe3a9c5a7, 1)              = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} ---
read returned -1, errno 4 (Interrupted system call), handler ran 1 time(s)

ERESTARTSYS is the kernel's internal "interrupted, restart if the handler allows it" result (it is ERESTARTSYS, not EINTR, that strace shows because it traces the kernel's side of the boundary). The program never sees that value: it sees EINTR or the result of the restarted call. If the handler's SA_RESTART flag is absent, the program sees EINTR; with the flag, the strace run of ./sig_demo r shows a second read call completing. If this step already contradicts the program's sigaction flags, the problem is in how the handler was installed (user code), not below.

Step 2: read the saved context in gdb

The gdb commands, one by one: break on_alarm stops when the handler starts; run starts the program; bt prints the stack, where frame 0 is the handler, frame 1 the kernel-built signal frame and frame 2 the code the signal interrupted; p/x ... prints a value in hex, here fields of the ucontext_t the kernel built (uc is the handler's third argument); x/2i $x30 examines (x) two instructions (i) at the address in x30, the link register, which is where the handler returns to; frame 2 selects the interrupted frame; x/3i $pc-8 shows three instructions starting 8 bytes before pc (AArch64 instructions are 4 bytes, so these are the two before pc and pc itself); info registers prints the named registers. set pagination off and set confirm off only stop gdb from pausing for prompts.

The gdb script:

set pagination off
set confirm off
break on_alarm
run
bt
echo \n== uc_mcontext as saved by the kernel\n
p/x ((ucontext_t *)uc)->uc_mcontext.pc
p/x ((ucontext_t *)uc)->uc_mcontext.regs[0]
echo \n== where the handler will return to (lr)\n
x/2i $x30
echo \n== the interrupted frame\n
frame 2
x/3i $pc-8
info registers pc x0 x8

Run as gdb -q -batch -x dbg.gdb ./sig_demo, without SA_RESTART. The transcripts in this and the next step leave out the Breakpoint 1 at 0x4009a0: file sig_demo.c, line 16. line that break prints (the address is specific to one build), gdb's startup notices, the Breakpoint 1, on_alarm (...) stop line with the source line under it (16 const ucontext_t *u = uc;), and the blank lines (the ones echo \n prints and the one gdb prints before each Breakpoint N, stop line). Their hexadecimal addresses vary from run to run: Docker's default seccomp profile stops gdb from disabling address-space randomisation (gdb warns Error disabling address space randomization: Operation not permitted), and with randomisation disabled the libc addresses are stable and look like 0x0000fffff7e728dc:

#0  on_alarm (sig=14, si=0xffffd5da9d10, uc=0xffffd5da9d90) at sig_demo.c:16
#1  <signal handler called>
#2  0x0000ffff8a9628dc in ?? () from /lib/aarch64-linux-gnu/libc.so.6
#3  0x0000ffff8a9628fc in ?? () from /lib/aarch64-linux-gnu/libc.so.6
#4  0x00000000004007e8 in main (argc=<optimized out>, argv=<optimized out>) at sig_demo.c:37
== uc_mcontext as saved by the kernel
$1 = 0xffff8a9628dc
$2 = 0xfffffffffffffffc
== where the handler will return to (lr)
   0xffff8aae08c0 <__kernel_rt_sigreturn>:	mov	x8, #0x8b                  	// #139
   0xffff8aae08c4 <__kernel_rt_sigreturn+4>:	svc	#0x0
== the interrupted frame
#2  0x0000ffff8a9628dc in ?? () from /lib/aarch64-linux-gnu/libc.so.6
   0xffff8a9628d4:	bl	0xffff8a962800
   0xffff8a9628d8:	svc	#0x0
=> 0xffff8a9628dc:	ldr	x19, [sp, #16]
pc             0xffff8a9628dc      0xffff8a9628dc
x0             0xfffffffffffffffc  -4
x8             0x3f                63

Reading it:

• Frame #1 <signal handler called> is gdb recognising the signal frame; it then continues to frames #2 to #4 correctly even though main has no frame pointer link, because it unwinds with CFI. Frames #2 and #3 are ?? because this libc has no symbols; the instructions still show the story.
• The saved pc (...28dc) is the instruction just after svc #0x0 (svc, supervisor call, is the AArch64 instruction that enters the kernel; the x86-64 equivalent is syscall), and saved x0 is -4, which is -EINTR. x8 is the system call number, 63, the AArch64 number of read. On this path the library wrapper will turn the negative x0 into -1 with errno = 4.
• The handler's link register x30 points at __kernel_rt_sigreturn, a two-instruction trampoline: mov x8, #139; svc #0 (system call 139 on AArch64 is rt_sigreturn).

With SA_RESTART (set args r), the same script prints a different saved state: saved pc is 0xffff85a328d8, the address of the svc instruction, and saved x0 is 0x3, the original first argument (the file descriptor). The kernel rewound the program counter by one instruction and put the argument back, so that returning from the handler re-executes the system call. That one-instruction difference (...28dc against ...28d8) is the evidence that distinguishes "restart" from "fail with EINTR" at the register level.

Step 3: prove who changed the state, by comparing before and after

A second gdb script stops at the interrupted instruction and compares registers before the handler runs and after sigreturn. Its key command is break *$pc: the * makes it a breakpoint at an exact address, here the svc instruction of the interrupted read inside libc. That instruction is inside libc's shared system call code, which the handler's own write also uses, so the breakpoint is hit twice:

set pagination off
set confirm off
break on_alarm
run
echo \n== interrupted frame before the handler body runs\n
frame 2
info registers pc sp x0 x19
break *$pc
continue
echo \n== stop 1: the handler's own write() goes through the same svc\n
bt 1
info registers sp x0
continue
echo \n== stop 2: back at the interrupted svc after sigreturn\n
info registers pc sp x0 x19

Run with set args r p (the handler overwrites the saved x0 with 42):

== interrupted frame before the handler body runs
#2  0x0000ffffb66228d8 in ?? () from /lib/aarch64-linux-gnu/libc.so.6
pc             0xffffb66228d8      0xffffb66228d8
sp             0xffffd2766c80      0xffffd2766c80
x0             0x3                 3
x19            0xffffb6791128      281473743130920
Breakpoint 2 at 0xffffb66228d8
Breakpoint 2, 0x0000ffffb66228d8 in ?? () from /lib/aarch64-linux-gnu/libc.so.6
== stop 1: the handler's own write() goes through the same svc
#0  0x0000ffffb66228d8 in ?? () from /lib/aarch64-linux-gnu/libc.so.6
sp             0xffffd27659f0      0xffffd27659f0
x0             0x4                 4
Breakpoint 2, 0x0000ffffb66228d8 in ?? () from /lib/aarch64-linux-gnu/libc.so.6
== stop 2: back at the interrupted svc after sigreturn
pc             0xffffb66228d8      0xffffb66228d8
sp             0xffffd2766c80      0xffffd2766c80
x0             0x2a                42
x19            0xffffb6791128      281473743130920

To tell the stops apart, read x0 and sp. The first stop is not the interrupted instruction: it is the handler's own write passing through the same svc (a different sp in the handler's frame area, x0 = 4 the pipe's write end). The second stop, after the handler returned through sigreturn, is the restarted read: pc, sp and the callee-saved x19 are identical to the values before the signal, which is what a correct round trip looks like, but x0 is 42, so the system call re-executed with a corrupted file descriptor and returned EBADF, matching the third program run. Because the kernel restored exactly the frame's contents, a changed register at this point means something wrote the frame: the handler through its ucontext_t argument, a buffer overrun inside the handler that writes upward into the frame (when the handler runs on the interrupted stack), or a wrong sigaltstack setup. A register that is unchanged here but wrong later was clobbered by code after the return.

Step 4: attribute the layer

Where omit-frame-pointer matters

Without a frame pointer, the compiler uses the register for ordinary values. On x86-64 the same thing happens in this function:

long mix(const long *p, long n)
{
    long a0 = 0, a1 = 1, a2 = 2, a3 = 3, a4 = 4, a5 = 5, a6 = 6, a7 = 7, a8 = 8, a9 = 9;
    for (long i = 0; i < n; i++) {
        a0 += p[i]; a1 ^= a0; a2 += a1; a3 ^= a2; a4 += a3;
        a5 ^= a4; a6 += a5; a7 ^= a6; a8 += a7; a9 ^= a8; a0 += a9;
    }
    return a0 + a1 + a2 + a3 + a4 + a5 + a6 + a7 + a8 + a9;
}

Compiled with gcc -O2 -fomit-frame-pointer -c (GCC 14.4, x86-64) and read back with objdump -d -M intel, the source has thirteen values in the loop (ten accumulators, p, n and i), and GCC reduces that to twelve (the ten accumulators, the pointer p, and an end pointer in r12 that replaces n and i), still more than the nine caller-saved general registers, so the compiler also uses the callee-saved rbx, rbp and r12, and rbp holds an accumulator (excerpt: only the instructions that mention rbp, with the machine-code byte column dropped, as --no-show-raw-insn does; the offsets are from this build):

  19:	push   rbp
  8a:	add    rbp,rdx
  8d:	xor    rbx,rbp
  ad:	add    rax,rbp
  b4:	pop    rbp

(CFI, call frame information, is the table in the binary that tells an unwinder how to find each caller's frame without a frame pointer.) A crash reporter or profiler that follows rbp links from inside the signal handler would read an accumulator as a frame address and print a corrupt stack, and may fault itself. That produces a convincing but false "the kernel boundary is broken" signal. The fix is to unwind through CFI (gdb and the libgcc unwinder do), or to build the code under investigation with -fno-omit-frame-pointer only to confirm that the backtrace, not the bug, was the problem. A related x86-64 point, separate from the diagnosis above: the System V ABI reserves a 128-byte red zone below rsp (scratch space a leaf function may use without moving rsp), and the ABI says signal and interrupt handlers must not modify it, and Linux honours that on x86-64 by moving the signal frame 128 bytes below rsp. Code built with -mno-red-zone, as kernel code is, never uses that area, so it does not depend on the guarantee.

Fix and verification

Once the layer is known: make the handler async-signal-safe (set a flag, write to a pipe, do no malloc or printf), install it with the intended SA_RESTART flag, wrap interrupted calls in a loop that retries on EINTR (or use SA_RESTART for calls where partial results are not a concern), never write the ucontext_t unless implementing something like a deliberate user-space context change, and use sigaltstack (which tells the kernel to run handlers on a separate, dedicated stack, selected per handler with SA_ONSTACK) with a large enough stack if the handler runs deep. Verify with the same scripts: the saved state must match one of the two valid shapes in Step 2, and the post-sigreturn registers must equal the pre-signal ones, repeated over many signal deliveries with the signal rate raised (setitimer, which arms a timer that raises SIGALRM repeatedly at a short interval) so the interruption lands at different instructions.

Direct answer

"Fails only when arguments are large or the call is variadic" is the signature of an ABI (application binary interface: the rules for where arguments, results and registers go at a call boundary) disagreement, because small scalar arguments are passed in plain registers with no special rules on every convention (in different registers per ABI: rdi, rsi, ... on System V, rcx, rdx, ... on Microsoft x64, x0 to x7 on AArch64), while large aggregates (structs) and variadic calls are exactly where conventions differ. The check is: stop at the call boundary on both sides, and compare what the caller actually put in registers and on the stack with what the callee's ABI says it should find. Do it in this order: (1) get the exact ABI and platform, (2) look at the caller's instructions before the call, (3) break at the callee's first instruction and read registers and the stack, (4) compare structure sizes and offsets from both compilations, (5) check callee-saved registers and stack alignment across the call.

1. Know the rules for this platform

Start with the one ABI that matches the machine where the failure happens, and read the others only if the library is also built for them: System V x86-64 for Linux and macOS on Intel or AMD, AAPCS64 for 64-bit Arm (Linux, Android; Apple's arm64 platforms follow it with documented deviations, including in how variadic arguments are passed, so read Apple's documentation when the failing machine is a Mac or iPhone), Microsoft x64 for Windows. The three all pass small scalar arguments in registers with no special handling, and differ on large structs and variadic calls, which is why those two cases separate them.

For x86-64 System V (Linux, macOS), as the ABI document states: integer arguments go in rdi, rsi, rdx, rcx, r8, r9; an aggregate (a struct, union or array treated as one value) larger than two eightbytes (an eightbyte is an 8-byte chunk of the value, so more than 16 bytes) is passed in memory, on the stack (the exception is a single 32- or 64-byte vector type such as __m256, which travels in a vector register); a function whose result is in memory gets a caller-provided buffer whose address is passed in rdi as a hidden first argument (an extra pointer parameter the C source never mentions), and returns that address in rax; for a variadic call (one declared with ..., like printf), al (the low byte of rax) must hold an upper bound (0 to 8) on the number of vector registers used, where the vector registers xmm0 to xmm7 are the ones that carry double arguments. For AArch64 (AAPCS64, the Arm 64-bit procedure call standard): a composite (the Arm document's word for a struct, union or array) larger than 16 bytes is copied by the caller to memory and replaced by a pointer to the copy (passed in x0 to x7 like any pointer), except that a homogeneous floating-point or short-vector aggregate (an HFA or HVA, for example a struct of four doubles) is passed in vector registers instead of being copied; a result that does not fit in registers is written to a caller-reserved block whose address is passed in x8. Microsoft x64 (Windows), as Microsoft's documentation describes it: first four arguments in rcx, rdx, r8, r9 (floating point in xmm0 to xmm3), the caller always reserves 32 bytes of shadow space (home space for those four register arguments) on the stack, any argument that is not 1, 2, 4 or 8 bytes is passed by reference (the caller passes the address of a copy instead of the value), and for variadic or unprototyped calls (calls through a declaration that lists no parameter types, such as the old-style int f();) floating-point values must be duplicated in the matching integer register.

2. What correct callers look like

struct small { long a, b; };
struct big   { long a, b, c; };

long take_small(struct small s);
long take_big(struct big b);
struct big make_big(long x);
int sum_doubles(int n, ...);

long call_small(void) { struct small s = {1, 2}; return take_small(s); }
long call_big(void)   { struct big b = {1, 2, 3}; return take_big(b); }
long call_make(void)  { struct big r = make_big(7); return r.c; }
int  call_va(void)    { return sum_doubles(2, 1.5, 2.5); }
int  call_nonva(int (*fp)(int, double)) { return fp(2, 1.5); }

x86-64 (gcc -O2 -S -masm=intel, GCC 14.4, emulated linux/amd64 container; directives, the .LFB/.LFE function-boundary labels and the read-only data definitions of .LC0 to .LC2 removed, and the listing is in the order GCC emitted it). Syntax is Intel, destination first. .LC0, .LC1 and .LC2 are labels for constants the compiler placed in read-only data (the constant pool), so a movsd xmm0, QWORD PTR .LC2[rip] loads a floating-point literal; movdqa and movups copy 16 bytes between an xmm register and memory (aligned and unaligned respectively):

call_small:
	mov	edi, 1
	mov	esi, 2
	jmp	take_small
call_big:
	sub	rsp, 72
	movdqa	xmm0, XMMWORD PTR .LC0[rip]
	mov	QWORD PTR [rsp+16], 3
	movups	XMMWORD PTR [rsp], xmm0
	call	take_big
	add	rsp, 72
	ret
call_make:
	sub	rsp, 40
	mov	esi, 7
	mov	rdi, rsp
	call	make_big
	mov	rax, QWORD PTR [rsp+16]
	add	rsp, 40
	ret
call_va:
	movsd	xmm1, QWORD PTR .LC1[rip]
	mov	edi, 2
	mov	eax, 2
	movsd	xmm0, QWORD PTR .LC2[rip]
	jmp	sum_doubles
call_nonva:
	mov	rax, rdi
	movsd	xmm0, QWORD PTR .LC2[rip]
	mov	edi, 2
	jmp	rax

Reading it. The 16-byte small travels in edi and esi. In call_big, sub rsp, 72 makes room for the 24-byte copy of big (written at [rsp] to [rsp+16]) and rounds the frame up so that rsp is a multiple of 16 at the call: on entry rsp is 8 modulo 16 because the caller's call pushed a return address, and 72 is 8 modulo 16. take_big finds the copy at rsp+8 after its own call pushes the return address. make_big receives rdi = rsp, the hidden result buffer, and the code reads r.c back from [rsp+16]. For the variadic call, eax (that is al) is set to 2 because two vector registers carry doubles. In call_nonva the call goes through a pointer whose prototype is not variadic, so no eax is set at all. If the callee really is variadic, it reads whatever al happens to hold. Its entry code is test al, al followed by a conditional skip over the stores of xmm0 to xmm7 into the register save area (a block on the callee's stack where va_arg later finds the register-passed values). Compiling this sum_doubles:

#include <stdarg.h>
int sum_doubles(int n, ...) {
    va_list ap; va_start(ap, n);
    double s = 0;
    for (int i = 0; i < n; i++) s += va_arg(ap, double);
    va_end(ap);
    return (int)s;
}

with the same flags gives this entry (directives and the .LFB0: label removed; abridged to the start of the function, where ... stands for the stores of xmm2 to xmm6, and the .L8: label that the je targets sits right after the last store, followed by the code that sets up the va_list):

sum_doubles:
	sub	rsp, 48
	test	al, al
	je	.L8
	movaps	XMMWORD PTR [rsp-88], xmm0
	movaps	XMMWORD PTR [rsp-72], xmm1
	...
	movaps	XMMWORD PTR [rsp+24], xmm7

With al = 0 the je jumps over all eight stores, the doubles are never saved and va_arg reads stale stack bytes.

AArch64 (gcc -O2 -S, GCC 14.4, native; directives, the .LFB/.LFE labels and the read-only data definition of the constant {1, 2, 3} (.LANCHOR0, .LC0) removed, and abridged to call_small, call_big and call_make, so call_va and call_nonva are not shown):

call_small:
	mov	x0, 1
	mov	x1, 2
	b	take_small
call_big:
	adrp	x1, .LANCHOR0
	add	x1, x1, :lo12:.LANCHOR0
	stp	x29, x30, [sp, -80]!
	mov	x29, sp
	add	x0, sp, 16
	ldp	x2, x3, [x1]
	stp	x2, x3, [sp, 16]
	ldr	x1, [x1, 16]
	str	x1, [sp, 32]
	bl	take_big
	ldp	x29, x30, [sp], 80
	ret
call_make:
	stp	x29, x30, [sp, -48]!
	mov	x0, 7
	mov	x29, sp
	add	x8, sp, 24
	bl	make_big
	ldr	x0, [sp, 40]
	ldp	x29, x30, [sp], 48
	ret

Syntax note: this is AArch64 assembly, where the destination is the first operand (as in Intel syntax) and [sp, -80]! means subtract 80 from sp first, then store (pre-index with writeback). Line by line in call_big: adrp/add form the address of the constant {1, 2, 3} in read-only data (.LANCHOR0); stp x29, x30, [sp, -80]! opens the frame by saving the frame pointer x29 and the link register x30 (the register holding the return address, which bl overwrites); ldp/stp and ldr/str copy the three members into the stack at sp+16 to sp+32; add x0, sp, 16 puts the address of that copy in x0; bl take_big calls; ldp x29, x30, [sp], 80 restores both registers and then adds 80 to sp (post-index). In call_make, add x8, sp, 24 is the result buffer and ldr x0, [sp, 40] reads member c (offset 16 from the buffer). Here the 24-byte struct is copied to the stack and x0 holds its address (add x0, sp, 16), and the result buffer address goes in x8. Same source, completely different mechanics: this is why a mismatch shows up differently per architecture.

3. A real mismatch, and what you see at the call boundary

Two compilation units disagree about the struct, as happens with a stale header against a rebuilt library:

/* lib_side.c */
struct big { long a, b, c; };

long take_big(struct big v) { return v.a + v.b + v.c; }

/* app_side.c */
#include <stdio.h>

struct big { long a, b; };   /* stale header: the library's struct has a third member */
long take_big(struct big v);

int main(void) {
    struct big v = {1, 2};
    printf("%ld\n", take_big(v));
    return 0;
}

Built with gcc -O1 -g -o app app_side.c lib_side.c (GCC 14.4, AArch64, native) the program dies with Segmentation fault. The two sides, from objdump -d --no-show-raw-insn run on the object files lib_side.o and app_side.o (each compiled with gcc -O1 -g -c), so offsets start at 0 and the bl target is not yet resolved (the // #n comments objdump appends to the mov lines, the seven instructions after the bl in main (shown as ...; they set up the printf call and return), the file format header lines, the Disassembly of section .text: lines, the blank lines and the 0000000000000000 address objdump prints before each function label are omitted; the two objects' listings are shown one after the other):

<take_big>:
   0:	ldp	x1, x2, [x0]
   4:	add	x1, x1, x2
   8:	ldr	x0, [x0, #16]
   c:	add	x0, x1, x0
   10:	ret
<main>:
   0:	stp	x29, x30, [sp, #-16]!
   4:	mov	x29, sp
   8:	mov	x0, #0x1
   c:	mov	x1, #0x2
  10:	bl	0 <take_big>
  ...

The callee treats x0 as a pointer to a 24-byte copy; the caller thinks the struct is 16 bytes and passes the values 1 and 2 in x0 and x1. Running the program under gdb 16.3 (Debian package, AArch64, gdb -q -batch with -ex run -ex bt -ex 'x/i $pc' -ex 'info registers x0 x1 x30' -ex 'p $_siginfo._sifields._sigfault.si_addr', so no prompts or command echoes appear) shows the fault at the faulting instruction. Left out of the output below: the start-up warning about address-space randomization, the thread-library lines, the blank line after them, and the source line gdb prints under the stop line; every other line is as printed:

Program received signal SIGSEGV, Segmentation fault.
take_big (v=<error reading variable: Cannot access memory at address 0x1>) at lib_side.c:4
#0  take_big (v=<error reading variable: Cannot access memory at address 0x1>) at lib_side.c:4
#1  0x0000000000400658 in main () at app_side.c:9
=> 0x400674 <take_big>:	ldp	x1, x2, [x0]
x0             0x1                 1
x1             0x2                 2
x30            0x400658            4195928
$1 = (void *) 0x1

(x30 is the link register, which holds the return address.) The tell is that x0, which the callee will dereference as an address, holds a small integer, and x1 holds the second member. This is the reading to look for: a register that the callee's ABI says is a pointer holding a value that is clearly not one.

On x86-64 the same mismatch compiles to a callee that reads [rsp+0x8], [rsp+0x10] and [rsp+0x18] (gcc -O1 in the emulated linux/amd64 container, then objdump -d --no-show-raw-insn -M intel: mov rax,QWORD PTR [rsp+0x10], add rax,QWORD PTR [rsp+0x8], add rax,QWORD PTR [rsp+0x18]), while the caller put the values in edi and esi and pushed nothing. Reading only those stack slots, which are not the caller's arguments, it returns a sum of stale stack words instead of faulting. Built from the same two source files and run in the emulated linux/amd64 container, the program exits normally and prints a large meaningless number (one sample run printed 422212454064320; the digits vary with what is on the stack). So the same stale header can show up as a crash on one architecture and as a quietly wrong number on another.

4. The checklist at the debugger

1. Break on the callee's first instruction (break *take_big stops before the first instruction; then info registers). On x86-64 System V, at entry [rsp] is the return address, rdi..r9 hold integer arguments, stack arguments start at [rsp+8], and rsp+8 should be 16-byte aligned (the ABI requires 16-byte alignment of the stack immediately before the call). On AArch64 read x0 to x7, x8, and sp.
2. For a large struct: is the argument register a valid pointer (x/4gx $x0), or, on x86-64, does x/6gx $sp show your struct at rsp+8?
3. For a variadic call on x86-64: look at al (p $al or the low byte of rax) at the callee entry. A value of 0 with float arguments, or garbage, means the caller used a non-variadic prototype or the library was declared without ....
4. For a returned struct: confirm the caller put a buffer address in rdi (x86-64) or x8 (AArch64), and that the callee returns it in rax on x86-64.
5. Compare layouts. Print sizeof and offsetof in both compilation units (gdb ptype /o struct big shows offsets and padding), and compile both sides with the same flags. For a Rust or assembly library, check that the Rust struct has #[repr(C)] (the attribute that asks Rust for C's field order and padding): the Rust reference says the default representation makes only the layout guarantees needed for soundness (field order is not one of them), and describes the C representation as designed for dual purposes, one of which is interoperability with C.
6. Callee-saved registers. Break after the call returns and compare rbx, rbp, r12 to r15 (x86-64 System V) or x19 to x28 (AArch64) with their values before the call. An assembly routine that uses one without saving it corrupts the caller in a way that appears far away.
7. For Windows targets, the checks change to rcx, rdx, r8, r9, 32 bytes of shadow space the caller must reserve, and by-reference passing for anything not 1, 2, 4 or 8 bytes. A hand-written assembly function that omits the shadow space makes the callee's spill of its register arguments overwrite the caller's frame.

The fix is to make one authoritative declaration (a shared header, or a generated binding) and rebuild both sides; for hand-written assembly, add a test that calls it with 16-byte and 24-byte structs and a variadic float argument.

Direct answer

A source-level backtrace lists functions, but the CPU only has addresses and a stack. When the compiler inlines a function (copies its body into the caller) there is no call, no return address and no stack frame for it (a frame is the stack region and bookkeeping one running function owns), so the debugger has to reconstruct "frames" from debug information. When the compiler turns a call at the end of a function into a plain jump (a tail call, so the callee returns straight to the caller's caller), the caller's frame is genuinely gone from the stack. So the trace can show functions that were never called as separate frames, miss functions that did run, and report variables as <optimized out>. To find out what actually executed, work from the machine code: disassemble the crash address, map it with the debug info including its inline chain, check the frame metadata, and compare with a build where inlining and tail calls are turned off.

A concrete program

crash.c has a chain main -> dispatch -> handle -> validate -> first_byte, where the last one reads through a null pointer. validate and first_byte are static inline, dispatch simply returns handle(r).

#include <stdio.h>

struct req { int len; const char *data; };

static inline int first_byte(const struct req *r) { return r->data[0]; }

static inline int validate(const struct req *r) {
    return r->len > 0 ? first_byte(r) : -1;
}

__attribute__((noinline)) int handle(const struct req *r) { return validate(r) + 1; }

__attribute__((noinline)) int dispatch(const struct req *r) { return handle(r); }

int main(int argc, char **argv) {
    (void)argv;
    struct req r = { argc, NULL };     /* len > 0, data == NULL */
    printf("%d\n", dispatch(&r));
    return 0;
}

Run in a gcc:14 container (GCC 14.4.0, GDB 16.3, aarch64 native; gdb cannot ptrace, the Linux system call a debugger uses to control another process, under amd64 emulation, so this is AArch64). Stack addresses differ from run to run, so the numbers below are one sample.

gcc -O0 -g crash.c -o crash_O0
gcc -O2 -g crash.c -o crash_O2
gcc -O2    crash.c -o crash_O2_nodebug
gdb -q -batch -ex run -ex bt ./crash_O0     # likewise for the other two

At -O0, every function is a real call and the backtrace is five frames with real return addresses:

#0  first_byte (r=0xfffff8820970) at crash.c:5
#1  0x0000000000400684 in validate (r=0xfffff8820970) at crash.c:8
#2  0x00000000004006a8 in handle (r=0xfffff8820970) at crash.c:11
#3  0x00000000004006c8 in dispatch (r=0xfffff8820970) at crash.c:13
#4  0x00000000004006f4 in main (argc=1, argv=0xfffff8820af8) at crash.c:18

At -O2 -g, the same five frames still print, but they are not equal in kind. Frames #1 and #2 have no address column: they are inlined frames, drawn from debug info, with no instructions or stack space of their own, and they share frame #0's pc 0x4006b0. Frame #3 has an address but is a tail-call frame (below). r=r@entry=0x... means the argument r has the same value now as at function entry, and <optimized out> means the compiler kept no copy of the value at this pc:

#0  0x00000000004006b0 in first_byte (r=<optimized out>) at crash.c:5
#1  validate (r=r@entry=0xffffd67fc710) at crash.c:8
#2  handle (r=r@entry=0xffffd67fc710) at crash.c:11
#3  0x00000000004006c8 in dispatch (r=r@entry=0xffffd67fc710) at crash.c:13
#4  0x0000000000400560 in main (argc=<optimized out>, argv=<optimized out>) at crash.c:18

Without -g (-O2, no debug info), the trace collapses to what the machine really has:

#0  0x00000000004006b0 in handle ()
#1  0x0000000000400560 in main ()

What the machine code says

objdump -d crash_O2 (raw instruction bytes removed; the complete code of handle and dispatch). The listing is AArch64 assembly, destination operand first: ldr w1, [x0] loads 32 bits from the address in x0 into w1, ldrb loads one byte, cmp sets the condition flags, b.le branches if less-or-equal (signed), b is an unconditional jump, and ret jumps to the address in the link register x30, the register a call instruction fills with the return address.

00000000004006a0 <handle>:
  4006a0:	ldr	w1, [x0]
  4006a4:	cmp	w1, #0x0
  4006a8:	b.le	4006bc <handle+0x1c>
  4006ac:	ldr	x0, [x0, #8]
  4006b0:	ldrb	w0, [x0]
  4006b4:	add	w0, w0, #0x1
  4006b8:	ret
  4006bc:	mov	w0, #0x0                   	// #0
  4006c0:	ret

00000000004006c4 <dispatch>:
  4006c4:	b	4006a0 <handle>

Reading it:

• first_byte and validate do not exist as code. Their bodies are the ldr w1, [x0] (test len), the ldr x0, [x0, #8] (load data) and the ldrb w0, [x0] (the faulting byte load) inside handle. The crash is at 0x4006b0, the ldrb.
• The b.le at 0x4006a8 jumps to 0x4006bc when len <= 0: that path returns 0, which is validate's -1 plus handle's + 1 folded into a constant at compile time. It never reads data, so it cannot crash.
• dispatch is a single b to handle: a tail call. It does not call, so it leaves no return address of its own on the stack. The return address in link register x30 is the one main set when it called dispatch, and it points back into main (0x400560; info registers x30 in gdb prints 0x400560).

Tools that expose this, on the same -O2 -g build:

(gdb) info frame
Stack level 0, frame at 0xfffffffff980:
 pc = 0x4006b0 in first_byte (crash.c:5); saved pc = 0x4006c8
 inlined into frame 1
 source language c.
 Arglist at unknown address.
 Locals at unknown address, Previous frame's sp in sp
(gdb) frame 3
#3  0x00000000004006c8 in dispatch (r=r@entry=0xfffffffff990) at crash.c:13
13	__attribute__((noinline)) int dispatch(const struct req *r) { return handle(r); }
(gdb) info frame
Stack level 3, frame at 0xfffffffff980:
 pc = 0x4006c8 in dispatch (crash.c:13); saved pc = 0xfffff7e1225c
 tail call frame, caller of frame at 0xfffffffff980
 source language c.
 Arglist at unknown address.
 Locals at unknown address, Previous frame's sp is 0xfffffffff980

These were run with gdb -q -batch -ex run -ex 'info frame' -ex 'frame 3' -ex 'info frame' ./crash_O2, and the transcript above is the full output after the crash report. The stack addresses and the saved pc inside the C library change from run to run; the 0x4006b0 and 0x4006c8 values and the words inlined into frame 1 and tail call frame do not.

How to read the two transcripts: in info frame, look for the words inlined into frame N (the frame is inlined, no stack of its own) and tail call frame (the frame was inferred, not on the stack). The saved pc in frame 0's output, 0x4006c8, is the address one instruction after the lone b at 0x4006c4 in dispatch: it is where a normal call from dispatch to handle would have returned. It is not in memory or in x30; gdb synthesizes it. The gdb manual says that on detecting a tail call gdb creates a fictitious call frame with the return address set up as if the caller had called the callee normally, using the compiler's DW_TAG_call_site records. That is why frame #3 prints 0x4006c8 (one past the jump, which happens to be the start of _fini, not code of dispatch) while the real return address, 0x400560 in main, appears in frame #4.

addr2line -i -f -e crash_O2 0x4006b0 lists the whole inline chain for that one instruction:

first_byte
/w/crash.c:5
validate
/w/crash.c:8
handle
/w/crash.c:11

readelf --debug-dump=info crash_O2 shows two DW_TAG_inlined_subroutine entries (DWARF, the debug-info format, records that make gdb print those frames). gdb's frame #3 is a "tail call frame": the debugger inferred a caller from debug-info call-site records, it is not on the stack, and it is absent in the build without -g.

For comparison, building with inlining and tail calls disabled, gcc -O2 -g -fno-inline -fno-optimize-sibling-calls crash.c, gives five frames with real return addresses in #1 to #3 (0x4006c4, 0x4006ec, 0x40070c), the same shape as -O0.

Method for a performance-critical or security-sensitive path

1. Take the crash or hot address (the faulting pc, or the sample address from perf) from the exact shipped binary, not from a rebuild.
2. Disassemble around it (x/10i $pc in gdb, objdump -d, or disassemble /s to interleave source lines) and name what each instruction does, as above.
3. Resolve the address with the inline chain (addr2line -i -f, or info frame for inlined into frame), using the debug info that matches the binary's build ID (a unique hash of the binary; readelf -n shows it) so the lines are not from a different build.
4. Check what the frames are: info frame distinguishes inlined, tail-call and normal frames, and treat <optimized out> as "not available", not as zero.
5. If you need a clean stack, compare with a debug build as above, but treat it as a different program: inlining changes what code runs, so a rebuilt -O0 binary can hide a bug that only exists in the optimized one.
6. Keep what matters visible in production builds: mark a function you want to appear in traces __attribute__((noinline)), and build with -g for the symbol file, since adding -g here left the instructions identical (a diff of the two disassemblies was empty).

Why this is worth the care

In a security-sensitive path, the source-level story ("check, then use") can differ from the executed code (the check merged into another function, reordered, or removed). In a performance-critical path, a profile that attributes time to handle includes the inlined validate and first_byte; use the inline-aware symbolization above to split it.