Assembly and Low-Level Language Fundamentals Questions

Direct answer

Inlining replaces a call with a copy of the callee's body at the call site. The compiler decides with a cost model: it estimates how much code size and compile time the copy adds against how much it saves (the call and return, argument shuffling, and the chance to optimize the body with the caller's known values), and it compares the result with size limits, adjusted for how hot the call is. Benefits: no call overhead, and constants and dead branches can be folded across the old boundary (folded means evaluated at compile time once the argument values are visible). Costs: larger code (more instruction-cache pressure, meaning the small fast memory next to the CPU that holds recently used instructions has to hold more, so more fetches miss it and go to slower memory; and larger binaries), longer compiles, and less faithful debugging and profiling. Link-time optimization (LTO) removes the "other file" barrier, so calls across source files become candidates. Profile-guided optimization (PGO) replaces guessed call frequencies with measured ones, so hot call sites inline more readily and cold ones stay as calls.

How the decision is made

For each call site the compiler checks, roughly in this order:

1. Can it? The body must be visible (same file, or LTO), the callee not marked noinline, and no property such as an address use or unusual calling pattern that prevents it. A function marked always_inline (a GCC attribute that demands inlining) skips the cost model.
2. How much does it grow the code? Small bodies, a function called from only one place, or a body that shrinks once the arguments are known are cheap. A static function with a single caller can be absorbed and its out-of-line copy dropped.
3. How hot is the site? (A hot site is one that runs very often, a cold site one that rarely or never runs.) A call inside a loop is worth more than one on an error path. Without a profile this is a guess from loop depth and branch heuristics.
4. Budget. GCC reports the limits it applied. In its units (an internal estimate of instructions, not exact machine instructions), the defaults on the GCC 14.4 used here, as printed by gcc --help=params -Q, are --param early-inlining-insns = 6 at -O2 (14 at -O3) for the first, early inlining pass, and --param max-inline-insns-auto = 15 at -O2 (30 at -O3) for functions not marked inline. A call is refused when the growth it would cause is above the limit.

A worked case uses the PGO source below, where mix is deliberately a long chain of arithmetic so that it is not tiny. -fopt-info-inline-all on it printed will not early inline: main/23->mix/22, growth 26 exceeds --param early-inlining-insns and not inlinable: main/23 -> mix/22, --param max-inline-insns-auto limit reached: inlining mix would grow main by 26, which is above both -O2 limits (6 and 15). So at -O2 without a profile all three calls to mix stay calls. At -O3 the same source, with limits of 14 and 30 (26 is below the 30 for the main pass), inlined the call inside the loop and left two calls to mix (the two under the if tests). Building -O2 with only those two parameters raised to -O3's values also left two calls. The only change between these builds is the size budget, which shows the size limit is the deciding quantity for a function this size.

The inline keyword is only a hint in C and C++; static inline mostly affects linkage rules (static gives each source file its own private copy of the function, so a definition in a header does not cause duplicate-symbol errors at link time), not the decision.

LTO: inlining across files

Two files, mathlib.c and app.c:

/* mathlib.c */
int clamp_add(int a, int b)
{
    int s = a + b;
    return s > 255 ? 255 : s;
}

/* app.c */
#include <stdio.h>

int clamp_add(int a, int b);

int main(void)
{
    int acc = 0;
    for (int i = 0; i < 1000; i++)
        acc = clamp_add(acc, i & 3);
    printf("%d\n", acc);
    return 0;
}

Built as gcc -O2 -c mathlib.c app.c and linked, objdump -d on main (GCC 14.4, AArch64 container) shows the call kept: bl 4006e0 <clamp_add> followed by bl 400500 <printf@plt> (the addresses depend on the link layout) (printf@plt is a small stub in the executable that jumps to the C library's printf). Built with gcc -O2 -flto -c ... and linked with -flto, main contains only the printf call: clamp_add was inlined into the loop. Both binaries print 255. Without LTO the compiler compiled app.c without ever seeing the body of clamp_add, so it could not copy it.

PGO: measured hotness

Profile-guided optimization is a two-build process: first build an instrumented program and run it on a representative input, and the run writes a profile (a .gcda file of counts: how many times each function and branch executed); then build again with the profile as input. mix below is a deliberately large function (ten rounds of shifts, xors and multiplies) so that it is above the default size limit, which makes the effect of the profile visible. The loop call is the hot site, and the two calls under if (argc > ...) are cold sites that never run in the training run.

#include <stdio.h>
#include <stdlib.h>

static unsigned mix(unsigned x)
{
    x ^= x >> 15; x *= 0x2c1b3c6dU;
    x ^= x >> 12; x *= 0x297a2d39U;
    x ^= x >> 15; x += 0x9e3779b9U;
    x ^= x << 7;  x *= 0x85ebca6bU;
    x ^= x >> 13; x *= 0xc2b2ae35U;
    x ^= x >> 11; x *= 0x7feb352dU;
    x ^= x >> 15; x *= 0x846ca68bU;
    x ^= x >> 15; x *= 0x2c1b3c6dU;
    x ^= x >> 12; x *= 0x297a2d39U;
    return x ^ (x >> 16);
}

int main(int argc, char **argv)
{
    unsigned h = 1;
    for (unsigned i = 0; i < 1000000; i++)       /* hot call site */
        h += mix(h + i);
    if (argc > 1)                                 /* cold call sites */
        h += mix((unsigned)atoi(argv[1]));
    if (argc > 2)
        h += mix((unsigned)atoi(argv[2]));
    printf("%u\n", h);
    return 0;
}

The sequence. The profile file is named after the output file and source (here app-mix_hot_cold.gcda), so the final build uses the same -o app and source name and GCC finds it. -fprofile-correction tells GCC to repair counts that do not add up, which happens when a threaded program updates the counters without locking:

gcc -O2 -o app mix_hot_cold.c                                   -> 3 calls to mix in main
gcc -O2 -fprofile-generate -o app mix_hot_cold.c && ./app       -> prints 1578539220, writes the profile
gcc -O2 -fprofile-use -fprofile-correction -o app mix_hot_cold.c -> 2 calls to mix in main

With the profile, -fopt-info-inline-optimized printed mix_hot_cold.c:22:14: optimized: Inlined mix/27 into main/23 which now has time 34000018.000000 and size 64, net change of +26. Line 22 is the loop call. Read the message as: mix/27 and main/23 are GCC's internal numbers for the two functions; after inlining, main has an estimated size of 64 units (so it was 64 - 26 = 38 before), net change of +26 is the growth, the same 26 that the cost model refused at -O2, and time 34000018.000000 is GCC's estimate of main's running time in its own units. It is dominated by the million loop iterations, which is why it is millions here, where the same build without a profile reports a time of only a few thousand (3402.12 at -O3); GCC does not document how it splits the figure per iteration, so read it as a relative estimate and not as cycles. The profile is what tells GCC that the loop really runs that many times; the same growth that was too much for an unknown call site is acceptable for a site the profile shows to be hot. The two cold sites on lines 24 and 26 remain bl instructions to mix. The profile run executed the loop a million times and the cold branches never, so the compiler spent code size where the time is. The run was on AArch64; the call counts come from objdump -d app.

Trade-offs and what flips the choice

• Optimizing for size (-Os) keeps the same parameter limits as -O2 (15 and 6 here) but treats every call site as cold, so an inline that grows the code is refused: on the PGO source above -Os printed call is cold and code would grow by 26 for each of the three sites, and all three stayed calls. A firmware build with a flash limit should check the final size after raising optimization, because inlining a function into many call sites multiplies its size.
• Hot loop, small function: inline. This is the case that justifies it, and it is where always_inline is reasonable.
• Large function called from many places: keep the call. Duplicated bodies grow the instruction footprint, which can slow the program through instruction-cache misses even though each call is cheaper.
• Debugging and profiling: inlined code has no stack frame of its own, so a profiler attributes its time to the caller unless it reads the inline debug information.
• PGO only helps if the training run resembles production. A profile from an unrepresentative input marks the wrong sites hot, and the build then optimizes the wrong paths.
• LTO increases link time and memory use; the benefit is largest for small helper functions in other files, as clamp_add shows.

Direct answer

The -O flags choose how many optimization passes the compiler runs and what it is willing to trade. -O0 does almost none (fastest compile, best debugging). -O1 and -O2 add safe transformations, with -O2 the usual release setting. -O3 adds more aggressive loop and cloning work that can grow code. -Os is -O2 tuned for size. Vectorization, mentioned throughout, means using SIMD (single instruction, multiple data) instructions that work on several values at once; on x86-64 these are the SSE instructions on 128-bit xmm registers, each holding four 32-bit floats or ints. IEEE here means IEEE 754, the standard that fixes how floating-point numbers are rounded, which is why strict builds give predictable sums. -Ofast is -O3 plus options that break strict IEEE and ISO C floating-point rules (fast math), so it can change numeric results. For a release build, use -O2 (or -Os when flash or cache is the constraint) and reach for -O3 or -Ofast only after measuring and, for -Ofast, after checking your numerics.

The levels side by side

Facts in this table come from the GCC manual's Optimize Options page and from running GCC 14.4 (x86-64, emulated under linux/amd64; -Q --help=optimizers lists which passes a level enables).

These flag lists are for reading, not for setting by hand: you normally choose the level and leave the individual passes alone. Passes enabled at -O3 and not at -O2 in GCC 14.4 include -floop-interchange (swap nested loops to walk memory in a cache-friendly order), -funswitch-loops (move a loop-invariant if out of the loop), -fpeel-loops (copy the first few iterations out of a loop), -fipa-cp-clone (clone a function so a constant argument can be propagated into the copy), -ftree-loop-distribution (split one big loop into several) and -fsplit-loops. -floop-unroll-and-jam (unroll an outer loop and fuse the copies of the inner loop) is also on at -O3 but not -O2. The unrolling, peeling and cloning in these lists are why -O3 code is larger. Passes -Ofast switches on beyond -O3 (the full -Q difference also includes -funsafe-math-optimizations, -fcx-limited-range and -fexcess-precision=fast) include the fast-math group: -fassociative-math (re-associate operands, so sums may be added in another order), -ffinite-math-only (assume no NaN or infinity), -freciprocal-math (use a reciprocal instead of dividing), -fno-signed-zeros (ignore the sign of zero), -fno-trapping-math (assume floating-point operations raise no user-visible traps), -fno-math-errno (do not set errno after single-instruction math functions such as sqrt) and -fallow-store-data-races (allow optimizations that may introduce new data races on stores). Of these, -fassociative-math is the one that changes the sum in the example below.

Worked example

The test program, compiled once per flag (add -fno-asynchronous-unwind-tables -fcf-protection=none for tidy listings):

#include <stdio.h>

static int square(int x) { return x * x; }

__attribute__((noinline)) int sum_sq(const int *a, int n)
{
    int s = 0;
    for (int i = 0; i < n; i++)
        s += square(a[i]);
    return s;
}

__attribute__((noinline)) float sum_f(const float *a, int n)
{
    float s = 0.0f;
    for (int i = 0; i < n; i++)
        s += a[i];
    return s;
}

int main(int argc, char **argv)
{
    (void)argv;
    int n = 4 * argc + 4;               /* 8 when run with no arguments */
    float f[8] = {1e8f, 1, 1, 1, -1e8f, 1, 1, 1};
    int v[8] = {1, 2, 3, 4, 5, 6, 7, 8};
    printf("sum_sq = %d\n", sum_sq(v, n));
    printf("sum_f  = %g\n", sum_f(f, n));
    return 0;
}

Sizes come from size (the text column covers the whole program including the C startup code common to every build), and objdump -d on the result:

The xmm count is the number of times objdump -d shows an xmm register inside sum_sq; any nonzero count means the compiler emitted SIMD code, and it says nothing about speed. A sum_sq loop at -O3 starts like this (Intel syntax, abridged): movdqu xmm0, XMMWORD PTR [rax] loads four ints, a few pmuludq/pshufd instructions square them four at a time, and paddd xmm2, xmm1 adds the four squares into four running totals, while add rax, 16 moves 16 bytes (four ints) per iteration.

sum_sq prints 204 at every level, as it must for integers. sum_f shows what -Ofast really changes. The exact sum is 6 (1e8 minus 1e8 plus six ones). Strict left-to-right float addition gives 3, because 1e8 plus 1 rounds back to 1e8 in single precision, losing three of the ones. With -ffast-math (so also -Ofast) the compiler reorders the additions into four vector lanes. The -Ofast loop of sum_f is movups xmm2, XMMWORD PTR [rax] (load four floats), add rax, 16, addps xmm0, xmm2 (add them lane by lane into four running totals). With the eight inputs 1e8, 1, 1, 1, -1e8, 1, 1, 1 the first pass puts 1e8, 1, 1, 1 in the lanes and the second adds -1e8, 1, 1, 1, so the lanes end as 1e8 - 1e8 = 0, 1 + 1 = 2, 2 and 2. Adding the lanes together gives 0 + 2 + 2 + 2 = 6: the big values cancel before any one is added to a small one, so no ones are lost. Neither answer is "wrong" under its flag; the second simply is not the sum the language rules define, and the difference depends on the order the compiler picks.

Debugging the same function in gdb on AArch64 (GDB 16.3), successive next commands at -O0 went through lines 7, 8 and 9 of the source, one line per step. At -O2 the breakpoint landed on line 8, the first next went to line 9, and the second went to line 3, the body of the inlined square, because there is no separate function any more. That is the practical debuggability cost of inlining: step and breakpoint locations stop matching what you wrote.

How to choose

• Daily debugging: -Og -g (-Og optimizes while keeping the debugging experience in mind, and the manual calls it the level of choice for the edit-compile-debug cycle; -g adds debug information) or -O0 when you need every variable.
• Release for servers and desktop: -O2. It adds nearly every optimization that does not trade space for speed, per the manual.
• Flash or instruction-cache limited firmware: -Os, then check that timing-critical loops still meet their deadlines.
• Numeric kernels: try -O3 and measure; do not assume it is faster, because the larger code can lose to -O2 on cache.
• -Ofast only for code where reassociation and the loss of NaN and infinity checks (the ISO C standard's floating-point rules, which -Ofast relaxes) are acceptable, with a test that compares results against a strict build.

Pitfalls

Code with undefined behaviour (signed overflow, out-of-bounds reads, uninitialized variables) often works at -O0 and fails at -O2 or -O3, because the optimizer is allowed to assume the behaviour never occurs; a bug that "disappears" at -O0 is a clue, not a fix. -ffast-math also assumes values are finite, so isnan style checks can be removed. And compile time is a property of the whole build: the manual's statement is that -O1 costs somewhat more time than -O0 and -O2 more again, not a fixed multiplier.

Direct answer

At -O0 the compiler translates each source line literally: every variable lives in a stack slot, is reloaded before each use, and both recursive calls remain. At -O2 it keeps values in registers, may inline the function into itself, and may rewrite one of the two recursive calls into a loop. The optimized listing is therefore shorter per call but can be far longer overall, has fewer or different calls, and no longer lines up with the source. The debugger behaves differently because the debug information has to describe values that move between registers, exist only in some address ranges, or no longer exist as separate frames.

The source and the listings

#include <stdio.h>
#include <stdlib.h>

unsigned long fib(unsigned n)
{
    if (n < 2)
        return n;
    return fib(n - 1) + fib(n - 2);
}

int main(int argc, char **argv)
{
    printf("%lu\n", fib(argc > 1 ? (unsigned)atoi(argv[1]) : 10));
    return 0;
}

The listings below are gcc -S output. The first two are AArch64 (GCC 14.4, run natively), showing only the fib function (main and the format-string data are omitted), in the GNU assembler syntax where the destination operand comes first: mov w20, w0 copies w0 into w20. x0 is a 64-bit register and w0 is its low 32 bits. [sp, -48]! is a pre-indexed address: the stack pointer is lowered by 48 bytes first and the access happens at the new address. ldp and stp load and store a pair of registers. Directives and .LFB/.LFE labels are removed, and the # comments to the right of the instructions were added by hand; the instructions themselves are as emitted. This is the -O0 listing:

fib:
	stp	x29, x30, [sp, -48]!    # save frame pointer and return address
	mov	x29, sp
	str	x19, [sp, 16]           # save a callee-saved register
	str	w0, [sp, 44]            # spill the argument n to memory
	ldr	w0, [sp, 44]            # ...and immediately reload it
	cmp	w0, 1
	bhi	.L2
	ldr	w0, [sp, 44]
	b	.L3
.L2:
	ldr	w0, [sp, 44]
	sub	w0, w0, #1
	bl	fib                     # fib(n - 1)
	mov	x19, x0
	ldr	w0, [sp, 44]
	sub	w0, w0, #2
	bl	fib                     # fib(n - 2)
	add	x0, x19, x0
.L3:
	ldr	x19, [sp, 16]
	ldp	x29, x30, [sp], 48
	ret

bhi is "branch if higher (unsigned)" and bls is "branch if lower or same (unsigned)"; both read the flags set by the preceding cmp. x19 to x28 are callee-saved registers: a function that uses one must save the caller's value and restore it before returning, which is why fib stores x19 at the top and reloads it at the end.

-O1 keeps the same two calls but holds n in a register (mov w19, w0, with no stores to a stack slot for n). -O2 -fno-inline (-fno-inline stops GCC copying a function's body into its callers, so fib stays a real function and the listing stays short) is different in kind:

fib:
	stp	x29, x30, [sp, -48]!
	mov	x29, sp
	stp	x19, x20, [sp, 16]
	mov	w20, w0
	stp	x21, x22, [sp, 32]
	mov	x21, 0                  # running sum of the results
	cmp	w0, 1
	bls	.L4
	sub	w0, w0, #2              # w0 = n - 2
	sub	w22, w20, #3            # w22 = n - 3
	and	w0, w0, -2              # clear the low bit: n - 2 rounded down to even
	sub	w19, w20, #1            # w19 = n - 1, the first argument to pass
	sub	w22, w22, w0            # w22 = loop stop value (-1 if n is even, 0 if odd)
	mov	x21, 0
.L3:
	mov	w0, w19
	sub	w19, w19, #2
	bl	fib                     # only ONE call site left
	add	x21, x21, x0
	cmp	w19, w22
	bne	.L3                     # the second call became a loop
	and	w20, w20, 1
.L4:
	add	x0, x21, w20, uxtw
	ldp	x19, x20, [sp, 16]
	ldp	x21, x22, [sp, 32]
	ldp	x29, x30, [sp], 48
	ret

Reading the loop. Calling fib on n means computing fib(n-1) + fib(n-2). Because fib(n-2) is itself fib(n-3) + fib(n-4), the compiler rewrites the second call as more calls with arguments n-3, n-5, and so on, which is a loop that steps the argument down by 2. The register w19 holds the next argument (n-1, then n-3, ...), x21 is the running total (an accumulator), and w22 is the value at which to stop. The four setup instructions compute that stop value: (n - 3) - ((n - 2) rounded down to even) is -1 for even n and 0 for odd n, so the loop ends after calling fib on every argument of the right parity from n-1 down to 1 or 2. What is left over after the loop is fib(1) for odd n or fib(0) for even n, which is n & 1; the and w20, w20, 1 line computes it and the final add includes it. Worked check with n = 6: the loop calls fib(5), fib(3), fib(1) (stops when the argument reaches -1) giving 5 + 2 + 1 = 8, plus 6 & 1 = 0, so 8, which is fib(6). Written this way, with a running total passed along instead of an addition after each call returns, the rewrite is called the accumulator form.

The same program at -O0 and -O2 -fno-inline on x86-64 (Intel syntax from -masm=intel, destination first; GCC 14.4 in a linux/amd64 container; only the fib function is shown, with main, the format-string data and all assembler directives such as the .cfi_* lines removed, the .LFB6 / .LFE6 labels removed, and the # comments, including the # -O0 and # -O2 -fno-inline tags after fib:, added by hand) shows the same pattern with different registers (edi is the argument, rbx, rbp and r12 are callee-saved):

fib:                              # -O0
	push	rbp
	mov	rbp, rsp
	push	rbx
	sub	rsp, 24
	mov	DWORD PTR [rbp-20], edi     # spill n to memory
	cmp	DWORD PTR [rbp-20], 1
	ja	.L2
	mov	eax, DWORD PTR [rbp-20]
	jmp	.L3
.L2:
	mov	eax, DWORD PTR [rbp-20]
	sub	eax, 1
	mov	edi, eax
	call	fib                         # fib(n - 1)
	mov	rbx, rax
	mov	eax, DWORD PTR [rbp-20]
	sub	eax, 2
	mov	edi, eax
	call	fib                         # fib(n - 2)
	add	rax, rbx
.L3:
	mov	rbx, QWORD PTR [rbp-8]
	leave
	ret

fib:                              # -O2 -fno-inline
	push	r12
	xor	r12d, r12d                  # running total = 0
	push	rbp
	mov	ebp, edi
	push	rbx
	cmp	edi, 1
	jbe	.L3
	mov	ebx, edi
.L2:
	lea	edi, [rbx-1]                # next argument
	sub	ebx, 2
	call	fib                         # the only call
	add	r12, rax
	cmp	ebx, 1
	ja	.L2                         # loop
	and	ebp, 1                     # leftover term: n & 1
.L3:
	mov	eax, ebp
	pop	rbx
	pop	rbp
	add	rax, r12
	pop	r12
	ret

Counting the instruction lines in the printed AArch64 listings (labels and directives are not instructions) gives 20 at -O0 and 26 at -O2 -fno-inline. Counting the same way in the full listings, -O1 has 17 instructions and plain -O2 has 179, with 2, 2, 1 and 1 bl fib instructions in the -O0, -O1, -O2 -fno-inline and -O2 builds respectively. All four builds print 0, 1, 8, 55 and 610 for inputs 0, 1, 6, 10 and 15. The plain -O2 build is large because GCC also inlines fib into itself several levels deep and still ends with a call; the nine breakpoint locations in the gdb output below are those inlined copies. The exact numbers depend on the GCC version and target, so treat them as an illustration of the pattern, not a constant.

What changes, and why

Why it is legal: only observable behaviour (the returned value and external effects) must be preserved, so the compiler may reorder, merge, duplicate or delete work as long as the result is the same.

Why the debugger looks different

Build with -g (debug info: DWARF tables that map addresses to source lines and say where each variable lives at each address) and run gdb -batch -ex 'break fib' -ex 'run 6' -ex 'bt' -ex 'info args' -ex 'continue 3' -ex 'bt 6' -ex 'info args' ./fib_g (GDB 16.3, AArch64 container; output trimmed to the relevant lines: the library-loading and ASLR warning lines, the blank lines, the echoed source lines (such as 6 if (n < 2)) and the second info args result are removed, and stack addresses such as argv=0x... differ from run to run).

-O0 -g:

Breakpoint 1 at 0x400694: file fib.c, line 6.
Breakpoint 1, fib (n=6) at fib.c:6
#0  fib (n=6) at fib.c:6
#1  0x000000000040070c in main (argc=2, argv=0xfffffffffb08) at fib.c:13
n = 6
Breakpoint 1, fib (n=3) at fib.c:6
#0  fib (n=3) at fib.c:6
#1  0x00000000004006b4 in fib (n=4) at fib.c:8
#2  0x00000000004006b4 in fib (n=5) at fib.c:8
#3  0x00000000004006b4 in fib (n=6) at fib.c:8
#4  0x000000000040070c in main (argc=2, argv=0xfffffffffb08) at fib.c:13

-O2 -g:

Breakpoint 1 at 0x400720: fib. (9 locations)
Breakpoint 1.1, fib (n=6) at fib.c:6
#0  fib (n=6) at fib.c:6
#1  0x0000000000400598 in main (argc=<optimized out>, argv=<optimized out>) at fib.c:13
Breakpoint 1.4, fib (n=3) at fib.c:6
#0  fib (n=3) at fib.c:6
#1  fib (n=4) at fib.c:8
#2  fib (n=<optimized out>) at fib.c:8
#3  fib (n=<optimized out>) at fib.c:8
#4  0x0000000000400598 in main (argc=<optimized out>, argv=<optimized out>) at fib.c:13

Reading the differences:

• "9 locations": the one source function now exists at nine addresses because copies were inlined into each other. One breakpoint covers all of them.
• Frames without an address (#1 fib (n=4) has no 0x... before it): these are inlined frames. The debugger reconstructs them from debug information; there is no call instruction and no stack frame behind them.
• <optimized out>: the value is not available at that program counter, because the register that held it was reused. It does not mean the program is wrong.
• Stepping at -O2 jumps between lines because the compiler interleaves instructions from different lines; at -O0 it goes in source order.

Pitfalls and recommendation

For source-level debugging build with -Og (GCC's optimization level meant to keep debugging usable) or -O0, and for a bug that appears only at -O2 debug the real optimized build: use disassemble /s and stepi rather than trusting line stepping. Diffing the two listings is for understanding the compiler, not for finding the bug: the difference between builds is expected everywhere, so only the one change that alters observable behaviour matters.

Direct answer

Four tools run in sequence: the preprocessor (expands #include and #define), the compiler proper (turns C into assembly text), the assembler (turns assembly into an object file, .o, which is an ELF file with unresolved references), and the linker (merges object files and libraries into the final ELF executable). Each question in the prompt is decided at a different stage:

Everything below was run with GCC 14.4 on AArch64 (native, gcc:14 container) against this file; the x86-64 lines come from building the same file with GCC 14.4 in an amd64 gcc:14 container. The defaults shown are this toolchain's; another distribution's compiler can enable PIE, canaries or full RELRO by default, so check the output rather than assuming.

#include <stdio.h>
#include <string.h>

#define LIMIT 8

static int helper_count;

__attribute__((visibility("default"))) int api_greet(const char *name)
{
    char buf[LIMIT];
    strcpy(buf, name);
    helper_count++;
    printf("hello %s (%d)\n", buf, helper_count);
    return helper_count;
}

int internal_total(void)
{
    return helper_count * 2;
}

int main(int argc, char **argv)
{
    return api_greet(argc > 1 ? argv[1] : "world") + internal_total();
}

Stage 1: preprocess

gcc -E greet.c expands macros and pastes in headers (about 1170 lines for this 25-line file: gcc -E greet.c | wc -l printed 1170 here). char buf[LIMIT]; becomes char buf[8];. Nothing about symbols, hardening or layout is decided here, except that -D flags change which code exists.

Stage 2: compile

The compiler picks instructions and also decides stack protection. A stack canary is a secret value placed between local buffers and the saved return address and checked before the function returns; if a buffer overflow overwrote it, the function calls __stack_chk_fail instead of returning. With default flags gcc -O2 -S greet.c emits no canary (grep -c stack_chk finds 0 matches). Adding -fstack-protector-strong makes api_greet (it has a local array) get one. Lines of greet.s (from gcc -O2 -fstack-protector-strong -S greet.c -o greet.s) that contain stack_chk, as grep -n stack_chk greet.s prints them (the number is the line in greet.s; the ... stands for two more matching lines, 40 and 43, the second adrp and the ldr of the guard):

18:	adrp	x2, __stack_chk_guard
19:	add	x2, x2, :lo12:__stack_chk_guard
...
58:	bl	__stack_chk_fail

Syntax key (AArch64, destination first): adrp x2, sym puts the address of the 4 KiB page containing sym in x2, add x2, x2, :lo12:sym adds the low 12 bits of the address (the offset inside that page), and bl is a call. The prologue builds the guard's address in x2 and copies the guard value into the frame; just before returning, the function loads the guard again (a second adrp and an ldr that the excerpt omits), compares it with the saved copy and branches to the bl __stack_chk_fail on a mismatch. On x86-64 the same protection looks different: GCC 14.4 reads the guard from thread-local storage, as mov rsi, QWORD PTR fs:40 in the prologue and sub rdx, QWORD PTR fs:40 in the epilogue (Intel syntax, as printed by gcc -masm=intel -S; GCC's default AT&T syntax prints movq %fs:40, %rsi and subq %fs:40, %rdx), followed by call __stack_chk_fail on a mismatch (call __stack_chk_fail@PLT when the file is built with -fPIE); there is no __stack_chk_guard symbol to relocate.

The decision is entirely a compile-time one, made per function, and only functions with vulnerable-looking locals get it under -strong.

The compiler also fixes position independence. Position-independent code (PIC) avoids baked-in absolute addresses so the loader can place the program anywhere. A relocation is a note the assembler leaves for the linker saying "patch this instruction with the final address of that symbol", and readelf -r greet.o lists them. The same call site differs in them (each excerpt line below is the first pair of readelf -rW relocation lines for __stack_chk_guard, cut down to its Type and Symbol Name columns: the Offset and Info columns and the trailing + 0 addend are left out, and the ... stands for the Symbol's Value column). With -fno-pic, the guard is reached through page-relative relocations (PREL in the name means the offset is computed from the instruction's own address, which assumes the symbol sits at a fixed distance from the code):

R_AARCH64_ADR_PREL_PG_HI21 ... __stack_chk_guard
R_AARCH64_ADD_ABS_LO12_NC  ... __stack_chk_guard

With -fPIE (position-independent executable code) it goes through the GOT (global offset table, a table of addresses filled in at load time): the instructions become adrp to the page of the symbol's GOT slot and an ldr that loads the symbol's address out of that slot, so the symbol may live anywhere, for example in a shared library:

R_AARCH64_ADR_GOT_PAGE     ... __stack_chk_guard
R_AARCH64_LD64_GOT_LO12_NC ... __stack_chk_guard

The change between the two builds is therefore that a direct address calculation became a load through a table the loader fills in. For a smaller reproduction, int get(void) { return counter; } with extern int counter; assembles to adrp plus ldr w0, [x0] with R_AARCH64_ADR_PREL_PG_HI21 and R_AARCH64_LDST32_ABS_LO12_NC under -fno-pic, and to adrp, ldr x0, [x0], ldr w0, [x0] with R_AARCH64_ADR_GOT_PAGE and R_AARCH64_LD64_GOT_LO12_NC under -fPIE. On x86-64 both builds of that function use R_X86_64_PC32 because x86-64 addresses data relative to the instruction pointer anyway; there the visible difference between the builds is mostly absolute 32-bit addresses, such as the one in the linker error below.

Stage 3: assemble

The assembler produces the object file and records each symbol with a binding (local, visible only inside this file, or global, visible to other files) and a visibility (whether a global symbol may be exported from the finished executable or library). nm greet.o (compiled with -O2 -fstack-protector-strong) shows api_greet, internal_total and main as T (global code), helper_count as lowercase b (local, zero-initialised because of static), and printf, strcpy, __stack_chk_fail, __stack_chk_guard as U (undefined, to be resolved by the linker). So static is a source-level decision that becomes a local symbol that no other file can reference. With -fvisibility=hidden the compiler marks the global symbols HIDDEN (readelf -sW: internal_total and main show GLOBAL HIDDEN; api_greet, which the source marks visibility("default"), stays GLOBAL DEFAULT). Hidden symbols still link across object files inside one executable or library, but the linker will not export them.

Stage 4: link

The linker resolves the U symbols, lays out sections and writes the ELF file. Its decisions here:

• Export table (the dynamic symbol table, .dynsym: the subset of symbols the loader and other libraries can see). Building a shared library from the same file, default flags exported api_greet internal_total main (from nm -D --defined-only); with -fvisibility=hidden only api_greet was exported. That is how visibility turns into what is visible in the final file.
• PIE. gcc -O2 greet.c produced ELF type EXEC (fixed addresses); gcc -fPIE -pie produced DYN (Position-Independent Executable file), with the loader free to choose a random base address (ASLR, address space layout randomisation) at each run. The flag has to be consistent at both stages: linking the -fno-pic object with -pie fails. A PIE is linked the way a shared object is: it is loaded at a random base, and a symbol such as __stack_chk_guard comes from libc at an address the linker cannot know. An instruction that assumes a fixed distance to the symbol (AArch64's page-relative adrp pair) or holds a 32-bit absolute address (x86-64's R_X86_64_32) could only be made right by rewriting code at load time, so the linker refuses; that is what can not be used when making a shared object and recompile with -fPIC mean:

/usr/bin/ld: nopic.o: relocation R_AARCH64_ADR_PREL_PG_HI21 against symbol `__stack_chk_guard@@GLIBC_2.17' which may bind externally can not be used when making a shared object; recompile with -fPIC

This is the first line of the linker's output; it is followed by an unresolvable R_AARCH64_ADR_PREL_PG_HI21 relocation line, final link failed: bad value and the collect2: error: ld returned 1 exit status line.

On x86-64 the same mistake prints, as its first line:

/usr/bin/ld: nopic.o: relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE

Here R_X86_64_32 is an absolute 32-bit address of the string constant.

• RELRO (relocation read-only: after the loader fills in the GOT it marks parts of the data read-only so a later exploit cannot overwrite them). The default link had a GNU_RELRO program header but no BIND_NOW flag (partial RELRO: with lazy binding, an imported function is resolved on its first call through the PLT, so its GOT entry must stay writable until then). Linking with -fPIE -pie -Wl,-z,relro,-z,now printed FLAGS BIND_NOW and FLAGS_1 Flags: NOW PIE (readelf -d; the same link without -fPIE -pie ends in Flags: NOW): every symbol is resolved at start-up, so the whole GOT can be made read-only (full RELRO). The linker records the request; the dynamic loader enforces it when the program starts.

What a reviewer can still learn from a stripped binary

strip removes the static symbol table (.symtab): readelf -S shows the section before and not after, and nm greet_stripped prints no symbols. It leaves the dynamic symbol table, which the loader needs. Running nm -D on the stripped hardened binary still lists the imports printf, strcpy, abort, __libc_start_main, __stack_chk_fail and __stack_chk_guard (and a few weak runtime hooks such as __cxa_finalize and __gmon_start__, left out of this list). From that alone a reviewer learns:

• the canary is present (the __stack_chk_* imports), and strcpy, an unbounded copy, is called: a place to look;
• PIE and RELRO status from readelf -h, -l, -d (the program headers and dynamic section are untouched);
• string constants such as hello %s (%d) (strings), which often reveal function purpose;
• all the machine code, with calls through the PLT (procedure linkage table, the stubs for imported functions) to strcpy, printf and __stack_chk_fail visible in objdump -d;
• a build ID only if one was requested: -Wl,--build-id produced a Build ID: line in the stripped file (a 40-hex-digit value, for example cef9c676...; the digits differ from build to build, so treat it as an example), whereas the default link here had none. A build ID lets you match a binary to its separate debug file.

What is gone: the names of local functions and static variables (helper_count, and the name of any static function), and, if the build used -g, the types and line numbers (kept in debug sections that strip also removes).

Recommendation

For a hardened Linux release: compile with -fstack-protector-strong -fPIE, link with -pie -Wl,-z,relro,-z,now, mark exported API functions explicitly and build with -fvisibility=hidden, keep an unstripped copy plus build ID for crash analysis and ship the stripped one. Then verify the result from the binary itself (readelf -h -l -d, nm -D) instead of trusting the flag list, because each flag acts at a different stage and a missing one at either stage silently weakens the result.

Direct answer

The debugger does not read your source: it reads the machine code and a separate table of debug information that says "at this instruction address, variable x lives in this register or stack slot, and these instructions came from this line". At -O0 that table is simple, because every variable has its own stack slot for the whole function and the instructions follow the source order. When the optimizer is on, it keeps a value in a register only while it is needed, reuses that register afterwards, merges or deletes computations, and interleaves instructions from different lines. The table then has to say "valid only between these two addresses", or nothing at all. So variables seem to disappear (<optimized out>), lines seem to run out of order, and a function can seem to have no frame of its own. The program still behaves the same; the picture the debugger can draw of it does not.

An analogy for a developer who expects one line, one step

Think of a recipe written line by line, and a cook who reads the whole recipe first, chops all the onions for steps 2, 5 and 7 together, and throws away a step that produces something nobody uses. Someone watching the kitchen with the recipe in hand sees step 5 happen before step 3 and no sign of step 4. The recipe (source) describes what the dish must be; it does not promise the order of the work. Optimizers are allowed to do exactly that, because only the observable result must stay the same.

A concrete session

#include <stdio.h>
#include <stdlib.h>

__attribute__((noinline)) unsigned rescale(const unsigned *v, int n)
{
    unsigned tag = (unsigned)rand();        /* result never read again */
    unsigned sum = 0;
    for (int i = 0; i < n; i++)
        sum += v[i] * 3;
    return sum;
}

int main(void)
{
    unsigned v[4] = {1, 2, 3, 4};
    printf("%u\n", rescale(v, 4));
    return 0;
}

tag receives a result that is never read again. Built twice with gcc -g (GCC 14.4, AArch64, run natively), and driven by gdb -batch -ex 'break rescale' -ex 'run' -ex 'info locals' -ex 'next' -ex 'next' -ex 'next' -ex 'next' -ex 'info locals' -ex 'print tag' -ex 'info address tag' ./rs (GDB 16.3; each run's Breakpoint 1 at ... line, the library-loading lines, and a closing line holding a single . after the info address output are removed, and the values shown for v and the uninitialized slots depend on the stack contents of that run):

-O0 -g:

Breakpoint 1, rescale (v=0xfffffffff980, n=4) at rescale.c:6
6	    unsigned tag = (unsigned)rand();        /* result never read again */
tag = 65535
sum = 65535
7	    unsigned sum = 0;
8	    for (int i = 0; i < n; i++)
9	        sum += v[i] * 3;
8	    for (int i = 0; i < n; i++)
i = 0
tag = 1804289383
sum = 3
$1 = 1804289383
Symbol "tag" is a complex DWARF expression:
     0: DW_OP_fbreg -12

-O2 -g:

Breakpoint 1, rescale (v=v@entry=0xfffffffff980, n=n@entry=4) at rescale.c:6
6	    unsigned tag = (unsigned)rand();        /* result never read again */
tag = <optimized out>
sum = <optimized out>
8	    for (int i = 0; i < n; i++)
9	        sum += v[i] * 3;
8	    for (int i = 0; i < n; i++)
9	        sum += v[i] * 3;
i = 1
tag = <optimized out>
sum = 3
$1 = <optimized out>
Symbol "tag" is optimized out.

What each difference means:

• Before its line runs, a variable holds garbage (tag = 65535, sum = 65535 at -O0): its stack slot has not been written yet. The debugger shows the slot, not an "unset" marker.
• tag = <optimized out> at -O2: rand() must still be called (it has side effects), but its result is not needed, so no register or slot keeps it. info address tag says Symbol "tag" is optimized out. There is nothing to show. At -O0, the same variable is a stack slot at frame offset -12 (DW_OP_fbreg -12, a DWARF location expression: "frame base minus 12").
• sum = <optimized out> at the start, then sum = 3 later: the variable has a location only in a range of addresses, after the instruction that creates it.
• Line 7 never appears in the -O2 stepping: unsigned sum = 0; generated no instruction of its own, so there is no place to stop. The loop's lines interleave (8, 9, 8, 9) because the compiler places the test and the body together.
• v=v@entry: GDB is saying "this is the value the argument had on entry", which it can recover even after the register was reused.

What to tell the developer, and what to do

1. The debugger is reading a map drawn for the unoptimized layout; the optimized code is correct, just not shaped like the source.
2. For source-level stepping, build with -Og or -O0 (or -O0 for only the file under investigation).
3. If the bug only exists in the optimized build, debug that build: disassemble /s shows source lines interleaved with instructions, info address var shows where the variable lives over which ranges, and stepi steps one instruction at a time.
4. Adding volatile to make a variable visible changes the code being debugged, so it can hide the bug it was added to find.