Kernel Architecture & OS Internals Questions
How an operating system kernel is structured and what it is responsible for: kernel subsystems (scheduler, memory manager, drivers, interrupt handling), kernel vs. user space and the cost of crossing between them, and the boot and initialization path. Includes the core mechanisms the kernel provides: process and thread lifecycle (creation, zombies and orphans, uninterruptible sleep) and context switching, processes vs. threads as an architectural choice, CPU scheduling (CFS, priorities, preemption models, real-time policies on a general-purpose kernel), virtual memory (address translation, multi-level page tables and entry formats, TLB misses and shootdowns, page faults, copy-on-write, memory-mapped files, userfaultfd, swapping, page replacement, kernel allocators, process address-space layout, page-level protection such as NX and ASLR as mechanisms, hugepages, page coloring, overcommit and the OOM killer as kernel mechanisms), interrupts and softirqs, and how loadable modules and device drivers extend the kernel. Includes worked exercises such as page-replacement traces, address-translation arithmetic and small page-table, TLB or scheduler simulations. Covers the concepts and internals, not microcontroller interrupt and ISR design, RTOS and real-time scheduling theory, day-to-day host administration, system-call and POSIX API semantics, OS-level performance tuning, or forensic and security analysis of a host.
Why do operating systems split execution into kernel space and user space? What can code in each mode do, and what actually happens in the CPU when a program needs something only the kernel can do?
Sample Answer
Direct answer
The CPU runs code in at least two privilege levels (x86 calls them rings, ring 0 being the most privileged and ring 3 the least; Arm calls them exception levels, EL1 for the kernel and EL0 for user code). The kernel (the core of the operating system that owns the hardware) runs in the privileged level, and ordinary programs run in the unprivileged one, called user space. The split exists so a buggy or malicious program cannot overwrite another program, read its memory, or reprogram the disk or the interrupt controller. When a program needs something only the kernel may do, it executes a special instruction (a system call, or syscall) that makes the CPU jump to a fixed kernel entry point at full privilege, and the kernel returns to the program when done.
What each mode can do
| Kernel mode (x86 ring 0, Arm exception level 1) | User mode (x86 ring 3, Arm exception level 0) | |
|---|---|---|
| Instructions | All, including loading the page-table base register (the register, CR3 on x86, that points at the top-level page table), masking interrupts (telling the CPU to hold off device signals), halting the CPU, reading and writing model-specific registers (configuration registers for CPU features, reachable only by privileged instructions), port I/O (the x86 in and out instructions that talk to devices through a separate numbered address space) | Ordinary arithmetic, loads, stores and jumps. A privileged instruction raises a fault, and the kernel typically kills the program with a signal |
| Memory | Sees all physical memory and every process's mappings | Only the pages its own page table marks user-accessible; the kernel half of the address space is present but the supervisor-only bit (a flag in each page-table entry marking the page as kernel-only) blocks access |
| Hardware | Talks to devices, programs the timer and interrupt controller | Cannot touch devices directly; asks the kernel through syscalls |
| Failure | A bug can crash or corrupt the whole machine | A bug kills that one process |
Things that genuinely require kernel privilege: creating or switching page tables (the data structures that map a program's addresses to physical memory), changing which process runs, enabling or disabling interrupts, accessing device registers, and opening files or sockets on the program's behalf.
What happens in the CPU on a syscall (x86-64 Linux)
- The program puts the syscall number in
rax(1 meanswrite) and arguments inrdi,rsi,rdx, then runs thesyscallinstruction. (C code normally does this through a libc wrapper such aswrite().) - The CPU, in one instruction, saves the return address in
rcxand the flags inr11, loads the instruction pointer from a model-specific register the kernel set at boot, raises the privilege level to ring 0, and masks some flags. It does not switch stacks by itself. - The kernel's entry code switches to this thread's kernel stack, saves the user registers, and indexes a table of handlers by the syscall number.
- The handler validates arguments (user pointers are checked and copied with special routines, because user memory cannot be trusted), does the work, and puts the result in
rax(on failure a negative errno, the numeric error code such as 2 for 'no such file'). sysret(orireton some paths) restores the user registers and drops back to ring 3 at the saved address.
Arm64 follows the same shape: svc raises an exception from EL0 to EL1, the syscall number goes in x8. Interrupts and faults (a page fault, a timer tick) enter the kernel through a similar privilege-raising path, but the CPU starts them rather than the program.
The crossing is not free: it is a pipeline-disrupting jump, a stack change and, on CPUs that need Meltdown mitigations, a page-table switch. (Meltdown is a CPU flaw that let user code speculatively read kernel memory; kernel page-table isolation removes most kernel mappings from the page tables a program runs on, so entering the kernel must swap in the full set.) So code that crosses constantly (one syscall per byte) is slow, and libraries batch work (buffered I/O) to cross less often. Some read-only data such as the clock can be served without entering the kernel at all through the vDSO (virtual dynamic shared object), a small kernel-provided library mapped into every process; this is why clock_gettime rarely shows up as a real syscall.
Observing the transitions
strace -f ./prog lists every syscall a program makes with arguments and return values (it uses the kernel's ptrace facility, a system call that lets one process observe and control another, so it slows the target). perf trace does the same with lower overhead. strace -c summarizes counts. These show user-to-kernel crossings only; they cannot show what the kernel does inside a call.
Worked example
printf("hi\n") writes into a user-space buffer. Only when the buffer is flushed does libc call write(1, buf, 3): one crossing for three bytes instead of three crossings. In registers that is rax=1 (write), rdi=1 (file descriptor 1, standard output), rsi=the address of buf, rdx=3 (the byte count), then syscall. The kernel handler checks that descriptor 1 is open, copies 3 bytes from the user buffer into the terminal or pipe, and returns 3 in rax.
Security consequence and pitfalls
- Everything user space believes about the system comes from syscalls the kernel answers. Code running in kernel mode (a malicious driver or loadable module, or a rootkit: malware that hides itself and its activity by running with kernel privilege) can therefore lie: it can filter the results of calls that list processes, files or network connections, so
ps,lsandnetstatshow a clean system. This is why a tool running above the kernel cannot reliably prove the kernel is clean, and why defenders use trusted-boot checks (firmware verifying a signature on each boot stage before running it), kernel integrity features or offline and out-of-band inspection. - The split protects against accidents and ordinary attackers, not against a kernel bug: a flaw that lets user code run in ring 0 (privilege escalation) defeats everything the split provides.
- Do not confuse "kernel mode" with "root". Root is a user identity enforced by kernel permission checks; root's code still runs in ring 3.
- Microkernels shrink the amount of code in ring 0 and push drivers into user-space servers, trading more crossings (messages) for a smaller trusted base.
What is a loadable kernel module and how does it differ from code built into the kernel? What does it mean for a kernel to be tainted, and what are the stability and security consequences of loading third-party modules?
Sample Answer
Direct answer
A loadable kernel module is a piece of kernel code compiled separately as a .ko file (an ELF object, ELF being the standard Linux executable and object file format) that can be inserted into and removed from a running kernel with insmod, modprobe and rmmod. Once loaded it is part of the kernel: same privilege, same address space, same crash domain. Code built into the kernel image (=y in the configuration) is always present from boot; a module (=m) is loaded on demand. A kernel is "tainted" when something has happened that makes its behaviour harder to trust or debug, such as loading a proprietary, out-of-tree or unsigned module; the state is a bitmask (one number whose individual binary digits are separate yes/no flags) readable from /proc/sys/kernel/tainted, and a nonzero value tells maintainers to be cautious with bug reports. Third-party modules carry stability risk (a bug panics the whole machine) and security risk (a malicious module is a rootkit (software that hides an intruder's presence) with full power), which signing and policy are meant to control.
Module versus built-in
| Built into the image | Loadable module | |
|---|---|---|
| Present | From boot, always | After insmod or modprobe, or automatic load |
| Needed for boot | Yes if it is required before the root filesystem can be mounted (unless it is in the initramfs) | Only if packaged in the initramfs |
| Memory | Always resident | Only when loaded; can be unloaded if nothing uses it |
| Trust | Covered by the signature of the kernel image | Verified separately if module signing is enabled |
| Update | New kernel and reboot | Replace the file and reload (no reboot, if unloadable) |
| ABI (the binary-level contract between compiled code and the kernel: structure layouts, function signatures) | Compiled together with the kernel | Must match the kernel it was built for |
The kernel has no stable internal interface, so a module must be built against the exact kernel (or compatible headers) it will run on: the module records a version string and, with CONFIG_MODVERSIONS, checksums of the kernel symbols it uses. That is why out-of-tree drivers are rebuilt by tools such as DKMS (Dynamic Kernel Module Support, which recompiles registered modules automatically when a new kernel is installed) on every kernel update. Modules can only use symbols the kernel exports; some are exported to GPL-compatible modules only, so a module without a GPL-compatible MODULE_LICENSE is treated as proprietary.
How loading happens: the loading process calls finit_module (or init_module), a system call that requires the CAP_SYS_MODULE capability (the Linux privilege bit that permits loading kernel modules, normally held only by root). The kernel loads the ELF image into kernel memory, resolves its symbols, checks the signature if enforcement is on, and runs its init function. At boot, modules come in three ways: from the initramfs (the early root image carrying the storage and filesystem drivers needed to mount the real root), from udev (the device manager daemon) loading by hardware alias (an identifier string a device advertises, matched to a module) when a device is discovered, and from lists in /etc/modules-load.d. Automatic loading can be suppressed with a modprobe blacklist or the modprobe.blacklist= boot parameter, but both are read by user-space modprobe, and a blacklist entry only makes modprobe ignore the module's internal hardware aliases (so alias-based loading by udev skips it): an explicit modprobe name or insmod can still load it. To refuse a module outright, use the kernel's own module_blacklist= boot parameter (read by the kernel itself: a comma-separated list of module names that the load path then rejects, whichever tool asked), or an install name /bin/false line in a modprobe.d file for loads that go through modprobe.
What taint means
Each cause sets a bit; the kernel documentation lists the flags. The ones that matter for modules are bit 0 (letter P, a proprietary module was loaded), bit 1 (F, a module was force-loaded), bit 12 (O, an externally built "out-of-tree" module was loaded) and bit 13 (E, an unsigned module was loaded); bit 7 (D) means the kernel has oopsed (hit a recoverable internal error and printed an "oops" report, as with a bad pointer dereference), and bit 9 (W) means it issued a warning. Decoding a value by hand:
12289=8192+4096+1=213+212+20
so bits 13, 12 and 0 are set: unsigned (E), out-of-tree (O) and proprietary (P). As a program:
FLAGS = {0: "P proprietary module loaded", 12: "O out-of-tree module loaded", 13: "E unsigned module loaded",
1: "F module force-loaded", 7: "D kernel oopsed or BUG hit", 9: "W kernel warning issued"}
def decode(value):
return [FLAGS.get(b, f"bit {b}") for b in range(19) if value >> b & 1]
for v in (0, 4097, 12289):
print(v, decode(v))
print(2**13 + 2**12 + 2**0)
Output:
0 []
4097 ['P proprietary module loaded', 'O out-of-tree module loaded']
12289 ['P proprietary module loaded', 'O out-of-tree module loaded', 'E unsigned module loaded']
12289
Taint is sticky: it stays until reboot even after the module is unloaded, because the damage a bad module could have done to kernel memory does not go away. Upstream developers generally treat reports from kernels tainted by proprietary modules with suspicion because they cannot inspect that code. Taint is information, not a block: the kernel keeps running.
Stability and security consequences
The two risks call for different controls: stability comes from where the module came from and how it is tested; security comes from signing, the one-way sysctl and who holds CAP_SYS_MODULE. The bullets below follow that split.
- Stability. There is no isolation. A null pointer or a missed lock in a third-party module can corrupt any kernel structure or panic the machine, and the fault can show up far from the cause. A module also ties you to a kernel version: every kernel upgrade means a rebuild and a test, and a GPU or storage driver that lags a release blocks the upgrade.
- Security. A loaded module runs at the highest privilege and can hide processes, hook system calls or read any memory, which is how kernel rootkits work. Controls: module signing (
CONFIG_MODULE_SIG; withoutCONFIG_MODULE_SIG_FORCEor themodule.sig_enforce=1boot parameter an unsigned module still loads but taints the kernel with E, with enforcement only validly signed modules load); the sysctlkernel.modules_disabled=1, which is one-way: after it is set, modules can be neither loaded nor unloaded until reboot; and restricting who holds CAP_SYS_MODULE. - Secure Boot. Secure Boot verifies the bootloader and kernel signature, and distribution kernels commonly extend that by refusing unsigned modules while it is on. To run a third-party driver such as an out-of-tree GPU or virtualization module you then sign it with your own key and enrol that key with the firmware's machine-owner-key (MOK) mechanism, using the
mokutiltool to queue the key for enrolment at the next boot, rather than disabling Secure Boot.
Recommendation
Prefer drivers that are in the mainline kernel, load only what the machine needs, enable signing with enforcement on production fleets, pin third-party modules to tested kernel versions, and record /proc/sys/kernel/tainted in your monitoring so a tainted host is visible before a crash report is filed. What would change this: if a vendor module is the only way to use the hardware, keep it on a fixed kernel and put the rebuild-and-test step into your upgrade pipeline.
For a pipeline that parses untrusted network input, enriches it and emits results, compare one process with many threads, several cooperating processes, and a hybrid. What do you gain and lose in performance, fault isolation and debuggability?
Sample Answer
Direct answer
For a pipeline that parses untrusted network input, I would choose the hybrid: a small, locked-down parser process (or a few, one per core or per tenant, where a tenant is one customer or team whose data the service handles) that does nothing but parse and validate, handing validated records over a narrow channel to a trusted process that enriches and emits using a thread pool. Threads in one address space are the fastest and simplest design, but a single memory-safety bug in the parser then gives an attacker the whole process: every tenant's data, credentials and the output path. Separate processes give each stage its own address space (the private virtual memory the kernel gives a process; see the Performance row of the table below for the cost of crossing between two of them), so a crash or exploit is contained to that process and can be restarted. The cost is copying data across the boundary and a harder-to-trace system. The hybrid spends that cost only at the one boundary where input is hostile.
A picture of the hybrid, with one record's journey:
network bytes --> [parser process, sandboxed] --validated record--> [trusted process]
- no files, no network (fixed-size struct: - thread pool enriches
- seccomp: read/write/exit tenant id, request id, - shared cache
- setrlimit: 256 MiB, 2 s CPU parsed fields) - emits results
- supervisor restarts on crash over a Unix socket
The raw bytes arrive only at the parser. It checks size, structure and types, and sends the trusted process a small record of already-checked fields (for example tenant id, request id, and the parsed values) over a pipe or Unix socket. If the parser crashes or is exploited, the attacker holds a process that can only read and write that one channel, and a supervisor (a small process that restarts workers that die) replaces it.
The three designs
| One process, many threads | Several cooperating processes | Hybrid (isolated parser, threaded trusted stage) | |
|---|---|---|---|
| Performance | Best: threads share memory, hand off a pointer; no copy or system call to pass a record | Each handoff is a copy through a pipe or socket (two copies and two system calls per message: the sender's write copies the bytes from its memory into a kernel buffer, and the receiver's read copies them from the kernel buffer into its own memory) or a shared-memory ring that avoids the copy but needs its own synchronisation | One boundary crossing per record, with the cheap thread handoff inside the trusted side |
| Fault isolation | None: a crash, an infinite loop that holds a lock, or an exploit affects every thread; one huge allocation can exhaust memory for all | Strong: a crash kills one process; the OS enforces separate memory; per-process limits (setrlimit, a system call that caps a process's memory, CPU time or open files, and cgroups, the kernel's resource-limit groups) cap memory and CPU; a supervisor restarts it | Strong where it matters: parser faults are contained and the parser can be sandboxed (no network, no filesystem, a seccomp syscall filter: a Linux mechanism that lets a process permanently restrict itself to a short list of allowed system calls, so even an exploited parser cannot open files or sockets); enrichment keeps shared caches cheaply |
| Debuggability | Weak for races: shared mutable state, heisenbugs (bugs that vanish or change when you try to observe them, typical of timing-dependent races); but one core dump, one debugger session, one profile | Clear crash story (one core dump per failing process, strace and gdb attach to one PID) but harder end-to-end tracing across processes: you need a request ID in every message and logs | Needs the same request ID, plus tests that kill the parser at random to prove restart works |
| Memory | Lowest: one heap, one copy of caches | Each process has its own heap and runtime. Code pages are shared (see below), private data is not | Parser processes are small; the big enrichment cache exists once |
Memory overhead, with stated assumptions
Assumptions for illustration only: 200 tenants, and each isolated worker carries 25 MiB of private memory (its heap plus runtime state) while the program binary and libraries are shared copy-on-write from one page-cache copy after fork. Process-per-tenant then costs:
200 tenants x 25 MiB private = 5,000 MiB (about 4.9 GiB)
By contrast, thread-per-request in one process costs one heap plus, per thread, a stack whose resident part is small (the default 8 MiB stack size is reserved address space, not resident memory, so 200 threads cost about 200 x the touched stack, not 200 x 8 MiB). The difference is the price of isolation, and you can shrink it: share the read-only loaded state by fork-ing from a prepared parent (pages stay shared until written), and use a pool of N workers with per-tenant routing rather than literally one process per tenant when tenants are many.
Blast radius (how much is damaged when one component fails or is compromised) per tenant: with process-per-tenant an exploit or crash affects that tenant only (and the memory it was given); with a shared pool, a crafted payload from tenant A that corrupts the heap exposes tenant B's data in the same address space. That blast-radius argument, not raw speed, is what decides the tenant case when payloads are hostile or tenants are mutually untrusting; for many small, mutually trusting internal tenants the cheaper pool is defensible.
Shared-memory queue vs pipe vs thread pool for hostile payloads at high rate
- Thread pool (shared heap): fastest, no isolation. Acceptable only if the parser is written in a memory-safe language (one where out-of-bounds reads and writes are checked or prevented, such as Rust, Go or Java) and has no
unsafeor native extension code, and you accept that logic bugs still leak across requests. - Pipe or Unix socket between processes: the kernel copies the bytes, which also gives you a natural validation point and flow control (a full pipe blocks the writer, giving back-pressure for free: a slow consumer automatically slows the producer instead of letting a queue grow without limit). Slower per message than shared memory.
- Shared-memory ring buffer (single producer, single consumer: one writer process and one reader process share a fixed circular array of message slots): avoids the copy and system calls at high message rates, but the consumer is now reading memory that a compromised producer can still modify while the consumer reads it (a time-of-check to time-of-use race, TOCTOU, called a double fetch when the reader reads the same field twice and the two reads can differ). Rules: copy a message out of the ring into private memory before validating it, never store pointers or offsets you trust inside the ring, bound every length, and use fixed-size slots. At high rate this is the right trade only when profiling shows the copy is the bottleneck.
Failure modes that worry me most with hostile input
- Memory-safety bug in the parser (buffer overflow, use-after-free): contained by process isolation plus a sandbox; catastrophic in the threaded design.
- Resource exhaustion: one payload that demands unbounded memory or CPU (decompression bombs, tiny compressed inputs that expand to gigabytes; deeply nested structures; regex backtracking, where a crafted string makes a regular expression take exponential time). Defences: hard limits on message size, nesting depth and per-message CPU time; per-process memory limits so the out-of-memory killer takes one worker, not the service.
- Poison messages and crash loops: a poison message is one that reliably crashes the worker that handles it; a crash loop is the supervisor restarting the worker only for the same message to kill it again. Record the payload ID before parsing, quarantine anything that crashed a worker twice, and cap restart rate.
- Slow or stalled senders occupying workers; use timeouts and a bounded queue that sheds load instead of growing.
- A compromised parser attacking its neighbour through the channel: the trusted side treats parser output as untrusted too, validating schema and sizes again.
- Debuggability under attack: keep the raw rejected payloads (size-capped) and a request ID per message so a crash can be reproduced from a core dump and its input.
Where this comes from, and when I would choose differently
Production systems already follow this shape: web browsers run the code that handles untrusted web pages (renderers) in separate sandboxed processes, so one malicious page cannot read another site's data; OpenSSH splits a privileged monitor from an unprivileged process that handles network data (privilege separation), so a bug in network parsing does not hand over root; nginx uses a master process with worker processes, so a crashed worker is replaced without stopping the server; and PostgreSQL uses a separate process per connection, so an ordinary error or a client disconnect affects only that session; but because its backends also share memory, a backend that crashes makes the postmaster terminate and restart every backend ("terminating connection because of crash of another server process"), which shows the limit of process isolation once processes share a memory region. Pick the single multi-threaded process if the input is trusted, latency is tight, the parser is memory-safe, and a restart of everything is cheap. Pick full process-per-tenant when tenants distrust each other or compliance demands separation. The hybrid is the default when input is hostile but throughput still matters.
How do page-level protection bits (present, read/write, user/supervisor, no-execute) enforce isolation between user and kernel space and between regions of one process? What attacks does W^X / NX stop, and what does it not stop?
Sample Answer
The bits. A page-table entry (PTE) is the hardware's record for one virtual page. On x86-64 the permission bits sit at fixed positions (bit numbers from Linux's arch/x86/include/asm/pgtable_types.h): bit 0 present, bit 1 read/write, bit 2 user/supervisor, bit 63 no-execute (called XD on Intel). The MMU (memory management unit, the CPU's translation hardware) checks them on every access during the page walk and caches the result in the TLB (a cache of recent translations).
- Present = 0: any access raises a page fault. The kernel uses this for demand paging, swapped-out pages, and guard pages (deliberately unmapped pages next to a stack or buffer).
- Read/write = 0: a write faults. Used for code, constants and the first step of copy-on-write.
- User/supervisor = 0: the page is supervisor-only. The CPU runs code at one of two privilege levels (the mode it is currently in): user mode for ordinary programs and kernel (supervisor) mode for the OS, and this bit says which mode may touch the page. So a user-mode (unprivileged) access faults even though the kernel's pages appear in the same address space. This is the core user versus kernel boundary: the kernel lives in the upper half of every process's address space, but user code gets a fault when it touches it.
- No-execute = 1: an instruction fetch from the page faults, while ordinary loads and stores still work.
The same user/kernel boundary is also hardened from the kernel's side. SMEP (supervisor mode execution prevention) stops the kernel executing user pages, so an attacker cannot trick the kernel into running code they placed in user memory. SMAP (supervisor mode access prevention) stops the kernel reading or writing user pages by accident, except in the kernel's explicit user-copy code. Since the Meltdown vulnerability (a CPU flaw that let user code read kernel memory speculatively), Linux can also use KPTI (kernel page-table isolation), which keeps two page-table sets so that most kernel mappings are absent while user code runs.
Isolation between regions of one process. The kernel creates each region with fixed permissions: code is read plus execute, read-only data is read, stack and heap are read plus write without execute, and the guard page below a stack is not present. A buggy write into code or a jump into the heap becomes an immediate fault instead of silent corruption.
What W^X / NX stops. W^X ("write xor execute") is a policy that no page is writable and executable at once, and NX is the hardware bit that enforces the execute half. Classic code injection puts machine code into a buffer on the stack or heap, then redirects control into it. The demo shows the fault. It builds a tiny function in a read/write page and jumps to it (the bytes are mov eax,42; ret on x86-64 and mov w0,#42; ret on AArch64, decoded below; run natively in an AArch64 Linux container, with the x86-64 branch also run there under emulation and printing the same lines, so the NX fault shown is not an x86 hardware observation):
#include <setjmp.h>
#include <signal.h>
#include <stdio.h>
#include <string.h>
#include <sys/mman.h>
static sigjmp_buf env;
static void on_segv(int s) { (void)s; siglongjmp(env, 1); }
int main(void) {
#if defined(__x86_64__)
unsigned char code[] = { 0xb8, 42, 0, 0, 0, 0xc3 }; /* mov eax,42 ; ret */
#else
unsigned char code[] = { 0x40, 0x05, 0x80, 0x52, 0xc0, 0x03, 0x5f, 0xd6 }; /* mov w0,#42 ; ret */
#endif
unsigned char *p = mmap(NULL, 4096, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
memcpy(p, code, sizeof code);
signal(SIGSEGV, on_segv);
if (sigsetjmp(env, 1) == 0) {
printf("jump into writable, non-executable page: returned %d\n", ((int (*)(void))p)());
} else {
printf("jump into writable, non-executable page: SIGSEGV (NX)\n");
}
mprotect(p, 4096, PROT_READ | PROT_EXEC); /* W^X flip: now executable, no longer writable */
printf("after mprotect(PROT_READ|PROT_EXEC): returned %d\n", ((int (*)(void))p)());
if (sigsetjmp(env, 1) == 0) { p[0] = 0x90; printf("write to R+X page: ok (unexpected)\n"); }
else printf("write to R+X page: SIGSEGV\n");
return 0;
}
jump into writable, non-executable page: SIGSEGV (NX)
after mprotect(PROT_READ|PROT_EXEC): returned 42
write to R+X page: SIGSEGV
Reading the demo:
- The bytes. On x86-64,
0xb8is the opcode for "move a 32-bit immediate intoeax", the next four bytes42, 0, 0, 0are that immediate in little-endian order (42), and0xc3isret. On AArch64 the four bytes0x40 0x05 0x80 0x52are the little-endian form of the instruction word0x52800540(mov w0, #0x2a, the return-value register set to 42) and0xc0 0x03 0x5f 0xd6is0xd65f03c0(ret);objdumpin a gcc:14 container decodes the AArch64 bytes asmov w0, #0x2aandret, andobjdumpin an amd64 container decodes the x86-64 bytes asmov $0x2a,%eaxandret. - The cast.
((int (*)(void))p)()reinterprets the addresspas a pointer to a function that takes nothing and returnsint, then calls it, so the CPU fetches an instruction fromp. mprotect. The system call that changes the permissions of an existing mapping; here it flips the page from read+write to read+execute.- The control flow.
signal(SIGSEGV, on_segv)registers a handler for the fault signal.sigsetjmp(env, 1)records a return point and returns 0 the first time; the jump into the non-executable page raises SIGSEGV, the handler callssiglongjmp(env, 1), and execution resumes at that samesigsetjmpcall, which now returns 1 (it "returns twice"), so theelsebranch prints "SIGSEGV (NX)". The second argument 1 saves the signal mask so SIGSEGV is deliverable again for the next fault.
The same bytes crash while the page is writable and run once the page is flipped to read plus execute, and the flipped page can no longer be written, which is W^X in both directions.
What it does not stop.
- Code reuse attacks (return-to-libc: overwrite a return address so the function returns into an existing library function such as
system; ROP, return-oriented programming: chain many short snippets of existing executable code that each end inret). The attacker never executes new bytes, so NX has nothing to block. - Calling mprotect or mmap with exec permission through such a chain, which makes the attacker's data executable. Mitigation is policy on top of NX: SELinux's execmem check (denies a process the right to create writable-and-executable memory) and seccomp filters (a per-process list of system calls it is allowed to make).
- Data-only attacks (overwriting a flag such as
is_admin), logic bugs, and information leaks. - JIT compilers legitimately need to produce code, so they use the flip pattern above (write the generated code into a read/write page, then
mprotectit to read plus execute, exactly as the demo does) or dual mappings (the same physical memory mapped twice, once writable and once executable), and an attacker who corrupts the compiler's output reaches code execution without breaking W^X. - It does not hide addresses: pair it with ASLR (randomized layout) so ROP chains must first learn addresses, and with control-flow integrity (checks that indirect jumps and returns only go to valid targets) and stack canaries (a secret value placed before a return address and checked before the function returns, so a stack overflow is detected) for the code-reuse side.
Treat NX as removing the easiest exploit class, not as a complete defence.
What is address space layout randomization, how does it work at the kernel level (what is randomised and when), why does it help against exploits, and how does it complicate reading crash reports and core dumps?
Sample Answer
What ASLR is. Address space layout randomization changes where a process's memory regions start each time the program is run, so an attacker cannot rely on fixed addresses.
What is randomised, and when. The kernel picks random offsets at execve time and keeps them for the life of the process. execve is the system call that replaces the calling process's program with a new one (the kernel builds a fresh process image: new stack, new mappings), and fork is the system call that creates a child that is a copy of the calling process. A sysctl is a kernel setting you read and change through /proc/sys or the sysctl command; kernel.randomize_va_space controls ASLR (kernel documentation): 0 is off; 1 randomises the mmap base, the stack, the vDSO page (a small kernel-provided shared object that user code calls for fast system calls) and the code start of PIE binaries; 2 additionally randomises the heap (the brk area, the classic heap that grows from the end of the program's data via the brk system call), and is the usual default (the kernel documentation gives 1 as the default when CONFIG_COMPAT_BRK is enabled and 2 when it is disabled, and the norandmaps boot parameter turns ASLR off). Concretely:
- the stack top, and the mmap region where shared libraries, anonymous mappings and thread stacks go;
- the heap start;
- the executable itself, but only if it was built as a position-independent executable (PIE,
-fPIE -pie); a non-PIE binary stays at a fixed address such as 0x400000, a common weak spot.
For the executable, the heap, the mmap region and the libraries the randomness is at page granularity: only the page-number bits are random, so the low 12 bits within a page never change. The stack is the exception: the kernel also subtracts a random sub-page amount from the initial stack pointer (on arm64arch_align_stacksubtracts a random value below the page size and then rounds down to a multiple of 16 bytes; x86 does the same with its own range), so stack addresses differ in their low digits too, in steps of 16 bytes. The amount of randomness (entropy, the number of random bits, so 2^bits possible positions) is set by/proc/sys/vm/mmap_rnd_bits(18 in the aarch64 Linux container used for the demo below; it is architecture and config dependent, and 32-bit systems get far less entropy). A child created byforkwithoutexechas the same layout as its parent, so a pre-forking server keeps one layout across all its workers until restarted.
Demo. Compile with PIE and run it several times in a Linux container (aarch64, kernel setting 2):
#include <stdio.h>
#include <stdlib.h>
#include <sys/mman.h>
int global_data;
int main(void) {
int local;
void *heap = malloc(16);
void *map = mmap(NULL, 4096, PROT_READ, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
printf("code=%p data=%p heap=%p mmap=%p libc_printf=%p stack=%p\n",
(void *)main, (void *)&global_data, heap, map, (void *)printf, (void *)&local);
return 0;
}
gcc -O0 -Wall -fPIE -pie aslr.c -o aslr, then run ./aslr three times:
code=0xaaaab38e07e4 data=0xaaaab390004c heap=0xaaaac13c02a0 mmap=0xffffa2671000 libc_printf=0xffffa24bcb40 stack=0xffffcdecbe4c
code=0xaaaad65e07e4 data=0xaaaad660004c heap=0xaaab0a2ad2a0 mmap=0xffffa46e2000 libc_printf=0xffffa452cb40 stack=0xffffc52dab5c
code=0xaaaac22807e4 data=0xaaaac22a004c heap=0xaaaac69592a0 mmap=0xffff89118000 libc_printf=0xffff88f6cb40 stack=0xffffe368c1bc
How to read these lines: in the three runs the code= value ends in 07e4 every time and data= ends in 004c, the heap= value ends in 2a0, and libc_printf= ends in b40. Only the higher digits of those four values change, because only whole pages move. The executable moves as one block: data - code is 0x1f868 in all three runs (for example 0xaaaab390004c - 0xaaaab38e07e4), and code - 0x7e4 gives the load base (0xaaaab38e0000 in the first run, ending in four zero digits, a page boundary). The mmap address sits near the top of the address space and moves by whole pages on every run. The stack address also changes on every run, and in its low digits too (e4c, b5c, 1bc above), because of the sub-page stack offset described earlier. So every region moves on each run. Built without -pie, the same program printed code=0x4006c4 on every run while heap, mmap and stack still moved.
Why it helps against exploits. Memory-corruption exploits need to know where something is: the address of a function to return to (return-to-libc: redirect a return into an existing library function; ROP, return-oriented programming: chain short snippets of existing code that each end in ret), of injected data, of a pointer to overwrite. With randomisation the attacker must first leak an address (via an information-disclosure bug, a flaw that makes the program reveal memory contents such as a pointer) or guess. A wrong guess usually crashes the process, which is noisy and, for a service that restarts with a new layout, resets the attacker's progress. ASLR is a probabilistic defence: it raises cost, and weak entropy (32-bit processes, fork-without-exec servers) can be brute-forced. It works together with NX (stack and heap memory is not executable), stack canaries (a secret value placed before a return address and checked on return, so a stack overflow is detected) and PIE; none is sufficient alone.
How it complicates crash reports and core dumps. A raw address in a log (SIGSEGV at 0xffffa24bcb40) means nothing next to a symbol file, and two crashes of the same bug show different addresses, so grouping by address fails. The way through:
- Log module-relative values: module name plus offset (instruction pointer minus load base), which ASan (AddressSanitizer, a compiler-inserted memory checker) reports and crash reporters do. Worked from the first demo run: the code address
0xaaaab38e07e4minus the load base0xaaaab38e0000is0x7e4. Thenaddr2line -f -e aslr 0x7e4(translate an address in a binary to a function name, and to a line if the binary has debug info) printsmainwhen run in the same container on a freshly built copy (nm aslrlistsmainat00000000000007e4). In gdb,info symbol 0x7e4does the same inside a debugger (lldb spells the lookupimage lookup --address). - Core files record the memory mappings, so a debugger on the same binary and libraries relocates automatically; keep the exact build ID (a unique hash stored in the binary that identifies the build, so you can match a crash to the right symbol file) and unstripped symbols (symbol names not removed from the file).
- Capture
/proc/<pid>/maps(or the loaded-module list) at crash time if you roll your own handler.
gdb disables ASLR for programs it launches by default (set disable-randomization on), which is why a bug can "not reproduce" under gdb but appear in production.
Turning it off temporarily for debugging. Per process: setarch -R ./prog (the -R flag is ADDR_NO_RANDOMIZE). Run in the demo container with seccomp unconfined, repeated setarch -R ./aslr runs printed identical addresses (code=0xaaaaaaaa07e4 ... stack=0xfffffffff9bc). In Docker, setarch -R is refused with "Operation not permitted" under Docker's default seccomp profile (a filter list of which system calls a container may make) and needed --security-opt seccomp=unconfined. System-wide: echo 0 | sudo tee /proc/sys/kernel/randomize_va_space, which weakens the whole host, so restore it afterwards.
That is every published Kernel Architecture & OS Internals question for Cybersecurity Engineer so far. Browse the other topics in this category, or practice this one interactively.