Kernel Architecture & OS Internals Questions

Design. With 4 KiB pages (2^12 bytes) in a 32-bit address space (2^32 bytes) there are 2^32 / 2^12 = 2^20 = 1,048,576 virtual pages. A single-level table is an array with one entry per virtual page number (VPN). A virtual address splits into a 20-bit VPN (the top bits) and a 12-bit offset (the low bits). Each entry is either a physical frame number or "not present". The physical side is a free list of frames: a plain Python list of the frame numbers not currently in use. Translate indexes the array by VPN; a missing entry raises a page fault (the exception the CPU delivers when a translation fails, handled by the kernel).

PAGE_SHIFT = 12
PAGE_SIZE = 1 << PAGE_SHIFT
NUM_VPAGES = 1 << (32 - PAGE_SHIFT)      # 1,048,576 entries

class PageFault(Exception):
    def __init__(self, vaddr, reason):
        super().__init__(f"page fault at {vaddr:#010x}: {reason}")

class PageTable:
    def __init__(self, num_frames):
        self.table = [None] * NUM_VPAGES       # entry: frame number or None (not present)
        self.free_frames = list(range(num_frames - 1, -1, -1))

    def map(self, vpn):
        if self.table[vpn] is not None:
            raise ValueError("already mapped")
        if not self.free_frames:
            raise MemoryError("out of physical frames")
        self.table[vpn] = self.free_frames.pop()
        return self.table[vpn]

    def unmap(self, vpn):
        frame = self.table[vpn]
        if frame is None:
            raise ValueError("not mapped")
        self.table[vpn] = None
        self.free_frames.append(frame)

    def translate(self, vaddr):
        vpn, offset = vaddr >> PAGE_SHIFT, vaddr & (PAGE_SIZE - 1)
        frame = self.table[vpn]
        if frame is None:
            raise PageFault(vaddr, "page not present")
        return (frame << PAGE_SHIFT) | offset

pt = PageTable(num_frames=4)
f = pt.map(0x00400)                      # virtual page 0x400 -> some frame
print("map vpn 0x400 -> frame", f)
print("translate 0x00400abc ->", hex(pt.translate(0x00400ABC)))
try:
    pt.translate(0x00500000)
except PageFault as e:
    print(e)
pt.unmap(0x00400)
try:
    pt.translate(0x00400ABC)
except PageFault as e:
    print("after unmap:", e)
for v in range(4):
    pt.map(v)
try:
    pt.map(99)
except MemoryError as e:
    print("map 5th page:", e)
print("entries:", NUM_VPAGES, "| 4-byte entries =", NUM_VPAGES * 4 // 2**20, "MiB per process")

Run with python sim.py (Python 3.12). Output:

map vpn 0x400 -> frame 0
translate 0x00400abc -> 0xabc
page fault at 0x00500000: page not present
after unmap: page fault at 0x00400abc: page not present
map 5th page: out of physical frames
entries: 1048576 | 4-byte entries = 4 MiB per process

Why the first physical address is 0xabc: the free list handed out frame 0, so (0 << 12) | 0xabc = 0xabc. After the unmap returned its frame, all four frames were free again; mapping VPNs 0 to 3 used all four, so the next map found the pool empty and failed. In a real kernel an empty pool triggers page reclaim or the OOM killer, not an exception to the caller.

Reading the constructor: list(range(num_frames - 1, -1, -1)) counts down from the last frame to 0 (with 4 frames it is [3, 2, 1, 0]). pop() removes and returns the last element of a list, so the first map gets frame 0, the next gets frame 1, and so on, and unmap pushes a freed frame back with append. The reversed order exists only to make the frames come out as 0, 1, 2, 3 for readable output.

Time cost. map, unmap and translate are all O(1): one array index and a free-list push or pop. A hardware walk of this table is one memory read per translation (plus the TLB, which caches recent translations to avoid even that).

Space cost. The table has 2^20 entries whether the process uses 3 pages or 3 million. With 4-byte entries (the size of 32-bit x86 entries) that is 4 MiB per process, and the Python list in this simulator holds 8-byte pointers, so about 8 MiB (sys.getsizeof([None] * 2**20) reports 8.0 MiB on 64-bit CPython 3.12). With 100 processes that is 400 MiB of page tables, mostly for empty entries. A typical process uses a few hundred KiB to a few hundred MiB scattered across a huge sparse range (code low, heap above it, stack at the top), so almost all entries are empty. For 64-bit it is impossible: a 48-bit address space with 4 KiB pages has 2^36 pages, and 2^36 entries of 8 bytes is 512 GiB per process.

What multi-level tables fix. Split the VPN into indices and allocate lower tables only for ranges that are used. Classic 32-bit x86 uses two levels (10 + 10 + 12 bits): a 1,024-entry page directory (4 KiB) whose entries point at 1,024-entry page tables (4 KiB each, each covering 4 MiB). A process with one used 4 MiB region, aligned to a 4 MiB boundary, needs 4 KiB + 4 KiB = 8 KiB of tables instead of 4 MiB (an unaligned region straddles two second-level tables and needs 12 KiB). x86-64 uses four levels of 512 entries (9 + 9 + 9 + 9 + 12 = 48 bits), each table exactly one 4 KiB page of 8-byte entries. The cost is time: a translation that misses the TLB needs one memory read per level (four on x86-64) instead of one, so caching structures and huge pages matter more. A 2 MiB page is 2 MiB / 4 KiB = 512 ordinary pages mapped by one entry in the level above the last, so the walk stops one level early: 3 table reads instead of 4 on x86-64, and one TLB entry instead of 512.

Extending to two levels. Replace table with a directory of 1,024 slots that are None until needed, each pointing to a lazily created list of 1,024 entries: dir_i = vpn >> 10, tbl_i = vpn & 0x3ff. translate checks the directory slot first and faults if it is None; unmap can free an empty second-level table.

Beyond a sketch. A real table also stores permission bits and present/accessed/dirty flags per entry, has one such table per process (the kernel also keeps bookkeeping for each physical frame, such as how many mappings point at it, so it knows when a frame can be freed), and sends faults to a handler that can load the page rather than only raising an error. The fixed pool here is what makes the memory-pressure path visible.

Direct answer

A process is a running program with its own address space, file-descriptor table, credentials and PID. A thread is a schedulable flow of execution inside a process. Threads of one process share the address space (code, globals, heap), the open files and the signal handlers, but each has its own registers, stack, thread ID and signal mask (the set of signals that thread currently blocks from being delivered). Choose threads when tasks need to share memory cheaply and a crash is acceptable to take down all of them. Choose several processes when you want fault or security isolation, or when sharing is minimal.

What the kernel tracks

On Linux the kernel schedules one task (task_struct) per thread. A "process" is a group of tasks that share resources. The clone system call creates a task, and flags decide what is shared. Each flag switches on one kind of sharing; a thread is a task created with all of them, and a separate process with none: CLONE_VM (same address space), CLONE_FILES (same descriptor table), CLONE_FS (same working directory), CLONE_SIGHAND (same signal handlers) and CLONE_THREAD (same process, so same PID as seen by getpid()). fork() is a clone that shares nothing: the child gets a copy-on-write copy of the address space (the kernel does not copy memory pages up front; parent and child share them read-only, and a page is copied only when one side writes to it), which is why it is cheap in proportion to the size of the page tables rather than the size of memory used.

On Windows the same split exists differently: a process object owns the address space and handle table, a thread is the unit of scheduling, and there is no fork; creating a process (CreateProcess) is heavier and starts a fresh image.

A tiny experiment shows the difference. A global counter starts at 0. A thread sets it to 100 and finishes; a forked child sets it to 200 and exits. The parent then prints it each time.

#include <pthread.h>
#include <stdio.h>
#include <sys/wait.h>
#include <unistd.h>

int counter = 0;                       /* a global: lives in the address space */

void *bump(void *arg) { counter = 100; return NULL; }

int main(void) {
    pthread_t t;
    pthread_create(&t, NULL, bump, NULL);   /* thread: same address space */
    pthread_join(t, NULL);
    printf("after thread: counter = %d\n", counter);

    counter = 0;
    pid_t pid = fork();                     /* process: child gets its own copy */
    if (pid == 0) { counter = 200; _exit(0); }
    waitpid(pid, NULL, 0);
    printf("after fork child: counter = %d\n", counter);
    return 0;
}

Compiled with gcc and run in a Linux container, it printed:

after thread: counter = 100
after fork child: counter = 0

The thread wrote into the one shared address space, so the parent sees 100. The child wrote into its own copy, so the parent still sees 0.

Process versus threads for a server

Recommendation. For a latency-sensitive service whose workers share a large in-memory cache, use threads (or an event loop on a few threads). For a component that handles untrusted input, such as a file parser, an image decoder, or a browser tab, run it in a separate low-privilege process (one given only the few permissions its job needs): if an attacker exploits a memory bug the damage stays in that process, and a crash restarts one worker. Real systems mix both: a browser uses a process per site and threads inside each, and Postgres uses a process per connection partly for isolation.

Worked example

A server decodes uploaded images. With thread-per-request, a malformed file that triggers a heap overflow (writing past the end of a heap buffer into neighbouring memory) runs the attacker's code with access to the process's encryption keys for HTTPS connections (the TLS protocol, unrelated to thread-local storage above), session cache and database credentials, all in the same address space, and a plain crash drops every in-flight request. With a pool of decoder processes running under a restricted user ID with a system-call filter (a kernel-enforced list of system calls the process may make, such as seccomp on Linux; anything else is refused or kills it), the same overflow reaches only that decoder's memory; the parent sees a child exit and starts a replacement. The price is copying each upload to the decoder through a pipe or a shared mapping.

Pitfalls and notes

• "Threads are always faster" ignores the locking they require; "processes are always safer" ignores IPC bugs and cost.
• Memory-forensics note: a memory image lists each thread as its own task entry, and because threads share one address space a dump of the process contains every thread's stack.
• Fork in a multithreaded program only copies the calling thread, which leaves locks held by other threads stuck in the child; call exec soon after.

The idea. Each process sees its own virtual address space: a private range of addresses that the CPU's MMU (memory management unit) translates to physical RAM through the process's page tables (the kernel's lookup tables of which virtual page is backed by which physical page, with read/write/execute permissions). The kernel divides that space into regions (called VMAs, virtual memory areas), each with a start, end, permissions and a backing. Two processes can use the same address for different data because their page tables differ.

Layout of a typical 64-bit Linux process, low addresses to high

Seeing it: a program that prints where its own variables live. Compiled and run in a Linux container (gcc 14.4.0, gcc -O0 -Wall -Wextra -fPIE -pie addr_regions.c -o addr_regions). The container is aarch64, so the absolute addresses are ARM-style; the same program on x86-64 puts user addresses below 0x0000800000000000.

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>

int global_data = 42;          /* .data */
int global_bss;                /* .bss */
const char ro_text[] = "ro";   /* .rodata */

static void where(const char *name, void *p) {
    unsigned long a = (unsigned long)p;
    FILE *f = fopen("/proc/self/maps", "r");
    char line[512];
    while (fgets(line, sizeof line, f)) {
        unsigned long lo, hi; char perms[8], rest[400] = "";
        if (sscanf(line, "%lx-%lx %7s %*s %*s %*s %399[^\n]", &lo, &hi, perms, rest) >= 3
            && a >= lo && a < hi) {
            printf("%-10s %p -> %lx-%lx %s %s\n", name, p, lo, hi, perms, rest);
            break;
        }
    }
    fclose(f);
}

int main(void) {
    int local = 1;
    void *small = malloc(100);
    void *big = malloc(1 << 20);
    void *anon = mmap(NULL, 1 << 20, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
    where("code", (void *)main);
    where("rodata", (void *)ro_text);
    where("data", &global_data);
    where("bss", &global_bss);
    where("heap", small);
    where("malloc1MiB", big);
    where("mmap", anon);
    where("libc", (void *)printf);
    where("stack", &local);
    return 0;
}

Output from one run (addresses change on every run because of ASLR):

code       0xaaaac40e0a74 -> aaaac40e0000-aaaac40e1000 r-xp /w/addr_regions
rodata     0xaaaac40e0b90 -> aaaac40e0000-aaaac40e1000 r-xp /w/addr_regions
data       0xaaaac4100070 -> aaaac4100000-aaaac4101000 rw-p /w/addr_regions
bss        0xaaaac4100078 -> aaaac4100000-aaaac4101000 rw-p /w/addr_regions
heap       0xaaaad7a5a2a0 -> aaaad7a5a000-aaaad7a7b000 rw-p [heap]
malloc1MiB 0xffff9043f010 -> ffff9033f000-ffff90540000 rw-p 
mmap       0xffff9033f000 -> ffff9033f000-ffff90540000 rw-p 
libc       0xffff9058cb40 -> ffff90540000-ffff906dc000 r-xp /usr/lib/aarch64-linux-gnu/libc.so.6
stack      0xffffc1495394 -> ffffc1475000-ffffc1496000 rw-p [stack]

Reading it: code and rodata share one r-xp mapping on this small program (the linker packed them together), data and bss share one rw-p mapping, the 100-byte malloc lands in [heap], and the 1 MiB malloc landed in an anonymous mapping next to the explicit mmap (large requests bypass the heap and ask the kernel for fresh pages), printf is in libc.so.6's executable mapping, and the stack is at the top.

Reading /proc/<pid>/maps. Each line is address perms offset dev inode pathname. offset is where in the backing file this mapping starts, dev is the device holding the file (major:minor numbers) and inode is the file's number on that device; anonymous mappings show 0 for these. The sscanf format in where() reads %lx-%lx (start and end of the range in hex), %7s (the perms string), then %*s %*s %*s (offset, dev and inode: the * means parse and discard them) and %399[^\n] (the rest of the line, the pathname). perms letters: r read, w write, x execute, and p private (copy-on-write) or s shared. The pathname is the backing file, or special names [heap], [stack], [vdso] (a small kernel-provided library for fast system calls), or blank for an anonymous mapping (from mmap or malloc).

From a faulting address to a region (ordered checks)

1. Get the faulting address. For a crash (SIGSEGV, the signal the kernel sends a process that makes an illegal memory access) it is in the signal information (the si_addr field of the siginfo_t structure the kernel passes to the signal handler) and debuggers print it. A core dump is the file the kernel writes at such a crash, holding the process's memory and registers, so you can inspect it afterwards. Also note whether the fault was a read, write or instruction fetch.
2. Get the mappings at the time of the crash. For a live process, read /proc/<pid>/maps. For a core dump, load it in gdb and use info proc mappings (the gdb documentation says process information is available from cores created on GNU/Linux; if your gdb version prints nothing, readelf -l core lists the memory segments the core recorded). Use the same binary and libraries that crashed, or addresses will not line up.
3. Find the line where start <= address < end. Then classify:
   • No line contains it: unmapped memory. A wild or freed pointer, or one already unmapped by munmap.
   • Between 0 and the first mapping, especially under 4 KiB: a NULL pointer plus a small field offset (the lowest addresses are unmapped on purpose: vm.mmap_min_addr, from 4 KiB up to commonly 64 KiB depending on configuration, keeps user mappings away from them).
   • Inside [heap] or an anonymous region: a heap bug such as use-after-free or an overrun; ask for a sanitizer run (AddressSanitizer, a compiler option that makes every heap access checked and reports out-of-bounds and use-after-free with the exact line) to confirm.
   • Just below the [stack] mapping's start, or at a guard region (a page or range deliberately mapped with no access, placed to catch overruns) next to a thread stack: stack overflow, usually runaway recursion or a huge local array.
   • Inside a file-backed mapping with a permission that forbids the access (a write to an r-x or r-- region such as .rodata, or a jump into rw- data): a permission fault. Writing to a string literal is the classic case.
   • Inside a library mapping: look up which library from the pathname, subtract the library's load base (the start of its first mapping, which normally begins at file offset 0) to get an offset inside the library, and map that offset to a symbol with addr2line -f -e <library> <offset> (prints the function name; -e names the file) or by finding the closest lower address in nm output (a listing of a file's symbols and their addresses).
   • An address in the kernel half or the non-canonical hole: user code cannot access either; this is usually a corrupted pointer (an overwritten pointer holding data such as ASCII text). On x86-64 an access to a non-canonical address raises a general-protection fault rather than a page fault, and the resulting SIGSEGV commonly reports si_addr as 0, so check the registers and the faulting instruction in the core rather than relying on si_addr alone.
4. Check the instruction pointer separately: the faulting data address says what was touched, the program counter says which code touched it. Both are needed.

Worked examples using the run above. Pretend each of these came out of si_addr (addresses are from that one run and are illustrative):

• 0xffff9058cb40: line ffff90540000-ffff906dc000 r-xp .../libc.so.6 contains it, and the access was an instruction fetch, so the crash is inside libc code. Offset = 0xffff9058cb40 - 0xffff90540000 = 0x4cb40, which you pass to addr2line -f -e libc.so.6 0x4cb40. This works here because, in this run, the executable mapping is libc's first mapping and starts at file offset 0 (its /proc/<pid>/maps offset column reads 00000000); on builds where a read-only mapping comes first, subtract the lowest start address among that library's mappings instead.
• 0xffffc1474ff8: no mapping contains it, and it is 8 bytes below the start of the [stack] line (0xffffc1475000 - 8). That points to a stack overflow: a push or a huge local array stepped past the bottom of the stack. One caveat about this illustration: the [stack] line in the run is only 132 KiB (0xffffc1475000 to 0xffffc1496000), and for an access just below a main-thread stack that is still under its size limit the kernel grows the mapping instead of faulting. A genuine overflow therefore shows a [stack] line already grown to the limit (ulimit -s, 8192 KiB by default) with the fault just below it or in the guard gap the kernel keeps below the stack, so read this example as the shape of the check, not as a faithful crash.
• 0xaaaac40e0b90 (the address of ro_text) on a write: the line aaaac40e0000-aaaac40e1000 r-xp contains it, but the perms have no w. That is a permission fault, the classic write to a read-only literal.
• 0x18: below every mapping and under 4 KiB, so a NULL pointer plus a field at offset 24 (0x18).

Leak investigation follow-up. Compare /proc/<pid>/maps (or pmap) over time: a growing [heap] points to malloc growth, many new anonymous regions point to leaking mmap calls or thread stacks, and growth inside one library's mappings points to that library.

The two schemes in one line each. Segmentation divides a program's memory into variable-length logical pieces (code, stack, heap...), and an address names a segment plus an offset inside it. Paging divides both virtual and physical memory into equal fixed-size blocks (pages and frames, 4 KiB on x86-64 and common ARM Linux) and an address names a page plus an offset inside it.

How an address is formed.

• Segmentation. The logical address is (segment selector, offset). The selector is a small number held in a segment register; the hardware uses it to find a segment descriptor in a table (on x86 the GDT, the global descriptor table, or the LDT, a per-task local one) holding a base (where the segment starts in memory), a limit (its size) and permissions. It checks offset <= limit, then computes linear = base + offset. Example: base 0x10000, limit 0x4000, offset 0x0ABC gives 0x10ABC; offset 0x5000 exceeds the limit and faults.
• Paging. A 32-bit virtual address with 4 KiB pages splits into a 20-bit virtual page number (VPN) and a 12-bit offset (4 KiB is 2^12). Example: 0x00403ABC has VPN 0x403 and offset 0xABC. The page table maps VPN 0x403 to a physical frame number, say 0x12345, so the physical address is (0x12345 << 12) | 0xABC = 0x12345ABC. The offset passes through unchanged.
• Both stages on x86 (32-bit protected mode). Take the segmentation example above: the segment stage gives the linear address 0x10ABC. The paging stage then treats 0x10ABC as a virtual address: VPN = 0x10ABC >> 12 = 0x10, offset = 0xABC. If the page table maps VPN 0x10 to frame 0x345, the physical address is (0x345 << 12) | 0xABC = 0x345ABC. This two-stage pipeline is what the classic textbook picture shows.

Hardware support.

• Segmentation needs segment registers and descriptor tables, plus a base-plus-limit adder and comparator on every access.
• Paging needs a page-table base register (CR3 on x86, a control register holding the physical address of the top-level table; TTBR0/TTBR1 on ARM, the two translation table base registers, which hold the tables for the lower and upper halves of the address space) that points at the current process's table, a hardware page-table walker (circuitry that reads the table entries in memory to translate an address on a TLB miss), and a TLB (translation lookaside buffer, a small cache of recent translations, without which every load would cost extra memory reads). Each page-table entry carries present, read/write, user/supervisor and no-execute bits.

Trade-offs.

Why Linux relies on paging. Fixed-size pages remove external fragmentation, so the kernel can place any page in any free frame, swap or reclaim single pages, load program files lazily, share one frame between processes and duplicate it only when written (copy-on-write), and set permissions at page granularity (read-only code, no-execute data, kernel-only pages). Isolation comes from giving each process its own page table. Segmentation cannot be switched off on x86: in 32-bit protected mode every memory access goes through a segment register, and the architecture has no mode that skips it. So Linux sets it up as flat: all segments have base 0 and cover everything, and in 64-bit mode the CPU ignores base and limit for the code, data and stack segment registers. The FS and GS segment bases are still used (the kernel documentation says they remain functional in 64-bit mode), so the one surviving use of segmentation is as a cheap base register. Example: each thread's FS base points at that thread's own block of thread-local storage, so the same instruction mov %fs:0x10, %rax (read the 8 bytes at FS base + 0x10) reads a different thread's memory in each thread, with no register spent on a pointer. The Linux kernel does the same with GS for per-CPU data (variables that exist once per CPU), so code can reach "this CPU's copy" without first asking which CPU it is on.

Recommendation and caveat. For a general-purpose multitasking OS with demand paging and many processes, choose paging: the cost is page-table memory and TLB pressure. Multi-level tables fix the memory: instead of one flat array with an entry for every virtual page (a 32-bit space with 4 KiB pages needs 2^20 entries, 4 MiB at 4 bytes each), the address is split into several indices and a lower-level table exists only for ranges the process actually uses. Huge pages ease the TLB pressure: a 2 MiB page is mapped by a single entry, so one TLB slot covers what 512 ordinary 4 KiB pages would need. Segmentation-style base and limit checks still make sense on small embedded parts with no MMU, where one contiguous region per task is acceptable. Pure segmentation also appears historically in early x86 and 16-bit systems, and x86 combined both (segment to linear address, then paging to physical), which is why the classic textbook picture shows two translation stages.

What mmap does inside the kernel. Mapping a file does not read it. The kernel creates a VMA (virtual memory area, a record "addresses X to Y map file F from offset O with these permissions") and leaves the page-table entries empty. No disk I/O happens yet. The file's data is served through the page cache (the kernel's in-RAM copy of file contents), which is the same cache read() and write() use.

First access. The first touch of a mapped page finds no page-table entry, so the CPU raises a page fault (an exception that transfers to the kernel). The fault handler finds the VMA, then asks the file's page cache for that page:

• Page already in the page cache (minor fault): the kernel just installs a page-table entry. No disk access, microseconds or less.
• Not cached (major fault): the kernel issues a read to storage and puts the faulting thread to sleep until it completes, which is milliseconds on a spinning disk or hundreds of microseconds on an SSD.
  Then the instruction restarts and succeeds. The kernel may also map several neighbouring cached pages at once. This is called fault-around: on a read fault the kernel maps up to a window of neighbouring pages that are already in the page cache (the window defaults to 64 KiB, which is 16 pages of 4 KiB, per mm/memory.c in the kernel source). In the demo below, reading all 64 pages of a mapped file increases the minor-fault count by only 4 (47 to 51 in the printed run), not 64, which is 64 pages / 16 pages per fault (output below).

Private versus shared.

• MAP_SHARED: all processes map the same page-cache pages. A write modifies the cache page, marks it dirty, and background writeback (or msync/fsync) later reaches the file. Other processes and read() see the change immediately.
• MAP_PRIVATE: reads share the page-cache page, but the first write triggers copy-on-write: the kernel copies the page into private anonymous memory and maps the copy writable. The file and other processes never see the change.

The demo (Linux aarch64 container; the 64-page file is created by the program). It also triggers SIGBUS, the signal the kernel sends a process when it touches a mapped page that the file can no longer supply, for example a page past the end of a file that was truncated after mapping.

#define _GNU_SOURCE
#include <fcntl.h>
#include <setjmp.h>
#include <signal.h>
#include <stdio.h>
#include <string.h>
#include <sys/mman.h>
#include <sys/resource.h>
#include <unistd.h>

static sigjmp_buf env;
static void on_bus(int sig) { (void)sig; siglongjmp(env, 1); }

static void faults(const char *label) {
    struct rusage r;
    getrusage(RUSAGE_SELF, &r);
    printf("%-34s minflt=%ld majflt=%ld\n", label, r.ru_minflt, r.ru_majflt);
}

int main(void) {
    const size_t PG = 4096, N = 64;
    faults("at start of main");
    int fd = open("data.bin", O_RDWR | O_CREAT | O_TRUNC, 0600);
    char page[4096];
    memset(page, 'A', sizeof page);
    for (size_t i = 0; i < N; i++) write(fd, page, PG);
    fsync(fd);
    faults("after creating the file");

    char *priv = mmap(NULL, N * PG, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0);
    char *shr  = mmap(NULL, N * PG, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
    faults("after mmap (no access yet)");
    volatile char c = 0;
    for (size_t i = 0; i < N; i++) c += shr[i * PG];
    faults("after reading 64 shared pages");
    priv[0] = 'P';           /* copy-on-write of a private page */
    shr[PG] = 'S';           /* write through to the page cache */
    char b[2];
    pread(fd, b, 1, 0);  printf("file byte 0 via pread: %c (private write)\n", b[0]);
    pread(fd, b, 1, PG); printf("file byte 4096 via pread: %c (shared write)\n", b[0]);

    signal(SIGBUS, on_bus);
    ftruncate(fd, 8 * PG);   /* another process shrinks the file */
    if (sigsetjmp(env, 1) == 0) {
        c += shr[3 * PG];    printf("page 3 (still in file): ok\n");
        c += shr[40 * PG];   printf("page 40: ok (unexpected)\n");
    } else {
        printf("page 40 (past new end of file): SIGBUS\n");
    }
    return 0;
}

at start of main                   minflt=38 majflt=0
after creating the file            minflt=47 majflt=0
after mmap (no access yet)         minflt=47 majflt=0
after reading 64 shared pages      minflt=51 majflt=0
file byte 0 via pread: A (private write)
file byte 4096 via pread: S (shared write)
page 3 (still in file): ok
page 40 (past new end of file): SIGBUS

How to read the demo: faults() prints how many minor and major page faults the process has taken so far (getrusage); a minor fault is one served from memory, a major fault one that needed disk I/O. The first line, minflt=38 majflt=0, is the baseline at the start of main: faults already taken while the program was being loaded (the dynamic loader and libc touching their own pages), before any mapping exists. The absolute baseline differs between runs and environments (it is larger when the program is started through a shell wrapper, and majflt can be 1 on a cold start when the program's own pages must be read from disk), so compare differences between lines, not absolute values. The jump from 38 to 47 while creating the file comes from the first printf and the file-writing calls touching new pages for the first time. mmap itself adds no faults (47 to 47), which shows that mapping reads nothing. The loop then reads one byte from each of the 64 pages. volatile char c stops the compiler from deleting loads whose result is never used. pread reads the file through the page cache with a normal system call, so it shows what the two writes did: the private write (priv[0]) was a copy-on-write copy visible only to this process, while the shared write (shr[PG]) changed the page-cache page that pread reads. For the truncation test, ftruncate(fd, 8 * PG) stands in for another process shrinking the file to 8 pages; the read of page 3 still works, the read of page 40 raises SIGBUS, the handler runs siglongjmp, and sigsetjmp (which returns 0 the first time and 1 after the jump back) takes the else branch so the program can report it instead of dying. The majflt count does not change through the mapping and reads.
The private write to byte 0 left the file at A; the shared write to byte 4096 reached it as S. (The program had just written the file, so its pages were already in the page cache, which is why reading them caused only minor faults. Exact counts vary by a fault or two between runs.)

Failure modes a service must expect.

1. Page-in stalls. Any memory access can block on disk, so a latency-sensitive thread (an event loop, a request handler holding a lock) stalls invisibly, with no system call in the profile. Under memory pressure clean pages are evicted, so a "fast" mapped file turns slow. Mitigations: madvise(MADV_WILLNEED) (a hint asking the kernel to start reading a range into the page cache now) or MAP_POPULATE (an mmap flag that pre-faults the whole mapping up front) to prefetch, mlock (pin pages in RAM so they cannot be evicted) for a small hot set, and watching major faults (majflt in /proc/<pid>/stat or perf stat -e major-faults).
2. File truncation. If anyone shrinks the file while it is mapped, touching a page beyond the new end delivers SIGBUS, as the demo shows (and the mmap(2) man page lists). There is no error return to check, because the failure happens in a plain load or store. An I/O error while paging in also produces SIGBUS. Defences: a SIGBUS handler with sigsetjmp/siglongjmp around risky accesses (as in the demo, which is fragile in a multi-threaded service), only mapping files nobody else can truncate (a file you own and lock, or a memfd, an anonymous in-memory file, with F_SEAL_SHRINK, a seal that makes shrinking it impossible), or using pread when you need clean error codes.
3. Unmap races. If one thread calls munmap while another still uses the pointer, the second thread gets SIGSEGV, or, worse, the address range is reused by a new mapping and it silently reads or corrupts unrelated data. Tie the unmap to a reference count, or to a quiescent point, meaning a moment when no thread can still be holding the pointer (an RCU-style epoch scheme waits until every thread has passed such a point; a shutdown phase is another).
4. Durability and visibility. A write to a MAP_SHARED page is not on disk when the store instruction returns; call msync(MS_SYNC) (flush the mapped range's dirty pages to the file and wait) or fsync where it matters. Writeback, the kernel's background writing of dirty pages to disk, can also make shutdown or checkpoint latency spiky.

Recommendation: use mmap for read-mostly files that you control and can prefetch, and treat truncation and stalls as real failure paths. Use read/write with your own buffer when you need predictable latency, error codes or exact durability.