Kernel Architecture & OS Internals Questions

What the D state means. Every task (a process or thread) the kernel schedules has a state. R is running or runnable, S is an interruptible sleep (waiting for something, and a signal can wake it), and D is uninterruptible sleep (the kernel flag is TASK_UNINTERRUPTIBLE; ps documents it as "uninterruptible sleep (usually I/O)" and proc(5) as "uninterruptible disk sleep"). A task in D has told the scheduler "do not run me until the event I am waiting for happens". It is off the CPU, so it uses no CPU time. That is why your service is dead while CPU usage is low.

Why it cannot be killed. A signal, including SIGKILL, is only acted on when the task is woken and about to return to user space. An uninterruptible sleep is not woken by signals, so kill -9 just leaves SIGKILL pending. It is delivered the moment the awaited event arrives, and never if the event never arrives. The design exists because kernel code in the middle of an operation (holding a lock, with a disk or network request in flight that will write into a buffer when it completes, halfway through a filesystem update) often cannot safely abort. Applications also assume that file I/O is not interrupted by signals. Linux added a middle state, TASK_KILLABLE (merged in 2.6.25), where only a fatal signal ends the sleep. The NFS client was the first code converted, but any wait that still uses the plain uninterruptible sleep stays unkillable. Two consequences: the task is not a zombie (Z, already dead, waiting for its parent to collect it), and for a plain uninterruptible wait (a request to a hung disk or a stuck driver) you cannot fix this by killing it; you must make the wait end. The wait name can tell you which kind you have: a wchan ending in _killable (such as the NFS ones below) means kill -9 will work even though ps shows D (a killable wait need not carry that suffix; the kernel_clone wait in the demo below is killable too). That matters for NFS: since 2.6.25 a hard-mounted NFS task waiting on a dead server sleeps in the killable form (nfs_wait_bit_killable and rpc_wait_bit_killable return -ERESTARTSYS when a fatal signal is pending in kernel source), so kill -9 normally clears those, although the mount itself stays hung and new tasks that touch it will pile up again.

Why load is high with idle CPUs. Linux's load average counts tasks that are runnable (R) plus tasks in D (the kernel adds the running and uninterruptible task counts in kernel/sched/loadavg.c). Fifty tasks stuck in D push load to 50 while the CPU sits idle. A high load with low CPU utilisation is a strong hint that you have a D-state pile-up, not a compute problem.

What I would check, in order.

1. Count and identify the stuck tasks, and whether they share a wait location:

ps -eo pid,stat,wchan:32,comm | awk 'NR==1 || $2 ~ /^D/'

wchan (wait channel) is the name of the kernel function where the task sleeps. If forty tasks all show the same NFS or block-layer function (the block layer is the kernel code between file systems and storage devices), you have one root cause. Illustrative output (not from a run) of a pile-up behind a dead NFS server; these waits are killable, so kill -9 would remove those processes, while the same pile-up behind a hung disk would show block-layer or driver wait functions that ignore it:

  PID STAT WCHAN                            COMMAND
 2210 D    nfs_wait_bit_killable            java
 2214 D    nfs_wait_bit_killable            java
 2301 D    rpc_wait_bit_killable            rsync
 ... 37 more lines, 36 of them nfs_wait_bit_killable ...

One function name repeated across nearly every line says the tasks are waiting on one thing, so the next step is that thing (the NFS mount), not each process.

2. Get the full kernel stack of one or two of them (needs root, and a kernel built with CONFIG_STACKTRACE, the option that lets the kernel record call stacks, per proc_pid_stack(5)):

cat /proc/<pid>/wchan; echo
cat /proc/<pid>/stack

Read it bottom to top: the bottom is the system call the application made (read, write, fsync, open), the top is where it is parked. Frames like nfs_* or rpc_* mean an NFS server or network stall, io_schedule (the kernel function that puts a task to sleep until its I/O finishes) under a filesystem function means waiting for a block device, a driver function means a hung device. A stuck NFS read might look like this illustrative stack (top frame first):

[<0>] rpc_wait_bit_killable+0x1c/0x60
[<0>] nfs_wait_on_request+0x40/0x90
[<0>] nfs_file_read+0x88/0xd0
[<0>] vfs_read+0x9c/0x1a0
[<0>] __x64_sys_read+0x1a/0x30

The bottom line is the read system call, the middle shows the NFS client, and the top shows it parked waiting on the RPC (remote procedure call) reply from the server.

3. Dump all blocked tasks at once, including stacks, to the kernel log:

echo w > /proc/sysrq-trigger     # sysrq ('magic SysRq', a kernel debugging interface, needs CONFIG_MAGIC_SYSRQ) 'w' = dump tasks in uninterruptible state
dmesg | tail -200

Also search dmesg for "blocked for more than ... seconds". The hung-task detector (a kernel watchdog thread that periodically looks for tasks stuck in D; it exists only in kernels built with CONFIG_DETECT_HUNG_TASK, which most distribution kernels enable) prints that warning with a stack for a task stuck in D longer than kernel.hung_task_timeout_secs (check your value with sysctl; 0 disables it). It skips tasks in the killable form of D (a comment in kernel/hung_task.c reads "skip the TASK_KILLABLE tasks -- these can be killed"), so a pile-up behind a hard NFS mount, whose waits are killable, can raise the load average without any such warning, while a plain uninterruptible wait (a hung disk or driver) does produce one.

4. Follow the stack to the resource. For storage: iostat -x 1 (is one device at 100% utilisation with a growing queue, aqu-sz, or is it at zero throughput because requests are not completing? Illustrative bad sign: %util 100 with r/s and w/s near 0 and aqu-sz large), dmesg for SCSI/NVMe resets, timeouts and multipath path failures (multipath means one disk reachable over several network or cable paths, and a failure of all paths stalls I/O), and lsblk / cat /proc/mounts for the device. For NFS: grep nfs /proc/mounts to see hard or soft, then check whether the server answers (ping; rpcinfo -p <server>, which lists the server's RPC services and hangs or errors if it is unreachable; nfsiostat, which shows per-mount operation counts and round-trip times, so a mount with requests outstanding and no replies is stalled). With the default hard mount (retry forever; the alternative soft gives up and returns an error after a timeout), NFS requests are retried indefinitely (nfs(5)), so a dead server freezes every task that touches the mount, silently, until it returns. For a driver hang: dmesg around the first error, and the stack's top function names the driver.

5. Find the common trigger: what changed just before the first stuck task (a storage failover, a network switch change, a server reboot, a full RAID rebuild (RAID: several disks combined into one volume), a new kernel module)? Look at the timestamp of the oldest blocked for more than message, when the waits are of the kind that produces one.

Fix. Restore the thing being waited on (bring the NFS server or the network path back, fix or fail over the storage path, reset the device). As soon as the awaited event completes, the tasks wake and any pending SIGKILLs are acted on, so stuck tasks usually disappear on their own. For tasks stuck on a dead NFS server specifically, kill -9 on the killable waits frees the processes without waiting for the server, but the mount keeps hanging every new access until it is fixed or detached. If the server is gone for good, a forced or lazy unmount (umount -f forces it; umount -l detaches the mount point immediately and cleans up later) can detach the mount, but be careful: it can lose unwritten data, and a task already blocked in the kernel may stay stuck. A reboot is the last resort. A soft NFS mount avoids the freeze but, as nfs(5) warns, can cause silent data corruption, so I would choose it only where responsiveness matters more than data integrity. For the service itself, put timeouts and a health check that does not touch the suspect mount, so the load balancer drains the node while you repair it.

A reproducible D state. This program parks itself in a killable D wait with vfork() (a variant of fork where the parent is suspended until the child execs or exits, because the child borrows the parent's memory) and has a monitor process look at it from outside. It is the easiest D state to produce safely on a laptop; it is a stand-in for the mechanics, and real incidents look like the NFS examples above.

// dstate.c: park a process in a killable D-state wait and look at it from outside.
// Build: gcc -O2 -Wall -Wextra dstate.c -o dstate     Run as root (needed for /proc/<pid>/stack)
#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/wait.h>

int main(void) {
    char pid[32];
    snprintf(pid, sizeof pid, "%d", (int)getpid());
    if (fork() == 0) {                      // monitor: inspects the parent while it is stuck
        sleep(1);
        execl("/bin/sh", "sh", "-c",
              "ps -o pid,stat,wchan:32,comm -p $0; grep State /proc/$0/status; "
              "echo \"wchan: $(cat /proc/$0/wchan)\"; head -3 /proc/$0/stack; "
              "kill -KILL $0; sleep 0.2; ps -p $0 >/dev/null && echo still-there || echo gone-after-SIGKILL",
              pid, (char *)NULL);
        _exit(127);
    }
    if (vfork() == 0) {                     // the parent is suspended until this child execs or exits
        sleep(5);
        _exit(0);
    }
    return 0;
}

Built and run as root in a --privileged gcc:14 container on arm64 (/proc/<pid>/stack needs root and CONFIG_STACKTRACE; without root the head line fails with Permission denied while the first three views still work), it prints the following, apart from the PID and a Killed notice from the shell:

    PID STAT WCHAN                            COMMAND
     15 D    kernel_clone                     dstate
State:	D (disk sleep)
wchan: kernel_clone
[<0>] kernel_clone+0x1a0/0x430
[<0>] __arm64_sys_clone+0x64/0x98
[<0>] do_el0_svc+0x70/0xd0
gone-after-SIGKILL

This shows what the three views look like (ps STAT D, /proc/<pid>/status "disk sleep", the stack with the system call at the bottom). Note that "disk sleep" is just the label for D and this wait involved no disk. This particular wait is killable, so kill -KILL removes the process at once (gone-after-SIGKILL). A true TASK_UNINTERRUPTIBLE wait, such as a request outstanding to a hung disk or a driver stuck in the kernel, would ignore it, and a dead NFS server is not reproduced here.

Direct answer

Firmware initializes the hardware and finds a bootloader; the bootloader loads the Linux kernel and an initial RAM filesystem (initramfs) into memory; the kernel decompresses itself, initializes the CPU, memory and devices, and runs /init from the initramfs; that code finds and mounts the real root filesystem and hands over to PID 1 (usually systemd), which starts services until a login prompt appears. The initramfs exists because the kernel often cannot reach the real root filesystem by itself: the drivers, disk encryption, RAID (several disks combined into one volume) or volume manager (software that carves a pool of disks into flexible logical volumes) needed to read it may be modules that live on that very disk.

Stage by stage

1. Firmware. On modern PCs this is UEFI (Unified Extensible Firmware Interface), the successor to legacy BIOS. It runs power-on self-test, initializes memory and basic devices, and picks a boot entry. With UEFI it reads a bootloader file (a .efi program) from the EFI System Partition, a small FAT filesystem (a simple, widely supported file system format that firmware can read). Legacy BIOS instead loads the first 512-byte sector of the boot disk and chains onward. Secure Boot, if enabled, makes the firmware verify the bootloader's signature before running it.
2. Bootloader (GRUB, systemd-boot, or the kernel itself acting as an EFI program). It shows a menu, loads the kernel image and the initramfs into RAM, builds the kernel command line (parameters such as root= and quiet), and jumps into the kernel.
3. Kernel start-up. The compressed image decompresses itself into memory. The kernel then sets up memory management, the interrupt table, the scheduler and timers, discovers hardware, and initializes built-in drivers. It mounts the initramfs as its first root and starts /init as the first user-space process.
4. initramfs stage. /init loads the missing driver modules (storage controller, filesystem, network for network root), assembles RAID or logical volumes (resizable virtual partitions built on top of one or more disks, managed by LVM), unlocks encrypted volumes (asking for a passphrase), finds the real root device, mounts it, and switches the root over to it (with switch_root, which also frees the initramfs memory). The kernel documentation describes the same idea for the older initrd mechanism: boot with a minimal set of built-in drivers, load the rest from a RAM disk, mount the real root, then run /sbin/init from it.
5. PID 1 and user space. On most distributions PID 1 is systemd. It mounts the remaining filesystems, brings up devices and the network, and starts units until it reaches the target named by default (a target is a named group of units that marks a system state: multi-user.target for text mode, graphical.target for a desktop).
6. Login prompt. A getty program (a small program that waits on a text console and starts a login; for example getty@tty1.service) opens a terminal and runs login, or a display manager shows a graphical login.

What is in memory at each handoff

Why an initramfs exists

Think of it as a tiny temporary operating system whose job is to reach the real one. The kernel binary should not carry every possible disk controller, filesystem and encryption driver, so distributions ship a generic kernel with most drivers as modules. A module for the root disk cannot be read from the root disk before the disk is readable. The initramfs breaks that circle: the bootloader can read it (it is a single file next to the kernel), and it carries exactly the modules and scripts needed for this machine.

Where things go wrong (and what each symptom tells you)

• No bootloader entry or "no boot device": firmware stage, boot order or ESP problem.
• Kernel panic "unable to mount root fs": initramfs lacks the driver or the root= parameter is wrong.
• Drops to an emergency shell asking for a passphrase or after a missing volume: initramfs stage.
• Boots then hangs in services: systemd stage; systemctl --failed and journalctl -b show why.

Pitfalls

• Confusing initramfs and initrd: initrd is an older block-device RAM disk; initramfs is a cpio archive (a simple file that bundles many files together, like a tar) that the kernel unpacks into a RAM-backed filesystem. Both serve the same purpose here.
• Assuming the kernel mounts the real root itself: on typical distributions it is the initramfs code that does it.
• A security note: this chain (firmware, bootloader, kernel, initramfs) is also where persistent malware hides (a tampered bootloader or a modified initramfs survives reinstalling programs), which is why signed boot chains matter.

The kernel's view. To the kernel a process's memory is a set of VMAs (virtual memory areas, each a start, an end and permissions). The stack and the heap are just two VMAs with different growth rules; nothing in the hardware says "stack".

Stack. At execve (the system call that starts a new program in a process) the kernel builds the initial stack near the top of user space (with argv, the environment variables, and the auxiliary vector, a table of facts the kernel passes to the program such as the page size) and marks the VMA as growing downward. When code pushes below the current bottom, the access faults, the kernel sees the address is just under a grow-down VMA and within the limit, and extends the VMA, which is how a stack grows without anyone asking. The limit is RLIMIT_STACK (a per-process resource limit that caps how large the main thread's stack may grow). The demo below printed 8192 KiB (8 MiB) in its container, the usual default. Beyond the limit, or into the kernel's protected gap, the fault becomes a SIGSEGV. That gap is stack_guard_gap, 256 pages (1 MiB with 4 KiB pages) by default in the kernel source mm/mmap.c (the stack_guard_gap= boot parameter changes it): the kernel refuses to let another mapping sit directly beneath the main stack, so a runaway stack cannot quietly grow into a neighbour. That is the guard for the one stack that grows on demand. Other threads' stacks work differently: they are fixed-size ordinary mmap regions that the C library allocates, and they do not grow, so the library puts a guard page (a page with no access rights) at the low end; touching it faults. So the two guards differ in who makes them and in which stack they protect: a kernel-enforced 1 MiB gap under the growable main stack, and a library-made one-page no-access region at the end of each fixed thread stack.

Heap. The classic heap is the program break (brk, the end of the process's data segment), which grows upward; larger or later allocations come from separate anonymous mmap regions. The C library's allocator (malloc) chooses between them, and its internals are a language-memory topic. The kernel's part is simply to add address ranges on request and to back them with frames lazily on first touch.

Why the failures look different. Stack growth is implicit: the program never asks, the access just lands in unmapped space, so exhaustion is a fault in the middle of a normal instruction. The kernel kills the process with SIGSEGV (a handler cannot run on the overflowed stack, so a program that wants to report the crash installs an alternate signal stack with sigaltstack, a separate small memory area the kernel switches to when running a signal handler). Heap growth is explicit: the program asks via malloc, which calls brk or mmap, and a refusal comes back as an error code (malloc returns NULL and errno, the C library's error-code variable, is set to ENOMEM, "out of memory"), which code can handle. The real exception is overcommit: the kernel often grants more memory than physically exists (it promises address space and only backs pages when they are touched), so it says yes and runs out of RAM later, and the failure becomes the OOM killer (the kernel's last resort that picks a process and kills it) sending SIGKILL when pages are touched, which no code can intercept.

Demo (Linux aarch64 container, built gcc -O0 -w stack.c -o stack):

#define _GNU_SOURCE
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/resource.h>
#include <unistd.h>

static void on_segv(int sig) {
    static const char m[] = "SIGSEGV: stack limit reached (handler runs on alternate stack)\n";
    (void)sig; write(1, m, sizeof m - 1); _exit(0);
}
static volatile char sink;
static long depth;
static void recurse(void) {
    volatile char buf[1024];
    buf[0] = 1; sink = buf[0];
    depth++;
    recurse();
    sink = buf[0];
}
int main(void) {
    struct rlimit rl;
    setvbuf(stdout, NULL, _IONBF, 0);
    getrlimit(RLIMIT_STACK, &rl);
    printf("stack limit: %lu KiB\n", (unsigned long)rl.rlim_cur / 1024);
    /* heap: cap address space, then ask for more than the cap */
    struct rlimit as = { 256UL << 20, 256UL << 20 };
    setrlimit(RLIMIT_AS, &as);
    void *p = malloc(512UL << 20);
    printf("malloc(512 MiB) under 256 MiB RLIMIT_AS: %s\n", p ? "ok" : "NULL (ENOMEM), program keeps running");
    static char altstack[16384];
    stack_t ss = { altstack, 0, sizeof altstack };
    sigaltstack(&ss, NULL);
    struct sigaction sa; memset(&sa, 0, sizeof sa);
    sa.sa_handler = on_segv; sa.sa_flags = SA_ONSTACK;
    sigaction(SIGSEGV, &sa, NULL);
    recurse();
    return 0;
}

stack limit: 8192 KiB
malloc(512 MiB) under 256 MiB RLIMIT_AS: NULL (ENOMEM), program keeps running
SIGSEGV: stack limit reached (handler runs on alternate stack)

Reading the demo: in recurse, volatile tells the compiler the stores to buf and sink must really happen, so it cannot optimise the recursion into a loop or drop the 1 KiB array; each call therefore consumes at least 1 KiB of stack, and 8 MiB runs out after at most about 8,000 calls. In main, setrlimit(RLIMIT_AS, ...) caps the total address space at 256 MiB (RLIMIT_AS is the resource limit on address-space size), so the following 512 MiB malloc must fail with NULL. For the handler: altstack is a static 16 KiB array, stack_t ss describes it to sigaltstack (base pointer, flags 0, size), and sigaction with SA_ONSTACK registers on_segv and tells the kernel to run it on that alternate stack instead of the exhausted one; without it the handler could not even start.
In this demo the recursion stops at the 8 MiB RLIMIT_STACK limit; the guard gap and thread guard pages described above give the same SIGSEGV when a stack runs into them. Removing SA_ONSTACK makes the program die with a plain segmentation fault (exit status 139) instead of printing the message. Heap exhaustion became a catchable NULL; unbounded recursion became a SIGSEGV that is only reportable because of the alternate stack.

Practical consequences. Large arrays belong on the heap, not in a local variable, and deep recursion should be bounded or turned into a loop; ulimit -s raises the main-thread limit but thread stack size is set when the thread is created (pthread_attr_setstacksize). To diagnose a stack overflow, look at the repeating frames in the backtrace and the crash address just below the stack VMA in /proc/<pid>/maps.

Direct answer

With overcommit, Linux lets a process reserve (commit) more virtual address space than there is RAM plus swap, because most reserved memory is never touched. A call like malloc or mmap therefore succeeds at once, and the real memory is only taken page by page when the program first touches each page (a page fault: the CPU traps to the kernel, which finds a physical frame and maps it). If the machine or the container's cgroup (control group: the kernel's resource-limit group) has no frame left at that moment, there is no error code to return to the faulting instruction, so the kernel runs the OOM killer (out-of-memory killer) and kills a process. That is why a process can die at a line of code that looks harmless, long after the allocation call that "succeeded". Containers matter because the overcommit mode is machine-wide, while the limit that kills a container is its cgroup's memory.max, and that limit counts more than the number ps calls resident size.

The three modes of vm.overcommit_memory

(From the kernel's overcommit documentation.)

For a concrete case in mode 0, on a machine with 16 GiB of RAM and 2 GiB of swap, one mmap of 20 GiB is refused (20 > 18) but a 17 GiB request is allowed, and so are several 10 GiB requests in a row, because each is checked alone and only the size of that one request is compared. That gap is the overcommit. Mode 1 would allow the 20 GiB request too.

Commit accounting. The kernel tracks Committed_AS: the sum of address space that the system has promised to processes. In mode 2 it compares that to CommitLimit and makes mmap, brk or fork return ENOMEM up front. CommitLimit is swap plus RAM times the ratio. Example from the Docker VM used for the runs below (values in kB, host-specific; read yours with grep -E 'MemTotal|SwapTotal|CommitLimit' /proc/meminfo and cat /proc/sys/vm/overcommit_ratio; the kernel also subtracts huge pages reserved for hugetlb from the RAM term):

CommitLimit = SwapTotal + MemTotal x 50 / 100
            = 17,462,192 + 16,413,624 x 0.50
            = 17,462,192 + 8,206,812 = 25,669,004

which matches the CommitLimit: 25669004 kB that /proc/meminfo printed there. The limit is only enforced in mode 2; in modes 0 and 1 it is informational.

Why allocation succeeds and the kill comes later

This script reserves 2 GiB of address space, touches only 64 MiB, and then (given a second argument) touches everything:

import mmap, os, sys

def status(label):
    f = {l.split(":")[0]: l.split(":")[1].strip() for l in open("/proc/self/status") if l.startswith(("VmSize", "VmRSS"))}
    print(f"{label:<34} VmSize={f['VmSize']:>12}  VmRSS={f['VmRSS']:>10}", flush=True)

print("overcommit_memory =", open("/proc/sys/vm/overcommit_memory").read().strip())
status("start")
big = mmap.mmap(-1, 2 * 1024**3, flags=mmap.MAP_PRIVATE | mmap.MAP_ANONYMOUS)   # reserve 2 GiB of address space
status("after mmap 2 GiB (untouched)")
mb = int(sys.argv[1])
for off in range(0, mb * 1024**2, 4096):                                         # touch only mb MiB
    big[off] = 1
status(f"after touching {mb} MiB")
if len(sys.argv) > 2:                                                            # keep going until the limit bites
    for off in range(mb * 1024**2, 2 * 1024**3, 4096):
        big[off] = 1
    status("touched all 2 GiB")

Run in a python:3.12-slim container with no limit (python overcommit.py 64), then with a 256 MiB cgroup limit (docker run --rm -m 256m --memory-swap 256m ... sh -c 'python overcommit.py 64 all; echo "exit status=$?"; cat /sys/fs/cgroup/memory.max; grep -E "^(oom|oom_kill) " /sys/fs/cgroup/memory.events'):

--- no limit, touch 64 MiB
overcommit_memory = 1
start                              VmSize=    13704 kB  VmRSS=   9612 kB
after mmap 2 GiB (untouched)       VmSize=  2110856 kB  VmRSS=   9676 kB
after touching 64 MiB              VmSize=  2110856 kB  VmRSS=  75332 kB
--- memory limit 256m, touch everything
overcommit_memory = 1
start                              VmSize=    13704 kB  VmRSS=   9640 kB
after mmap 2 GiB (untouched)       VmSize=  2110856 kB  VmRSS=   9704 kB
after touching 64 MiB              VmSize=  2110856 kB  VmRSS=  75360 kB
Killed
exit status=137
268435456
oom 1
oom_kill 1

The command pieces: -m 256m caps the container's memory at 256 MiB (this becomes memory.max); --memory-swap 256m sets memory plus swap to the same figure, so no swap is allowed and the limit is hard; sh -c '...' runs several commands in order inside the container; echo "exit status=$?" prints the exit code of the Python run; cat /sys/fs/cgroup/memory.max prints the limit in bytes; and the grep pulls the oom and oom_kill counters out of memory.events, a file of counters the kernel keeps per cgroup.

Read it like this: VmSize (virtual size, the address space reserved) jumped by about 2 GiB at mmap while VmRSS (resident set size, the pages actually in RAM) barely moved. The mmap of 2 GiB succeeded inside a 256 MiB container because the limit counts used pages, not reserved ones. The process was killed (exit status 137 = 128 + signal 9) only when touching pushed usage to memory.max = 268,435,456 bytes, and memory.events recorded oom 1 and oom_kill 1. The Docker VM used for these runs was in mode 1, so this run shows mode 1 plus a cgroup limit, not the default heuristic mode.

Why the resident size can look modest and the process still dies

memory.max limits memory.current, which per the cgroup v2 documentation includes more than anonymous memory: page cache (file contents the kernel keeps in RAM), kernel data structures such as dentries (cached directory-entry lookups) and inodes (the kernel's per-file records), and network socket buffers (queues of data waiting to be sent or read). Things ps RSS for one process does not show:

• Other processes in the same cgroup. The limit is shared by every process in the container, including sidecars, shells and forked workers.
• Page cache and tmpfs. tmpfs is a file system that lives entirely in RAM, and /dev/shm is a tmpfs directory used for shared memory between processes. Files written, files read, and anything under /dev/shm or a tmpfs mount are charged to the cgroup. Clean cache can be reclaimed, but dirty or shared-memory pages cannot be dropped quickly.
• Kernel memory: page tables for a large mapping, slab objects (small fixed-size kernel allocations such as the dentries and inodes above), socket buffers.
• A small numeric case. A container with a 512 MiB limit runs one process with 200 MiB RSS, but it also wrote a 250 MiB log file (250 MiB of page cache) and holds 30 MiB of tmpfs and socket buffers. memory.current is 200 + 250 + 30 = 480 MiB. The clean 250 MiB of cache can be reclaimed, so this is survivable, and even dirty log pages can be reclaimed once they are written back (reclaim stalls the allocating thread meanwhile). What cannot be reclaimed is tmpfs and shared memory when there is no swap, plus kernel memory: if the tmpfs share grows past what the cache reclaim can make room for, the kernel kills a process even though ps still shows 200 MiB.
• Memory that was not resident when you looked: a spike between samples, or pages touched after your last measurement.

Also, the OOM killer picks a victim by its OOM score (a badness number the kernel computes per process, mainly from how much memory it uses, adjustable through oom_score_adj), so with several processes in a cgroup the one killed is not necessarily the one that grew. By default a cgroup OOM kills one process, not the whole group, unless memory.oom.group is set to 1, which makes the kernel kill every process in that cgroup together.

What this means for containers

• Set the memory limit from measured memory.current at peak load plus headroom, not from the process RSS. Watch memory.events for oom_kill, and know that memory.high throttles with heavy reclaim but never invokes the OOM killer, so it makes a good soft ceiling below memory.max. Of the cgroup files, memory.max (the hard limit) and memory.events (the evidence) are the two to know first; memory.high and memory.oom.group are tuning.
• Managed runtimes (the JVM, Go, Node) need their heap caps set from the container limit, not the machine's RAM, or they size themselves for the host and are killed at the cgroup line.
• Do not rely on vm.overcommit_memory=2 to protect one container: it is machine-wide and makes unrelated fork and mmap calls fail (a large process forking needs a second copy's worth of commit even though copy-on-write would share it). Use cgroup limits for per-container isolation and overcommit settings for the host's global policy.
• Exit status 137 and an event in memory.events are the evidence to look for when a container "just disappears".

The idea. Each process sees its own virtual address space: a private range of addresses that the CPU's MMU (memory management unit) translates to physical RAM through the process's page tables (the kernel's lookup tables of which virtual page is backed by which physical page, with read/write/execute permissions). The kernel divides that space into regions (called VMAs, virtual memory areas), each with a start, end, permissions and a backing. Two processes can use the same address for different data because their page tables differ.

Layout of a typical 64-bit Linux process, low addresses to high

Seeing it: a program that prints where its own variables live. Compiled and run in a Linux container (gcc 14.4.0, gcc -O0 -Wall -Wextra -fPIE -pie addr_regions.c -o addr_regions). The container is aarch64, so the absolute addresses are ARM-style; the same program on x86-64 puts user addresses below 0x0000800000000000.

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>

int global_data = 42;          /* .data */
int global_bss;                /* .bss */
const char ro_text[] = "ro";   /* .rodata */

static void where(const char *name, void *p) {
    unsigned long a = (unsigned long)p;
    FILE *f = fopen("/proc/self/maps", "r");
    char line[512];
    while (fgets(line, sizeof line, f)) {
        unsigned long lo, hi; char perms[8], rest[400] = "";
        if (sscanf(line, "%lx-%lx %7s %*s %*s %*s %399[^\n]", &lo, &hi, perms, rest) >= 3
            && a >= lo && a < hi) {
            printf("%-10s %p -> %lx-%lx %s %s\n", name, p, lo, hi, perms, rest);
            break;
        }
    }
    fclose(f);
}

int main(void) {
    int local = 1;
    void *small = malloc(100);
    void *big = malloc(1 << 20);
    void *anon = mmap(NULL, 1 << 20, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
    where("code", (void *)main);
    where("rodata", (void *)ro_text);
    where("data", &global_data);
    where("bss", &global_bss);
    where("heap", small);
    where("malloc1MiB", big);
    where("mmap", anon);
    where("libc", (void *)printf);
    where("stack", &local);
    return 0;
}

Output from one run (addresses change on every run because of ASLR):

code       0xaaaac40e0a74 -> aaaac40e0000-aaaac40e1000 r-xp /w/addr_regions
rodata     0xaaaac40e0b90 -> aaaac40e0000-aaaac40e1000 r-xp /w/addr_regions
data       0xaaaac4100070 -> aaaac4100000-aaaac4101000 rw-p /w/addr_regions
bss        0xaaaac4100078 -> aaaac4100000-aaaac4101000 rw-p /w/addr_regions
heap       0xaaaad7a5a2a0 -> aaaad7a5a000-aaaad7a7b000 rw-p [heap]
malloc1MiB 0xffff9043f010 -> ffff9033f000-ffff90540000 rw-p 
mmap       0xffff9033f000 -> ffff9033f000-ffff90540000 rw-p 
libc       0xffff9058cb40 -> ffff90540000-ffff906dc000 r-xp /usr/lib/aarch64-linux-gnu/libc.so.6
stack      0xffffc1495394 -> ffffc1475000-ffffc1496000 rw-p [stack]

Reading it: code and rodata share one r-xp mapping on this small program (the linker packed them together), data and bss share one rw-p mapping, the 100-byte malloc lands in [heap], and the 1 MiB malloc landed in an anonymous mapping next to the explicit mmap (large requests bypass the heap and ask the kernel for fresh pages), printf is in libc.so.6's executable mapping, and the stack is at the top.

Reading /proc/<pid>/maps. Each line is address perms offset dev inode pathname. offset is where in the backing file this mapping starts, dev is the device holding the file (major:minor numbers) and inode is the file's number on that device; anonymous mappings show 0 for these. The sscanf format in where() reads %lx-%lx (start and end of the range in hex), %7s (the perms string), then %*s %*s %*s (offset, dev and inode: the * means parse and discard them) and %399[^\n] (the rest of the line, the pathname). perms letters: r read, w write, x execute, and p private (copy-on-write) or s shared. The pathname is the backing file, or special names [heap], [stack], [vdso] (a small kernel-provided library for fast system calls), or blank for an anonymous mapping (from mmap or malloc).

From a faulting address to a region (ordered checks)

1. Get the faulting address. For a crash (SIGSEGV, the signal the kernel sends a process that makes an illegal memory access) it is in the signal information (the si_addr field of the siginfo_t structure the kernel passes to the signal handler) and debuggers print it. A core dump is the file the kernel writes at such a crash, holding the process's memory and registers, so you can inspect it afterwards. Also note whether the fault was a read, write or instruction fetch.
2. Get the mappings at the time of the crash. For a live process, read /proc/<pid>/maps. For a core dump, load it in gdb and use info proc mappings (the gdb documentation says process information is available from cores created on GNU/Linux; if your gdb version prints nothing, readelf -l core lists the memory segments the core recorded). Use the same binary and libraries that crashed, or addresses will not line up.
3. Find the line where start <= address < end. Then classify:
   • No line contains it: unmapped memory. A wild or freed pointer, or one already unmapped by munmap.
   • Between 0 and the first mapping, especially under 4 KiB: a NULL pointer plus a small field offset (the lowest addresses are unmapped on purpose: vm.mmap_min_addr, from 4 KiB up to commonly 64 KiB depending on configuration, keeps user mappings away from them).
   • Inside [heap] or an anonymous region: a heap bug such as use-after-free or an overrun; ask for a sanitizer run (AddressSanitizer, a compiler option that makes every heap access checked and reports out-of-bounds and use-after-free with the exact line) to confirm.
   • Just below the [stack] mapping's start, or at a guard region (a page or range deliberately mapped with no access, placed to catch overruns) next to a thread stack: stack overflow, usually runaway recursion or a huge local array.
   • Inside a file-backed mapping with a permission that forbids the access (a write to an r-x or r-- region such as .rodata, or a jump into rw- data): a permission fault. Writing to a string literal is the classic case.
   • Inside a library mapping: look up which library from the pathname, subtract the library's load base (the start of its first mapping, which normally begins at file offset 0) to get an offset inside the library, and map that offset to a symbol with addr2line -f -e <library> <offset> (prints the function name; -e names the file) or by finding the closest lower address in nm output (a listing of a file's symbols and their addresses).
   • An address in the kernel half or the non-canonical hole: user code cannot access either; this is usually a corrupted pointer (an overwritten pointer holding data such as ASCII text). On x86-64 an access to a non-canonical address raises a general-protection fault rather than a page fault, and the resulting SIGSEGV commonly reports si_addr as 0, so check the registers and the faulting instruction in the core rather than relying on si_addr alone.
4. Check the instruction pointer separately: the faulting data address says what was touched, the program counter says which code touched it. Both are needed.

Worked examples using the run above. Pretend each of these came out of si_addr (addresses are from that one run and are illustrative):

• 0xffff9058cb40: line ffff90540000-ffff906dc000 r-xp .../libc.so.6 contains it, and the access was an instruction fetch, so the crash is inside libc code. Offset = 0xffff9058cb40 - 0xffff90540000 = 0x4cb40, which you pass to addr2line -f -e libc.so.6 0x4cb40. This works here because, in this run, the executable mapping is libc's first mapping and starts at file offset 0 (its /proc/<pid>/maps offset column reads 00000000); on builds where a read-only mapping comes first, subtract the lowest start address among that library's mappings instead.
• 0xffffc1474ff8: no mapping contains it, and it is 8 bytes below the start of the [stack] line (0xffffc1475000 - 8). That points to a stack overflow: a push or a huge local array stepped past the bottom of the stack. One caveat about this illustration: the [stack] line in the run is only 132 KiB (0xffffc1475000 to 0xffffc1496000), and for an access just below a main-thread stack that is still under its size limit the kernel grows the mapping instead of faulting. A genuine overflow therefore shows a [stack] line already grown to the limit (ulimit -s, 8192 KiB by default) with the fault just below it or in the guard gap the kernel keeps below the stack, so read this example as the shape of the check, not as a faithful crash.
• 0xaaaac40e0b90 (the address of ro_text) on a write: the line aaaac40e0000-aaaac40e1000 r-xp contains it, but the perms have no w. That is a permission fault, the classic write to a read-only literal.
• 0x18: below every mapping and under 4 KiB, so a NULL pointer plus a field at offset 24 (0x18).

Leak investigation follow-up. Compare /proc/<pid>/maps (or pmap) over time: a growing [heap] points to malloc growth, many new anonymous regions point to leaking mmap calls or thread stacks, and growth inside one library's mappings points to that library.