System Calls & the Kernel Interface Questions

seccomp ("secure computing mode") restricts the set of syscalls a process, and everything it later forks or execs (filters are inherited and can only be narrowed further, never removed), is allowed to make. seccomp-bpf is the modern form: the filter is an actual BPF (Berkeley Packet Filter, the same in-kernel bytecode VM originally built for packet filtering) program that the kernel runs against every syscall's number and arguments before it executes, returning an action such as SECCOMP_RET_ALLOW, SECCOMP_RET_KILL_PROCESS (terminate immediately), SECCOMP_RET_ERRNO (fake a specific error without ever running the syscall), SECCOMP_RET_TRACE (hand the decision to a ptrace-attached tracer), or SECCOMP_RET_LOG (allow the syscall, but log the fact that this filter rule fired, via the kernel audit subsystem).

1. How seccomp filters work with BPF programs

Because it's a real (restricted) program, not just a static list, a seccomp-bpf filter can inspect syscall arguments too, not only the syscall number: e.g. allow ioctl() for a specific request code but deny it for others, or allow open()/openat() unconditionally while still denying ptrace() outright. The filter is compiled and loaded via seccomp(2) (or, in practice, almost always through the higher-level libseccomp library rather than hand-written BPF bytecode, since getting the raw bytecode right, including 32-bit-vs-64-bit architecture checks, is notoriously easy to get subtly wrong in ways that quietly leave a bypass).

2. A safe rollout plan

• Discover the syscalls the service actually needs. Run the service under strace -f -c (or attach with strace against a running instance) across a representative workload, and critically, across a long enough window to hit the rarely-exercised paths: startup, graceful shutdown, log rotation, signal handling, a dependency's periodic health-check or retry path, and at least one full deploy cycle. Coverage gaps during discovery are the single most common cause of a seccomp rollout killing a production process days after it "worked in testing," because the missing syscall was one that only fires under a rare condition (an error path, a weekly cron-triggered branch, a library upgrade that started using a new syscall).
• Build the whitelist. Take the discovered syscall set, add the universally-needed baseline (read, write, exit_group, rt_sigreturn, mmap, brk, futex, and similar), and build the filter with libseccomp's seccomp_rule_add() API rather than raw BPF, specifically to avoid the architecture-check and encoding mistakes that make hand-rolled filters a common source of accidental bypasses.
• Test in "log mode." Deploy with the default action set to SECCOMP_RET_LOG instead of a kill/deny action, so anything not on the whitelist is logged (visible via auditd/ausearch -m seccomp or the kernel log) but still allowed to run. Leave it running in this mode across the full range of real production traffic, scheduled jobs, and at least one deploy, for long enough (days, ideally spanning a weekly job cycle) that a rare code path showing up as an unexpected syscall in the log is a near-certainty before you ever flip to enforcement.
• Enforce gradually, keep a fast rollback. Once the log shows zero unexpected syscalls over the observation window, switch the default action to deny (SECCOMP_RET_ERRNO returning EPERM for a softer failure the application can potentially catch, or SECCOMP_RET_KILL_PROCESS for syscalls you're confident should never legitimately happen, like ptrace or mount in a sandboxed helper). Roll it out to a canary subset of instances first, not the whole fleet at once, and keep the rollback path a config toggle rather than a code deploy, since a false-positive kill under enforcement is a hard, immediate outage, not a degraded-mode failure.
• Keep capturing deny events after enforcement. Wire SECCOMP_RET_LOG/audit denies (or SECCOMP_RET_ERRNO failures surfaced through application error logs) into your normal observability, so a new code path that needs a syscall not on the whitelist, introduced later by an ordinary code change, a dependency upgrade, or a new glibc version (a real example: newer glibc versions started using clone3 in some paths where older versions used clone), shows up as a signal before it becomes a production incident, not after.

The raw status isn't the exit code

waitpid()'s status output parameter is a packed, platform-defined bitfield, not directly "the exit code." The exact bit layout isn't part of the portable API (which is precisely why the macro family exists), so correct code always decodes it through WIFEXITED/WEXITSTATUS/WIFSIGNALED/WTERMSIG/WCOREDUMP/WIFSTOPPED/WSTOPSIG, never by inspecting the raw integer's bits directly. I'll show the raw bytes below purely to illustrate the packing, not as something to rely on in real code.

Decoding, with real measured examples

I forked three children (a clean exit, a nonzero exit, and a fatal signal with a core dump) and printed the raw status alongside the decoded macros:

static void run_case(const char *label, void (*fn)(void)) {
    pid_t pid = fork();
    if (pid == 0) { fn(); _exit(0); }
    int status;
    waitpid(pid, &status, 0);
    printf("[%s] pid=%d raw_status=0x%04x", label, pid, status);
    if (WIFEXITED(status)) {
        printf(" WIFEXITED=1 WEXITSTATUS=%d\n", WEXITSTATUS(status));
    } else if (WIFSIGNALED(status)) {
        printf(" WIFSIGNALED=1 WTERMSIG=%d(%s) WCOREDUMP=%d\n",
               WTERMSIG(status), strsignal(WTERMSIG(status)), !!WCOREDUMP(status));
    }
}
/* case_clean: _exit(0); case_error: _exit(17); case_crash: raise(SIGSEGV); */

MEASURED output (Debian 12, ulimit -c unlimited):

[clean-exit] pid=5126 raw_status=0x0000 WIFEXITED=1 WEXITSTATUS=0
[nonzero-exit] pid=5127 raw_status=0x1100 WIFEXITED=1 WEXITSTATUS=17
[fatal-signal] pid=5128 raw_status=0x008b WIFSIGNALED=1 WTERMSIG=11(Segmentation fault) WCOREDUMP=1

• WIFEXITED true means the child called exit()/_exit()/returned from main() normally (regardless of value); WEXITSTATUS extracts the 8-bit exit code, note only 8 bits, 0 to 255, is the on-the-wire Unix exit-status width, so a language runtime that lets you exit(300) will wrap silently and you'd see 44, not 300. The 17 case above lands as 0x11 in the packed status's upper byte, and decodes cleanly back to 17.
• WIFSIGNALED true means the child was terminated BY a signal, it never reached its own exit path at all. WTERMSIG gives which signal; WCOREDUMP tells you whether the kernel actually wrote a core file, which requires the signal to be one of the core-dumping set (SIGSEGV/SIGABRT/SIGBUS/SIGQUIT and a few others), ulimit -c/RLIMIT_CORE to permit it, and enough disk/permissions for the dump to actually land. Above, 0x8b decodes as signal 11 (SIGSEGV) in the low 7 bits plus the 0x80 coredump bit set.
• WIFSTOPPED/WSTOPSIG only matter if you passed WUNTRACED (or you're a ptrace-based tracer): the child was merely stopped, not terminated, still alive, resumable with SIGCONT. A plain supervisor that never passes WUNTRACED simply never sees this case, which is usually what you want, you don't want a job-control-suspended child mistaken for one that exited.

Mapping to the restart / log / quarantine decision

• WIFEXITED and WEXITSTATUS == 0: success, normal bookkeeping, no restart needed.
• WIFEXITED with a known "won't-succeed-on-retry" convention code (say, your own agreed convention that exit code 2 means "bad config, deterministic failure"): log at high severity, do NOT restart (retrying a deterministic failure just burns cycles and floods the logs), alert a human instead.
• WIFEXITED with any OTHER nonzero code: treat as transient (a network hiccup, momentary resource contention), log it, restart with backoff, but track a per-worker restart counter so a worker that fails EVERY time still eventually trips a "restart budget exhausted, stop and page someone" threshold instead of crash-looping forever.
• WIFSIGNALED with SIGTERM/SIGINT: usually an intentional, externally requested stop (an operator, or your own supervisor issuing it as part of a controlled restart). Don't treat this as a crash on its own; correlate against whether YOUR supervisor is the one that sent it.
• WIFSIGNALED with SIGSEGV/SIGABRT/SIGBUS/SIGFPE/SIGILL (the classic "bug or tampering" set): this is the case that most warrants the "quarantine the host" branch, especially when WCOREDUMP == 1 (there's an actual core file to investigate) AND the SAME crashing signal recurs across successive restarts of the same worker. One crash might be transient memory corruption from unusual input; the SAME signal repeatedly after restart is a much stronger signal that something is deterministically wrong on that host specifically, a corrupted binary, a bad dependency version, or an exploit attempt targeting that worker, worth pulling the host for forensics rather than restarting it back into the same failure indefinitely.

Implementation note

Gate this on a per-worker restart counter with exponential backoff, and always persist the RAW exit detail (the exit code or the signal name and whether it dumped core, not just "it failed") into whatever your incident timeline uses, "worker died" without the why is close to useless hours into an on-call investigation.

A futex (fast userspace mutex) is the Linux primitive underneath essentially every userspace lock (pthread_mutex_t, semaphores, condition variables): the actual lock state is a plain integer word living in userspace, and the kernel's futex(2) syscall is only involved when there's contention.

1. The userspace fast path

In the uncontended case, acquiring and releasing a futex-backed lock never enters the kernel at all: the locking code does an atomic compare-and-swap on the futex word directly in userspace (e.g. "if the word is 0 (unlocked), set it to the caller's identity and proceed"), and unlocking is the reverse atomic operation. This is why uncontended locking is essentially free, on the order of a single atomic instruction, with no syscall overhead.

2. When the kernel's wait/wake is actually invoked

The kernel only gets involved on the slow path: when a thread's atomic compare-and-swap fails because the lock is already held, it calls futex(FUTEX_WAIT, addr, expected_value, ...). This call atomically re-checks that the word at addr still equals expected_value and, only if so, puts the calling thread to sleep on a kernel-maintained wait queue keyed off that memory location (internally, addresses are hashed into a global futex hash table). That atomic check-then-sleep is what avoids a lost-wakeup race: without it, the value could change (the lock could be released) in the gap between the userspace thread deciding to sleep and the kernel actually parking it, and the wakeup that was meant for it would have nothing to wake. The thread releasing the lock, if it noticed contention (typically via a "contended" bit or count it maintains in the same word), calls futex(FUTEX_WAKE, addr, n) to wake up to n waiters parked on that address.

3. Wait-queue semantics

Threads blocked via FUTEX_WAIT on the same address sit on that address's wait queue in the kernel; FUTEX_WAKE only wakes threads that are actually parked there right now; if no thread happens to be sleeping on that address at the moment of the wake (e.g. it hasn't reached its FUTEX_WAIT call yet), the wake is simply a no-op, which is fine precisely because the userspace protocol re-checks the actual lock word's value rather than trusting that a wake implies success. Per the futex(2) man page, a FUTEX_WAIT return of 0 doesn't even guarantee the wake was "for you": spurious wakeups can happen (e.g. from unrelated code that previously used the same memory address), so correct callers always re-check the userspace lock word after waking, not just trust the return value.

4. How futexes show up in strace

Running strace -f (following forked/cloned threads) against a multi-threaded program under lock contention shows lines like:

futex(0x7f2a1c0009d0, FUTEX_WAIT_PRIVATE, 2, NULL) = 0
futex(0x7f2a1c0009d0, FUTEX_WAKE_PRIVATE, 1) = 1

The _PRIVATE suffix (FUTEX_PRIVATE_FLAG) is a performance hint meaning the futex is only ever used within a single process's threads (not shared across processes via shared memory), which lets the kernel skip some address-space bookkeeping it would otherwise need for a possibly cross-process futex. A FUTEX_WAIT returning -1 ETIMEDOUT means a timed wait expired without a wake; -1 EAGAIN means the value at the address didn't match expected_value at check time (the caller lost the race and should retry in userspace rather than sleep).

5. Diagnosing many threads blocked in FUTEX_WAIT with latency spikes

Many threads parked in FUTEX_WAIT means heavy lock contention: threads are repeatedly hitting the slow path instead of acquiring uncontended. To diagnose:

• Confirm the shape. strace -f -c -T (or attach to a running process) on the service to see how much time and how many calls are going into futex() relative to everything else; a large fraction of wall time in FUTEX_WAIT with modest CPU usage says the bottleneck is serialization, not raw compute.
• Check thread state. ps -eLo pid,tid,state,wchan (wchan: the kernel function name the thread is blocked inside, useful for seeing what specifically it's waiting on) or /proc/<pid>/task/*/status shows each thread's state; a futex wait normally shows as interruptible sleep (state S), which is a useful contrast against D state (uninterruptible sleep, meaning the thread is blocked inside the kernel, usually on I/O, and cannot even be interrupted by a signal until the wait completes). Seeing threads in D state alongside the futex waiters usually points at an I/O bottleneck feeding the contention, not the locking itself.
• Get the actual call stack. The futex address alone doesn't say which mutex or which code path is contended; a stack-sampling profiler (perf record/perf top, or a language-level profiler like py-spy for Python, async-profiler for the JVM) attached during the contention window will show the specific lock acquisition site, which is usually the fastest way from "many futex waits" to "this specific global lock/connection pool/cache mutex."
• Look for a lock convoy or an undersized shared resource. Common root causes: one hot global lock protecting a data structure that many threads touch on every request, a connection pool or semaphore sized well below the actual concurrency level, or a single-writer structure (like a language runtime's global interpreter lock analog) being hit far more often than intended.
• Fix by reducing contention, not by adding CPU. Shrinking the critical section, sharding the lock (e.g. per-shard mutexes instead of one global one), or increasing the size of the constrained resource (pool/semaphore) addresses the actual bottleneck; more CPU cores don't help serialized code waiting on a single lock.

What a zombie is

A zombie (also called "defunct") is a process that has already called _exit() (or been killed) and released essentially everything it owned, memory, file descriptors, and any other resources, but whose EXIT STATUS entry is still sitting in the kernel's process table because the parent hasn't yet called wait()/waitpid() to collect it. It isn't "still running" in any meaningful sense, no CPU time, no memory footprint worth mentioning, it's a lightweight bookkeeping record (essentially a PID plus an exit status) that POSIX requires the kernel to keep around until a parent explicitly retrieves it, because a parent is always entitled to learn how its child exited.

How it's created

Process A fork()s child B; B exits (normally, via exit()/_exit(), or terminated by a signal); the kernel converts B into a zombie and delivers SIGCHLD to A. If A never calls wait()/waitpid(), no signal handler installed, ignoring SIGCHLD without the SA_NOCLDWAIT auto-reap behavior, or simply stuck/busy and never getting around to it, B remains a zombie indefinitely, until A itself exits, at which point B is reparented (to init/PID 1 historically, or to whatever process has registered itself as a "subreaper" via prctl(PR_SET_CHILD_SUBREAPER) on modern Linux), and that new parent reaps it.

Detection

ps aux (or more precisely, ps -eo pid,ppid,stat,cmd) shows Z in the STAT column with the command name suffixed <defunct>. MEASURED, a zombie I deliberately created and captured: ps -o pid,ppid,stat,cmd -p 5195 showed 5195 5193 Z [zombie_demo] <defunct>. The raw kernel-level view is /proc/<pid>/stat, whose third whitespace-separated field is a single character giving process state; cat /proc/5195/stat | awk '{print $3}' confirmed the literal character Z. It's worth explicitly distinguishing a zombie (Z) from a similar-looking but unrelated state, D (uninterruptible sleep): D means a process is blocked inside a kernel operation, usually disk or storage I/O, or a driver wait, that cannot even be interrupted by a signal, it is still fully alive, still consuming resources, whereas Z is a dead process consuming almost nothing but a table slot. A fast host-wide sweep is ps -eo stat= | grep -c Z; pstree -p is the useful next step, since it shows WHICH parent owns a cluster of zombies, which is the process you actually need to fix.

Consequences

Each zombie's process-table entry occupies one PID out of a finite, GLOBAL PID space (/proc/sys/kernel/pid_max, classically 32768 on older/legacy Linux configurations, tunable much higher, up to roughly 4 million on 64-bit systems, and many modern distros ship a higher default out of the box; treat 32768 as the traditional textbook figure, not a universal constant, since it varies by distro and kernel configuration). Practically: a long-running process that forks repeatedly without reaping any children will, given enough time, exhaust available PIDs for the ENTIRE HOST, not just for itself, since PID space is shared system-wide, at which point fork() anywhere on the box starts failing with EAGAIN, an outage that looks completely unrelated to its actual root cause unless whoever's investigating knows to check ps/pstree for a zombie pileup first.

Reaping strategies for long-running services/supervisors

• A blocking wait()/waitpid(pid, &status, 0) right after each fork(), fine when the supervisor only ever has one child in flight at a time, wrong the moment it needs to do anything else concurrently.
• A SIGCHLD handler that loops on waitpid(-1, &status, WNOHANG) until it returns 0 (nothing left to reap right now) or -1/ECHILD (no children at all). The loop matters because SIGCHLD, like other standard (non-realtime) signals, is not queued and can coalesce: several children exiting in quick succession can result in a single handler invocation, so a handler that reaps only one child per call will systematically leak zombies under load, exactly when the supervisor is busiest.
• Explicit signal(SIGCHLD, SIG_IGN) (or sigaction with SA_NOCLDWAIT) when the exit status genuinely doesn't matter, this tells the kernel to auto-reap children the instant they exit, so they never become zombies at all. Correct for pure fire-and-forget children; wrong the moment you need the exit code or signal to decide restart/log/quarantine behavior.

The concrete operational motivation for a scanning/sandboxing daemon

A daemon that spawns children specifically for scanning or sandboxing, a security scanner forking a per-file analysis process, say, must ACTIVELY reap them: waitpid(-1, &status, WNOHANG) on every pass through its own event loop, or a SIGCHLD handler doing the same loop, or those scan-worker zombies accumulate silently. The dangerous part is that they accumulate WITHOUT tripping the metrics anyone's actually watching: a zombie costs almost no CPU or memory, so the daemon's own health checks look completely normal while ps/PID-table headroom quietly degrades in the background. During an actual incident, a responder who tails logs and checks top/CPU/RSS sees a "healthy" daemon and, separately, a mysteriously unresponsive host that can no longer fork anything, and the connection between the two isn't obvious unless you specifically know to look for unreaped children. That's exactly the incident-response blind spot this creates: the thing that's actually broken doesn't show up anywhere near the alert that fired.

Four things are being asked here: how errno gets set, the thread-safety gap between strerror() and strerror_r(), a concrete format for a diagnostic line, and how to collect these cheaply on a hot path.

1. How errno is set

At the raw kernel ABI level there's no separate "error" signal, a failing syscall returns a small negative number that IS -errno (permission denied, EACCES, comes back as -13 in rax). The libc wrapper (open(), read(), etc) checks for that negative-in-range return, negates it, stores it into the calling thread's errno (a thread-local variable since the pthreads era, implemented as a macro around *__errno_location(), so two threads failing concurrently don't stomp each other's error code), and returns plain -1 to your code. Two things worth being precise about: errno is only meaningful immediately after a call that actually failed, a successful call is not required to reset it to 0, so checking errno after a call you didn't verify failed is a bug in itself; and almost any libc call, not just the one you're checking, is free to overwrite errno internally (even on its own success path), so you must read errno before doing anything else, including calling another libc function to help you log the error.

2. strerror() vs strerror_r(): the thread-safety gap

strerror(errnum) returns a char* into a buffer that is commonly a single, static, process-wide buffer (or in older/less careful implementations, genuinely process-wide state). It is not thread-safe: one thread's strerror() call can be overwritten by a concurrent (or even a subsequent, same-thread) call before you've finished using the string, a classic scenario in multi-threaded logging code where several worker threads all hit an error around the same time and each one's log line ends up describing a DIFFERENT error than the one that actually happened to it.

strerror_r() is the fix, but it has a landmine of its own: it has two incompatible signatures, and which one you get depends on feature-test macros, not on anything visible at the call site.

• The XSI/POSIX version (int strerror_r(int errnum, char *buf, size_t buflen), active when _POSIX_C_SOURCE >= 200112L and _GNU_SOURCE is NOT defined) always writes the message into your buffer and returns 0 on success or an error number.
• The GNU-specific version (char *strerror_r(int errnum, char *buf, size_t buflen), active whenever _GNU_SOURCE is defined, which is common, since many codebases turn it on repo-wide for unrelated GNU extensions) may return a pointer to glibc's OWN internal, immutable string INSTEAD of writing anything into your buffer at all.

I verified this concretely rather than trusting memory of the man page: with _GNU_SOURCE defined, I filled a 64-byte buffer with a sentinel byte, called strerror_r(EACCES, buf, sizeof(buf)), and checked both the return pointer and the buffer contents afterward.

#define _GNU_SOURCE
#include <stdio.h>
#include <string.h>
#include <errno.h>

int main(void) {
    char buf[64];
    memset(buf, 'X', sizeof(buf));
    char *msg = strerror_r(EACCES, buf, sizeof(buf));
    printf("return ptr == buf? %s\n", msg == buf ? "yes" : "no (static string returned)");
    printf("message: %s\n", msg);
    printf("buf untouched? %s\n", buf[0] == 'X' ? "yes" : "no");
    return 0;
}

MEASURED output (Debian 12, glibc 2.36):

return ptr == buf? no (static string returned)
message: Permission denied
buf untouched? yes

The buffer was never written to at all; the return value pointed somewhere else entirely. Code that assumes "the buffer now holds my message" (for example strerror_r(errno, buf, sizeof buf); snprintf(logline, N, "%s", buf);) will silently log garbage or stale bytes on any build with _GNU_SOURCE defined, while working fine on a strict-POSIX build, exactly the kind of environment-dependent bug the "runs fine locally, breaks in prod's build config" pattern produces. The rule: always format from the return value of strerror_r(), never assume the buffer was filled.

3. Formatting a diagnostic message

Include all three named pieces so the message is both human-readable and machine-greppable/joinable against errno tables:

char ebuf[128];
char *msg = strerror_r(errno, ebuf, sizeof(ebuf));
fprintf(stderr, "open(\"%s\") failed: errno=%d (%s: %s)\n",
        path, errno, "EACCES", msg);
/* -> open("/var/log/app.log") failed: errno=13 (EACCES: Permission denied) */

Prefer symbolic errno constants over hardcoded numbers in your OWN comparisons (if (errno == EACCES), not if (errno == 13)), since the numeric values, while stable on Linux/x86-64 in practice, are not guaranteed identical across every POSIX platform; but DO log the numeric value alongside the symbol, because grepping/aggregating logs by a stable integer is more robust across log-parsing tooling than string-matching human text that a locale setting could translate.

4. Minimal-impact error collection on high-throughput paths

Formatting a string (via snprintf/strerror_r) on every single failure is real, avoidable cost if a hot path fails thousands of times a second (a busy proxy hitting EAGAIN on a non-blocking socket, for instance, that's not even a "real" error: it just means "nothing available right now, try again," not a genuine failure). The pattern that keeps the hot path cheap:

• On the hot path itself, do the minimum: increment a fixed-size, lock-free counter array indexed by errno value (atomic_fetch_add(&errno_counts[errno], 1)), no string formatting, no I/O, no allocation.
• Rate-limit the actual log/string-formatting work: log the FIRST occurrence of a new errno immediately (so nothing is silently missed), then only log again once per time window (say, once per second) or every Nth occurrence thereafter, with the count of suppressed occurrences included in that log line.
• Prefer structured/binary logging over formatted text on the hot path: record the syscall name (as an integer id or interned string pointer, not a fresh snprintf) and the raw errno value, and defer the human-readable string formatting to whatever consumes the log (a separate low-priority thread, or offline at query time), rather than paying strerror_r's cost inline on every occurrence.
• Avoid heap allocation on the error path if the process is truly high-throughput: even an infrequent-relative-to-total-traffic error path can execute thousands of times a second on a big enough fleet, and malloc/free add lock contention and unpredictable latency that a pre-sized thread-local scratch buffer avoids entirely.