System Calls & the Kernel Interface Questions

Design first: the state machine and the syscalls at each transition

A supervisor for a small worker pool is a state machine per worker, STARTING -> RUNNING -> (BACKOFF -> STARTING | STOPPED), driven entirely by process-lifecycle syscalls:

• STARTING -> RUNNING: fork(2) (create the child), then in the child, execve(2) (replace its image with the worker binary; if execve itself fails, e.g. ENOENT, the child must _exit() with a distinguishable code rather than falling through into the supervisor's own code, since a failed exec leaves the fork'd copy still running your supervisor's logic, a classic footgun).
• RUNNING -> (BACKOFF or STOPPED): waitpid(2) (or wait4) to learn the worker's fate, decoded via the WIFEXITED/WIFSIGNALED/WEXITSTATUS/WTERMSIG macro family, since a clean exit, a nonzero exit, and a signal-terminated exit each call for a different restart-versus-alert decision. This is also the single call that PREVENTS zombies: every exited worker must be waited on, or its entry lingers in the process table indefinitely.
• BACKOFF -> STARTING: a timed delay before the next attempt, nanosleep(2) (looping on EINTR, since a signal shouldn't silently truncate the backoff window), with the delay itself computed by simple exponential growth (double the wait each consecutive failure), capped, and reset to the base value after a sustained period of healthy running.
• -> STOPPED: reached either on a clean exit (status 0) or once a restart-count budget is exhausted, at which point the supervisor stops attempting restarts and surfaces the failure rather than crash-looping forever.

Reference implementation (verified, not illustrative)

supervisor.c, built and run exactly as shown, no modifications needed to reproduce:

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <signal.h>
#include <string.h>
#include <errno.h>
#include <time.h>

typedef enum { ST_STARTING, ST_RUNNING, ST_BACKOFF, ST_STOPPED } state_t;

static const char *state_name(state_t s) {
    switch (s) {
        case ST_STARTING: return "STARTING";
        case ST_RUNNING:  return "RUNNING";
        case ST_BACKOFF:  return "BACKOFF";
        case ST_STOPPED:  return "STOPPED";
    }
    return "?";
}

static void sleep_ms(int ms) {
    struct timespec ts = { ms / 1000, (long)(ms % 1000) * 1000000L };
    /* Restart on EINTR: a signal delivered mid-sleep should not truncate
     * our backoff window silently. */
    while (nanosleep(&ts, &ts) == -1 && errno == EINTR) { }
}

int main(int argc, char **argv) {
    if (argc < 2) {
        fprintf(stderr, "usage: %s <worker-path> [args...]\n", argv[0]);
        return 2;
    }
    const int max_restarts = 4;
    int restarts = 0;
    int backoff_ms = 200;
    state_t state = ST_STARTING;

    while (state != ST_STOPPED) {
        printf("[supervisor] state=%s restarts=%d\n", state_name(state), restarts);

        pid_t pid = fork();
        if (pid < 0) { perror("fork"); return 1; }

        if (pid == 0) {
            /* Child: exec the worker. execve() replaces this process's
             * image in place -- no new PID, same fd table unless we
             * change it, environment fully controlled by us here. */
            execv(argv[1], &argv[1]);
            /* Only reached if execve(2) itself failed (e.g. ENOENT). */
            perror("execv");
            _exit(127);
        }

        /* Parent: block until the child changes state. Plain waitpid()
         * here because there's nothing else to do until the one worker
         * we started exits -- a supervisor watching N workers would
         * instead reap in a SIGCHLD handler or an event loop. */
        state = ST_RUNNING;
        int status = 0;
        pid_t reaped = waitpid(pid, &status, 0);
        if (reaped < 0) { perror("waitpid"); return 1; }

        if (WIFEXITED(status) && WEXITSTATUS(status) == 0) {
            printf("[supervisor] pid=%d exited cleanly (status 0), done\n", (int)reaped);
            state = ST_STOPPED;
        } else if (WIFEXITED(status)) {
            printf("[supervisor] pid=%d exited with code %d\n", (int)reaped, WEXITSTATUS(status));
            if (++restarts > max_restarts) {
                printf("[supervisor] restart budget exhausted, giving up\n");
                state = ST_STOPPED;
            } else {
                state = ST_BACKOFF;
                printf("[supervisor] backing off %d ms before restart #%d\n", backoff_ms, restarts);
                sleep_ms(backoff_ms);
                backoff_ms *= 2; /* exponential backoff, no cap needed for this demo */
                state = ST_STARTING;
            }
        } else if (WIFSIGNALED(status)) {
            printf("[supervisor] pid=%d killed by signal %d%s\n", (int)reaped,
                   WTERMSIG(status), WCOREDUMP(status) ? " (core dumped)" : "");
            if (++restarts > max_restarts) { state = ST_STOPPED; }
            else { state = ST_BACKOFF; sleep_ms(backoff_ms); backoff_ms *= 2; state = ST_STARTING; }
        }
    }
    printf("[supervisor] final state=STOPPED, total restarts=%d\n", restarts);
    return 0;
}

Driver, flaky_worker.sh, which fails twice via a counter file, then succeeds, so the restart-with-backoff path is exercised deterministically:

#!/bin/sh
COUNTER=/tmp/flaky_worker_count
n=0
[ -f "$COUNTER" ] && n=$(cat "$COUNTER")
n=$((n + 1))
echo "$n" > "$COUNTER"
if [ "$n" -le 2 ]; then
    echo "flaky_worker: attempt $n, failing on purpose"
    exit 1
fi
echo "flaky_worker: attempt $n, succeeding"
rm -f "$COUNTER"
exit 0

Build and run:

$ gcc -Wall -o supervisor supervisor.c
$ chmod +x flaky_worker.sh
$ ./supervisor ./flaky_worker.sh

MEASURED output (Debian 12, real run):

flaky_worker: attempt 1, failing on purpose
flaky_worker: attempt 2, failing on purpose
flaky_worker: attempt 3, succeeding
[supervisor] state=STARTING restarts=0
[supervisor] pid=5020 exited with code 1
[supervisor] backing off 200 ms before restart #1
[supervisor] state=STARTING restarts=1
[supervisor] pid=5021 exited with code 1
[supervisor] backing off 400 ms before restart #2
[supervisor] state=STARTING restarts=2
[supervisor] pid=5023 exited cleanly (status 0), done
[supervisor] final state=STOPPED, total restarts=2

I also confirmed no zombies were left behind for THIS supervisor's own children afterward (ps -eo pid,ppid,stat,cmd | grep defunct showed none owned by the supervisor's PID), which is the point of the design: every exit is reaped through waitpid, so nothing lingers.

Extending to N concurrent workers

This single-worker version blocks on one waitpid(pid, ...) because it only has one child at a time. A supervisor managing several workers concurrently swaps that for either: a SIGCHLD handler that loops waitpid(-1, &status, WNOHANG) until nothing's left to reap (necessary because SIGCHLD delivery can coalesce, one signal for several near-simultaneous exits), reading which specific pid came back to look up which worker's state machine to advance; or, on modern Linux, pidfd_open(2) per child plus epoll, which sidesteps SIGCHLD/PID-reuse ambiguity entirely by giving each worker its own pollable file descriptor, the natural next step once you're past a handful of workers.

The investigation comes down to reading the SHAPE of the process tree and the IDENTITY of what's actually being executed at each hop, then comparing both against what's normal for this host.

What a fork bomb looks like

A fork bomb is defined by rate and self-similarity, not by what any individual process does. Symptoms: an extremely high, near-instantaneous rate of new PIDs, potentially thousands per second; the SAME short-lived binary re-forking itself recursively (the parent and the child are typically running the identical program, sometimes literally the same shell one-liner re-invoking itself); and system-wide symptoms that show up FAST, fork() starting to fail everywhere on the host with EAGAIN/ENOMEM because the global PID space or resource limits are being exhausted, CPU pegged near 100% from pure scheduling/context-switch overhead even though no individual process is doing meaningful work (no real file I/O, no network activity, nothing but forking). Confirming this: ps -eo pid,ppid,etimes,cmd --sort=start_time | tail -50 (or pstree -p) shows a large cluster of processes all born within the same second or two, almost all running the same command, with a shallow but extremely WIDE tree (one ancestor with an enormous number of direct or near-direct descendants), rather than a deep chain.

What a malware loader / process-chaining attack looks like

The opposite shape: a NARROW, sequential chain, parent -> child -> grandchild, where each hop exec()s into a DIFFERENT binary than the one before it ("process chaining," commonly discussed as part of "living off the land" tradecraft). The telltale signal is fork() immediately followed by execve() into an unrelated program, especially when: the target binary lives in a WRITABLE, non-standard location (/tmp, /dev/shm, a user's Downloads directory) rather than a normal system binary path; or it's a legitimate, trusted system binary being invoked with unusual, scripted-looking arguments that a human wouldn't type interactively (a base64-encoded inline script, a curl/wget piped directly into a shell interpreter, a download-then-exec two-step); or a process that has no legitimate reason to spawn a shell at all suddenly does, a web server worker process forking sh -c "...", or a document viewer (PDF reader, Office app) spawning a script interpreter, is a strong standalone signal on its own, since normal automation for those specific parent processes essentially never legitimately does that.

What normal automation looks like, as the baseline to compare against

A stable, REPEATING, predictable tree shape: a cron job or systemd timer forking the same handful of children on a fixed schedule; a CI runner spawning build-tool subprocesses in a consistent pattern run after run. Children exec well-known binaries from standard system paths (/usr/bin, /usr/local/bin, not a temp directory). The fork rate is moderate and clearly correlated with an actual trigger (a cron entry firing, a webhook arriving), not continuous, unbounded self-replication. And critically, the specific argv patterns and binary paths involved are ones you'd expect to have SEEN BEFORE on this host if you have any historical baseline (fleet-wide EDR telemetry, prior audit logs), whereas malicious activity commonly shows a first-time-ever binary path, argv combination, or parent/child pairing for that specific host.

Concrete investigative steps

1. pstree -p <suspect_root_pid> first, to see the actual tree SHAPE at a glance: wide-and-shallow (fork bomb candidate) versus narrow-and-deep with binary changes at each hop (loader candidate) versus a familiar, repeating pattern (probably fine).
2. ps -eo pid,ppid,lstart,etime,cmd for exact start times and full command lines, WHEN things started (clustered instantaneously, or spread out on a schedule) and WHAT was actually typed as arguments (a base64 blob, a download URL, a -enc/-EncodedCommand-style obfuscation flag are all strong indicators).
3. Don't trust the process name alone: argv[0] (what shows up as the "command" in a casual ps) can be rewritten by the process itself to look innocuous. Cross-check /proc/<pid>/exe (a symlink that resolves to the ACTUAL binary inode being executed, harder to spoof after the fact than an in-process argv rewrite) and /proc/<pid>/cmdline (the real argv the kernel recorded at exec time) against what ps displayed; a mismatch between a friendly-looking process name and a suspicious /proc/<pid>/exe target is itself a finding.
4. Check /proc/<pid>/status for PPID (confirm the actual parentage rather than trusting a displayed process name) and cross-reference against who/last/authentication logs: does this chain trace back to an interactive login session, a legitimate cron entry, or a network-facing service process, and if it's the latter, does that service have any legitimate reason to be spawning a shell or a script interpreter AT ALL? A network-facing daemon forking a shell is close to always worth escalating on its own.
5. Recognize the limits of reasoning from the CURRENT process tree alone: fork()/execve() calls aren't logged anywhere by default, so if the malicious chain has already partially exited by the time you're looking, the tree you're inspecting is incomplete. This is exactly why Linux audit (auditctl -a exit,always -F arch=b64 -S execve) or an eBPF (a Linux kernel facility that lets small sandboxed monitoring programs run in-kernel, the basis for many modern tracing/EDR tools)-based EDR agent recording exec events needs to be enabled BEFORE an incident, not reached for during one; reconstructing a process-chaining attack cleanly from an execve audit trail is straightforward, reconstructing it purely from ps snapshots and whatever remnants are still running when you happen to look is much harder and can miss steps that already completed and exited.

The short version, up front: never trust what readdir() reports about an entry on its own; before acting on any entry, re-check it through the SAME already-open directory descriptor the listing came from, rather than a freshly resolved path, so a rename or a symlink swapped in between the listing and the action can't redirect what you actually touch. The rest of this answer works through why, in detail.

How the three calls work together

opendir(path) opens a directory as a stream and returns a DIR * handle; internally this typically performs an open() with O_DIRECTORY on the path and wraps the resulting file descriptor. dirfd(3) recovers that underlying descriptor from the DIR *, which is what makes it possible to pass the same, already-resolved directory to openat()/fstatat() calls for confinement (restricting every later lookup to stay inside the directory you already hold open, so a rename or a symlink swapped in elsewhere in the filesystem can't redirect the lookup outside it) and TOCTOU avoidance, rather than opening the directory a second time by path.

readdir(dirp) returns one struct dirent per call, advancing the stream's internal position, until it returns NULL at end-of-stream, or on error. Both cases return NULL, so the caller must reset errno to 0 immediately before calling readdir() and check errno afterward to tell "reached the end" apart from "the read itself failed."

closedir(dirp) closes the underlying descriptor and frees the stream's associated memory.

What you can and cannot trust from an entry

struct dirent gives you d_name (the entry's filename within the directory) and, on filesystems that populate it, d_type, a cheap hint of the entry's type (DT_REG, DT_DIR, DT_LNK, and so on) read straight from the directory entry itself, without a separate stat() call. Two things limit how far you can trust this:

• POSIX explicitly permits d_type to be DT_UNKNOWN on filesystems that don't track entry type in the directory entry itself (some network filesystems, and some older or non-native filesystem types). Code that switches on d_type without a fallback for DT_UNKNOWN silently mishandles every entry on such a filesystem.
• Far more importantly for a security tool: d_name/d_type reflect the directory's state at the MOMENT readdir() returned that entry. By the time code later acts on that name (opens it, stats it, deletes it), the entry may have been renamed, deleted, or replaced by something else at the same name: the same TOCTOU (time-of-check to time-of-use) race that affects any path-based stat() call. POSIX also gives no guarantee about entries added or removed by a concurrent writer during enumeration: such entries may be seen once, more than once, or not at all, and no ordering is promised at all.

Additional checks before acting on an entry in a security tool

Don't treat d_type as authoritative; revalidate. The correct revalidation uses fstatat(dirfd, d_name, &st, AT_SYMLINK_NOFOLLOW) (AT_SYMLINK_NOFOLLOW: fail rather than silently follow the entry if it turns out to be a symlink, so you inspect the entry itself, not whatever it points at) against the SAME directory file descriptor the stream was opened on (via dirfd(3)), rather than re-resolving a fresh path from the filesystem root, which would just reopen a second, independent TOCTOU window. If the tool is going to actually open the entry, do so via openat(dirfd, d_name, O_NOFOLLOW) (the equivalent guard for open: fail rather than follow a symlink) and fstat() the resulting descriptor rather than stat()-then-open() on the name, the same general discipline of checking the fd you already hold rather than a re-resolved path. Finally, treat every entry as potentially attacker-influenced whenever the directory is writable by a lower-privileged principal or an untrusted input source, since an adversary watching the enumeration in real time can rename or replace entries in the gap between your readdir() call and whatever the tool does with that name next.

A worked trace

Say a directory /incoming contains three entries: report.pdf (a regular file), logs (a subdirectory), and evil (a symlink an attacker planted, pointing at /etc/shadow). A loop over it looks like:

DIR *d = opendir("/incoming");
int dfd = dirfd(d);
struct dirent *e;
errno = 0;
while ((e = readdir(d)) != NULL) {
    struct stat st;
    /* Revalidate through the SAME dfd, never a fresh path. */
    if (fstatat(dfd, e->d_name, &st, AT_SYMLINK_NOFOLLOW) < 0) continue;
    if (!S_ISREG(st.st_mode)) continue;   /* skips "logs" (a directory) and "evil" (a symlink, not a regular file) */
    int fd = openat(dfd, e->d_name, O_RDONLY | O_NOFOLLOW);
    if (fd < 0) continue;                 /* defense-in-depth: catches a TOCTOU swap between the two checks above */
    /* fd now safely refers to the exact entry just revalidated */
    close(fd);
    errno = 0;
}
closedir(d);

For report.pdf: d_type, if populated, reads DT_REG; fstatat(..., AT_SYMLINK_NOFOLLOW) confirms S_ISREG; openat() with O_NOFOLLOW succeeds and hands back an fd bound to exactly that file. For logs: S_ISREG is false, so the loop skips it before ever calling openat(). For evil: fstatat(..., AT_SYMLINK_NOFOLLOW) reports on the entry itself rather than whatever it points at, so it comes back S_ISLNK, not S_ISREG: the !S_ISREG check rejects it right there, and openat() is never reached for this entry at all (verified by running this exact loop against a real evil -> /etc/shadow symlink: it prints "not a regular file, skip" and never reaches the openat() line). The openat(..., O_NOFOLLOW) guard on the next line is not what stops evil in this static trace; it earns its keep in a DIFFERENT case the trace above does not exercise: a race where an entry passes the fstatat() check as a genuine regular file and is then swapped for a symlink in the gap before openat() runs a moment later. fstatat() and openat() are two separate syscalls with a window between them, and O_NOFOLLOW on the openat() call is what closes that specific window: the case that matters whenever the directory is writable by an untrusted principal, as noted above.

Why this is a live attack surface, not a theoretical edge case

The vulnerable pattern is: enumerate a directory tree with readdir(), filter entries down to "regular files" using d_type and/or stat(), then separately open() and hash the same path string. Every gap between "decided this is safe to hash" and "actually opened it" is a TOCTOU (time-of-check to time-of-use) window, and in malware analysis the directory tree being walked is explicitly adversarial input: the sample itself, or anything an attacker controls in a shared analysis environment, can plant content specifically designed to defeat or redirect the scanner. This isn't a hardening nicety here, it's the expected threat model.

The specific tricks to defend against

• Symlink tricks: an entry named to look like an innocuous sample (report.pdf, invoice.exe) that is actually a symlink to /etc/shadow, a device file, or a named pipe designed to block the scanner forever (a denial-of-service against the tool itself), or a symlink to an enormous sparse file designed to exhaust disk or memory when "hashed."
• Renamed-path tricks: the entry readdir() reported gets deleted and a DIFFERENT object created under the same name before the tool acts on it, possible whenever the directory being scanned is live and writable, whether by the sample's own partial execution, by another process sharing the analysis filesystem, or by an operator re-triaging concurrently.
• Concurrent filesystem changes: directories scanned on a live system, as opposed to a static forensic image, can have entries added, removed, or replaced mid-walk; POSIX gives no ordering or consistency guarantee for readdir() under concurrent modification, so entries can be seen twice, missed entirely, or observed in an inconsistent state relative to each other.

A resistant design

• Walk using directory FILE DESCRIPTORS, not path strings. Open the top directory once with open(path, O_DIRECTORY); for every subdirectory encountered, descend via openat(parent_dirfd, name, O_DIRECTORY | O_NOFOLLOW) rather than concatenating a path string, so a symlink planted where a subdirectory was expected is rejected instead of followed.
• For each entry, don't trust d_type as authoritative (it can be DT_UNKNOWN, and even when populated it reflects a moment in time that has already passed by the time you act). Instead of a check-then-open sequence, openat(dirfd, name, O_RDONLY | O_NOFOLLOW) directly: if the entry is a symlink, this call fails with ELOOP (too many levels of symbolic links encountered, the same errno O_NOFOLLOW produces when it hits a symlink directly) and the entry can be flagged and skipped rather than silently redirected.
• fstat() the resulting fd, never fstatat/stat on the name a second time, and verify S_ISREG before reading a single byte, so the object being hashed is guaranteed to be the exact fd already held open, with zero remaining path re-resolution for anything to hijack.
• Compare (st_dev, st_ino) (device and inode, from the same fd's fstat()) against a set of already-visited identifiers to detect hardlink loops or the same file linked into the tree more than once, avoiding both infinite loops and double-counted results.
• Don't trust st_size blindly before reading: a pathological entry (a sparse file reporting a huge logical size, or a FIFO with a stale or meaningless size field) can turn a naive "read st_size bytes" into a resource-exhaustion or hang condition. Read in bounded chunks with an overall size and time budget, and use non-blocking I/O or a read timeout so a FIFO planted specifically to block forever can't stall the whole walk.
• Run the entire scan under explicit resource limits (a wall-clock timeout per file, RLIMIT_FSIZE (a per-process resource limit capping the largest file the process may write, to stop the scanner itself from being tricked into writing an unbounded amount of data), memory caps), treating the sample directory as hostile input for the whole duration of the walk, not just at the point of opening each file, since the scanner's own resource consumption is as much a part of the attack surface as the data it reads.

This is the classic building block behind every shell pipeline (cmd1 | cmd2) and every popen()-style API: fork() a child, wire the child's stdout to one end of a pipe(), and have the parent read the other end. The code below compiles and runs unchanged; it launches echo as the child and captures its output in the parent.

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/wait.h>
#include <string.h>

int main(void) {
    int pipefd[2];
    if (pipe(pipefd) == -1) {
        perror("pipe");
        return 1;
    }

    pid_t pid = fork();
    if (pid == -1) {
        perror("fork");
        return 1;
    }

    if (pid == 0) {
        /* Child: writes to the pipe's write end via stdout */
        close(pipefd[0]);              /* close unused read end */
        if (dup2(pipefd[1], STDOUT_FILENO) == -1) {
            perror("dup2");
            _exit(1);
        }
        close(pipefd[1]);              /* now redundant, avoid a leaked extra fd */

        execlp("echo", "echo", "hello from child", NULL);
        perror("execlp");              /* only reached if exec fails */
        _exit(1);
    }

    /* Parent */
    close(pipefd[1]);                  /* close unused write end: required so read() sees EOF */

    char buf[256];
    ssize_t n;
    printf("parent received: ");
    while ((n = read(pipefd[0], buf, sizeof(buf) - 1)) > 0) {
        buf[n] = '\0';
        fputs(buf, stdout);
    }
    close(pipefd[0]);

    int status;
    waitpid(pid, &status, 0);
    if (WIFEXITED(status)) {
        printf("child exited with status %d\n", WEXITSTATUS(status));
    }
    return 0;
}

Compiled with cc -Wall -Wextra -o pipe_demo pipe_demo.c and run as ./pipe_demo, this prints, deterministically:

parent received: hello from child
child exited with status 0

Steps needed to avoid deadlocks and descriptor leaks

• Close the unused end of the pipe in each process, immediately after fork(). A pipe() gives you two file descriptors; after fork(), both the parent and the child have both ends open. If the parent doesn't close its copy of the write end (pipefd[1]), the pipe still has a live writer from the kernel's point of view even after the child exits and closes its own copy, so the parent's read() loop never sees EOF (end of file) and blocks forever, i.e. deadlocks waiting for a byte that will never come. Symmetrically, the child closing its unused read end (pipefd[0]) isn't strictly required to avoid this deadlock, but it prevents a descriptor leak and keeps the child from accidentally being able to read its own output back.
• Read before (or concurrently with) waiting, not after. A pipe has a finite kernel buffer (commonly 64 KiB on Linux). If the child writes more than that and the parent calls waitpid() before it starts reading, you get a two-sided deadlock: the child blocks in write() because the pipe buffer is full and nothing is draining it, while the parent blocks in waitpid() waiting for a child that can't make progress until it's read from. The code above avoids this by reading in a loop until EOF before calling waitpid(), which is safe here because the child's total output is small; for a child that could write an unbounded amount, you'd want the read loop and the wait to be interleaved (e.g. via non-blocking reads inside a poll()/select() loop, or a dedicated reader thread) rather than assuming a single blocking read loop will always finish before you need to reap the child.
• Always reap the child. waitpid() (or wait()) after the read loop both retrieves the exit status and prevents the child from becoming a zombie (an exited process still holding a slot in the process table because nobody collected its status).
• Check every syscall's return value. pipe(), fork(), dup2(), and close() can all fail; in particular a failed dup2() in the child that goes unchecked would leave the child writing to its original stdout (e.g. the terminal) instead of the pipe, which looks like "no output was captured" rather than an obvious crash.