System Calls & the Kernel Interface Questions

All three are ways to create a new thread of execution, but they trade off differently on what gets shared, how much it costs, and how easy they are to misuse. Think of them as a spectrum from "share nothing" to "share almost everything" to "you choose exactly what to share."

fork(2): full, independent process via copy-on-write

The child gets its own copy-on-write address space and its own duplicated fd table; nothing is shared going forward except the underlying open file descriptions those fds still point at. Cost is proportional to page-table size at fork time, plus whatever COW page faults actually happen afterward. Use case: the general-purpose default any time you want a genuinely independent process, most commonly immediately followed by execve(). Hazard: the "memory doubling" concern, virtual-memory accounting (not necessarily physical RAM) that can make fork() fail with ENOMEM on a very large process under strict overcommit settings, and the real doubling that DOES happen if the child touches most of its inherited memory before it execs or exits.

vfork(2): no address-space copy at all, at the cost of a sharp restriction

vfork() predates copy-on-write being cheap enough on the hardware of its era; on modern Linux it's essentially clone() with CLONE_VFORK semantics: the child runs IN the parent's actual address space (no copy, not even a COW page-table setup), and the PARENT IS SUSPENDED, literally stopped, until the child calls execve() or _exit(). Because there's no copying whatsoever, it's the fastest of the three for the specific "about to exec anyway" case. The hazard is sharp and easy to trip: since the child shares the parent's memory, any write it makes (a local variable the compiler decided to spill onto the shared stack, a function call that isn't on the sanctioned minimal list) can corrupt the parent's state, and the parent can't even observe or intervene because it's suspended. The safe usage pattern is essentially "touch nothing except setting up the arguments for the immediately-following exec/_exit call." There's a second, distinct vfork hazard worth naming explicitly, since it's the actual deadlock hazard among the three (the memory-corruption hazard above is a different failure mode, silent corruption, not a hang): because the parent is unconditionally SUSPENDED from the moment vfork() returns in the child until that child calls execve() or _exit(), if the child never reaches either of those (it blocks on a syscall, the target binary doesn't exist and the error-handling path forgets to call _exit(), it deadlocks on its own, or it's simply killed by something that leaves it stopped rather than terminated), the parent is suspended right along with it, indefinitely. That's a real deadlock of the entire calling process, not corrupted data, and it's a second, independent reason the "touch nothing but the exec setup" discipline matters: code that's careful never to corrupt shared memory can still hang the parent forever just by not reaching exec/_exit promptly. Use case today: mostly as an internal implementation detail, posix_spawn() uses a CLONE_VFORK-style fast path on Linux specifically for the fork-then-immediately-exec pattern, where the restriction doesn't matter because the library, not arbitrary application code, is the only thing running in that narrow shared-memory window. Hand-writing raw vfork() calls in application code is now rare and generally discouraged in favor of posix_spawn().

clone(2): the general primitive, sharing is an explicit bitmask, not a preset

clone() is what BOTH fork() and thread creation (pthread_create) are actually built on. Instead of "share nothing" or "share everything", you pass explicit flags for exactly what to share: CLONE_VM (share the address space, this is the core of what makes a "thread" a thread rather than a process), CLONE_FILES (share the fd table itself, not just duplicate it, so closing an fd in one "thread" closes it for all of them), CLONE_SIGHAND (share signal handler dispositions). Use case 1, thread-like behavior: pthread_create() is, under the hood, a clone() call with CLONE_VM | CLONE_FILES | CLONE_SIGHAND (plus more) set, a new PID/TID with its own stack, but sharing memory and fds with the "parent" thread, which is exactly the threading model. Use case 2, container namespaces: the namespace flags, CLONE_NEWPID (own view of the process-ID tree, so the new process can be PID 1 inside it), CLONE_NEWNET (own network stack/interfaces), CLONE_NEWNS (own mount table), CLONE_NEWUTS (own hostname), CLONE_NEWIPC (own System V IPC objects and POSIX message queues, isolated from the host's), CLONE_NEWUSER (own UID/GID mapping), are the literal mechanism container runtimes (Docker, runc, systemd-nspawn) build on: a process that gets a kernel-enforced, scoped VIEW of system resources, with no hypervisor or hardware virtualization involved at all. Hazards: it's the lowest-level and easiest to get subtly wrong of the three. Sharing CLONE_FILES between things you intended to be more independent than true threads means one side closing an fd affects the other unexpectedly. And creating a PID namespace has a real, load-bearing consequence people forget: when a process's parent exits before it does, the kernel reparents that orphaned child to PID 1 (or to a registered subreaper), and PID 1 is expected to eventually wait() on those reparented children so their zombie table entries don't linger forever, exactly the job a normal init process does for every ordinary orphan on the host. A freshly created CLONE_NEWPID namespace's own PID 1 inherits that identical responsibility, scoped to its own descendants, and if the image running as that PID 1 was never written to reap children (a bare application binary, or literally a sleep command used as a container entrypoint), zombies accumulate silently for the container's entire lifetime. I've reproduced this directly: a bare sleep infinity used as a container's PID 1 left every test program's exited-but-unreaped children as permanent zombies, exactly why real container images commonly ship a tiny init like tini/dumb-init as PID 1 instead of the application binary directly.

Summary, as a decision

Need a fully independent process, most commonly about to exec? Use fork() (or better, posix_spawn()). Need the absolute minimum overhead for a fork-then-immediately-exec sequence and you're implementing something like posix_spawn() itself, not writing application code? vfork()/CLONE_VFORK, and even then, prefer letting posix_spawn() do it for you. Need actual OS threads sharing memory? clone(CLONE_VM|...), which is what pthread_create already does, you rarely call this directly. Need container-style isolation, a process with its own view of PIDs/network/mounts? clone() with the relevant CLONE_NEW* namespace flags, and remember that whoever ends up as PID 1 in that namespace now owns reaping duties it may not have been written to fulfill.

setrlimit(2) lets you cap how much of a given resource a process may consume, and the kernel enforces the limit at the point the process tries to exceed it. For memory specifically, the two limits people reach for are RLIMIT_AS and RLIMIT_RSS, and only one of them actually does anything on modern Linux.

1. How setrlimit(2) constrains memory

setrlimit(resource, &rlim) takes a resource identifier and a struct rlimit { rlim_t rlim_cur; rlim_t rlim_max; } (a soft limit the process can raise up to the hard limit, and a hard limit only a privileged process can raise). A process typically sets its own limits at startup, or a parent/supervisor (a systemd unit's LimitAS=, or a shell's ulimit) sets them before exec, since limits are inherited across fork(2) and preserved across execve(2).

2. RLIMIT_AS vs RLIMIT_RSS

RLIMIT_AS caps the size, in bytes, of the process's total virtual address space: every brk()/mmap()/mremap() call that would push the process's mapped address space past the limit fails with ENOMEM, and automatic stack growth past the limit raises SIGSEGV. This is enforced synchronously and reliably by the kernel, and it counts reserved virtual memory, not memory actually backed by RAM: a large mmap(MAP_NORESERVE) region, a generous heap arena a language runtime reserves up front, or a big anonymous mapping that's mostly untouched all count fully against RLIMIT_AS even if their resident footprint is tiny.

RLIMIT_RSS looks like the natural fit ("limit the resident set, i.e. the physical RAM actually in use") but per the setrlimit(2) man page it has effect only on Linux kernels in the 2.4.x series before 2.4.30, and even there it only affected madvise(MADV_WILLNEED) behavior. On every Linux kernel in production today, setting RLIMIT_RSS is a silent no-op: getrlimit() will report back whatever value you set, but the kernel does not use it to constrain, reclaim, or kill anything. Reaching for RLIMIT_RSS to fix an OOM-killed worker will look like it did something (no error, the limit "sets") and change nothing.

3. Consequences of overly strict limits

Because RLIMIT_AS counts virtual, not resident, memory, an overly strict RLIMIT_AS causes failures well before the process is actually short on physical RAM. A worker that reserves a 1 GiB heap arena at startup (common for JVM-style runtimes, and for allocators that reserve large arenas per thread) will fail to even start under a 512 MiB RLIMIT_AS, even though its live-object footprint might be 100 MiB. The failure mode is also abrupt and hard to diagnose from the outside: malloc() returns NULL or the process is killed by SIGSEGV on stack growth, which looks like "random crash" rather than "hit a configured limit," unless someone thinks to check /proc/<pid>/limits or the process's own ENOMEM handling. Debugging/instrumentation tools that inflate virtual (not resident) footprint, like AddressSanitizer's shadow memory (extra memory a sanitizer reserves to track the validity of every real byte allocated, often several times the real footprint), can also blow through an RLIMIT_AS sized for the un-instrumented binary.

4. Choosing sensible limits while preserving SLOs

Given RLIMIT_RSS doesn't work, the practical Linux mechanism for "cap this worker's actual physical memory use" is a cgroup (control group) memory controller, not a process rlimit: memory.max is a hard ceiling that gets the cgroup OOM-killed (only processes in that cgroup, not a global OOM-killer sweep across unrelated processes on the host) when exceeded, and memory.high is a soft throttle that triggers reclaim/backpressure on the cgroup before the hard kill, giving the workload a chance to shed load gracefully instead of being killed outright.

To size these: measure the worker's actual peak resident usage under representative load, e.g. VmHWM (high-water-mark RSS) from /proc/<pid>/status, or a cgroup's memory.peak. Say that measurement comes back as 900 MiB peak RSS (a MEASURED figure, not an estimate). Set memory.high around 1.1x that peak (about 1.0 GiB) as the soft throttle point, and memory.max around 1.3-1.5x the peak (900 MiB x 1.3 = 1,170 MiB, about 1.14 GiB, up to 900 MiB x 1.5 = 1,350 MiB, about 1.32 GiB, at the 1.5x end) as the hard kill ceiling, giving headroom for legitimate spikes (a larger-than-usual request, a GC pause building up garbage) without either constantly throttling in the normal case or letting one runaway worker starve the whole host. If you also want an outer RLIMIT_AS backstop (useful against pathological virtual-memory leaks, e.g. an mmap loop that never frees), set it generously above what the runtime is known to reserve at startup, not near the measured RSS figure, since AS and RSS are measured on different bases and conflating them is exactly how you get spurious ENOMEM at start-up. Finally, validate under load: run the same load test that produced the 900 MiB measurement against the new limits and confirm the SLO (e.g. p99 latency, error rate) holds at the soft-throttle boundary, then keep watching memory.high throttling events and OOM-kill counts in production so the limit gets revisited as the workload's real memory profile changes.

A process's identity for permission checks is a set of credentials, not one number, and execve(2) (the syscall that replaces a process's memory image with a new program) has specific, well-defined rules for updating them. This answer covers the credential model, the setuid/setgid security model, why interpreter scripts are a special case, and what to actually do operationally.

1. Process credentials and how execve() updates them

Every process carries (at least) three UIDs and three GIDs: the real UID (RUID, "who actually launched this"), the effective UID (EUID, "whose permissions are checked for most operations right now"), and the saved set-user-ID (saved UID, "the privilege this process is allowed to switch back to"). Linux adds a fourth, the filesystem UID (FSUID), used only for filesystem access checks and normally tracks EUID automatically.

On an ordinary execve() of a non-setuid file, RUID, EUID, and saved UID are all preserved unchanged from the calling process. The setuid/setgid bits change that: if the file being executed has the set-user-ID bit (mode bit S_ISUID) set, a successful execve() sets the new process's EUID to the file owner's UID, and also updates the saved UID to that same value, while RUID stays whatever it was before (typically the invoking user). The set-group-ID bit (S_ISGID) does the analogous thing to EGID using the file's group owner. This is exactly how /usr/bin/passwd works: it is owned by root and has the setuid bit set, so when an ordinary user runs it, RUID stays that user's UID (so the program can still tell who invoked it) but EUID becomes 0 (root), which is what lets the program write to /etc/shadow, a file the invoking user has no direct permission to touch.

2. The security model for setuid binaries

A setuid-root binary is a deliberate, narrow hole punched through the normal permission model: "this specific program, and only this program, may act as root on the invoking user's behalf." That model only holds if the program is written to close every path an attacker could use to make it act as root on their behalf instead of the intended narrow task. Concretely that means: sanitize or drop dangerous inherited environment variables (PATH, IFS, LD_PRELOAD, LD_LIBRARY_PATH) before doing anything, because a setuid program that shells out or dynamically links using an attacker-controlled PATH or LD_PRELOAD can be tricked into running attacker code with root's privileges; validate every argument and avoid unbounded buffers (a setuid binary with a stack-based buffer overflow is a direct local-root exploit, since the overflow runs with the effective, elevated privileges); drop privileges (via setuid()/setresuid(), not just seteuid(), because seteuid() alone leaves the saved UID unchanged, so a compromised process could call seteuid(0) again and reclaim the privilege it appeared to have dropped; setresuid() also updates the saved UID, closing that path) the moment the elevated action is done, rather than running the whole program's lifetime at EUID 0; and never trust the RUID caller's other ambient state (open file descriptors, working directory, umask) without re-checking it under the elevated identity. Every historical local-root CVE class (format-string bugs, TOCTOU races on files the program then opens as root, PATH-based library or helper hijacking) is dangerous specifically because the vulnerable code is running setuid.

3. Interpreters and shebang lines

Modern Linux does not honor the setuid bit on an interpreter script (a file starting with #!/path/to/interpreter), even if the script's file itself has the bit set and is owned by root. The reason is a classic time-of-check-to-time-of-use (TOCTOU) race: the kernel identifies the file as a script and hands the path to the interpreter to open and read; between the kernel's permission check on the script and the interpreter actually opening that path a moment later, an attacker with write access to the containing directory can swap the script out from under it (via a symlink race or a rename), so the interpreter ends up executing attacker-controlled content with the privilege the kernel granted to the original, legitimate script. Because that race is essentially unfixable at the shebang layer, Linux (and most modern Unixes) simply drops the setuid/setgid bit's effect when the target is an interpreter script; the interpreter itself starts at the caller's real privilege, not the elevated one. (Some older systems shipped a dedicated setuid-safe wrapper like suidperl for this exact gap; it is deprecated/removed on current distributions precisely because it kept reintroducing this class of bug.) The practical takeaway: if you need setuid behavior, it has to live in a compiled binary, not a shell/Python/Perl script with the bit set on the script file itself.

4. Operational recommendations for third-party binaries needing elevated privileges

• Prefer Linux file capabilities over setuid-root: setcap cap_net_bind_service+ep /usr/local/bin/myproxy grants only the ability to bind ports below 1024, not full root, which shrinks the blast radius of any bug in that binary to "can bind low ports," not "can do anything."
• If you must run something as root, scope it with sudo to the exact command and arguments (a sudoers entry with a fixed command line, not NOPASSWD: ALL), not blanket root shell access.
• Run the binary in a container or with systemd service hardening: User=/DynamicUser=yes to avoid literal root, NoNewPrivileges=yes to block it from gaining more privilege than it started with (this is the same effect as PR_SET_NO_NEW_PRIVS), CapabilityBoundingSet= to cap which capabilities it could even acquire, and a seccomp profile to restrict which syscalls it can issue at all.
• Verify provenance before granting any of the above: checksum/signature-verify the binary, and if it's closed-source, treat it as untrusted input to your privilege boundary, not a trusted extension of it.
• If it insists on the setuid bit itself, audit whether that's actually required (many "needs root" third-party tools only need one specific capability, e.g. raw sockets or CAP_NET_ADMIN) and replace the setuid bit with the matching file capability wherever possible, since capabilities are strictly narrower than "full effective UID 0."
• One useful cross-check: setuid and setgid processes cannot be attached to with ptrace(2) (the debugging/tracing syscall) by an unprivileged tracer, by design, specifically so a normal user can't attach a debugger to a setuid-root process and manipulate its already-elevated memory. If you find yourself needing to strace a setuid binary to understand it, that itself is a sign to do the investigation as root or to test with a non-setuid copy in a sandboxed environment instead of fighting the protection.