System Calls & the Kernel Interface Questions

Why rename() enables atomic rotation

rename(oldpath, newpath), when both paths are on the same filesystem, is a single atomic kernel operation: the directory entry newpath is made to point at oldpath's inode, and if newpath already existed, it is atomically replaced (unlinked) as part of that same operation. There is no window where newpath is momentarily missing, and no window where a reader opening newpath could see a partially-written file at that name. This is exactly the property log rotation relies on: rename(current.log, current.log.1) moves the old log out of the way in one step, with no interval where the log path doesn't resolve to something.

What happens to already-open file descriptors after the rename

An fd that was already open on the file before the rename keeps referring to the SAME underlying inode regardless of what any directory entry now says, because rename() only changes which name in the directory tree points at that inode; it does not touch any process's open file descriptors. So a long-lived process that opened current.log before rotation and is still writing through that fd keeps writing to the RENAMED file, now called current.log.1, because the fd was bound to the inode at open time, not to the path string. This is precisely why rename-based rotation requires the writing application to reopen the log file (conventionally on receiving SIGHUP) after rotation; without that reopen, the application's writes silently continue landing in the rotated-away file forever, and the freshly created current.log stays empty from that writer's perspective even though the file on disk with that name is brand new.

When copytruncate is required instead

If the writing process cannot be signaled or otherwise made to reopen its log fd (a third-party binary with no reopen hook, or a process that can't safely be restarted), copytruncate is the fallback: copy the current file's content out to the rotated name, then ftruncate() the ORIGINAL file back to zero length in place, so the still-open fd keeps writing to the same inode and the same name. The caveat: after truncation the fd's write offset is unchanged unless the fd was opened with O_APPEND (in which case O_APPEND makes every write() reposition to the file's current end before writing, regardless of the fd's stored offset, so each write recomputes its position from the file's actual current end), so without O_APPEND the next writes can land at the old offset past the new end-of-file, leaving a gap of null bytes until the application's position catches up on its own. Copytruncate is also not atomic the way rename is: there is a brief window between the copy and the truncate where a concurrent write can land in a region already captured by the copy, or be lost from both copies, so it is a fallback for when rename's atomicity genuinely isn't available, not a strict improvement on it.

A robust rotation strategy

Prefer rename() plus a signal-driven reopen (a SIGHUP handler that calls close() then open(path, O_APPEND | O_CREAT, mode) on the log path) as the primary strategy, since it is atomic and loses no log data; fall back to copytruncate, always paired with O_APPEND on the writer's fd, only when the writer genuinely cannot be made to reopen. Keep enough rotated generations for the retention policy, and compress older generations asynchronously so the rotation step itself stays fast and doesn't block the writer.

A second worked example built on the same atomicity fact: safe config replacement

The same rename() guarantee is used for a related but distinct problem: publishing a new version of a config file (potentially secret-bearing) without ever exposing a partially-written or wrongly-permissioned file at the real path. The pattern: write the new content to a TEMPORARY file in the SAME directory as the target (same filesystem is required here too, since rename() across filesystems is not atomic and in fact fails with EXDEV), fsync() that temp file so its content is durable on disk before it's ever visible under the real name, then fchmod()/fchown() the temp file to the FINAL intended mode and ownership BEFORE the rename, not after. That ordering matters specifically for secrets: if the mode is fixed up after the rename instead of before, the file is briefly visible at its real, well-known name with an overly permissive mode, a real exposure window even if it lasts only milliseconds, and even a very short window is enough for a concurrently running process that happens to be watching that path to read it. Then rename(tmp, target) atomically publishes the new content, and finally fsync() the containing DIRECTORY's own file descriptor as well, a commonly missed step, since the directory entry update itself needs its own fsync to be durable against a crash; without it, a crash immediately after rename could leave the directory entry pointing at the old inode again on some filesystems after recovery, even though the rename() call itself returned success.

setrlimit(2) lets you cap how much of a given resource a process may consume, and the kernel enforces the limit at the point the process tries to exceed it. For memory specifically, the two limits people reach for are RLIMIT_AS and RLIMIT_RSS, and only one of them actually does anything on modern Linux.

1. How setrlimit(2) constrains memory

setrlimit(resource, &rlim) takes a resource identifier and a struct rlimit { rlim_t rlim_cur; rlim_t rlim_max; } (a soft limit the process can raise up to the hard limit, and a hard limit only a privileged process can raise). A process typically sets its own limits at startup, or a parent/supervisor (a systemd unit's LimitAS=, or a shell's ulimit) sets them before exec, since limits are inherited across fork(2) and preserved across execve(2).

2. RLIMIT_AS vs RLIMIT_RSS

RLIMIT_AS caps the size, in bytes, of the process's total virtual address space: every brk()/mmap()/mremap() call that would push the process's mapped address space past the limit fails with ENOMEM, and automatic stack growth past the limit raises SIGSEGV. This is enforced synchronously and reliably by the kernel, and it counts reserved virtual memory, not memory actually backed by RAM: a large mmap(MAP_NORESERVE) region, a generous heap arena a language runtime reserves up front, or a big anonymous mapping that's mostly untouched all count fully against RLIMIT_AS even if their resident footprint is tiny.

RLIMIT_RSS looks like the natural fit ("limit the resident set, i.e. the physical RAM actually in use") but per the setrlimit(2) man page it has effect only on Linux kernels in the 2.4.x series before 2.4.30, and even there it only affected madvise(MADV_WILLNEED) behavior. On every Linux kernel in production today, setting RLIMIT_RSS is a silent no-op: getrlimit() will report back whatever value you set, but the kernel does not use it to constrain, reclaim, or kill anything. Reaching for RLIMIT_RSS to fix an OOM-killed worker will look like it did something (no error, the limit "sets") and change nothing.

3. Consequences of overly strict limits

Because RLIMIT_AS counts virtual, not resident, memory, an overly strict RLIMIT_AS causes failures well before the process is actually short on physical RAM. A worker that reserves a 1 GiB heap arena at startup (common for JVM-style runtimes, and for allocators that reserve large arenas per thread) will fail to even start under a 512 MiB RLIMIT_AS, even though its live-object footprint might be 100 MiB. The failure mode is also abrupt and hard to diagnose from the outside: malloc() returns NULL or the process is killed by SIGSEGV on stack growth, which looks like "random crash" rather than "hit a configured limit," unless someone thinks to check /proc/<pid>/limits or the process's own ENOMEM handling. Debugging/instrumentation tools that inflate virtual (not resident) footprint, like AddressSanitizer's shadow memory (extra memory a sanitizer reserves to track the validity of every real byte allocated, often several times the real footprint), can also blow through an RLIMIT_AS sized for the un-instrumented binary.

4. Choosing sensible limits while preserving SLOs

Given RLIMIT_RSS doesn't work, the practical Linux mechanism for "cap this worker's actual physical memory use" is a cgroup (control group) memory controller, not a process rlimit: memory.max is a hard ceiling that gets the cgroup OOM-killed (only processes in that cgroup, not a global OOM-killer sweep across unrelated processes on the host) when exceeded, and memory.high is a soft throttle that triggers reclaim/backpressure on the cgroup before the hard kill, giving the workload a chance to shed load gracefully instead of being killed outright.

To size these: measure the worker's actual peak resident usage under representative load, e.g. VmHWM (high-water-mark RSS) from /proc/<pid>/status, or a cgroup's memory.peak. Say that measurement comes back as 900 MiB peak RSS (a MEASURED figure, not an estimate). Set memory.high around 1.1x that peak (about 1.0 GiB) as the soft throttle point, and memory.max around 1.3-1.5x the peak (900 MiB x 1.3 = 1,170 MiB, about 1.14 GiB, up to 900 MiB x 1.5 = 1,350 MiB, about 1.32 GiB, at the 1.5x end) as the hard kill ceiling, giving headroom for legitimate spikes (a larger-than-usual request, a GC pause building up garbage) without either constantly throttling in the normal case or letting one runaway worker starve the whole host. If you also want an outer RLIMIT_AS backstop (useful against pathological virtual-memory leaks, e.g. an mmap loop that never frees), set it generously above what the runtime is known to reserve at startup, not near the measured RSS figure, since AS and RSS are measured on different bases and conflating them is exactly how you get spurious ENOMEM at start-up. Finally, validate under load: run the same load test that produced the 900 MiB measurement against the new limits and confirm the SLO (e.g. p99 latency, error rate) holds at the soft-throttle boundary, then keep watching memory.high throttling events and OOM-kill counts in production so the limit gets revisited as the workload's real memory profile changes.

Semantics

O_APPEND, set at open() time (or via fcntl(fd, F_SETFL, ... | O_APPEND) afterward), changes what a subsequent write(2) on that fd does: instead of writing at the fd's current offset, the kernel atomically seeks to the current end-of-file and performs the write there, as one combined operation. "Atomically" is the operative word: without O_APPEND, an application that wants append behavior would have to lseek(fd, 0, SEEK_END) and then write() as two separate calls, leaving a window between them where another writer could extend the file, so the seek would find a stale end-of-file position and the write would land in the wrong place, potentially overwriting the other writer's just-added data. O_APPEND removes that window by making "find the end, then write" a single kernel operation.

Is write() atomic for multiple processes appending?

Yes, but only per individual write(2) call, and only on local filesystems. For a single write() call to a file opened O_APPEND on a local filesystem such as ext4 or xfs, Linux guarantees no other process's write (whether or not that other process also has O_APPEND set) can land in the middle of it: each write() call is atomically positioned and committed relative to the file's end. What is NOT guaranteed is atomicity ACROSS multiple separate write() calls: if one logical log line is emitted via three separate write() calls (say, timestamp, then message, then newline), another process's single write() can interleave between any of those three calls, producing a torn, interleaved line in the file even though every individual write() itself was atomic.

There is also a well-known real-world gap: on NFS, particularly older NFS protocol versions, O_APPEND atomicity across different CLIENT machines is NOT guaranteed, because the "find the end of file" step is negotiated over the network and a race window exists between clients that does not exist for local filesystems. This bites teams that mount a shared NFS directory as a common log target for multiple hosts and assume the same atomicity they'd get locally.

Implications for multi-process logging, and recommendations

Multiple processes can safely append to the SAME file without corrupting each other's data, provided every writer emits one complete log record (line, JSON object, whatever the unit is) via a SINGLE write() call, on a local filesystem, with O_APPEND set on the fd. The ordering BETWEEN processes' records is not guaranteed (whichever write() call the kernel services first wins that position, so wall-clock write order and file order can diverge slightly under heavy contention), but that's a liveness/ordering nuance, not a corruption risk, and is normally acceptable for logs since each line still carries its own timestamp.

Concrete recommendations:

• Always open the log fd with O_APPEND; never use a separate lseek(SEEK_END) followed by write(), which reintroduces exactly the race O_APPEND exists to close.
• Format the entire record into one buffer in memory first, and emit it with a SINGLE write() call, rather than writing the timestamp, message, and newline as three separate calls.
• Keep individual records to a size the underlying filesystem will genuinely commit as one write (very large single writes can, in principle, still be split by some layers); most logging libraries cap or truncate oversized single lines defensively rather than relying on an unbounded guarantee.
• For high-volume or many-writer scenarios, prefer routing through a single log daemon or socket (syslog, journald) that serializes writes itself, rather than having N processes race directly on one shared file at all.
• Do not rely on O_APPEND's cross-process atomicity for a log target mounted over NFS; use per-host files or a dedicated network log sink instead.

What a zombie is

A zombie (also called "defunct") is a process that has already called _exit() (or been killed) and released essentially everything it owned, memory, file descriptors, and any other resources, but whose EXIT STATUS entry is still sitting in the kernel's process table because the parent hasn't yet called wait()/waitpid() to collect it. It isn't "still running" in any meaningful sense, no CPU time, no memory footprint worth mentioning, it's a lightweight bookkeeping record (essentially a PID plus an exit status) that POSIX requires the kernel to keep around until a parent explicitly retrieves it, because a parent is always entitled to learn how its child exited.

How it's created

Process A fork()s child B; B exits (normally, via exit()/_exit(), or terminated by a signal); the kernel converts B into a zombie and delivers SIGCHLD to A. If A never calls wait()/waitpid(), no signal handler installed, ignoring SIGCHLD without the SA_NOCLDWAIT auto-reap behavior, or simply stuck/busy and never getting around to it, B remains a zombie indefinitely, until A itself exits, at which point B is reparented (to init/PID 1 historically, or to whatever process has registered itself as a "subreaper" via prctl(PR_SET_CHILD_SUBREAPER) on modern Linux), and that new parent reaps it.

Detection

ps aux (or more precisely, ps -eo pid,ppid,stat,cmd) shows Z in the STAT column with the command name suffixed <defunct>. MEASURED, a zombie I deliberately created and captured: ps -o pid,ppid,stat,cmd -p 5195 showed 5195 5193 Z [zombie_demo] <defunct>. The raw kernel-level view is /proc/<pid>/stat, whose third whitespace-separated field is a single character giving process state; cat /proc/5195/stat | awk '{print $3}' confirmed the literal character Z. It's worth explicitly distinguishing a zombie (Z) from a similar-looking but unrelated state, D (uninterruptible sleep): D means a process is blocked inside a kernel operation, usually disk or storage I/O, or a driver wait, that cannot even be interrupted by a signal, it is still fully alive, still consuming resources, whereas Z is a dead process consuming almost nothing but a table slot. A fast host-wide sweep is ps -eo stat= | grep -c Z; pstree -p is the useful next step, since it shows WHICH parent owns a cluster of zombies, which is the process you actually need to fix.

Consequences

Each zombie's process-table entry occupies one PID out of a finite, GLOBAL PID space (/proc/sys/kernel/pid_max, classically 32768 on older/legacy Linux configurations, tunable much higher, up to roughly 4 million on 64-bit systems, and many modern distros ship a higher default out of the box; treat 32768 as the traditional textbook figure, not a universal constant, since it varies by distro and kernel configuration). Practically: a long-running process that forks repeatedly without reaping any children will, given enough time, exhaust available PIDs for the ENTIRE HOST, not just for itself, since PID space is shared system-wide, at which point fork() anywhere on the box starts failing with EAGAIN, an outage that looks completely unrelated to its actual root cause unless whoever's investigating knows to check ps/pstree for a zombie pileup first.

Reaping strategies for long-running services/supervisors

• A blocking wait()/waitpid(pid, &status, 0) right after each fork(), fine when the supervisor only ever has one child in flight at a time, wrong the moment it needs to do anything else concurrently.
• A SIGCHLD handler that loops on waitpid(-1, &status, WNOHANG) until it returns 0 (nothing left to reap right now) or -1/ECHILD (no children at all). The loop matters because SIGCHLD, like other standard (non-realtime) signals, is not queued and can coalesce: several children exiting in quick succession can result in a single handler invocation, so a handler that reaps only one child per call will systematically leak zombies under load, exactly when the supervisor is busiest.
• Explicit signal(SIGCHLD, SIG_IGN) (or sigaction with SA_NOCLDWAIT) when the exit status genuinely doesn't matter, this tells the kernel to auto-reap children the instant they exit, so they never become zombies at all. Correct for pure fire-and-forget children; wrong the moment you need the exit code or signal to decide restart/log/quarantine behavior.

Absorbed: the concrete operational motivation for a scanning/sandboxing daemon

A daemon that spawns children specifically for scanning or sandboxing, a security scanner forking a per-file analysis process, say, must ACTIVELY reap them: waitpid(-1, &status, WNOHANG) on every pass through its own event loop, or a SIGCHLD handler doing the same loop, or those scan-worker zombies accumulate silently. The dangerous part is that they accumulate WITHOUT tripping the metrics anyone's actually watching: a zombie costs almost no CPU or memory, so the daemon's own health checks look completely normal while ps/PID-table headroom quietly degrades in the background. During an actual incident, a responder who tails logs and checks top/CPU/RSS sees a "healthy" daemon and, separately, a mysteriously unresponsive host that can no longer fork anything, and the connection between the two isn't obvious unless you specifically know to look for unreaped children. That's exactly the incident-response blind spot this creates: the thing that's actually broken doesn't show up anywhere near the alert that fired.

inotify(7) lets a process ask the kernel to notify it, as a readable fd, when specific files or directories change, which is the standard building block for "reload the config when it changes" without polling.

1. Using inotify to watch config and trigger a reload

inotify_init1() creates an inotify fd; inotify_add_watch(fd, path, mask) registers a watch on a path with a bitmask of event types (IN_MODIFY, IN_CREATE, IN_DELETE, IN_MOVED_FROM, IN_MOVED_TO, IN_CLOSE_WRITE, and others). For a config-reload use case, IN_CLOSE_WRITE (fired when a file opened for writing is closed) is usually the more reliable signal than IN_MODIFY: many editors and deployment tools write a config in several small writes, so IN_MODIFY can fire repeatedly mid-write, and a reload triggered on the first one can read a half-written file. The service reads events off the inotify fd (each event is a struct inotify_event, optionally followed by a filename) in an event loop, typically plugged into the same epoll loop as everything else since the inotify fd is just another readable fd, and debounces/coalesces bursts of events (a single "save" in many editors fires several) before actually triggering the reload.

2. Resource limits

/proc/sys/fs/inotify/max_user_watches caps the number of watches a single real user ID can hold across all its processes; it's per-user, not per-process, so a host running several services that each watch a directory tree can collectively exhaust it even though no individual service looks excessive. inotify is not recursive: watching a directory tree means adding one watch per subdirectory yourself, so watching a large tree can consume watches fast. There's also a per-user max_user_instances limit (how many inotify fds/instances a user can hold) and a max_queued_events limit on the kernel's event queue for a given instance; if events arrive faster than the consumer drains them, the queue can overflow, which surfaces as a single IN_Q_OVERFLOW event (with no per-file detail) and silent loss of the events that overflowed. A consumer that relies purely on the event stream without ever cross-checking has no way to tell it missed something unless it specifically watches for IN_Q_OVERFLOW and reacts by doing a full resync (re-reading the config from disk unconditionally) rather than trusting the stream was complete.

3. Race conditions: moved and renamed files

inotify watches track a specific inode (the filesystem's internal file identity), not a path string. The standard "atomic config swap" pattern many tools use, write a new version to a temp file, then rename() it over the original path, breaks a watch placed directly on the original file: after the rename, the watch you had is now watching the old inode (which the rename replaced at that path), and typically fires IN_MOVE_SELF/IN_ATTRIB once and then the watch becomes invalid, it does not automatically start watching whatever new inode now occupies that path. The correct pattern is to watch the containing directory, not the file itself, for IN_MOVED_TO/IN_CREATE/IN_CLOSE_WRITE on the filename you care about, and re-open (and, if you want live updates on that specific file going forward, re-watch) the target whenever it reappears at that path. There's also a residual time-of-check-to-time-of-use gap between "receive the event" and "actually read the file": the file can be rewritten again in the interval, so a defensive reloader re-checks a checksum or mtime, or is simply willing to reload again on the next event rather than assuming the read it just did is final.

4. Alternatives for large-scale monitoring with lower overhead

fanotify(7) can watch an entire mount rather than needing one watch per path, gets richer events (including, in newer modes, permission events that can allow/deny an access before it completes), and avoids inotify's per-directory watch-count overhead for monitoring broad filesystem activity; it's the mechanism behind most Linux-based AV/DLP-style file-activity monitoring, though historically required CAP_SYS_ADMIN (a Linux capability: a narrow, named slice of root's privilege that can be granted independently of full root access; later kernels added narrower FAN_REPORT_* modes that need less). eBPF-based watchers, hooking VFS-layer tracepoints or LSM (Linux Security Module) hooks such as security_file_open, can observe filesystem activity system-wide with much lower per-watch overhead than inotify's per-path bookkeeping and without the max_user_watches ceiling at all, at the cost of real complexity: kernel-version sensitivity, BPF program verification, and typically CAP_BPF/CAP_SYS_ADMIN (often effectively root) to load the program. For "reload my own service's config directory," inotify on the containing directory is the right, simple tool; fanotify or eBPF-based watching is for a different problem, broad, system-wide or security-monitoring-grade file-activity visibility, not a single service's config-reload path.