System Calls & the Kernel Interface Questions

High syscall volume with high user CPU points at overhead from crossing the user/kernel boundary itself (each syscall costs a mode switch, argument validation, and often a context-switch-adjacent cost), not from the underlying work being expensive. The same symptom is often reported as an elevated context-switch rate rather than a raw syscalls/sec counter, since every syscall that blocks (or every mode transition, depending on what your monitoring actually samples) shows up there too; treat "syscalls/sec is high" and "context-switch rate is high" as two views of the same underlying problem, and profile with tools that show you which specific syscalls, not just the aggregate rate.

1. Profiling which syscalls dominate

• strace -c -p <pid> (or wrapping the process at startup) gives a summary table of syscalls by count and total time, cheaply for a quick look, but strace's own ptrace-based tracing overhead is significant and can itself distort timing under a syscall-heavy workload; treat its numbers as directionally correct, not precise, at high syscall rates.
• perf trace or perf stat -e 'syscalls:sys_enter_*' uses kernel tracepoints instead of ptrace stops, so it has far lower overhead than strace and is safer to run against a live, loaded production process for a short sampling window.
• bpftrace (eBPF-based) is the right tool for both low-overhead sampling and answering more specific questions live: bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm, args->id] = count(); }' tallies syscalls by process name and number with minimal overhead, and can be scoped to a specific PID or run continuously as a lightweight production diagnostic in a way strace cannot.
• sysdig gives a similar system-call-level view with richer contextual filtering (by container, by file, by network connection) if the environment is containerized.

2. Concrete changes to reduce syscall overhead, each with its trade-off

• Batching with vectored I/O (writev/readv). If the workload currently issues many small write()/read() calls per logical operation (e.g. writing a header, then a body, then a trailer as three separate write()s), writev()/readv() combine them into a single syscall carrying multiple buffers. Trade-off: you need to gather the buffers to hand to the vectored call up front, and it doesn't help if the data genuinely isn't available yet (you can't vector data you haven't produced), so it's a pure win for "I have N buffers ready right now and issue them one at a time out of habit," not for a fundamentally serial producer.
• User-space buffering. Accumulate several logical writes into an application-level buffer and flush it with one syscall once it's full or a timer expires (the same idea stdio's buffered FILE* does automatically). Trade-off: this trades latency for throughput, since a given write now waits for either the buffer to fill or the flush timer, whichever comes first, typically a few milliseconds of added latency in exchange for a large reduction in syscall count; it's the wrong choice for a latency-sensitive path where every write must be visible immediately (e.g. a synchronous acknowledgment).
• Zero-copy APIs (sendfile, splice, vmsplice). For "read from one fd and write to another" workloads (serving a static file to a socket, proxying), sendfile()/splice() move data kernel-to-kernel without ever copying it through a userspace buffer, cutting both syscall count and CPU spent copying. Trade-off: they only apply to that specific fd-to-fd shape; they don't help if the application needs to inspect or transform the data in userspace along the way, since the data never becomes addressable to the application in that path.
• Reducing per-request syscall count directly. Sometimes the real fix is upstream of any of the above: caching a value that's currently being re-fetched with a syscall on every request (e.g. calling gettimeofday()/clock_gettime() far more often than the use case actually needs precision for), or reusing connections/file descriptors instead of opening and closing them per request. Trade-off: caching introduces staleness that has to be bounded and reasoned about explicitly.

3. Validating each change safely in production

For each candidate fix, measure the same syscall-rate profile before and after on the same basis (e.g. syscalls per request, not just an aggregate syscalls/sec that also moves with traffic volume), roll it out to a small canary fraction of instances first, and watch both the mechanism-level signal (syscall rate, CPU in the kernel vs. user, context-switch rate) and the actual SLO (p99 latency, error rate, throughput) together, since a change that cuts syscalls but adds latency (buffering) or memory (larger buffers) can violate the SLO it was meant to protect even while "succeeding" on the metric it targeted. Roll back on regression in either dimension, not just the syscall count.

Worked example, on a consistent basis. If the service handles 5,000 requests/second and each request currently issues 20 syscalls (5,000 req/sec x 20 syscalls/req = 100,000 syscalls/sec, matching the observed rate), batching those 20 per-request calls into, say, 5 via writev()/buffering brings per-request syscalls from 20 down to 5, which at the same 5,000 req/sec cuts total syscall rate from 100,000/sec to 25,000/sec, a 4x reduction, all computed on the same per-request-times-request-rate basis. The cost is whatever latency the batching/buffering step adds to each request; that has to be measured and checked against the SLO, not assumed to be free.

pidfd_open(2) and pidfd_send_signal(2)

pidfd_open(pid, flags) returns a file descriptor ("pidfd") that refers to a SPECIFIC process instance, not to a PID number. pidfd_send_signal(pidfd, sig, info, flags) sends a signal through that fd instead of through a raw PID (as kill(2) does).

The race pidfds solve

PIDs are recycled. On Linux, the kernel by default won't reuse a PID number until it wraps back around through the whole PID space (bounded by pid_max), but on a busy, long-running, high-fork-rate system, that wraparound genuinely happens, and it happens FASTER the smaller pid_max is configured. The dangerous window is: you learn a process's PID (say, you're a supervisor thread that just forked worker #482), some time passes (maybe you're processing other events, maybe you got descheduled), the process exits AND gets reaped by someone else, and its PID number gets reissued to a completely unrelated new process, all before you get around to acting on the PID you originally recorded. If you then call kill(482, SIGTERM), or worse, waitpid(482, ...), you are now operating on a DIFFERENT process that happens to share that number, not the one you meant. This is especially live in a MULTITHREADED supervisor: one thread reads a PID out of a shared table, hands off to another thread to act on it later, and the gap between "read the PID" and "act on the PID" is exactly the race window, no amount of careful locking around the shared table closes the race, because the kernel-side PID-to-process mapping can change independently of your process's own locks.

A pidfd closes this: it's a file descriptor bound to the kernel's internal struct pid for the specific process instance you opened it against, at the moment you called pidfd_open. If that process exits and its PID number is reissued to someone else, the pidfd keeps pointing at the original (now-dead) process; operations through it (pidfd_send_signal, poll, waitid(P_PIDFD, ...)) either act correctly on the original process or fail with ESRCH once it's fully reaped, they never silently retarget to the new process wearing the old PID number. That's the concrete guarantee: identity is bound to the fd, not re-derived from a number that can be recycled.

Reference implementation (verified, not illustrative)

pidfd_demo.c, forks a child, opens a pidfd for it, signals it THROUGH the pidfd, waits for exit via poll(), and reaps it via waitid(P_PIDFD, ...):

#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <signal.h>
#include <poll.h>
#include <sys/wait.h>
#include <sys/pidfd.h>
#include <sys/types.h>
#include <string.h>
#include <errno.h>
#include <time.h>

int main(void) {
    pid_t child = fork();
    if (child < 0) { perror("fork"); return 1; }

    if (child == 0) {
        printf("[child %d] sleeping until signaled\n", getpid());
        fflush(stdout);
        /* Infinite sleep standing in for the child's real work in this demo;
           written as a goto loop rather than while(1) purely as a style choice. */
        pause_loop:
        sleep(30);
        goto pause_loop;
    }

    /* Parent: open a pidfd bound to this exact child process instance. */
    int pidfd = pidfd_open(child, 0);
    if (pidfd < 0) { perror("pidfd_open"); return 1; }
    printf("[parent] opened pidfd=%d for child pid=%d\n", pidfd, child);

    usleep(200000); /* let the child print & settle */

    /* Signal the child THROUGH THE FD, not through its raw pid. */
    if (pidfd_send_signal(pidfd, SIGTERM, NULL, 0) < 0) {
        perror("pidfd_send_signal");
        return 1;
    }
    printf("[parent] sent SIGTERM via pidfd_send_signal(pidfd=%d)\n", pidfd);

    /* poll() the pidfd: it becomes readable (POLLIN) exactly when the
     * process exits. */
    struct pollfd pfd = { .fd = pidfd, .events = POLLIN };
    int pr = poll(&pfd, 1, 5000);
    if (pr < 0) { perror("poll"); return 1; }
    if (pr == 0) { fprintf(stderr, "timed out waiting for exit\n"); return 1; }
    printf("[parent] pidfd is readable: child has exited (revents=0x%x)\n", pfd.revents);

    /* Reap through the pidfd via waitid(P_PIDFD, ...) so the child never
     * lingers as a zombie. */
    siginfo_t info;
    memset(&info, 0, sizeof(info));
    if (waitid(P_PIDFD, pidfd, &info, WEXITED) < 0) { perror("waitid"); return 1; }

    printf("[parent] reaped pid=%d code=%d status=%d (%s)\n",
           info.si_pid, info.si_code, info.si_status,
           info.si_code == CLD_KILLED ? "killed by signal" :
           info.si_code == CLD_EXITED ? "exited normally"  : "other");

    close(pidfd);
    return 0;
}

Build and run (needs glibc >= 2.36 for the pidfd_open/pidfd_send_signal wrapper declarations in <sys/pidfd.h>, and a kernel >= 5.4 for P_PIDFD in waitid, when that support was added):

$ gcc -Wall -o pidfd_demo pidfd_demo.c
$ ./pidfd_demo

MEASURED output (Debian 12, glibc 2.36, real run):

[child 5066] sleeping until signaled
[parent] opened pidfd=3 for child pid=5066
[parent] sent SIGTERM via pidfd_send_signal(pidfd=3)
[parent] pidfd is readable: child has exited (revents=0x1)
[parent] reaped pid=5066 code=2 status=15 (killed by signal)

code=2 is CLD_KILLED, status=15 is SIGTERM's numeric value, both consistent with the child having been terminated by the signal delivered through the pidfd, exactly as intended, and the reap happened with no zombie left behind.

Why this matters specifically for multithreaded supervisors

Beyond closing the PID-reuse race, pidfds let a supervisor watching many children put every one of their pidfds into a SINGLE epoll (a Linux mechanism for waiting on many file descriptors at once and finding out which ones became ready) set. That replaces the alternative of relying on one process-wide SIGCHLD signal (which can coalesce multiple near-simultaneous exits into a single delivery, and racing multiple threads all calling waitpid(-1, ...) concurrently to figure out which child fired is genuinely awkward to get right) with a design where each epoll_wait() readiness event unambiguously names, via which pidfd became readable, exactly which worker exited, no signal-coalescing ambiguity, no -1-wildcard reaping race between threads, and no PID-reuse window to worry about, since the fd itself is the identity.

select(2), poll(2), and epoll(7) all answer the same question ("which of these file descriptors are ready for I/O right now"), but they differ in exactly the dimension that matters at scale: how the cost of asking scales with the number of watched fds versus the number of ready fds.

1. Complexity and the fd_set limit

select() represents the watched set as a fixed-size bitmap (fd_set), sized by FD_SETSIZE, typically 1024 on Linux/glibc. That's a hard ceiling on the numeric value of any fd you can watch, not just a count limit: if your process happens to have an fd numbered 1025 open (easy to hit with enough concurrent connections over the process's lifetime), you literally cannot pass it to select(), regardless of how few fds are active at once. Every call is O(n) in the number of watched fds: the kernel scans the whole bitmap, and userspace has to rebuild the fd_set before every call since select() mutates it in place. poll() replaces the bitmap with an array of struct pollfd, removing the FD_SETSIZE ceiling, but is still O(n) per call: the kernel scans the whole array, and the whole array is copied user-to-kernel and kernel-to-user on every single call regardless of how many fds are actually ready. Neither scales to watching tens of thousands of mostly-idle connections, since both pay for the full watched set on every call. epoll() splits registration from waiting: epoll_ctl() registers a fd with a persistent, kernel-side interest list once, and epoll_wait() returns only the fds that are actually ready, an O(1)-amortized cost per ready event rather than O(n) over the whole watched set.

2. Kernel/user interactions

select/poll are stateless from the kernel's perspective: each call is a fresh, complete description of what to watch, so the kernel has to walk that whole description every time, and the copy-in/copy-out of the (possibly large) fd list happens on every call too. epoll keeps that state inside the kernel across calls, in the epoll instance created by epoll_create1(); epoll_ctl(EPOLL_CTL_ADD/MOD/DEL, ...) mutates that persistent interest list, and epoll_wait() just asks "what's ready," with no need to re-describe the whole set. This is the mechanical reason epoll's cost tracks the number of events, not the number of watched fds.

3. Pitfalls implementing an event loop, including edge-triggered gotchas

epoll has two readiness-reporting modes. Level-triggered (LT, the default) behaves like select/poll: as long as data remains, epoll keeps reporting the fd as ready on every epoll_wait() call. Edge-triggered (ET, EPOLLET) reports a fd exactly once, at the moment it transitions from not-ready to ready. The classic ET bug: if a handler reads one chunk from a socket, sees there's more data buffered, but moves on to the next fd "to be fair" instead of draining fully, the socket buffer still has bytes in it, but no new edge (transition) will occur until more data arrives from the peer, so that handler is never woken for the leftover bytes and the connection silently stalls. The only correct pattern under ET is: set the fd non-blocking, and loop read()/write() until the call returns -1/EAGAIN, then go back to epoll_wait(). LT mode avoids that specific class of bug by re-notifying every time, at the cost of the notification overhead LT and select/poll share.

4. Which to choose for tens of thousands of concurrent connections

epoll, and it isn't close. For a service holding, say, 50,000 concurrent connections where only a few hundred are active in any given instant, select/poll pay for scanning and copying all 50,000 watched descriptors on every call, while epoll_wait() only returns and processes the few hundred that are actually ready, roughly two orders of magnitude less work per call at that scale, on the same basis (cost per readiness-check call). The only reasons to still reach for poll() are portability (epoll is Linux-specific) or a genuinely small, low-fd-count workload where the difference doesn't matter.

5. The same pattern one layer up: Python's asyncio

Python's asyncio event loop is a good example of this exact reasoning showing up above the raw syscall layer. On Linux, the selectors module asyncio builds on prefers epoll (selectors.EpollSelector) over select/poll for precisely the C10K-style reasoning above (the classic C10K problem: how to serve ten thousand simultaneous connections without the per-connection cost exploding): a single-threaded event loop juggling thousands of open sockets needs cost proportional to ready connections, not watched ones. This is also why asyncio uses one (or a small, fixed number of) OS thread(s) running an epoll-based reactor rather than a thread-per-connection model: each OS thread carries real overhead (a default stack reservation, scheduler and context-switch cost), so thousands of threads for thousands of mostly-idle connections would reproduce the same scaling problem epoll was built to avoid, just moved from "syscall cost" to "thread/scheduling cost." As for the edge-triggered pitfall: asyncio's default selector-based loop runs epoll in level-triggered mode specifically so individual callback authors don't have to implement the drain-to-EAGAIN discipline themselves; the trade-off is the LT overhead described above. That pitfall doesn't disappear from the ecosystem, though, it moves to wherever a lower layer does use edge-triggered mode directly, e.g. a C extension or an alternative event-loop implementation (such as one built on libuv) that embeds its own edge-triggered epoll internals: any code integrated at that layer still has to fully drain a readable fd on each wakeup, or it will reproduce the exact same "stalled connection" bug described above, just one level of abstraction further from the raw syscall.

What umask() does

umask(mask) sets a per-process file mode creation mask. It doesn't create or restrict files directly; instead, it's subtracted from the permission bits an application REQUESTS at file or directory CREATION time, for calls like open(path, O_CREAT, mode) and mkdir(path, mode). The actual permission bits the kernel grants are computed as requested_mode & ~umask, i.e. any bit set in the umask is cleared from the requested mode, never added. Crucially, umask only ever removes permission bits from what was requested; it has no effect on chmod() of an existing file, and it doesn't change the mode value the application itself passes in, only what the kernel ultimately grants at the moment of creation.

A concrete example

With the common default umask 0022, a call to open(path, O_CREAT, 0666) produces a file with mode 0644 (0666 & ~0022 = 0644): group and other lose the write bit, owner keeps read/write. The same umask applied to mkdir(path, 0777) yields a directory with mode 0755.

Why it matters for tools that create reports, temp files, or secret-bearing artifacts

umask is inherited from the parent process, typically the shell or the service manager that launched the tool, and that inherited value is not something the tool's own code controls unless it explicitly overrides it. That makes it an ambient, environment-dependent setting: an interactive shell might run with umask 0022, while a cron job, a systemd service, or a container entrypoint can easily run with a different, sometimes looser umask (0002, or in some misconfigured base images, 0000) depending on how it was configured. A tool that requests a "reasonably conservative-looking" mode like 0644 and trusts the ambient umask to tighten it further is at the mercy of whatever environment happens to invoke it. In an environment with a loose or zero umask, a secret-bearing file the developer believed would end up 0600 can instead end up world-readable, not because the code did anything wrong in isolation, but because it relied on an environmental factor it never verified.

Practical guidance

For anything sensitive, don't rely on the ambient umask at all. Either call umask(0) explicitly at process startup and then pass the EXACT desired mode to every subsequent open()/mkdir() call, removing the dependency on the caller's environment entirely, or fchmod() the file descriptor immediately after creation to the exact intended mode, before any content is ever written to it, so there is no window where a wrong (looser) mode is live on a file that might already contain sensitive data. The fchmod-on-fd approach mirrors the same act-on-the-descriptor-not-the-path discipline that generally defends against TOCTOU races: it fixes the permission on the specific object you already hold open, rather than depending on a race-prone or environment-dependent side effect happening correctly on its own.

Why rename() enables atomic rotation

rename(oldpath, newpath), when both paths are on the same filesystem, is a single atomic kernel operation: the directory entry newpath is made to point at oldpath's inode, and if newpath already existed, it is atomically replaced (unlinked) as part of that same operation. There is no window where newpath is momentarily missing, and no window where a reader opening newpath could see a partially-written file at that name. This is exactly the property log rotation relies on: rename(current.log, current.log.1) moves the old log out of the way in one step, with no interval where the log path doesn't resolve to something.

What happens to already-open file descriptors after the rename

An fd that was already open on the file before the rename keeps referring to the SAME underlying inode regardless of what any directory entry now says, because rename() only changes which name in the directory tree points at that inode; it does not touch any process's open file descriptors. So a long-lived process that opened current.log before rotation and is still writing through that fd keeps writing to the RENAMED file, now called current.log.1, because the fd was bound to the inode at open time, not to the path string. This is precisely why rename-based rotation requires the writing application to reopen the log file (conventionally on receiving SIGHUP) after rotation; without that reopen, the application's writes silently continue landing in the rotated-away file forever, and the freshly created current.log stays empty from that writer's perspective even though the file on disk with that name is brand new.

When copytruncate is required instead

If the writing process cannot be signaled or otherwise made to reopen its log fd (a third-party binary with no reopen hook, or a process that can't safely be restarted), copytruncate is the fallback: copy the current file's content out to the rotated name, then ftruncate() the ORIGINAL file back to zero length in place, so the still-open fd keeps writing to the same inode and the same name. The caveat: after truncation the fd's write offset is unchanged unless the fd was opened with O_APPEND (in which case O_APPEND makes every write() reposition to the file's current end before writing, regardless of the fd's stored offset, so each write recomputes its position from the file's actual current end), so without O_APPEND the next writes can land at the old offset past the new end-of-file, leaving a gap of null bytes until the application's position catches up on its own. Copytruncate is also not atomic the way rename is: there is a brief window between the copy and the truncate where a concurrent write can land in a region already captured by the copy, or be lost from both copies, so it is a fallback for when rename's atomicity genuinely isn't available, not a strict improvement on it.

A robust rotation strategy

Prefer rename() plus a signal-driven reopen (a SIGHUP handler that calls close() then open(path, O_APPEND | O_CREAT, mode) on the log path) as the primary strategy, since it is atomic and loses no log data; fall back to copytruncate, always paired with O_APPEND on the writer's fd, only when the writer genuinely cannot be made to reopen. Keep enough rotated generations for the retention policy, and compress older generations asynchronously so the rotation step itself stays fast and doesn't block the writer.

A second worked example built on the same atomicity fact: safe config replacement

The same rename() guarantee is used for a related but distinct problem: publishing a new version of a config file (potentially secret-bearing) without ever exposing a partially-written or wrongly-permissioned file at the real path. The pattern: write the new content to a TEMPORARY file in the SAME directory as the target (same filesystem is required here too, since rename() across filesystems is not atomic and in fact fails with EXDEV), fsync() that temp file so its content is durable on disk before it's ever visible under the real name, then fchmod()/fchown() the temp file to the FINAL intended mode and ownership BEFORE the rename, not after. That ordering matters specifically for secrets: if the mode is fixed up after the rename instead of before, the file is briefly visible at its real, well-known name with an overly permissive mode, a real exposure window even if it lasts only milliseconds, and even a very short window is enough for a concurrently running process that happens to be watching that path to read it. Then rename(tmp, target) atomically publishes the new content, and finally fsync() the containing DIRECTORY's own file descriptor as well, a commonly missed step, since the directory entry update itself needs its own fsync to be durable against a crash; without it, a crash immediately after rename could leave the directory entry pointing at the old inode again on some filesystems after recovery, even though the rename() call itself returned success.