System Calls & the Kernel Interface Questions

A file descriptor (fd) and the open file description it points to are two different objects in the kernel, and conflating them is where a lot of subtle bugs (and a lot of secure-design opportunities) live.

1. What a file descriptor actually is

A file descriptor is a small non-negative integer that is an index into a per-process table (the file descriptor table). Each entry in that table holds two things: a pointer to an open file description (below) and a small set of per-descriptor flags, of which the important one is FD_CLOEXEC (close-on-exec, whether this specific descriptor should be closed automatically across an execve). The integer itself has no meaning outside the process: fd 5 in process A and fd 5 in process B can point to completely unrelated files.

2. What the open file description (OFD) holds

Every successful open(2) creates a new open file description in a system-wide, kernel-internal table. The OFD is what actually holds the state that matters for reading/writing: the current file offset, the file status flags set at open time (O_APPEND, O_NONBLOCK, O_SYNC, etc, read/written via fcntl's F_GETFL/F_SETFL), and a reference to the underlying inode. The fd table entry is just a pointer to one of these.

3. Why they differ: dup() versus a fresh open()

This is the discriminator that trips people up. If you call open("log.txt", ...) twice, you get two fds and two SEPARATE open file descriptions, each with its own independent offset, even though both point at the same underlying inode: writes through one do not advance the other's offset.

If instead you take a single fd and duplicate it with dup(fd), dup2(fd, newfd), or via fork() (which duplicates the whole fd table into the child), the new fd number is a DIFFERENT integer but points at the SAME open file description. The two descriptors share one offset, so a write through either one advances the position both will read/write from next, and a fcntl(F_SETFL) status-flag change made through one is visible through the other (they are the same OFD). What is NOT shared even between dup()'d descriptors is FD_CLOEXEC: that flag lives in the per-descriptor table entry, not the OFD, so each descriptor number has its own close-on-exec setting even when two descriptors refer to the same open file.

A concrete example: dup2(log_fd, 1) makes stdout (fd 1) share log_fd's open file description. Anything the process (or a child that inherits fd 1) writes to stdout now lands in the log file at whatever position that shared OFD's offset says, which is exactly the plumbing shell redirection (cmd > log.txt) relies on.

4. Why this distinction matters for secure design

A file descriptor is a capability-like handle: once you hold an fd, you no longer need to re-resolve a pathname to use it, and the kernel has already done the permission check once, at open time. Passing a descriptor to another component (across fork/exec inheritance, or across processes via SCM_RIGHTS on a Unix domain socket) hands over exactly the access that one already-open, already-checked object represents, with no way for the receiver to substitute a different file the way it could if you'd handed over a path string instead, since re-resolving a path between the check and the use is exactly the TOCTOU (time-of-check to time-of-use) class of bug, where the resolved target can change in the gap between when you check it and when you actually use it. Designing an interface around "here is an fd, already opened read-only, already scoped to the one file you need" is a strictly narrower grant of trust than "here is a path and permission to open things under it," and it is why privilege-separated daemons (open the file as the privileged parent, hand the fd to an unprivileged worker) are a standard pattern.

There are seven things packed into this question: the difference between a function call and a syscall, which instruction crosses the boundary, the register calling convention for arguments and the return value, how the CPU's context changes, why the syscall costs more, why that overhead matters for low-latency/high-throughput design, and which measurements you'd collect. Take them in order.

1. Function call vs system call

A plain function call (f(x)) stays inside the same protection domain: the CPU just pushes a return address and jumps, the callee runs with the same privilege level and the same page tables as the caller, and the CPU's branch predictor can speculate across the call/return pair. A system call (syscall) is a request from unprivileged user code (x86-64 protection ring 3) to the kernel (ring 0) to do something only the kernel is allowed to do: touch hardware, another process's memory, the filesystem, the network stack. It requires an actual privilege-level transition, not just a jump.

2. The instruction and the register convention

On Linux/x86-64, the transition instruction is syscall (not the older 32-bit int 0x80 software-interrupt/trap-gate mechanism, which x86-64 keeps only for backward compatibility). syscall is a fast, hardware-assisted entry point: the CPU reads a kernel-configured model-specific register (MSR) for the target instruction pointer instead of walking an interrupt descriptor table (the table the CPU consults, indexed by interrupt/trap number, to find the handler address for the older int-based trap mechanism), which is why it replaced int 0x80 as the 64-bit default.

The calling convention (defined by the Linux/x86-64 ABI (Application Binary Interface, the fixed contract for how registers and the stack are used at a call boundary), distinct from the ordinary C function-call ABI):

• rax holds the syscall number going in, and the return value coming out.
• Arguments go in rdi, rsi, rdx, r10, r8, r9 in that order (up to 6 args). Note it's r10, not rcx: the syscall instruction itself clobbers rcx (return address) and r11 (flags) as part of the hardware mechanism, so the kernel ABI substitutes r10 for the 4th argument to avoid colliding with them.
• On the raw kernel ABI (below libc), there is no separate "did it fail" register. Success returns a value >= 0 in rax; failure returns a small negative number that IS -errno (for example, permission denied, EACCES = 13, comes back as rax = -13). The kernel checks the return value against the range -4095..-1 to decide "this looks like an error" versus "this is a legitimate large return value" (some syscalls legitimately return large unsigned-looking values, like a file offset).

This is exactly where errno comes from, and it's worth being precise about it because misreading it is a real production bug class. The libc wrapper (e.g. read(), open()) is the thing that takes that raw negative kernel return, negates it, stores it into the thread-local errno variable, and returns plain -1 to your C code. So errno is a libc-level abstraction, not something the kernel maintains for you. This matters because errno lets you distinguish a recoverable condition from a real failure that both look identical at the -1 level: read() returning -1 with errno == EINTR (interrupted by a signal, nothing actually went wrong, safe to retry) looks the same at the return-value level as read() returning -1 with errno == EACCES (permission genuinely denied). Code that checks only if (ret == -1) { fail(); } without reading errno will treat a transient signal interruption as a hard failure and abort work that would have succeeded on retry, or worse, code that assumes "any -1 is retryable" will spin forever retrying a real permission failure instead of surfacing it, silently masking a misconfiguration. And code that doesn't check the return value at all can treat a short/partial write() as if the whole buffer went out.

3. How context changes during the transition

On syscall entry the CPU: switches the current privilege level to ring 0; loads a per-CPU kernel stack pointer (via swapgs, which swaps in the kernel's private per-CPU data segment base); and the hardware itself saves the user return address into rcx and flags into r11. Everything else, the rest of the general-purpose registers, is saved manually by the kernel's entry assembly onto the kernel stack (into a struct pt_regs), because the kernel handler is about to run C code that will clobber them. The kernel then dispatches to the right syscall handler via the syscall number in rax, runs it (often touching kernel data structures, taking locks, doing I/O), and on the way out reverses all of this with sysret, switching the CPU back to ring 3, restoring the saved registers, and resuming user code right after the syscall instruction.

4. Why this is more expensive than a function call

Every one of those steps costs real cycles that an intra-process call skips entirely: the privilege-level switch itself, the extra register save/restore beyond what a normal call/return does, and on top of that, most production kernels also carry Meltdown/Spectre mitigations (like kernel page-table isolation, KPTI) that can force a page-table (CR3) reload and a partial TLB (translation lookaside buffer, the CPU's cache of virtual-to-physical address translations) flush on the transition, which shows up as extra cache/TLB misses right after the call returns. None of that exists for a same-domain function call, which is why the gap is roughly two orders of magnitude, not a small constant factor.

MEASURED, on this machine (an ARM64 Linux container under virtualization, not native x86-64, so treat the absolute nanosecond figures as illustrative, not portable, but the ratio is the well-known order of magnitude): calling a trivial non-inlined local function averaged 0.8 ns/call over 2,000,000 iterations; forcing a real syscall (getpid() via syscall(SYS_getpid), so it can't be short-circuited) averaged about 76-78 ns/call over the same iteration count, roughly a 95-100x ratio. On native x86-64 with all mitigations enabled, published figures for a minimal syscall (like getpid) commonly cite tens of nanoseconds, and it's meaningfully higher than that on hardware with KPTI active versus a machine where the CPU isn't vulnerable to the relevant speculative-execution bugs; both figures are still one to two orders of magnitude above a same-process call.

5. Why this matters for low-latency / high-throughput design

The overhead is small per call, but it's a fixed tax that scales with syscall count, not with the amount of useful work done, so it disproportionately hurts designs that issue many small syscalls. Concretely: suppose a service does 4 syscalls per request (say, one read(), one write(), and two socket-related calls) and serves 50,000 requests/sec. That's 200,000 syscalls/sec. At roughly 100 ns of pure transition overhead per syscall, that's 200,000 x 100 ns = 20,000,000 ns = 20 ms of CPU time spent purely on syscall entry/exit, per second of wall-clock time, i.e. about 2% of one CPU core, before any of the actual I/O work the syscalls exist to do. That number gets much worse if the service is chattier (small, unbuffered writes; polling loops that call a syscall per iteration instead of blocking), and it directly inflates tail latency (p99), not just average CPU, because every syscall is also a scheduling point where the kernel can decide to run something else instead of returning to you promptly. This is exactly why high-throughput systems favor batching APIs that fold many logical operations into one transition: readv/writev (scatter-gather I/O in one call), sendmmsg/recvmmsg (multiple datagrams per call), and io_uring (a shared ring buffer so many I/O operations can be submitted and completed with a handful of syscalls total, sometimes near-zero in polling mode).

6. Measurements to collect to validate the impact

• strace -c <cmd> for a syscall-count and time-per-syscall-type breakdown. Useful for counts and relative weight; be careful trusting its absolute timing numbers, since strace works by attaching via ptrace(2) (a debugging/tracing syscall that lets one process inspect and control another) and stopping the traced process at every syscall entry and exit, which itself adds two extra context switches per call, so a busy syscall path measured under strace looks slower than in production.
• perf stat -e 'syscalls:sys_enter_*' or perf trace, using kernel tracepoints rather than ptrace, much lower observer overhead than strace, safe to sample against a live service briefly.
• /proc/<pid>/status, the voluntary_ctxt_switches / nonvoluntary_ctxt_switches fields, as a cheap proxy for how often the process is giving up the CPU (a blocking syscall is a common voluntary-switch trigger).
• An eBPF (a Linux kernel facility for running small sandboxed programs in-kernel, which bpftrace is built on)/bpftrace syscall-count histogram (e.g. bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }'), low enough overhead to leave running in production continuously rather than only during a one-off investigation.
• Finally, and most important: end-to-end p50/p99 request latency and CPU-per-request before and after any change aimed at reducing syscall count. Micro-benchmarking syscall cost in isolation tells you the theoretical tax; it doesn't prove that tax is what's limiting this service, so the change has to be validated against the real request-latency metric, not just the syscall count going down.

One more nuance worth knowing when you're reasoning about "syscall cost": not every libc call that looks like a syscall actually traps into the kernel. The vDSO (virtual dynamic shared object, a small page of code the kernel maps read-only into every process's address space) lets a handful of read-only, timing-sensitive calls, like clock_gettime(), gettimeofday(), and getcpu(), be served entirely in user space by reading a kernel-updated memory page, with no ring transition at all. So "how many syscalls does my hot path make" and "how many syscall instructions actually execute" can differ, and strace will show you the vDSO-served ones as ordinary calls with suspiciously fast, uniform timings, or sometimes not at all, depending on the glibc version.

There are two ways to put a file descriptor (fd, the small integer handle the kernel hands back for an open file, socket, pipe, etc.) into non-blocking mode, then a real race condition that bites people who use the wrong one, then the class of bugs that shows up once the fd actually is non-blocking and you have to handle the 'nothing to do right now' signal correctly.

1. Setting non-blocking mode with fcntl(2) on an fd you already have

fcntl (file control) is the general-purpose syscall for inspecting and changing properties of an already-open fd. The file status flags (which include O_NONBLOCK, O_APPEND, and a few others) are read and written as a bitmask, so the correct idiom is read-modify-write, not a blind set, because a blind fcntl(fd, F_SETFL, O_NONBLOCK) silently clears every other flag that was already set on that fd (for example O_APPEND on a log file you're also writing to):

int flags = fcntl(fd, F_GETFL, 0);
if (flags == -1) { perror("fcntl(F_GETFL)"); return -1; }

if (fcntl(fd, F_SETFL, flags | O_NONBLOCK) == -1) {
    perror("fcntl(F_SETFL)");
    return -1;
}

2. Setting non-blocking mode with O_NONBLOCK at open(2) time

If you control the call that creates the fd, you can request non-blocking mode up front instead of a separate fcntl call afterward:

int fd = open("/tmp/myfifo", O_RDONLY | O_NONBLOCK);

Note that O_NONBLOCK is meaningful mainly for FIFOs (named pipes, a filesystem object two processes use to talk to each other without a shared parent), sockets, and terminal/character devices. On a regular disk file it is effectively a no-op: POSIX regular-file reads and writes are defined to not return EAGAIN, so O_NONBLOCK does not make disk I/O asynchronous.

3. The race condition: open() then fcntl(), versus doing it atomically

A race condition is a bug where correctness depends on the relative timing of two things that can happen in either order. Here the bug shows up when you create an fd in blocking mode and only afterward call fcntl to flip on O_NONBLOCK, leaving a window between creation and that fcntl call during which the fd is still fully blocking.

The clearest version of this is accept(). A single-threaded reactor calling accept() then fcntl(connfd, F_SETFL, ... | O_NONBLOCK) looks safe, but the moment you hand connfd off to a worker thread or another event loop for load balancing before that fcntl call runs, that worker can issue a blocking read() on what it assumes is a non-blocking socket and stall a thread that was supposed to never block. The two operations (accept, then set-nonblocking) are not atomic, so anything with access to the fd in between can observe the pre-fcntl (blocking) state.

An even harder version of the same race exists on FIFOs, and there fcntl-after-open cannot fix it at all: opening a FIFO for reading in blocking mode does not even return until a writer opens the other end. If your intent was 'open it non-blocking so I don't stall waiting for a peer,' calling fcntl after open() is too late, because open() itself is the thing that blocks.

How to avoid it: request the flag atomically at creation, using the Linux syscalls built for exactly this

• open(2) already supports this directly: pass O_NONBLOCK in the flags argument, no follow-up fcntl needed (shown above).
• socket(2) has a Linux extension: socket(AF_INET, SOCK_STREAM | SOCK_NONBLOCK, 0) creates the socket already non-blocking.
• accept4(2) is the fix for the accept() race specifically: accept4(listenfd, addr, addrlen, SOCK_NONBLOCK) sets the flag on the same syscall that creates the connected fd, so there is no window where another thread can see it blocking.
• pipe2(2) does the same for pipes: pipe2(fds, O_NONBLOCK) instead of pipe() followed by two fcntl calls.

(These *2/*4 variants need _GNU_SOURCE or _DEFAULT_SOURCE defined before the includes on glibc, since they're Linux-specific, not POSIX. This is the same atomicity motivation as O_CLOEXEC/SOCK_CLOEXEC, which close an analogous race where a forked child could inherit and leak an fd into an exec'd program before the parent got a chance to mark it close-on-exec.)

4. Bugs from mishandling EAGAIN/EWOULDBLOCK

Once an fd is genuinely non-blocking, a read/write/accept call that would otherwise have to wait instead returns -1 immediately and sets errno to EAGAIN (or EWOULDBLOCK: on Linux the two are numerically identical, errno 11, but POSIX does not guarantee that on every platform, so portable code checks errno == EAGAIN || errno == EWOULDBLOCK). This is not a failure, it is the kernel's way of saying 'no data/space right now, try again later.' The bugs cluster around three mistakes:

• Treating EAGAIN as a real error. Logging it as a failure, tearing down the connection, or retrying in a tight spin loop instead of going back to the event loop and waiting for the next readiness notification.
• Not looping until EAGAIN with edge-triggered epoll (EPOLLET). epoll is Linux's readiness-notification API for watching many fds at once; it can run in level-triggered mode (EPOLLLT, the default: it keeps telling you 'still readable' every time you ask, as long as data remains) or edge-triggered mode (EPOLLET: it tells you exactly once, at the moment the fd transitions from not-ready to ready). With EPOLLET, if you read one chunk, see there's more, and just move on to the next fd in your event loop 'to be fair,' the socket buffer still has bytes in it but no new edge is coming until more data arrives from the peer, so you never get woken up again for that leftover data. The connection appears to silently stall. The only correct pattern under EPOLLET is: read (or write) in a loop until the call returns -1/EAGAIN, and only then go back to epoll_wait.
• Forgetting to unsubscribe from EPOLLOUT once a deferred write drains. When a write() returns a short count or EAGAIN because the send buffer is full, the standard pattern is to buffer the remainder and register interest in EPOLLOUT so you're notified when there's room again. The corresponding bug is forgetting to remove EPOLLOUT interest once the buffered data has fully drained: the socket is writable almost all the time, so epoll_wait keeps returning immediately for that fd on every iteration, burning CPU in what looks like a busy loop with no work actually happening.
• Confusing EINTR with EAGAIN. EINTR (call interrupted by a signal before any I/O happened) is a different case that also needs a retry, but retrying it correctly means simply re-issuing the same call, not re-arming epoll interest or treating it as 'no data.' Code that lumps EINTR into the same branch as EAGAIN either busy-loops or misses real data.

Checklist for a strace/code review of this pattern: confirm the non-blocking flag was set atomically at creation (or read-modify-write via fcntl if not); confirm every read/write path checks for EAGAIN/EWOULDBLOCK and treats it as 'go back to the event loop,' not an error; confirm any EPOLLET consumer loops to EAGAIN on every readiness event; confirm EPOLLOUT interest is added when a write is short and removed once the buffer is empty.

select(2), poll(2), and epoll(7) all answer the same question ("which of these file descriptors are ready for I/O right now"), but they differ in exactly the dimension that matters at scale: how the cost of asking scales with the number of watched fds versus the number of ready fds.

1. Complexity and the fd_set limit

select() represents the watched set as a fixed-size bitmap (fd_set), sized by FD_SETSIZE, typically 1024 on Linux/glibc. That's a hard ceiling on the numeric value of any fd you can watch, not just a count limit: if your process happens to have an fd numbered 1025 open (easy to hit with enough concurrent connections over the process's lifetime), you literally cannot pass it to select(), regardless of how few fds are active at once. Every call is O(n) in the number of watched fds: the kernel scans the whole bitmap, and userspace has to rebuild the fd_set before every call since select() mutates it in place. poll() replaces the bitmap with an array of struct pollfd, removing the FD_SETSIZE ceiling, but is still O(n) per call: the kernel scans the whole array, and the whole array is copied user-to-kernel and kernel-to-user on every single call regardless of how many fds are actually ready. Neither scales to watching tens of thousands of mostly-idle connections, since both pay for the full watched set on every call. epoll() splits registration from waiting: epoll_ctl() registers a fd with a persistent, kernel-side interest list once, and epoll_wait() returns only the fds that are actually ready, an O(1)-amortized cost per ready event rather than O(n) over the whole watched set.

2. Kernel/user interactions

select/poll are stateless from the kernel's perspective: each call is a fresh, complete description of what to watch, so the kernel has to walk that whole description every time, and the copy-in/copy-out of the (possibly large) fd list happens on every call too. epoll keeps that state inside the kernel across calls, in the epoll instance created by epoll_create1(); epoll_ctl(EPOLL_CTL_ADD/MOD/DEL, ...) mutates that persistent interest list, and epoll_wait() just asks "what's ready," with no need to re-describe the whole set. This is the mechanical reason epoll's cost tracks the number of events, not the number of watched fds.

3. Pitfalls implementing an event loop, including edge-triggered gotchas

epoll has two readiness-reporting modes. Level-triggered (LT, the default) behaves like select/poll: as long as data remains, epoll keeps reporting the fd as ready on every epoll_wait() call. Edge-triggered (ET, EPOLLET) reports a fd exactly once, at the moment it transitions from not-ready to ready. The classic ET bug: if a handler reads one chunk from a socket, sees there's more data buffered, but moves on to the next fd "to be fair" instead of draining fully, the socket buffer still has bytes in it, but no new edge (transition) will occur until more data arrives from the peer, so that handler is never woken for the leftover bytes and the connection silently stalls. The only correct pattern under ET is: set the fd non-blocking, and loop read()/write() until the call returns -1/EAGAIN, then go back to epoll_wait(). LT mode avoids that specific class of bug by re-notifying every time, at the cost of the notification overhead LT and select/poll share.

4. Which to choose for tens of thousands of concurrent connections

epoll, and it isn't close. For a service holding, say, 50,000 concurrent connections where only a few hundred are active in any given instant, select/poll pay for scanning and copying all 50,000 watched descriptors on every call, while epoll_wait() only returns and processes the few hundred that are actually ready, roughly two orders of magnitude less work per call at that scale, on the same basis (cost per readiness-check call). The only reasons to still reach for poll() are portability (epoll is Linux-specific) or a genuinely small, low-fd-count workload where the difference doesn't matter.

5. The same pattern one layer up: Python's asyncio

Python's asyncio event loop is a good example of this exact reasoning showing up above the raw syscall layer. On Linux, the selectors module asyncio builds on prefers epoll (selectors.EpollSelector) over select/poll for precisely the C10K-style reasoning above (the classic C10K problem: how to serve ten thousand simultaneous connections without the per-connection cost exploding): a single-threaded event loop juggling thousands of open sockets needs cost proportional to ready connections, not watched ones. This is also why asyncio uses one (or a small, fixed number of) OS thread(s) running an epoll-based reactor rather than a thread-per-connection model: each OS thread carries real overhead (a default stack reservation, scheduler and context-switch cost), so thousands of threads for thousands of mostly-idle connections would reproduce the same scaling problem epoll was built to avoid, just moved from "syscall cost" to "thread/scheduling cost." As for the edge-triggered pitfall: asyncio's default selector-based loop runs epoll in level-triggered mode specifically so individual callback authors don't have to implement the drain-to-EAGAIN discipline themselves; the trade-off is the LT overhead described above. That pitfall doesn't disappear from the ecosystem, though, it moves to wherever a lower layer does use edge-triggered mode directly, e.g. a C extension or an alternative event-loop implementation (such as one built on libuv) that embeds its own edge-triggered epoll internals: any code integrated at that layer still has to fully drain a readable fd on each wakeup, or it will reproduce the exact same "stalled connection" bug described above, just one level of abstraction further from the raw syscall.

All three exist to move data between two file descriptors without the kernel having to hand that data to userspace in between, but they differ in which fd types they accept and how much they actually save.

sendfile(2)

Copies data from one fd directly to another entirely inside the kernel, with the classic and still most common use being file-to-socket: serving a static asset from disk out over a TCP connection. The application never sees the bytes, which removes the read-into-userspace-buffer-then-write-to-socket round trip and the associated copies. Historically the Linux contract was out_fd must be a socket and in_fd must be a regular (or otherwise mmap-able) file; that is still the mental model to reach for even though later kernels loosened some restrictions. It is not universally "zero-copy" in the strictest sense: for a network destination the kernel still moves the data from the page cache into the socket buffer, just without the extra userspace hop, though with scatter-gather-capable NICs that final hop can be a genuine hardware-level zero-copy.

splice(2)

More general: moves data between two file descriptors where at least ONE of them must be a pipe. This is why splice can do things sendfile cannot, like proxying between two sockets, which requires two splice calls chained through an intermediate pipe (socket to pipe, then pipe to the other socket), since splice cannot go directly socket-to-socket. SPLICE_F_NONBLOCK and other flags control blocking behavior per call. sendfile is effectively implementable in terms of splice through an internal pipe, which is part of why splice is the more general primitive.

tee(2)

Duplicates data between two pipes WITHOUT consuming it from the source: the bytes remain available to be read again from the source pipe by whatever would otherwise have read them, while a copy is also made available on the second pipe. The typical use is tapping a stream to send it to two consumers without draining it from the first, for example duplicating a live data feed so a monitoring/logging path can see the same bytes a primary consumer is processing. Despite the shared name, this is a different mechanism from the interactive shell's tee command, which reads from stdin and writes to both stdout and a file using ordinary read/write, not necessarily tee(2) underneath, though the intent (duplicate a stream without consuming it) is the same idea.

Limitations and when the complexity is worth it

All three lose the ability to inspect or transform the bytes in flight: if you need to do anything to the data as it passes through (TLS encryption/decryption, compression, application-level framing, content rewriting), the kernel cannot run your logic on bytes it never hands to you, so these primitives only help for pure pass-through transfers. Fd-type restrictions matter in practice too: a TLS-terminated connection historically could not be the target of sendfile/splice without kernel TLS (kTLS) support doing the encryption in-kernel, since otherwise the plaintext has to pass through userspace for the TLS layer to touch it.

The complexity is worth paying for in I/O-bound services moving large volumes of pass-through data: static asset/file servers, reverse proxies, and log/data shippers where CPU time and memory bandwidth spent on userspace copies is a measurable fraction of total cost. It is not worth it for small responses (well under whatever the constant per-syscall/setup overhead is for the extra plumbing, such as splice's intermediate pipe) or anywhere the data must be transformed in-process anyway.

Monitoring signals that tell you it paid off

Compare like-for-like before/after on identical hardware and load: system CPU time (sys%) per GB transferred should drop measurably, since the whole point is fewer userspace copies; context-switch rate (vmstat cs) and memory-bandwidth/cache-miss counters should also drop. Track p50/p99 latency and throughput under realistic concurrency rather than a single-request microbenchmark, since syscall-plumbing changes like an intermediate pipe for splice can show a net loss for small transfers even while showing a clear win for large ones; a canary rollout comparing CPU utilization at matched request rates is the credible signal, not a synthetic single-shot benchmark claiming the copy was eliminated.