System Calls & the Kernel Interface Questions
The boundary between user space and the kernel: how programs request privileged services through system calls, the user/kernel mode transition, and the semantics of core POSIX calls such as fork, exec, wait, open, read, write, and stat. Covers syscall numbers, arguments, return values and errno, and how libc wrappers relate to the underlying trap. This is the foundational interface for all systems programming on Unix/Linux.
Write C code or pseudocode for a function that copies data from one file descriptor to another robustly. It must correctly handle partial reads, partial writes, EINTR, and end-of-file without corrupting the destination.
Sample Answer
A robust fd-to-fd copy has to get three things right that a naive read()/write() pair does not: a read() can return fewer bytes than requested even before EOF (a "partial read", common on pipes, sockets, and slow devices), a write() can likewise write fewer bytes than you asked for (a "partial write", common once the kernel's internal buffers are under pressure), and either call can be interrupted by a signal and return -1 with errno == EINTR even though nothing actually went wrong, in which case the correct response is simply to retry, not to treat it as a real error.
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <errno.h>
#include <string.h>
#include <fcntl.h>
/* Copies all bytes from fd_in to fd_out. Returns 0 on success, -1 on error (errno set). */
int copy_fd(int fd_in, int fd_out) {
char buf[8192];
for (;;) {
ssize_t n_read = read(fd_in, buf, sizeof(buf));
if (n_read < 0) {
if (errno == EINTR) continue; /* interrupted before any data was read: retry */
return -1; /* real error, errno already set by read() */
}
if (n_read == 0) break; /* EOF: nothing left to copy */
ssize_t total_written = 0;
while (total_written < n_read) {
ssize_t n_written = write(fd_out, buf + total_written, n_read - total_written);
if (n_written < 0) {
if (errno == EINTR) continue; /* retry the same write, no data lost */
return -1;
}
total_written += n_written; /* advance past a short/partial write */
}
}
return 0;
}
int main(void) {
int fd_in = open("src.txt", O_RDONLY);
if (fd_in < 0) { perror("open src.txt"); return 1; }
int fd_out = open("dst.txt", O_WRONLY | O_CREAT | O_TRUNC, 0644);
if (fd_out < 0) { perror("open dst.txt"); close(fd_in); return 1; }
if (copy_fd(fd_in, fd_out) < 0) {
perror("copy_fd");
close(fd_in);
close(fd_out);
return 1;
}
close(fd_in);
if (close(fd_out) < 0) { perror("close dst.txt"); return 1; }
printf("copy complete\n");
return 0;
}
Run it from an empty directory:
$ printf 'alpha\nbeta\ngamma\n' > src.txt
$ gcc -Wall -Wextra -o copy_fd copy_fd.c
$ ./copy_fd
copy complete
$ diff src.txt dst.txt && echo IDENTICAL
IDENTICAL
(Verified: this exact source compiles cleanly and produces this exact output on Linux, gcc 13.)
Why each piece is necessary, not defensive-programming theater
- The inner
whileloop onwrite()is not optional: a singlewrite()call is only guaranteed to accept as much as it accepts, and silently ignoring the return value (or treating it as all-or-nothing) is the single most common source of silent data corruption in copy routines, because it manifests as "usually works" until the destination is a pipe, a socket, or the source system is under memory pressure. - Retrying on EINTR must retry the SAME write with the SAME remaining data (
buf + total_written,n_read - total_written), not restart the whole buffer, or bytes get duplicated. - EOF is
read()returning exactly 0, not a negative return; conflating "0 bytes read" with an error is a common off-by-one in these loops. - Closing
fd_outand checking its return value matters because a buffered filesystem's actual write errors (out of disk space, quota) can surface onclose()rather than on the precedingwrite()calls, depending on the filesystem and any write-back caching involved; checking only the write() calls and ignoring close()'s return value can mask a failed copy that looks like it succeeded.
Walk through what happens at the OS level when a program calls a system call such as read() and it returns -1 with EINTR. How should robust systems code handle this, and why does it matter for long-running monitoring or automation software?
Sample Answer
What happens at the OS level
read() is a blocking syscall: when there's nothing to read yet (say, a socket with no data pending), the calling thread is put into an interruptible sleep state and taken off the CPU entirely, the kernel doesn't spin waiting. Two things can wake it back up: the condition it's waiting for becomes true (data arrives), or a signal is delivered to the process while it's asleep.
When a signal arrives mid-read(), the kernel has to unwind the syscall before it can run the signal handler in user space (a signal handler is itself just user-mode code, so control has to return to ring 3 first). At that point the kernel has a choice, controlled by whether SA_RESTART was set when the signal handler was installed via sigaction(2):
- Without
SA_RESTART(the default): the kernel abandons the syscall, delivers the signal (the handler runs), and when the handler returns, control resumes in your program right after theread()call, withread()having returned-1anderrnoset toEINTR. Internally, before the signal is even delivered, the kernel marks the syscall's exit status with an internal restart code (ERESTARTSYS), which only turns into a real, user-visibleEINTRonce the kernel confirmsSA_RESTARTis not set for that handler; if you strace this, you can actually see the internal marker before it resolves. - With
SA_RESTART: the kernel transparently re-issues the sameread()call after the handler returns, so your code never observes the interruption at all for that call.
A real transcript
I built a small program where a child process blocks in read() on an empty pipe, installs a SIGUSR1 handler without SA_RESTART, and the parent sends SIGUSR1 before any data exists, then writes the real payload a moment later. Running it under strace -f (the child is pid 5086, fd 3 is the pipe's read end):
5086 rt_sigaction(SIGUSR1, {sa_handler=0x..., sa_mask=[], sa_flags=0}, NULL, 8) = 0
5086 read(3, <unfinished ...>
5085 kill(5086, SIGUSR1) = 0
5086 <... read resumed>0x..., 63) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
5086 --- SIGUSR1 {si_signo=SIGUSR1, si_code=SI_USER, si_pid=5085, si_uid=0} ---
5086 write(2, "[child] read() interrupted, errn"..., 50) = 50
5086 read(3, <unfinished ...>
5085 write(4, "hello", 5) = 5
5086 <... read resumed>"hello", 63) = 5
(MEASURED: this is a real strace -f transcript from a Debian 12 container; the leading numbers are thread/process IDs, and I trimmed the dynamic-linker startup noise for readability.) You can see the exact sequence: read() blocks, kill() delivers SIGUSR1, the kernel's internal ERESTARTSYS marker resolves to a genuine interruption because there's no SA_RESTART, the child's own log line confirms it observed errno == EINTR, and it calls read() again, which this time succeeds once real data ("hello") is written.
How robust code should handle it
The standard, correct pattern is a retry loop that checks specifically for EINTR and nothing else:
ssize_t n;
do {
n = read(fd, buf, sizeof(buf));
} while (n < 0 && errno == EINTR);
if (n < 0) {
/* a real error: EACCES, EBADF, ECONNRESET, etc. */
perror("read");
}
The loop must check errno specifically, not just "retry on any negative return", because a genuine error like EACCES (permission denied) or ECONNRESET (peer reset the connection) also returns -1 and blindly retrying those would spin forever or mask a real failure that needs to be surfaced.
Why it matters for long-running monitoring/automation software
A daemon that watches log files, tails a socket, or shells out to helper tools typically installs signal handlers for its own lifecycle management, SIGHUP to reload config, SIGCHLD to reap children, SIGTERM to shut down gracefully. Every one of those handlers, once installed, makes every blocking syscall anywhere else in that process a candidate for EINTR, even in code paths that have nothing to do with the signal's purpose. A monitoring agent that doesn't handle EINTR on its core read() loop will intermittently, and non-reproducibly under normal testing, crash or drop data the moment its own reload/reap machinery fires in production, exactly the kind of "works in dev, flaky in prod" bug that's hard to trace back to its cause without knowing this mechanism.
How can mprotect(), mlock(), and page permissions be used to reduce the exposure of secrets in memory? What are the limitations of these primitives, especially after a fork() or during a crash?
Sample Answer
mprotect() and mlock() are page-table-level and memory-residency controls respectively, and both can meaningfully shrink the window during which a secret sitting in memory is exposed to an attacker, but neither is a substitute for actually protecting the secret's content (encryption, zeroing), and both have real gaps around fork() and crashes.
1. Using mprotect(), mlock(), and page permissions to reduce exposure
mprotect(addr, len, prot) changes the permission bits (PROT_READ, PROT_WRITE, PROT_NONE) on an already-mapped range of pages. The useful pattern for a secret buffer: allocate it via mmap() (not a stack variable or a heap chunk sitting inside a larger arena, since you can only mprotect whole pages), mark it PROT_NONE whenever it's not actively in use, and flip it to PROT_READ (or PROT_READ|PROT_WRITE while writing the secret in) only for the specific window it's needed. Any bug elsewhere in the process (a wild pointer write, an accidental log statement over that buffer's range) that tries to touch the buffer while it's PROT_NONE faults immediately instead of silently succeeding, which both prevents accidental exposure and turns a whole class of memory-corruption bugs into an instantly visible crash rather than a silent leak. Guard pages (a PROT_NONE page placed immediately before and after the secret buffer) catch adjacent buffer overflows the same way.
mlock()/mlockall() pin the pages in physical RAM so the kernel will never write them to swap. This matters because swap is ordinary, usually unencrypted disk storage; without mlock(), a secret sitting in memory can be paged out under memory pressure and then persist on disk long after the process exits, recoverable later by anyone with access to that disk (including, notably, a forensic examiner imaging the drive after the fact, which is exactly the scenario this control is meant to prevent). Production use of mlock() typically needs either CAP_IPC_LOCK or a raised RLIMIT_MEMLOCK (the default is often small, on the order of 64 KiB, specifically to stop unprivileged processes from locking down large amounts of RAM), so this needs to be provisioned deliberately, not assumed to just work.
2. Limitations after fork()
fork() gives the child a copy-on-write view of the entire parent address space, which by default includes the secret buffer, whatever its mprotect() permission bits say. Those page-table bits themselves are also copied to the child, so a PROT_NONE buffer stays PROT_NONE in the child, which helps, but the bigger issue is the narrow window between fork() and any subsequent execve() in the child: until the exec actually happens, the secret is still mapped into the child's address space, and if the child crashes, is ptraced (attached to with the process-tracing syscall), or is inspected via /proc/<pid>/mem during that window, the secret is exposed there too, independent of what the parent's protection bits say. The fix is madvise(addr, len, MADV_DONTFORK) on the secret region before forking: this excludes that specific range from the child's copy of the address space entirely, so a child that doesn't need the secret never has it mapped in the first place, closing the exposure window instead of just narrowing it.
3. Limitations during a crash
A crash (SIGSEGV, SIGABRT, etc.) by default triggers a core dump, and the kernel's core-dump writer bypasses mprotect() permission bits entirely: it dumps the raw contents of every mapped page it's configured to include, PROT_NONE or not, so a secret you carefully kept unreadable at runtime can still end up written out in plaintext inside the core file on disk the moment the process crashes. mprotect(PROT_NONE) and mlock() are runtime access-control and residency controls; they say nothing about what a full memory dump captures. The mitigation is madvise(addr, len, MADV_DONTDUMP) on the secret region, which explicitly excludes those pages from any subsequent core dump regardless of the process-wide /proc/[pid]/coredump_filter setting; for a belt-and-suspenders approach, disabling core dumps for the process entirely via setrlimit(RLIMIT_CORE, {0,0}) or prctl(PR_SET_DUMPABLE, 0) removes the risk outright, though PR_SET_DUMPABLE(0) also blocks ptrace() attachment to the process, which is usually the point but does complicate live debugging.
4. What these primitives don't do
Worth being explicit about: none of this encrypts or zeroes the secret. mprotect(PROT_NONE) is purely a page-table permission check enforced by the CPU's MMU (memory management unit); it does not stop a kernel-level compromise, a DMA-capable attack (DMA: Direct Memory Access, where a peripheral device reads/writes RAM directly, bypassing the CPU's normal page-permission checks entirely), or anyone with CAP_SYS_PTRACE/root reading /proc/<pid>/mem directly, since that path goes around normal userspace protection checks. And mlock() only guarantees the page won't be paged out under normal swap pressure; depending on how a given system implements hibernate (suspend-to-disk), a locked page can still end up written to the hibernation image on disk. Treat mprotect/mlock/MADV_DONTFORK/MADV_DONTDUMP as reducing the exposure window and the number of paths that leak the secret, not as a guarantee that it never touches disk or is never readable by a sufficiently privileged attacker; actually sensitive material (private keys, etc.) still wants explicit zeroing after use and, where the threat model calls for it, encryption at rest even in memory.
Compare execve(), execl(), execvp(), and posix_spawn(). When would you choose one over another in a security-sensitive application that launches helper programs or scripts?
Sample Answer
All four end up calling the same underlying syscall, execve(2), the ONLY true exec syscall on Linux (along with its sibling execveat); execl, execv, execlp, execvp, and posix_spawn are all libc-level conveniences built on top of it. What actually differs between them is what you're forced to specify explicitly versus what gets inherited or looked up implicitly, and in a security-sensitive launcher, that difference is the whole ballgame.
execve(2): the raw syscall, and the only one with fully explicit environment
execve(path, argv, envp) takes an explicit pathname (no PATH search), an explicit argv array, and, critically, an explicit envp array that you build yourself rather than inheriting. That last part is the one that matters most for security: environment variables are a well-known injection/tampering vector (LD_PRELOAD and LD_LIBRARY_PATH can redirect what code actually runs inside a "trusted" binary, PATH affects anything that does its own internal lookups, IFS and other shell-interpreted variables matter if the target is itself a shell script). execve() lets you hand the child a deliberately minimal, allowlisted environment instead of whatever the calling process happened to have accumulated. Most explicit, most auditable, the right default for anything security-sensitive.
execl(3) / execlp(3): convenient, but implicit environment (and implicit PATH for the p variant)
execl(path, arg0, arg1, ..., NULL) is a variadic wrapper, nice when you have a small, fixed, compile-time-known argument list. But it has no envp parameter at all, it implicitly uses the CALLER's current environ, exactly the property a hardened launcher does not want, since it silently passes through whatever the parent's environment contains, including anything an earlier, less-trusted step in the process's life might have set. execlp() adds a second problem on top: it does a PATH search to resolve a bare filename, meaning it trusts the (also inherited, also potentially attacker-influenceable) PATH variable to find the right binary. This is the single riskiest behavior in the whole exec family for a security tool: if PATH has been tampered with, or even just misconfigured to include a writable, earlier-searched directory, execlp("some-tool", ...) can execute an attacker-planted binary instead of the one you meant, a classic confused-deputy vulnerability.
execvp(3): the array-argument sibling of execlp, same PATH-search risk
Same implicit-environment-inheritance and PATH-search behavior as execlp, just with an argv array instead of a variadic list, more convenient when the argument list is built at runtime (a shell implementing a pipeline, general CLI tooling that legitimately wants to honor the user's PATH). Fine for trusted, interactive contexts; wrong for launching a helper on behalf of untrusted input, or in any privilege-sensitive code path, for the same PATH-hijack reason as execlp. The concrete rule: pass an absolute path, never a bare command name, whenever the caller of your launcher isn't fully trusted.
posix_spawn(3): a higher-level, structured API that's a strong default for hardened launchers
posix_spawn() is a POSIX API that, on Linux glibc, is implemented internally via clone(CLONE_VM | CLONE_VFORK, ...) + execve() for the common case (falling back to a plain fork+exec when the requested spawn attributes require it), getting close to vfork()-level speed without exposing your own application code to vfork()'s "don't touch shared memory" footgun, since the library, not your code, does the small amount of work in that narrow window. Two features make it a particularly good fit for a hardened launcher specifically: posix_spawn_file_actions_t lets you specify fd close/dup/open operations that happen IN THE CHILD before the exec, atomically as part of the spawn call, so you can express "close every fd except stdin/stdout/stderr" or "redirect stdout to this specific log fd" without a manual fork-then-fixup window where a signal or a race could leave things in an inconsistent state; and posix_spawnattr_t lets you set process attributes (signal mask, process group, and on Linux, POSIX_SPAWN_SETSID for detaching a session) as part of the same atomic call, rather than as separate steps after fork that a bug could skip.
The recommendation, concretely
For a security-sensitive application launching helper programs or scripts: use execve() directly with an explicit, minimal envp you construct (an allowlist of exactly the variables the helper needs, nothing inherited by default), or equivalently posix_spawn() with the same explicit envp plus posix_spawn_file_actions_t for fd hygiene, which gets you the same guarantees with a friendlier, more atomic API and slightly better performance for the common case. Never use execlp/execvp with a bare (non-absolute) command name when the input isn't fully trusted, since that hands PATH resolution, and therefore "which binary actually runs", to an environment variable you don't fully control. And regardless of which call you use, always pass an ABSOLUTE path to the target binary in a security context, don't rely on any implicit search at all.
During an incident, you notice a suspicious process tree spawning rapidly and executing different binaries. How would you use your understanding of fork(), exec(), and process parent-child relationships to distinguish normal automation from a fork bomb or malware loader?
Sample Answer
The investigation comes down to reading the SHAPE of the process tree and the IDENTITY of what's actually being executed at each hop, then comparing both against what's normal for this host.
What a fork bomb looks like
A fork bomb is defined by rate and self-similarity, not by what any individual process does. Symptoms: an extremely high, near-instantaneous rate of new PIDs, potentially thousands per second; the SAME short-lived binary re-forking itself recursively (the parent and the child are typically running the identical program, sometimes literally the same shell one-liner re-invoking itself); and system-wide symptoms that show up FAST, fork() starting to fail everywhere on the host with EAGAIN/ENOMEM because the global PID space or resource limits are being exhausted, CPU pegged near 100% from pure scheduling/context-switch overhead even though no individual process is doing meaningful work (no real file I/O, no network activity, nothing but forking). Confirming this: ps -eo pid,ppid,etimes,cmd --sort=start_time | tail -50 (or pstree -p) shows a large cluster of processes all born within the same second or two, almost all running the same command, with a shallow but extremely WIDE tree (one ancestor with an enormous number of direct or near-direct descendants), rather than a deep chain.
What a malware loader / process-chaining attack looks like
The opposite shape: a NARROW, sequential chain, parent -> child -> grandchild, where each hop exec()s into a DIFFERENT binary than the one before it ("process chaining," commonly discussed as part of "living off the land" tradecraft). The telltale signal is fork() immediately followed by execve() into an unrelated program, especially when: the target binary lives in a WRITABLE, non-standard location (/tmp, /dev/shm, a user's Downloads directory) rather than a normal system binary path; or it's a legitimate, trusted system binary being invoked with unusual, scripted-looking arguments that a human wouldn't type interactively (a base64-encoded inline script, a curl/wget piped directly into a shell interpreter, a download-then-exec two-step); or a process that has no legitimate reason to spawn a shell at all suddenly does, a web server worker process forking sh -c "...", or a document viewer (PDF reader, Office app) spawning a script interpreter, is a strong standalone signal on its own, since normal automation for those specific parent processes essentially never legitimately does that.
What normal automation looks like, as the baseline to compare against
A stable, REPEATING, predictable tree shape: a cron job or systemd timer forking the same handful of children on a fixed schedule; a CI runner spawning build-tool subprocesses in a consistent pattern run after run. Children exec well-known binaries from standard system paths (/usr/bin, /usr/local/bin, not a temp directory). The fork rate is moderate and clearly correlated with an actual trigger (a cron entry firing, a webhook arriving), not continuous, unbounded self-replication. And critically, the specific argv patterns and binary paths involved are ones you'd expect to have SEEN BEFORE on this host if you have any historical baseline (fleet-wide EDR telemetry, prior audit logs), whereas malicious activity commonly shows a first-time-ever binary path, argv combination, or parent/child pairing for that specific host.
Concrete investigative steps
pstree -p <suspect_root_pid>first, to see the actual tree SHAPE at a glance: wide-and-shallow (fork bomb candidate) versus narrow-and-deep with binary changes at each hop (loader candidate) versus a familiar, repeating pattern (probably fine).ps -eo pid,ppid,lstart,etime,cmdfor exact start times and full command lines, WHEN things started (clustered instantaneously, or spread out on a schedule) and WHAT was actually typed as arguments (a base64 blob, a download URL, a-enc/-EncodedCommand-style obfuscation flag are all strong indicators).- Don't trust the process name alone:
argv[0](what shows up as the "command" in a casualps) can be rewritten by the process itself to look innocuous. Cross-check/proc/<pid>/exe(a symlink that resolves to the ACTUAL binary inode being executed, harder to spoof after the fact than an in-process argv rewrite) and/proc/<pid>/cmdline(the real argv the kernel recorded at exec time) against whatpsdisplayed; a mismatch between a friendly-looking process name and a suspicious/proc/<pid>/exetarget is itself a finding. - Check
/proc/<pid>/statusforPPID(confirm the actual parentage rather than trusting a displayed process name) and cross-reference againstwho/last/authentication logs: does this chain trace back to an interactive login session, a legitimate cron entry, or a network-facing service process, and if it's the latter, does that service have any legitimate reason to be spawning a shell or a script interpreter AT ALL? A network-facing daemon forking a shell is close to always worth escalating on its own. - Recognize the limits of reasoning from the CURRENT process tree alone:
fork()/execve()calls aren't logged anywhere by default, so if the malicious chain has already partially exited by the time you're looking, the tree you're inspecting is incomplete. This is exactly why Linux audit (auditctl -a exit,always -F arch=b64 -S execve) or an eBPF (a Linux kernel facility that lets small sandboxed monitoring programs run in-kernel, the basis for many modern tracing/EDR tools)-based EDR agent recording exec events needs to be enabled BEFORE an incident, not reached for during one; reconstructing a process-chaining attack cleanly from an execve audit trail is straightforward, reconstructing it purely frompssnapshots and whatever remnants are still running when you happen to look is much harder and can miss steps that already completed and exited.
Unlock Full Question Bank
Get access to all 30 System Calls & the Kernel Interface interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.