System Calls & the Kernel Interface Questions

A ptrace-based tracer works by intercepting a target process at every syscall boundary and inspecting its registers and memory while it's stopped. The code below is a complete, minimal execve()-logging tracer that compiles and runs unchanged on Linux/aarch64 (verified in a clean Linux/aarch64 container, gcc 13): it traces the process it launches, and any time that process calls execve(), it prints the path and argv it's about to exec.

/* Minimal ptrace-based execve() tracer. Built and verified on Linux/aarch64.
 * On Linux/x86_64: regs.regs[8] (syscall number) becomes orig_rax, regs.regs[0]
 * (1st arg) becomes rdi, regs.regs[1] (2nd arg) becomes rsi; struct user_pt_regs +
 * PTRACE_GETREGSET becomes struct user_regs_struct + PTRACE_GETREGS. See
 * "AArch64 register layout and the x86_64 equivalent" below the code for why
 * each register holds what it holds; unlike the aarch64 path below, this
 * x86_64 mapping has not itself been compiled and run. */
#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <errno.h>
#include <sys/ptrace.h>
#include <sys/wait.h>
#include <sys/uio.h>
#include <asm/ptrace.h>   /* struct user_pt_regs */
#include <elf.h>

#define MAX_ARGS 16
#define MAX_ARGLEN 256
#define SYS_execve_arm64 221
#define NT_PRSTATUS_ 1

/* Read a NUL-terminated string out of the traced process's address space,
 * one machine word at a time via PTRACE_PEEKDATA. */
static char *read_traced_string(pid_t pid, unsigned long addr) {
    static char buf[MAX_ARGLEN];
    size_t i = 0;
    while (i < MAX_ARGLEN - sizeof(long)) {
        errno = 0;
        long word = ptrace(PTRACE_PEEKDATA, pid, addr + i, NULL);
        if (word == -1 && errno != 0) break;
        memcpy(buf + i, &word, sizeof(long));
        if (memchr(&word, 0, sizeof(long)) != NULL) break;
        i += sizeof(long);
    }
    buf[MAX_ARGLEN - 1] = '\0';
    return buf;
}

int main(int argc, char **argv) {
    if (argc < 2) {
        fprintf(stderr, "usage: %s <program> [args...]\n", argv[0]);
        return 1;
    }

    pid_t child = fork();
    if (child == -1) { perror("fork"); return 1; }

    if (child == 0) {
        /* Child asks to be traced, then execs the target. That exec
         * delivers the first SIGTRAP the parent's initial waitpid catches. */
        ptrace(PTRACE_TRACEME, 0, NULL, NULL);
        execvp(argv[1], &argv[1]);
        perror("execvp");
        _exit(1);
    }

    int status;
    waitpid(child, &status, 0);   /* catch the exec-triggered SIGTRAP */
    ptrace(PTRACE_SETOPTIONS, child, 0, PTRACE_O_TRACESYSGOOD);

    int at_syscall_entry = 1;
    while (1) {
        if (ptrace(PTRACE_SYSCALL, child, NULL, NULL) == -1) break;
        waitpid(child, &status, 0);

        if (WIFEXITED(status)) {
            printf("child exited with status %d\n", WEXITSTATUS(status));
            break;
        }
        /* PTRACE_O_TRACESYSGOOD (set above) ORs 0x80 into SIGTRAP specifically on
         * a syscall-stop, so this is how the tracer tells "stopped because of a
         * syscall" apart from "stopped because of an unrelated real SIGTRAP." */
        if (!WIFSTOPPED(status) || WSTOPSIG(status) != (SIGTRAP | 0x80)) {
            continue; /* a real signal, not a syscall-stop: skip in this minimal demo */
        }

        if (at_syscall_entry) {
            struct user_pt_regs regs;
            struct iovec iov = { .iov_base = &regs, .iov_len = sizeof(regs) };
            ptrace(PTRACE_GETREGSET, child, (void *)(long)NT_PRSTATUS_, &iov);
            if (regs.regs[8] == SYS_execve_arm64) {
                unsigned long path_addr = regs.regs[0];
                unsigned long argv_addr = regs.regs[1];
                printf("execve(\"%s\") args:", read_traced_string(child, path_addr));
                for (int i = 0; i < MAX_ARGS; i++) {
                    errno = 0;
                    long argp = ptrace(PTRACE_PEEKDATA, child,
                                        argv_addr + (unsigned long)i * sizeof(long), NULL);
                    if (argp == 0 || (argp == -1 && errno != 0)) break;
                    printf(" \"%s\"", read_traced_string(child, (unsigned long)argp));
                }
                printf("\n");
                fflush(stdout); /* flush before the traced child can write its own output */
            }
        }
        at_syscall_entry = !at_syscall_entry;
    }
    return 0;
}

Compiled with gcc -Wall -Wextra -o tracer tracer.c and run as ./tracer /bin/sh -c 'exec /bin/echo hello world', this prints, deterministically:

execve("/bin/echo") args: "/bin/echo" "hello" "world"
hello world
child exited with status 0

(The target here is a shell that itself execs /bin/echo, using the same process, so the tracer catches a genuine execve() made by the traced program. Plain /bin/echo would never trigger the output line, since echo itself never calls execve().)

AArch64 register layout, syscall-stop detection, and the x86_64 equivalent

The code above relies on two pieces of ABI (application binary interface) knowledge worth stating explicitly rather than leaving implicit in bare register indices.

Syscall-stop detection. ptrace(PTRACE_SETOPTIONS, child, 0, PTRACE_O_TRACESYSGOOD) tells the kernel to OR an extra bit (0x80) into the stop signal specifically when the tracee stops because it entered or exited a syscall, as opposed to stopping for an unrelated real signal (including a genuine SIGTRAP the tracee raised itself, e.g. from a breakpoint instruction). WSTOPSIG(status) != (SIGTRAP | 0x80) is exactly that check: without PTRACE_O_TRACESYSGOOD there is no reliable way to tell "this stop is a syscall boundary" apart from "this stop is something else," and the at_syscall_entry toggle would have nothing trustworthy to key off.

AArch64 syscall calling convention. On AArch64 Linux, a syscall's arguments are passed in registers x0 through x5, and the syscall number goes in x8. struct user_pt_regs (populated via PTRACE_GETREGSET with NT_PRSTATUS) exposes those as the regs[] array: regs.regs[8] is x8 (the syscall number, compared against SYS_execve_arm64), regs.regs[0] is x0 (the first argument, execve()'s path_addr), and regs.regs[1] is x1 (the second argument, execve()'s argv_addr). Nothing more exotic is happening: the code is reading the ABI-defined argument registers straight out of the register-set snapshot ptrace hands back.

The x86_64 equivalent, by name, not just position. On x86_64 Linux, the first two integer/pointer arguments to a syscall are rdi and rsi (the System V AMD64 calling convention's first two argument registers), so path_addr becomes regs.rdi and argv_addr becomes regs.rsi. The syscall number is not in rax: it's saved separately in orig_rax specifically because rax itself gets overwritten with the syscall's return value once the syscall actually runs, so by the time you observe the syscall-exit stop, rax no longer tells you which syscall just ran, only orig_rax still does. So the substitution the code comment calls for is precisely: regs.regs[8] becomes regs.orig_rax, regs.regs[0] becomes regs.rdi, regs.regs[1] becomes regs.rsi, read via PTRACE_GETREGS into a struct user_regs_struct (from <sys/user.h>) instead of PTRACE_GETREGSET/struct user_pt_regs, and the syscall-number constant becomes __NR_execve (59 on x86_64, from <sys/syscall.h>) instead of SYS_execve_arm64 (221). The SIGTRAP | 0x80 syscall-stop check and the rest of the event loop are architecture-independent and need no change.

This x86_64 mapping is standard, documented Linux/x86_64 ABI and ptrace behavior, not a guess, but unlike the aarch64 code above it has not itself been compiled and run for this answer: only the aarch64 path carries that verification. Treat it as a correct substitution to make and verify yourself, not as machine-checked output.

Attaching to an already-running process

The demo above uses fork() + PTRACE_TRACEME + execvp() to trace a freshly-launched program, which is convenient when the tracer gets to start the target. But the question specifically asks for attaching to a process that's already running, which is a different pair of primitives, and it is just as reproducible to demonstrate as the fork+TRACEME case: launch a target in the background first (exactly the situation an "attach to an existing PID" scenario is always in), note its PID, then attach to it. Below is a second, complete program that does exactly that, compiled and run the same way as the first, and also verified in a clean Linux/aarch64 container, gcc 13:

#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <errno.h>
#include <sys/ptrace.h>
#include <sys/wait.h>
#include <sys/uio.h>
#include <asm/ptrace.h>

#define MAX_ARGS 16
#define MAX_ARGLEN 256
#define SYS_execve_arm64 221

static char *read_traced_string(pid_t pid, unsigned long addr) {
    static char buf[MAX_ARGLEN];
    size_t i = 0;
    while (i < MAX_ARGLEN - sizeof(long)) {
        errno = 0;
        long word = ptrace(PTRACE_PEEKDATA, pid, addr + i, NULL);
        if (word == -1 && errno != 0) break;
        memcpy(buf + i, &word, sizeof(long));
        if (memchr(&word, 0, sizeof(long)) != NULL) break;
        i += sizeof(long);
    }
    buf[MAX_ARGLEN - 1] = '\0';
    return buf;
}

int main(int argc, char **argv) {
    if (argc < 2) { fprintf(stderr, "usage: %s <pid>\n", argv[0]); return 1; }
    pid_t target = (pid_t)atoi(argv[1]);

    /* PTRACE_ATTACH (or the gentler PTRACE_SEIZE, which doesn't stop the
     * target immediately) requires either matching real/effective UID
     * between tracer and tracee, or CAP_SYS_PTRACE, and is also subject to
     * the Yama LSM's ptrace_scope sysctl, which on many distributions
     * restricts attaching to only a process's own children by default. */
    if (ptrace(PTRACE_ATTACH, target, NULL, NULL) == -1) {
        perror("PTRACE_ATTACH");
        return 1;
    }
    int status;
    waitpid(target, &status, 0);   /* the ATTACH-induced stop: there is no
                                       exec-trap to synchronize on here,
                                       since the target was already running
                                       arbitrary code when we attached. */
    ptrace(PTRACE_SETOPTIONS, target, 0, PTRACE_O_TRACESYSGOOD);

    int at_syscall_entry = 1;
    while (1) {
        if (ptrace(PTRACE_SYSCALL, target, NULL, NULL) == -1) break;
        waitpid(target, &status, 0);
        if (WIFEXITED(status)) {
            printf("target exited with status %d\n", WEXITSTATUS(status));
            break;
        }
        if (!WIFSTOPPED(status) || WSTOPSIG(status) != (SIGTRAP | 0x80)) {
            continue;
        }
        if (at_syscall_entry) {
            struct user_pt_regs regs;
            struct iovec iov = { .iov_base = &regs, .iov_len = sizeof(regs) };
            ptrace(PTRACE_GETREGSET, target, (void *)(long)1 /* NT_PRSTATUS */, &iov);
            if (regs.regs[8] == SYS_execve_arm64) {
                unsigned long path_addr = regs.regs[0];
                unsigned long argv_addr = regs.regs[1];
                printf("execve(\"%s\") args:", read_traced_string(target, path_addr));
                for (int i = 0; i < MAX_ARGS; i++) {
                    errno = 0;
                    long argp = ptrace(PTRACE_PEEKDATA, target,
                                        argv_addr + (unsigned long)i * sizeof(long), NULL);
                    if (argp == 0 || (argp == -1 && errno != 0)) break;
                    printf(" \"%s\"", read_traced_string(target, (unsigned long)argp));
                }
                printf("\n");
                fflush(stdout);
            }
        }
        at_syscall_entry = !at_syscall_entry;
    }
    return 0;
}

Compiled with gcc -Wall -Wextra -o attach_tracer attach_tracer.c and run against an already-running background process (started with bash -c 'sleep 1; exec /bin/echo already-running-attach-worked' &, its PID passed as the argument), this prints, deterministically:

execve("/bin/echo") args: "/bin/echo" "already-running-attach-worked"
already-running-attach-worked
target exited with status 0

The mechanics: after PTRACE_ATTACH, the tracer waitpid()s for the attach-induced stop, sets PTRACE_O_TRACESYSGOOD exactly as before, then reuses an identical syscall-entry/exit loop; PTRACE_SYSCALL/PTRACE_GETREGSET/PTRACE_PEEKDATA don't care whether the tracee arrived via PTRACE_TRACEME or PTRACE_ATTACH, only the setup differs. In a real incident-response or security-tooling scenario the target PID is normally already known (from ps, a monitoring alert, or a specific process under investigation), so this attach-based path, not the fork+TRACEME one, is the code you'd actually reach for; fork+TRACEME is the right choice only when the tracer also controls how the target gets launched, the way strace prog does by default.

Performance and security limitations of ptrace in production

• Overhead. Every traced syscall costs (at minimum) two extra context switches between tracer and tracee (one stop at syscall-entry, one at syscall-exit) plus a waitpid() round trip in the tracer, on top of whatever PTRACE_PEEKDATA/PTRACE_GETREGSET calls the tracer makes to actually inspect the state. For a syscall-heavy workload, this is not a small tax, tracing every syscall of a busy process can slow it down by an order of magnitude or more, which is why ptrace-based tracing (unlike eBPF-based tracing) is a diagnostic/debugging tool, not something you'd leave attached to a production service continuously.
• Single tracer per process. Only one tracer can be attached to a given process at a time (via the standard ptrace API); a debugger and a tracing tool can't both attach simultaneously.
• Setuid/setgid exemption. The kernel refuses to let an unprivileged tracer ptrace()-attach to a process running a setuid or setgid binary: allowing that attach would let a tracer with only ordinary-user credentials read and rewrite the memory of a process running with privilege the tracer itself doesn't have, defeating the whole point of the setuid/setgid mechanism. This is a real limitation if the thing you need to trace legitimately runs with elevated privilege; you either need to be privileged yourself (typically root, or CAP_SYS_PTRACE) or you're locked out of tracing it this way.
• PTRACE_PEEKDATA's word-at-a-time cost. Reading a string out of the target's memory one machine word per syscall, as the demo above does, is simple to reason about but slow for large reads; process_vm_readv(2) (a newer, purpose-built syscall for cross-process memory reads) does the same job in far fewer syscalls and is the better choice for a tracer that needs to read substantial amounts of target memory, not just short argv strings.
• It's a debugging primitive, not a hardened sandbox boundary. A ptrace-based interception layer (e.g. denying/rewriting syscalls via PTRACE_SYSCALL plus SECCOMP_RET_TRACE) is significantly more fragile and slower than a seccomp-bpf filter for actually restricting what a process can do; ptrace is the right tool for observing and debugging, seccomp is the right tool for enforcing a policy, and conflating the two (using ptrace as your actual security boundary rather than as an inspection tool) is a common mistake.

Why this is a live attack surface, not a theoretical edge case

The vulnerable pattern is: enumerate a directory tree with readdir(), filter entries down to "regular files" using d_type and/or stat(), then separately open() and hash the same path string. Every gap between "decided this is safe to hash" and "actually opened it" is a TOCTOU (time-of-check to time-of-use) window, and in malware analysis the directory tree being walked is explicitly adversarial input: the sample itself, or anything an attacker controls in a shared analysis environment, can plant content specifically designed to defeat or redirect the scanner. This isn't a hardening nicety here, it's the expected threat model.

The specific tricks to defend against

• Symlink tricks: an entry named to look like an innocuous sample (report.pdf, invoice.exe) that is actually a symlink to /etc/shadow, a device file, or a named pipe designed to block the scanner forever (a denial-of-service against the tool itself), or a symlink to an enormous sparse file designed to exhaust disk or memory when "hashed."
• Renamed-path tricks: the entry readdir() reported gets deleted and a DIFFERENT object created under the same name before the tool acts on it, possible whenever the directory being scanned is live and writable, whether by the sample's own partial execution, by another process sharing the analysis filesystem, or by an operator re-triaging concurrently.
• Concurrent filesystem changes: directories scanned on a live system, as opposed to a static forensic image, can have entries added, removed, or replaced mid-walk; POSIX gives no ordering or consistency guarantee for readdir() under concurrent modification, so entries can be seen twice, missed entirely, or observed in an inconsistent state relative to each other.

A resistant design

• Walk using directory FILE DESCRIPTORS, not path strings. Open the top directory once with open(path, O_DIRECTORY); for every subdirectory encountered, descend via openat(parent_dirfd, name, O_DIRECTORY | O_NOFOLLOW) rather than concatenating a path string, so a symlink planted where a subdirectory was expected is rejected instead of followed.
• For each entry, don't trust d_type as authoritative (it can be DT_UNKNOWN, and even when populated it reflects a moment in time that has already passed by the time you act). Instead of a check-then-open sequence, openat(dirfd, name, O_RDONLY | O_NOFOLLOW) directly: if the entry is a symlink, this call fails with ELOOP (too many levels of symbolic links encountered, the same errno O_NOFOLLOW produces when it hits a symlink directly) and the entry can be flagged and skipped rather than silently redirected.
• fstat() the resulting fd, never fstatat/stat on the name a second time, and verify S_ISREG before reading a single byte, so the object being hashed is guaranteed to be the exact fd already held open, with zero remaining path re-resolution for anything to hijack.
• Compare (st_dev, st_ino) (device and inode, from the same fd's fstat()) against a set of already-visited identifiers to detect hardlink loops or the same file linked into the tree more than once, avoiding both infinite loops and double-counted results.
• Don't trust st_size blindly before reading: a pathological entry (a sparse file reporting a huge logical size, or a FIFO with a stale or meaningless size field) can turn a naive "read st_size bytes" into a resource-exhaustion or hang condition. Read in bounded chunks with an overall size and time budget, and use non-blocking I/O or a read timeout so a FIFO planted specifically to block forever can't stall the whole walk.
• Run the entire scan under explicit resource limits (a wall-clock timeout per file, RLIMIT_FSIZE (a per-process resource limit capping the largest file the process may write, to stop the scanner itself from being tricked into writing an unbounded amount of data), memory caps), treating the sample directory as hostile input for the whole duration of the walk, not just at the point of opening each file, since the scanner's own resource consumption is as much a part of the attack surface as the data it reads.

Launching a helper binary as a different user is a privilege-boundary crossing, and the review has to cover four surfaces, environment, file descriptors, path resolution, and the privilege-drop sequence itself, because a mistake in any one of them can turn "run this with less privilege" into either a leak from the parent or an escalation for the child. Concretely: say the agent, running as root, needs to launch /opt/agent/helper as the unprivileged service account svc_helper. Each section below works through that same scenario.

Environment variables

Review the FULL environment being handed to the child, not just assume a bare invocation is clean. Specifically: LD_PRELOAD/LD_LIBRARY_PATH can inject arbitrary shared objects into the child's address space if it's a dynamically linked binary and the loader honors them for that target; note that the dynamic linker specifically ignores these variables for a binary that is genuinely setuid/setgid on disk, as a kernel/glibc protection, but that protection does nothing if it's YOUR wrapper doing the privilege drop via setuid()/execve() rather than the target binary itself being marked setuid. PATH matters if the exec is by bare binary name rather than a fully-qualified path: an attacker-controlled or unexpectedly ordered PATH entry can substitute an entirely different binary. Any application-specific variables (config paths, feature flags, anything that might carry a secret or influence which code path the child trusts) need review for whether they should cross the boundary at all. The general fix: don't pass the caller's environment wholesale; construct an explicit, minimal, allowlisted environment and pass it via execve()'s envp argument directly, rather than relying on execvp/execlp variants that implicitly inherit the calling process's environ. Concretely: svc_helper's environment should be built from scratch, not forwarded from root, so nothing root-specific (an LD_PRELOAD set for a root-only debugging session, a PATH entry pointing at a root-writable directory) survives into the child.

File descriptors

Review every fd open in the parent for whether it should be marked close-on-exec via FD_CLOEXEC/O_CLOEXEC. A listening socket, an open log file, or a credential-bearing pipe left inheritable hands the child, which may be running with DIFFERENT privilege than the parent, a capability it was never meant to have; depending on the direction of the privilege change, an inherited fd can be a leak (child is less privileged, sees something it shouldn't) just as easily as an escalation (child somehow retains a handle a correctly-dropped process shouldn't have). Concretely for /opt/agent/helper: every socket and log-file descriptor the agent's root process holds needs to be marked FD_CLOEXEC before the fork/exec, so svc_helper's process doesn't inherit a handle the root identity opened.

Path-resolution rules

Use an ABSOLUTE, fully-qualified path to the helper binary, never a bare name resolved via PATH search, for the reason above. Verify that the binary's own file and every parent directory in its path up to the root are not writable by the target (lower-privileged) user, or by whichever user the calling process itself runs as if that's not fully trusted, since a writable directory anywhere in the resolution chain lets an attacker substitute the binary, or swap a directory component with a symlink, before the exec actually runs. If the path is built from any user input or a relative path at all, resolve it via openat() anchored on an already-open directory descriptor with O_NOFOLLOW, rather than trusting a raw, re-resolvable path string. Concretely: exec the absolute path /opt/agent/helper, never a bare helper resolved through svc_helper's PATH, and confirm /opt, /opt/agent, and the binary itself are not writable by svc_helper or any other unprivileged account.

Privilege transitions

Get the ORDER right, and get it EXACTLY right, because every step in this sequence still needs root's privilege to succeed, so anything placed after the UID drop silently fails instead of doing what you intended: first setgroups(0, NULL) (or initgroups("svc_helper", svc_helper_gid) if the target account should retain its own supplementary groups) to clear whatever supplementary groups root's process carried, THEN setgid(svc_helper_gid), THEN setresuid(svc_helper_uid, svc_helper_uid, svc_helper_uid) last, immediately before execve() of /opt/agent/helper. setgroups() has to run before setresuid() specifically, not merely "sometime before execve()": setgroups() itself requires CAP_SETGID (effectively, root), so if it is called after the UID has already been dropped, it fails with EPERM, and if that return value isn't checked, the process carries on believing groups were cleared when in fact the OLD, privileged supplementary group list is still attached. Dropping the user ID first, especially from root, removes the privilege needed to still change the group ID afterward, which silently leaves the process running with the OLD, more privileged group even though the UID looks correctly dropped, a subtle bug that passes a naive "is my UID what I expect" check while leaving real privilege behind. Use setgroups()/initgroups() to explicitly clear or correctly set the supplementary group list; a very common miss is dropping the primary UID/GID while leaving root's supplementary groups intact, which silently retains privileged group-based access the review would otherwise have assumed was gone. Verify the drop actually took, as a defensive check, by attempting to REGAIN the original privilege immediately after dropping it; a correctly-dropped process should fail that attempt, and a process where it unexpectedly succeeds has caught a partial-drop bug before it matters. Confirm no Linux capabilities (a capability is a narrow, named unit of root's privilege, such as CAP_NET_ADMIN or CAP_NET_BIND_SERVICE, that can be granted to a binary or process without granting full root) linger beyond what the child's job requires, since capabilities can persist across a UID change in ways that are easy to miss, particularly when file capabilities are set on the target binary: the kernel re-evaluates those file capabilities at execve() time and can grant them to the resulting process regardless of its UID, similar in effect to the setuid bit, so a stray file capability on /opt/agent/helper would hand svc_helper's process that privilege no matter how carefully the UID/GID drop above was done. Finally, decide explicitly whether the child should inherit the parent's working directory, umask, and resource limits, since by default all of these cross the exec boundary unchanged unless the parent resets them first.

The investigation comes down to reading the SHAPE of the process tree and the IDENTITY of what's actually being executed at each hop, then comparing both against what's normal for this host.

What a fork bomb looks like

A fork bomb is defined by rate and self-similarity, not by what any individual process does. Symptoms: an extremely high, near-instantaneous rate of new PIDs, potentially thousands per second; the SAME short-lived binary re-forking itself recursively (the parent and the child are typically running the identical program, sometimes literally the same shell one-liner re-invoking itself); and system-wide symptoms that show up FAST, fork() starting to fail everywhere on the host with EAGAIN/ENOMEM because the global PID space or resource limits are being exhausted, CPU pegged near 100% from pure scheduling/context-switch overhead even though no individual process is doing meaningful work (no real file I/O, no network activity, nothing but forking). Confirming this: ps -eo pid,ppid,etimes,cmd --sort=start_time | tail -50 (or pstree -p) shows a large cluster of processes all born within the same second or two, almost all running the same command, with a shallow but extremely WIDE tree (one ancestor with an enormous number of direct or near-direct descendants), rather than a deep chain.

What a malware loader / process-chaining attack looks like

The opposite shape: a NARROW, sequential chain, parent -> child -> grandchild, where each hop exec()s into a DIFFERENT binary than the one before it ("process chaining," commonly discussed as part of "living off the land" tradecraft). The telltale signal is fork() immediately followed by execve() into an unrelated program, especially when: the target binary lives in a WRITABLE, non-standard location (/tmp, /dev/shm, a user's Downloads directory) rather than a normal system binary path; or it's a legitimate, trusted system binary being invoked with unusual, scripted-looking arguments that a human wouldn't type interactively (a base64-encoded inline script, a curl/wget piped directly into a shell interpreter, a download-then-exec two-step); or a process that has no legitimate reason to spawn a shell at all suddenly does, a web server worker process forking sh -c "...", or a document viewer (PDF reader, Office app) spawning a script interpreter, is a strong standalone signal on its own, since normal automation for those specific parent processes essentially never legitimately does that.

What normal automation looks like, as the baseline to compare against

A stable, REPEATING, predictable tree shape: a cron job or systemd timer forking the same handful of children on a fixed schedule; a CI runner spawning build-tool subprocesses in a consistent pattern run after run. Children exec well-known binaries from standard system paths (/usr/bin, /usr/local/bin, not a temp directory). The fork rate is moderate and clearly correlated with an actual trigger (a cron entry firing, a webhook arriving), not continuous, unbounded self-replication. And critically, the specific argv patterns and binary paths involved are ones you'd expect to have SEEN BEFORE on this host if you have any historical baseline (fleet-wide EDR telemetry, prior audit logs), whereas malicious activity commonly shows a first-time-ever binary path, argv combination, or parent/child pairing for that specific host.

Concrete investigative steps

1. pstree -p <suspect_root_pid> first, to see the actual tree SHAPE at a glance: wide-and-shallow (fork bomb candidate) versus narrow-and-deep with binary changes at each hop (loader candidate) versus a familiar, repeating pattern (probably fine).
2. ps -eo pid,ppid,lstart,etime,cmd for exact start times and full command lines, WHEN things started (clustered instantaneously, or spread out on a schedule) and WHAT was actually typed as arguments (a base64 blob, a download URL, a -enc/-EncodedCommand-style obfuscation flag are all strong indicators).
3. Don't trust the process name alone: argv[0] (what shows up as the "command" in a casual ps) can be rewritten by the process itself to look innocuous. Cross-check /proc/<pid>/exe (a symlink that resolves to the ACTUAL binary inode being executed, harder to spoof after the fact than an in-process argv rewrite) and /proc/<pid>/cmdline (the real argv the kernel recorded at exec time) against what ps displayed; a mismatch between a friendly-looking process name and a suspicious /proc/<pid>/exe target is itself a finding.
4. Check /proc/<pid>/status for PPID (confirm the actual parentage rather than trusting a displayed process name) and cross-reference against who/last/authentication logs: does this chain trace back to an interactive login session, a legitimate cron entry, or a network-facing service process, and if it's the latter, does that service have any legitimate reason to be spawning a shell or a script interpreter AT ALL? A network-facing daemon forking a shell is close to always worth escalating on its own.
5. Recognize the limits of reasoning from the CURRENT process tree alone: fork()/execve() calls aren't logged anywhere by default, so if the malicious chain has already partially exited by the time you're looking, the tree you're inspecting is incomplete. This is exactly why Linux audit (auditctl -a exit,always -F arch=b64 -S execve) or an eBPF (a Linux kernel facility that lets small sandboxed monitoring programs run in-kernel, the basis for many modern tracing/EDR tools)-based EDR agent recording exec events needs to be enabled BEFORE an incident, not reached for during one; reconstructing a process-chaining attack cleanly from an execve audit trail is straightforward, reconstructing it purely from ps snapshots and whatever remnants are still running when you happen to look is much harder and can miss steps that already completed and exited.

A process's identity for permission checks is a set of credentials, not one number, and execve(2) (the syscall that replaces a process's memory image with a new program) has specific, well-defined rules for updating them. This answer covers the credential model, the setuid/setgid security model, why interpreter scripts are a special case, and what to actually do operationally.

1. Process credentials and how execve() updates them

Every process carries (at least) three UIDs and three GIDs: the real UID (RUID, "who actually launched this"), the effective UID (EUID, "whose permissions are checked for most operations right now"), and the saved set-user-ID (saved UID, "the privilege this process is allowed to switch back to"). Linux adds a fourth, the filesystem UID (FSUID), used only for filesystem access checks and normally tracks EUID automatically.

On an ordinary execve() of a non-setuid file, RUID, EUID, and saved UID are all preserved unchanged from the calling process. The setuid/setgid bits change that: if the file being executed has the set-user-ID bit (mode bit S_ISUID) set, a successful execve() sets the new process's EUID to the file owner's UID, and also updates the saved UID to that same value, while RUID stays whatever it was before (typically the invoking user). The set-group-ID bit (S_ISGID) does the analogous thing to EGID using the file's group owner. This is exactly how /usr/bin/passwd works: it is owned by root and has the setuid bit set, so when an ordinary user runs it, RUID stays that user's UID (so the program can still tell who invoked it) but EUID becomes 0 (root), which is what lets the program write to /etc/shadow, a file the invoking user has no direct permission to touch.

2. The security model for setuid binaries

A setuid-root binary is a deliberate, narrow hole punched through the normal permission model: "this specific program, and only this program, may act as root on the invoking user's behalf." That model only holds if the program is written to close every path an attacker could use to make it act as root on their behalf instead of the intended narrow task. Concretely that means: sanitize or drop dangerous inherited environment variables (PATH, IFS, LD_PRELOAD, LD_LIBRARY_PATH) before doing anything, because a setuid program that shells out or dynamically links using an attacker-controlled PATH or LD_PRELOAD can be tricked into running attacker code with root's privileges; validate every argument and avoid unbounded buffers (a setuid binary with a stack-based buffer overflow is a direct local-root exploit, since the overflow runs with the effective, elevated privileges); drop privileges (via setuid()/setresuid(), not just seteuid(), because seteuid() alone leaves the saved UID unchanged, so a compromised process could call seteuid(0) again and reclaim the privilege it appeared to have dropped; setresuid() also updates the saved UID, closing that path) the moment the elevated action is done, rather than running the whole program's lifetime at EUID 0; and never trust the RUID caller's other ambient state (open file descriptors, working directory, umask) without re-checking it under the elevated identity. Every historical local-root CVE class (format-string bugs, TOCTOU races on files the program then opens as root, PATH-based library or helper hijacking) is dangerous specifically because the vulnerable code is running setuid.

3. Interpreters and shebang lines

Modern Linux does not honor the setuid bit on an interpreter script (a file starting with #!/path/to/interpreter), even if the script's file itself has the bit set and is owned by root. The reason is a classic time-of-check-to-time-of-use (TOCTOU) race: the kernel identifies the file as a script and hands the path to the interpreter to open and read; between the kernel's permission check on the script and the interpreter actually opening that path a moment later, an attacker with write access to the containing directory can swap the script out from under it (via a symlink race or a rename), so the interpreter ends up executing attacker-controlled content with the privilege the kernel granted to the original, legitimate script. Because that race is essentially unfixable at the shebang layer, Linux (and most modern Unixes) simply drops the setuid/setgid bit's effect when the target is an interpreter script; the interpreter itself starts at the caller's real privilege, not the elevated one. (Some older systems shipped a dedicated setuid-safe wrapper like suidperl for this exact gap; it is deprecated/removed on current distributions precisely because it kept reintroducing this class of bug.) The practical takeaway: if you need setuid behavior, it has to live in a compiled binary, not a shell/Python/Perl script with the bit set on the script file itself.

4. Operational recommendations for third-party binaries needing elevated privileges

• Prefer Linux file capabilities over setuid-root: setcap cap_net_bind_service+ep /usr/local/bin/myproxy grants only the ability to bind ports below 1024, not full root, which shrinks the blast radius of any bug in that binary to "can bind low ports," not "can do anything."
• If you must run something as root, scope it with sudo to the exact command and arguments (a sudoers entry with a fixed command line, not NOPASSWD: ALL), not blanket root shell access.
• Run the binary in a container or with systemd service hardening: User=/DynamicUser=yes to avoid literal root, NoNewPrivileges=yes to block it from gaining more privilege than it started with (this is the same effect as PR_SET_NO_NEW_PRIVS), CapabilityBoundingSet= to cap which capabilities it could even acquire, and a seccomp profile to restrict which syscalls it can issue at all.
• Verify provenance before granting any of the above: checksum/signature-verify the binary, and if it's closed-source, treat it as untrusted input to your privilege boundary, not a trusted extension of it.
• If it insists on the setuid bit itself, audit whether that's actually required (many "needs root" third-party tools only need one specific capability, e.g. raw sockets or CAP_NET_ADMIN) and replace the setuid bit with the matching file capability wherever possible, since capabilities are strictly narrower than "full effective UID 0."
• One useful cross-check: setuid and setgid processes cannot be attached to with ptrace(2) (the debugging/tracing syscall) by an unprivileged tracer, by design, specifically so a normal user can't attach a debugger to a setuid-root process and manipulate its already-elevated memory. If you find yourself needing to strace a setuid binary to understand it, that itself is a sign to do the investigation as root or to test with a non-setuid copy in a sandboxed environment instead of fighting the protection.