Operating System Internals and Administration Questions

Fundamental and advanced operating system concepts that underlie system administration across platforms. Topics include process and thread management, scheduling and concurrency, memory management and swapping, virtual memory and page replacement, input output and disk performance, file system architecture and semantics, system call interfaces, kernel parameters and tuning, authentication and permission models, boot and initialization sequences, monitoring and system performance analysis, and general techniques for debugging and diagnosing systemic operating system issues. Candidates should be able to explain not only how to perform administrative tasks but why the underlying mechanisms behave as they do and how design choices affect performance and reliability.

EasyTechnical

64 practiced

Explain Linux CPU scheduling basics: Completely Fair Scheduler (CFS), 'nice' values, SCHED_FIFO/SCHED_RR (real-time) policies, and CPU affinity. As an SRE, when and how would you use 'taskset', 'chrt', or cgroups to tune scheduling for critical services?

Sample Answer

Linux CPU scheduling basics:- CFS (Completely Fair Scheduler): default Linux scheduler for normal processes. It models fairness using virtual runtime (vruntime) and distributes CPU time proportional to weight. Short tasks get scheduled quickly; long-running tasks accumulate vruntime and yield. Goal: low latency + fairness across CPUs/threads.- 'nice' values: user-space priority hint for CFS. Range -20 (highest priority) to +19 (lowest). Lower nice -> larger weight -> more CPU share. Use sparingly for priority control of non-RT tasks.- Real-time policies (SCHED_FIFO, SCHED_RR): kernel-level RT scheduling with strict priorities (0–99). SCHED_FIFO is non-preemptive among same priority (runs until blocks or yields); SCHED_RR adds time quantum rotation among same-priority tasks. RT tasks can starve normal tasks — use only when necessary (low-latency, deterministic work).- CPU affinity: binds a process/thread to specific CPU cores to improve cache locality or isolate critical workloads.

Practical SRE tools and when to use them:- taskset: set or query CPU affinity. Use to pin latency-sensitive services away from noisy CPUs or colocate with hardware interrupts.Example:

bash

# pin PID 1234 to CPUs 0 and 1
taskset -pc 0,1 1234

- chrt: set real-time scheduling policy/priority. Use for real-time workloads (audio, packet processing) or emergency fixes to reduce latency; be careful to avoid starving system daemons.

bash

# run command with SCHED_RR priority 50
sudo chrt -r 50 ./my_process
# change existing PID
sudo chrt -f -p 10 1234   # SCHED_FIFO priority 10

- cgroups (v2): control groups for resource and scheduling tunables (cpus, cpu.weight, rtprio for v1). Use for production: isolate containers, limit CPU shares, set cpuset for pinning, and cap RT usage to prevent system starvation. Example policies: - Use cpuset to reserve cores for system vs service. - Use cpu.max/cpu.weight to limit or prioritize CPU usage. - With cgroup v1, set cpu.rt_runtime_us to limit RT runtime.

Best practices:- Prefer CFS tuning + nice and cgroups for multi-tenant production isolation.- Reserve a small core set for RT or pinned critical services; limit RT runtime via cgroups to avoid starvation.- Monitor with top/htop, pidstat, perf, and trace scheduling issues before applying RT policies.- Document changes and automate via orchestration (systemd, Kubernetes node configs) so scheduling tweaks are reproducible and reversible.

MediumTechnical

117 practiced

Explain virtual memory internals: TLB (translation lookaside buffer), page tables, soft vs hard page faults, and TLB shootdown on SMP systems. Describe the performance implications of TLB misses and strategies (hugepages, locality, memory layout) you would use to reduce TLB pressure for a latency-sensitive service.

Sample Answer

Virtual memory maps virtual addresses used by processes to physical RAM via page tables. The CPU caches recent translations in the TLB (Translation Lookaside Buffer) — a small, fully/associatively-mapped cache of (VA→PA) entries. On a TLB hit the CPU proceeds quickly; on a miss it walks the page table (hardware or assisted by OS) to refill the TLB.

Page tables: multi-level (e.g., x86-64 4/5-level) trees where each level indexes bits of the virtual address. Entries include permissions, present bit, dirty/accessed bits, and can point to hugepages (2MB/1GB) to reduce levels.

Soft vs hard page faults:- Soft fault: page is present in RAM but not mapped in the process’s page tables/has no TLB entry or requires copy-on-write fixups. The OS can handle it without disk I/O — low latency.- Hard fault: page not in RAM; OS must read from backing store (swap or file) — high latency (ms to hundreds of ms).

TLB shootdown on SMP: when the kernel changes mappings (e.g., unmap, update PTEs) it must invalidate corresponding TLB entries on all CPUs that might hold them. The OS sends IPIs or uses architectures’ global/invalidate mechanisms. Shootdowns are synchronous and expensive — they stall cores and serialize mapping updates, increasing latency under frequent mappings/unmappings.

Performance implications of TLB misses: each miss costs a page-table walk (tens to hundreds of cycles) or, worse, a hard page fault. High TLB pressure increases CPU cycles, causes pipeline stalls, and increases tail latencies — critical for latency-sensitive services.

Strategies to reduce TLB pressure (practical for SREs):- Use hugepages (2MB/1GB): reduces number of entries, lowers TLB pressure and page-walk depth. Use hugetlbfs or transparent hugepages carefully (measure fragmentation and latency spikes).- Improve locality: ensure working set fits in RAM and in TLB reach by organizing data access (hot data contiguous, avoid random sparse pointers). Cache-friendly data structures (arrays vs linked lists).- Memory layout: preallocate and touch pages (numa-aware allocation, madvise(MADV_WILLNEED)) to avoid on-demand page faults and remote NUMA penalties. Align frequently accessed structures to page/hugepage boundaries.- Reduce page table churn: avoid frequent mmap/munmap or unmap/remap loops; reuse allocations or use memory pools/arenas to minimize shootdowns.- CPU pinning and NUMA affinity: keep threads and memory on same NUMA node to avoid cross-node page mappings and remote TLB effects.- Monitor: track minor/major fault rates, page-walks, and TLB-miss counters via perf, /proc/vmstat, and NUMA tools; measure tail latency changes when toggling hugepages or layout changes.

Trade-offs: hugepages reduce fragmentation and TLB pressure but increase internal memory waste and can complicate allocation. Transparent hugepages are easier but can introduce unpredictable compaction pauses. Always benchmark under production-like load and monitor tail latency before rolling out changes.

HardTechnical

63 practiced

Sketch an eBPF or bpftrace program that counts syscalls per PID and records syscall latencies (histogram). Describe probe points you would attach to, how you would aggregate to avoid high-cardinality explosion, how to export metrics to Prometheus, and techniques to limit overhead in production.

Sample Answer

Approach summary:- Attach entry/exit probes to sys_enter and sys_exit (or syscall:sys_enter_* / syscall:sys_exit_* tracepoints) to count syscalls per PID and measure latency. Use eBPF maps with per-cpu storage and logarithmic histogram buckets. Reduce cardinality by grouping by container/cgroup or by keeping only top-N PIDs with an LRU map or sampling.

bpftrace sketch (concise):

bash

# capture start time on syscall entry (tracepoint)
tracepoint:syscalls:sys_enter_* 
/ pid != 0 /
{
  @start[tid] = nsecs;
  @count[comm, pid] = sum(1);        // coarse counter per PID+comm
}

# on exit compute latency and record histogram
tracepoint:syscalls:sys_exit_*
/ @start[tid] /
{
  $lat = (nsecs - @start[tid]) / 1000; // microseconds
  delete(@start[tid]);

  // use log2 buckets to avoid many bins
  @lat_hist[pid] = hist(log($lat+1)/log(2));  // bpftrace hist helper
  // optionally use aggregated by container: @lat_hist[cgroup_id]
}

Key design choices and reasoning:- Probe points: use tracepoints (tracepoint:syscalls:sys_enter_* and sys_exit_*) — safer and lower overhead than kprobes and stable across kernels. For specific syscall names use the wildcard or individual tracepoints.- Latency measurement: store per-thread start timestamp on entry keyed by tid (thread id) to avoid races; compute delta on exit.- Aggregation & cardinality control: - Aggregate by cgroup_id or container ID to avoid per-process explosion (use bpf_get_current_cgroup_id() in C/eBPF). - Keep coarse per-PID counters only for PIDs in a configured whitelist (high-value services) or maintain an LRU map (BPF_MAP_TYPE_LRU_HASH) limited to N entries for "top" PIDs. - Use logarithmic (power-of-two) histogram buckets to reduce bin count while preserving distribution shape. - Use per-cpu maps or per-cpu arrays to avoid contention. - Optionally sample 1/N syscalls to lower traffic while preserving distribution.

Exporting to Prometheus:- Run a small user-space exporter that: 1) Periodically (e.g., every 10s) polls eBPF maps (via libbpf, BCC, or bpftool) and reads counters/histograms. 2) Converts counts into Prometheus metrics: counters for syscall counts (labels: pid, comm, cgroup) and Prometheus histogram buckets derived from your eBPF histogram. Use Prometheus client library to expose /metrics.- Use existing projects like prometheus-bpf-exporter or write a lightweight Go/Python reader using github.com/iovisor/gobpf/elf or libbpfgo that translates eBPF histograms into Prometheus histogram (buckets + sum + count). For high-cardinality labels, only export top-K PIDs or aggregate by service label.

Techniques to limit overhead in production:- Prefer tracepoints over kprobes; use minimal logic inside eBPF (timestamp store, increment map).- Use per-cpu maps and BPF_MAP_TYPE_PERCPU_HASH or PERCPU_ARRAY to reduce atomic ops.- Use LRU or bounded maps (BPF_MAP_TYPE_LRU_HASH) to limit memory.- Filter in-kernel: restrict to specific cgroup/pid/comm at probe time to avoid recording all processes.- Sampling: only record 1 of N syscalls or user-space decide sampling windows.- Reduce histogram resolution: log2 buckets, coarser buckets for very high-throughput syscalls.- Use ring buffer or perf event to push heavy payloads to user-space rather than large maps.- Test under load in staging, measure overhead (CPU, latency) and tune sample rate and map sizes accordingly.

Edge cases & safety:- Ensure start timestamps cleaned up (delete on exit) to avoid memory leak if threads exit abnormally.- Handle non-matched enter/exit (e.g., signals) by pruning stale entries periodically.- Respect eBPF verifier limits: keep stack usage small, avoid loops in C program (bpftrace helper loops are OK).

This design balances observability (counts + latency distribution) with production safety (bounded maps, aggregation, sampling) and provides a clear path to export useful, low-cardinality Prometheus metrics.

EasyTechnical

107 practiced

Explain Linux file permission models and umask. Describe how user/group/other permissions, SUID/SGID, and the sticky bit behave. As an SRE, how would you debug and fix a recurring 'Permission denied' error on a service that runs as a specific user and accesses shared files?

Sample Answer

Linux file permissions use three classes — user(owner), group, other — each with read(r=4), write(w=2), execute(x=1). Permissions are typically shown as rwxrwxrwx or octal 0755. The umask is a process-level mask that removes permission bits from newly created files/dirs: default mode for files is 666, for dirs 777; umask (e.g., 0022) subtracts bits, so file becomes 644, dir 755.

Special bits:- SUID (set-user-ID, bit 4000): when set on an executable, process runs with file owner’s UID. Useful for privilege elevation (e.g., passwd) but risky if binary is exploitable.- SGID (set-group-ID, bit 2000): on executables, runs with file’s group. On directories, newly created files inherit the directory’s group rather than the creator’s.- Sticky bit (bit 1000): on directories (e.g., /tmp), prevents users from deleting/renaming files they don’t own even if dir is writable by all.

Debugging a recurring "Permission denied" for a service user:1. Reproduce and collect context: exact failing syscall (open/stat) from logs, journalctl, or service logs.2. Check effective UID/GID of process (ps -o uid,gid, euid,egid) and thread credentials (cat /proc/<pid>/status).3. Inspect file/dir permissions and ownership (ls -l, namei -l /path/to/file) and any ACLs (getfacl).4. Verify SELinux/AppArmor contexts (sestatus, getenforce; ls -Z) or container mount options (ro vs rw).5. Check parent directory execute (x) bits — missing x on parent blocks access even if file perms allow.6. Look at umask of the creating process; if files are created too restrictive, change creator’s umask or have creation code set explicit modes (open with proper mode or chmod after create).7. Fixes: adjust ownership (chown), group membership (usermod -aG), permissions (chmod), set SGID on shared dir to force group inheritance and set a permissive umask (e.g., 002) for the creator, or add ACL entries (setfacl) for the service user. If SELinux is blocking, add a policy or correct context. After change, test and monitor. For recurring issues, automate the correct state via config management and add alerting when permission anomalies occur.

MediumTechnical

75 practiced

Write a Python script (pseudocode acceptable) that monitors a given PID's RSS over time. The script should sample every 30 seconds, record RSS, detect if RSS grows by more than 20% over a 10-minute window, handle PID reuse, and print an alert with the process command line and owner. Describe limitations and how you'd run this at scale.

Sample Answer

Approach: sample /proc/<pid> every 30s, keep a sliding window of the last 10 minutes (20 samples). For robustness against PID reuse, record the process starttime (from /proc/[pid]/stat field 22) and only compare samples with the same starttime. If RSS rises >20% from the window's oldest sample to current, print an alert including PID, starttime, RSS values, command line and owner.

python

#!/usr/bin/env python3
import time, os, pwd
from collections import deque

SAMPLE_INTERVAL = 30            # seconds
WINDOW_SECONDS = 10 * 60        # 10 minutes
MAX_SAMPLES = WINDOW_SECONDS // SAMPLE_INTERVAL
THRESHOLD = 0.20                # 20%

def read_proc_stat(pid):
    p = f"/proc/{pid}"
    try:
        # get RSS (kB) from /proc/<pid>/status VmRSS
        with open(os.path.join(p, "status")) as f:
            lines = f.readlines()
        rss = None
        for ln in lines:
            if ln.startswith("VmRSS:"):
                rss = int(ln.split()[1])  # in kB
                break
        # get starttime (field 22) and cmdline
        with open(os.path.join(p, "stat")) as f:
            stat = f.read().split()
        starttime = stat[21]
        with open(os.path.join(p, "cmdline")) as f:
            cmd = f.read().replace("\0", " ").strip()
        owner = pwd.getpwuid(os.stat(p).st_uid).pw_name
        return int(rss) if rss is not None else 0, starttime, cmd, owner
    except Exception:
        return None  # process gone or permission error

def monitor(pid):
    window = deque(maxlen=MAX_SAMPLES)  # store tuples (timestamp, rss_kb, starttime)
    while True:
        info = read_proc_stat(pid)
        ts = time.time()
        if info is None:
            print(f"[{time.ctime()}] PID {pid} not present or inaccessible")
            window.clear()
        else:
            rss, starttime, cmd, owner = info
            window.append((ts, rss, starttime))
            # find oldest sample with same starttime
            same_proc_samples = [s for s in window if s[2] == starttime]
            if len(same_proc_samples) >= 2:
                old_ts, old_rss, _ = same_proc_samples[0]
                new_ts, new_rss, _ = same_proc_samples[-1]
                # compare growth
                if old_rss > 0 and (new_rss - old_rss) / old_rss > THRESHOLD:
                    print(f"[ALERT {time.ctime()}] PID {pid} (start {starttime}) RSS grew >{THRESHOLD*100}%: "
                          f"{old_rss}kB -> {new_rss}kB. cmd='{cmd}' owner={owner}")
                    # optional: take action, e.g., notify, capture heap, restart
        time.sleep(SAMPLE_INTERVAL)

if __name__ == "__main__":
    import sys
    if len(sys.argv) != 2:
        print("usage: monitor_rss.py <pid>")
        raise SystemExit(1)
    monitor(int(sys.argv[1]))

Key points:- Uses VmRSS (resident set) from /proc/<pid>/status in kB.- Guard against PID reuse with starttime from /proc/<pid>/stat.- Sliding window length = 10 minutes (20 samples at 30s).

Limitations:- Requires permission to read /proc/<pid> (root or same user).- /proc VmRSS is Linux-specific and reports current RSS only.- Short sampling interval may miss very short-lived spikes between samples.- Starttime resolution ties to boot-time jiffies; in rare cases of collisions it may not fully prevent reuse confusion.

Running at scale:- Don’t run one script per PID on many hosts. Instead: - Integrate into existing monitoring (Prometheus exporter or node_exporter textfile) to expose rss metrics per PID or per service. - Push metrics to a centralized TSDB and run alerts (Prometheus alerting rule comparing t-10m to now). - Use service labels (systemd/container IDs) rather than PIDs where possible. - Run with limited privileges (capabilities) and aggregate efficiently (batch reads of /proc) to reduce overhead.- Add rate-limiting, backoff, logging, and automated remediation hooks (notify PagerDuty, capture profiles, restart).

Unlock Full Question Bank

Get access to hundreds of Operating System Internals and Administration interview questions and detailed answers.

Join thousands of developers preparing for their dream job.