Kernel Architecture & OS Internals Questions

In Unix-like filesystems an inode is the metadata record for a file: it stores ownership, permissions, timestamps, link count and pointers to data blocks. Data blocks are the on-disk storage units that contain file contents. A file exists as an inode + zero or more data blocks.

Why df and du differ

• df reports filesystem-level free space (based on free data blocks and reserved blocks tracked in the superblock). It shows space available to the whole filesystem.
• du walks directories and sums actual disk usage of files (blocks allocated to files).
  Differences arise because:
• Deleted-but-still-open files free space in directory listings (du) but remain allocated until the process closes the file — df still shows them used.
• Sparse files have logical size larger than allocated blocks; du shows allocated blocks, df accounts for actual free/used space.
• Filesystem reserved blocks (e.g., root reserve on ext4) reduce space shown as available to non-root users; df -h shows total/available taking reserve into account.

Investigation steps for “no space left” when disk seems empty

1. Run df -h and df -i to check block and inode exhaustion.
2. Use du -sh /path and du -xsh / to compare against df; check mounts (mount, findmnt) — maybe you're looking at a different mount.
3. Find large or deleted-open files: lsof +L1 to list deleted but open files consuming space; pfiles/fuser as alternative.
4. Check for reserved blocks (tune2fs -l /dev/.. or dumpe2fs) and whether root sees free space.
5. Check hidden mountpoints: a directory with files may be masked by a later mount (umount suspicious mounts, then du).
6. Look for many small files/inode exhaustion: tune inode ratio or remove files.
7. If needed, reboot or restart offending process after freeing/deleting to release space, and clean up logs, rotated files, tmp directories, or expand filesystem/LVM.

Best practices

• Monitor both inode and block usage, alert on deleted-but-open files, use log rotation and tmpclean, reserve appropriate free space, and document mount points.

Definition & key difference:

• Process: independent OS entity with its own virtual address space, file descriptors, and kernel metadata.
• Thread: execution context (lightweight) within a process sharing the process address space and resources.

Isolation (address space):

• Processes are isolated — separate virtual memory; bugs (e.g., segfault, heap corruption) in one process don’t directly corrupt another.
• Threads share the same address space; a bad pointer in one thread can corrupt the whole process.

Scheduling unit:

• On Linux, the kernel schedules tasks (threads) — each thread has its own pid/tid and can be scheduled independently. Processes are groups of threads (same mm_struct).

Memory sharing:

• Threads share heap, globals, and open fds. Processes do not share memory unless explicit (shared memory, mmap, pipes).

Context-switch costs:

• Thread switch is cheaper (no TLB shootdown for mm change) because mm_struct is shared. Process switch is heavier (switching address space, possible TLB flush).

IPC approaches:

• Between processes: pipes, UNIX sockets, shared memory, message queues — explicit and safer boundaries.
• Between threads: direct memory access + synchronization primitives (mutexes, condition variables); faster but requires careful locking.

When to prefer multiple processes:

• Reliability/safety: isolate failures (sandboxing, crash isolation, restart per-process).
• Security: least privilege, different user namespaces or seccomp per process.
• Memory leaks: isolate growth per worker; can recycle by restarting process.
• Language/runtime limitations: when runtime has global interpreter lock or unsafe native libs.

When to prefer multithreaded workers:

• High-performance low-latency workloads requiring cheap context switches and shared in-memory caches without serialization cost.
• Tight coordination/state sharing where copying/IPC overhead would dominate.
• IO-bound workloads where many threads can utilize blocking syscalls efficiently.

Trade-offs summary:

• Processes = stronger isolation, higher IPC cost, heavier context switches.
• Threads = lower latency and lower IPC cost, less isolation, higher risk of cross-thread bugs. Choose based on reliability/security needs versus performance requirements.

Linux OOM killer picks a victim by scoring processes based on how much memory they use and other heuristics; the highest score is killed to free memory. Key fields:

• oom_score: read-only per-process value in /proc/<pid>/oom_score. It’s a kernel-computed integer (higher = more likely to be killed) that reflects the process’s memory footprint and priority.
• oom_score_adj: writable per-process adjustment in /proc/<pid>/oom_score_adj (range -1000..+1000). It biases the kernel score: -1000 makes a process immune, positive values increase kill likelihood. Systemd exposes this via ProtectKernelModules/oomScoreAdjust.
• /proc/<pid>/oom_score shows the final score the OOM killer uses; monitoring it helps identify at-risk processes.

As an SRE, mitigate frequent OOMs and protect critical services:

1. Immediate protections

   • Set oom_score_adj = -1000 for critical daemons (e.g., kube-apiserver, etcd) so they’re last to be killed.
   • Use cgroups/v2 or systemd slices to isolate memory (memory.max) so one workload can’t exhaust host memory.
   • In Kubernetes: set requests/limits, use QoS Guaranteed/Burstable appropriately, and set eviction thresholds to protect system pods via kube-system priorityClass and podPriority.

2. Root cause & capacity

   • Add monitoring/alerts for memory pressure, OOM events (dmesg/kern.log), and high /proc/*/oom_score.
   • Profile leaking processes (perf, pmap, heap profilers) and fix leaks or excessive caches.
   • Adjust overcommit policy (vm.overcommit_memory and vm.overcommit_ratio) thoughtfully; consider adding swap or tuning vm.min_free_kbytes.
   • Scale capacity: add nodes, increase instance sizes, or limit memory-intensive batch jobs.

3. Runtime controls & safety nets

   • Enable oom_kill_allocating_task if appropriate (kernel option) to kill the allocation-hitting task.
   • Use out-of-memory daemons (earlyoom, systemd-oomd) to preemptively kill low-priority processes before catastrophic OOM.
   • Apply resource quotas and limit ranges in clusters to prevent noisy neighbors.

4. Process-level best practices

   • Set intelligent JVM/DB memory limits, use memory-aware configuration (heap sizes, cache caps).
   • Graceful degradation: health checks that stop accepting work before memory exhaustion.

Why this works: combining immediate hard protections (oom_score_adj, cgroups, k8s priorities) prevents critical services from being selected, while monitoring, profiling, and capacity planning eliminate the recurring root causes that trigger OOMs.

Approach summary: use perf for a low-overhead, kernel-aware stack-sampling run and generate a flamegraph; use bpftrace for quick production-safe one-liners when you need dynamic, PID-filtered profiles or to sample kernel events. Keep sampling low (50–200 Hz), filter to the service PID/comm, run short captures (30–120s), and generate flamegraphs to find hot stacks (leaf vs aggregated stacks).

Perf sequence (recommended):

# 1) Record stacks for PID (prefer -g for callchains). Frequency 99 Hz is common.
sudo perf record -F 99 -p <PID> -g -- sleep 60

# 2) Convert to script
sudo perf script > out.perf

# 3) Collapse stacks and generate SVG (from FlameGraph repo)
git clone https://github.com/brendangregg/FlameGraph.git
./FlameGraph/stackcollapse-perf.pl out.perf > out.folded
./FlameGraph/flamegraph.pl out.folded > perf_flamegraph.svg

bpftrace quick sampling (one-liner):

# Sample user+kernel stacks at ~101Hz for specific command name
sudo bpftrace -e 'profile:hz:101 /comm == "myservice"/ { @[ustack()] = count(); }' -o bpf.out & sleep 60; kill $!

# bpf.out contains counts; convert to folded stacks (simple script or bpftrace's `printf` can format)
# Example: using Brendan Gregg's flamegraph scripts if you collect folded format.

Or a more direct bpftrace that prints folded stacks:

sudo bpftrace -e 'profile:hz:97 /pid == PID/ { printf("%s\n", ustack()); }' > stacks.txt
# Post-process stacks.txt into folded format (replace spaces with ';', aggregate counts) then run flamegraph.pl

Sampling frequency guidance:

• 50–200 Hz typical. Lower (50–100 Hz) reduces overhead; higher (>200 Hz) increases fidelity but can add measurable overhead and noise.
• For production, start at ~97–100 Hz for 30–60s; increase only if you miss short-lived hotspots.

Low-overhead practices:

• Filter to PID or command (/pid or /comm) so you only sample your service.
• Short captures (30–120s) covering representative load.
• Avoid very high frequencies or system-wide captures in production.
• Use perf record -e cycles or -e cpu-clock for CPU-bound work if callchains are secondary.

Interpreting flamegraphs / hotspots:

• Width = aggregated time spent in that function (inclusive). Taller stacks show call paths.
• Leaf-heavy wide boxes indicate functions consuming CPU directly (good optimization targets).
• Deep stacks indicate time is distributed down a call path or in syscalls — inspect leaf to find hotspots.
• Distinguish user vs kernel frames (perf shows kernel frames). If kernel time dominates, look at syscalls, I/O, scheduler.
• Verify symbols: ensure binaries have debug symbols or use addr2line/perf map for JIT (e.g., Java/Go). For containers, run perf on host and map container PID to host namespace or run perf in the container with CAP_SYS_ADMIN/CAP_SYS_PTRACE.
• Validate with multiple captures and with different sampling rates; correlate with metrics (CPU, latency).

Extra tips:

• Use perf top for quick live hotspots, then perf record for flamegraph.
• For managed runtimes, use runtime-specific maps (perf map for Java/Go) or flamegraph helpers (async-profiler, pprof) to get accurate Java frames.
• Keep captures and artifacts for post-incident analysis; annotate service version and load conditions.

Clarify the problem

• The usual pattern for atomic replacement is: write new content to a tmp file, fsync it, then rename(tmp, target). On POSIX local filesystems rename() is atomic (observers either see old or new name). The race appears when concurrent open() calls target the file while rename is in progress.
  Why NFS (and some network filesystems) break this
• NFSv3 has weak semantics: clients cache directory/attribute info and the server-side “atomic rename” may not be observed consistently by all clients. Some NFS versions implement rename as a sequence of ops; clients may still hold old filehandles; an open() racing with rename can succeed on different clients at different times producing transient inconsistencies. Also POSIX advisory locks (flock/fcntl) are not reliably supported or coherent across all NFS implementations.
  Design goals
• Ensure: (1) clients never observe a partially written file, (2) one-source of truth for the current file, (3) works across network filesystems and multiple hosts.
  Recommended strategies (practical, SRE-focused)

1. Use an external coordinator (preferred for distributed systems)

• Acquire a distributed lock via etcd/consul/zookeeper/Redis (SETNX with TTL) before deploy or rotation.
• Steps:
  1. Acquire lock L for target resource.
  2. Write tmp file on the shared store, fsync.
  3. Rename tmp→target (or update pointer metadata).
  4. Release lock L.
• Why: single lock owner serializes open/rename; clients that need consistent reads either check lock or use a two-phase protocol.

2. Use versioned files + indirection (robust when rename semantics are weak)

• Write versioned file: target.vN (fsynced).
• Update a separate small metadata pointer (e.g., a single “current” object in the key-value store or a small file on a well-behaved metadata service) that points to versioned filename. The metadata update is done under the distributed lock. Clients read pointer then open the referenced file.
• Advantage: no requirement that rename on NFS is atomic, pointer update is serialized by lock.

3. For systems without an external coordinator: use atomic create + O_EXCL for lockfiles, but be cautious

• Attempt to create a lockfile via open(lockfile, O_CREAT|O_EXCL). If success, you hold the lock; else retry. O_EXCL is atomic at the server but behavior can vary with caching—so test against your NFS version. Always fsync directory after creating/removing lockfile where possible.
• Use timeouts, stale-lock detection, and owner info in lockfile.

4. Log rotation specifics

• Prefer signalling processes to reopen (e.g., SIGHUP) rather than replacing files clients still write to.
• If you must rotate on NFS: instruct writers to open files by name for each write, use versioned files + pointer, or use copytruncate as last resort (risky for lost log lines).

Practical checklist

• Prefer external consensus-based locks for multi-host atomicity.
• Always fsync file and parent directory when possible before publishing.
• Avoid relying solely on POSIX locks across NFS.
• Add health/timeout to avoid deadlocks from crashed holders.
• Test on your exact NFS/version under load and failure modes.

This approach guarantees that only one deploy/rotation proceeds at a time and that readers see a consistent pointer to a fully-written file, avoiding transient inconsistent open() observations on NFS.