Kernel Architecture & OS Internals Questions

Brief overview / what to check first

• Per-process limit: ulimit -n (shell) and /proc/<pid>/limits
• System-wide limit: /proc/sys/fs/file-max and /proc/sys/fs/file-nr
• Find offending processes with lsof or /proc/<pid>/fd counts

Commands to discover current limits

• Per-shell (current shell):

ulimit -n

• For a specific PID:

cat /proc/1234/limits | grep "Max open files"

• System-wide limit and allocation:

cat /proc/sys/fs/file-max
cat /proc/sys/fs/file-nr

Find processes exhausting FDs

• Top offenders via lsof:

lsof -nP | awk '{print $2}' | sort | uniq -c | sort -nr | head

• Per-PID FD counts (human readable):

for pid in $(ls /proc | grep -E '^[0-9]+$'); do
  cnt=$(ls -1 /proc/$pid/fd 2>/dev/null | wc -l)
  [ "$cnt" -gt 0 ] && echo "$cnt $pid $(ps -p $pid -o comm=)"
done | sort -rn | head

• Check network sockets (if leak suspected):

ss -s
ss -pan | grep <pid>

Increase limits persistently

• System-wide (kernel)

  • Temporary:
  bash

  sysctl -w fs.file-max=200000
  echo 200000 > /proc/sys/fs/file-max
  

  • Persistent across reboots: add to /etc/sysctl.conf or create a file in /etc/sysctl.d/
  bash

  # /etc/sysctl.d/99-file-max.conf
  fs.file-max = 200000
  

  then sysctl --system.

• For a systemd-managed service (recommended)

  • Create override drop-in:
  bash

  mkdir -p /etc/systemd/system/myservice.service.d
  cat > /etc/systemd/system/myservice.service.d/limits.conf <<'EOF'
  [Service]
  LimitNOFILE=65536
  EOF
  systemctl daemon-reload
  systemctl restart myservice
  

  • You can also set in the unit file's [Service] section directly, but prefer drop-in so package updates won't overwrite.

• For all shell users (persistent)

  • Edit limits.conf or add file in limits.d:
  bash

  # /etc/security/limits.d/90-nofile.conf
  *               soft    nofile          65536
  *               hard    nofile          131072
  

  • Ensure PAM applies it (Debian/Ubuntu):
  bash

  grep -E "pam_limits.so" /etc/pam.d/* || echo "session required pam_limits.so" >> /etc/pam.d/common-session
  

  • Note: systemd user sessions may ignore PAM limits; prefer systemd service LimitNOFILE for services started by systemd.

Verification after change

sysctl fs.file-max
systemctl show myservice --property=LimitNOFILE
su - someuser -c 'ulimit -n'
cat /proc/<pid>/limits | grep "Max open files"

Notes / trade-offs

• Raising system limits allows more FDs but masks leaks — still investigate root cause.
• Prefer per-service LimitNOFILE for systemd-managed daemons; use limits.conf for interactive/logged-in users.

Detection — monitoring first

• Start with system-level metrics: collectd/Prometheus + node_exporter, track RSS, VIRT, OOM events, and open files. Alert when RSS grows beyond baseline or when RSS/VIRT diverge.
• Quick live checks:

ps -o pid,user,%mem,rss,vsz,cmd -p <pid>
top -p <pid>

Inspect /proc and pmap

• Check smaps for detailed mappings and leaking anon pages:

pmap -x <pid>
sudo cat /proc/<pid>/smaps | awk '/AnonHugePages|Anon/ {print}'

• /proc/<pid>/status gives VmRSS/VmSize and Threads.

Heap profilers

• Native C/C++: run under valgrind massif or heaptrack to get allocation hotspots:

valgrind --tool=massif --massif-out-file=ms.out ./daemon
ms_print ms.out
# or
heaptrack ./daemon
heaptrack_gui heaptrack.*.gz

• Use massif to see growth over time; heaptrack gives object-level callstacks.

gdb heap analysis

• Attach gdb, use heap introspection (libc debug malloc), or iterate /proc mappings; take core and inspect with gdb/malloctools to find large arenas.

Java-specific

• Use jcmd/jmap to get heap dumps and jstat/jcmd GC for live metrics; analyze with VisualVM or Eclipse MAT:

jmap -dump:live,format=b,file=heap.hprof <pid>
jcmd <pid> GC.heap_info

Python-specific

• Use tracemalloc to trace allocations, objgraph/guppy to find refcycles:

import tracemalloc
tracemalloc.start()
# snapshot = tracemalloc.take_snapshot()

Mitigation & operational fixes

• Immediate: restart service, enable graceful restarts, tune OOM killer, add swap or cgroups memory limit.
• Long-term: fix allocation hotspots found by profilers, reduce retention (cache TTLs), close file descriptors, and add unit/integration tests with stress harness to reproduce leak.

Reasoning: monitor to detect trend, use /proc and pmap to confirm, apply profilers to locate root cause, use language tools for higher-level leaks, and apply operational controls to reduce blast radius while fixing code.

Situation & Goal
I’m responding as the on-call Systems Administrator to suspected malicious kernel module (rootkit). Goal: detect, contain, forensically capture evidence, verify kernel integrity, and restore trust with minimal service disruption and preserved evidence.

Detection Signals

• Unexpected module names or duplicate initials:

lsmod | egrep -v "^(Module|Module)"     # inspect loaded modules
cat /proc/modules

• Unusual /proc entries or altered timestamps:

ls -la /proc | head
grep -R "hide" /proc/modules  # heuristic search
dmesg | tail -n 200

• Network anomalies:

netstat -tunap; ss -tunap

Containment

• Isolate host from network (remove from cluster, disable external interfaces) but keep management plane for forensics.
• If safe and module is unloadable, remove with:

rmmod suspicious_module

only after capturing evidence and ensuring no dependent services; otherwise leave in-memory to preserve state.

Forensic Capture

• Volatile state:

lsmod; cat /proc/modules; dmesg > /tmp/dmesg.out
ss -tunap > /tmp/conn.out; ps aux > /tmp/ps.out
volatility or LiME memory dump for full RAM image

• Persistent:

tar czf /tmp/fs_snapshot.tgz /etc /var/log /boot
dd if=/dev/sda of=/tmp/disk_image.dd bs=4M conv=sync,noerror

• Hash all artifacts (sha256sum) and document chain-of-custody.

Verify Kernel Integrity

• Check module signature enforcement:

cat /proc/sys/kernel/modules_disabled

• Validate using IMA/Appraisal logs, dm-verity for immutable partitions, and compare kernel and modules against known good packages (apt/rpm) and signed kernels.
• Use evmctl/ima_measurements and journalctl -t ima.

Restore Trust

• Rebuild on clean media: reinstall kernel and packages, reprovision host from known-good image, rotate keys and credentials accessible from the host, re-image rather than in-place repair when rootkit confirmed.
• Post-restore: enable module signature enforcement, enable IMA/Appraisal and dm-verity where feasible, implement monitoring (integrity checks, audit rules), and perform full network/credential sweep.

Lessons & Preventive Controls

• Enforce secure boot + module signature, centralized logging, immutable boot, regular integrity scans, limited key holders for kernel signing, and documented IR runbook.

Overview (Unix-like vs Windows)

• Unix-like: a process moves between states: running (on CPU), runnable/ready (eligible for CPU), sleeping (waiting I/O or timer — interruptible or uninterruptible), stopped (SIGSTOP / debugger), zombie (exited but parent hasn't waited), and init adoption.
• Windows: similar concepts but different names: Running, Ready, Waiting (blocked on I/O/objects), Suspended, Terminated. The kernel handles wait blocks and thread context.

State transitions (common paths)

• Runnable -> Running when scheduler picks it.
• Running -> Sleeping when it waits for I/O/timer.
• Sleeping -> Runnable when event occurs.
• Running -> Stopped via signal or debugger attach.
• Running -> Exited -> Zombie (Unix) until parent calls wait(); then kernel frees it.

Inspecting processes

• Linux examples:

ps -eo pid,ppid,state,cmd | head
top              # interactive, shows R,S,D,Z states
systemctl status  # service state via systemd
cat /proc/<pid>/status

• Windows examples (PowerShell):

Get-Process | Format-Table Id,Handles,CPU,StartTime,Responding
Get-Process -Id <pid> | Select-Object *
Get-CimInstance Win32_Process | Select ProcessId,ParentProcessId,ExecutionState

Why zombies happen in production

• Parent never calls wait()/waitpid() due to bug or design (e.g., long-running parent ignoring SIGCHLD).
• Parent crashed before reaping children.
• Misconfigured supervisor or PID 1 in containers not reaping orphaned processes.
• High churn of short-lived processes without proper reaper leads to many zombies and PID table exhaustion.

Operational fixes

• Restart or fix parent process, implement proper SIGCHLD handler or use double-fork/daemonize pattern, ensure init/systemd or a supervisor (systemd, runit) adopts and reaps children. Monitor with scripts/alerts for zombies.

Definition & presentation to processes
Virtual memory gives each process a private linear address space (e.g., 0..2^48-1 on x86-64) that the OS maps onto physical RAM + swap. Processes use virtual addresses; the OS and MMU translate them to physical frames, so processes behave as if they have contiguous memory even when memory is fragmented.

Address space & virtual→physical mapping

• Address space: per-process view of memory (code, heap, stack, mmap regions).
• Mapping: page-sized chunks (commonly 4 KiB) map virtual page numbers (VPN) to physical frame numbers (PFN) via page tables maintained by the OS.

Page tables & multi-level page tables

• Single-level tables are large; multi-level (e.g., 4-level in x86-64) break the VPN into indices for hierarchical tables to save memory: only levels for used ranges are allocated.

TLB (Translation Lookaside Buffer)

• CPU cache of recent VPN→PFN translations. Hits are fast; misses require walking page tables (hardware or software-assisted) costing many cycles.

Page-fault sequence (typical, Linux-like)

1. Process accesses a VPN not in TLB → TLB miss.
2. Hardware/page-table walker probes page tables; if entry absent or not present bit set, a page fault trap to kernel.
3. Kernel checks VMA/mmap to see if access is valid.
4. If valid but page swapped out: locate swap slot or file, read page into a free frame (may invoke page reclaim/evict), update page table entry (set PFN, present bit).
5. If invalid access → send SIGSEGV.
6. Kernel updates TLB (or lets CPU reload) and resumes process.

Why virtual memory helps (isolation & utilization)

• Isolation: per-process page tables enforce access permissions (user vs kernel, read/write/exec), preventing accidental/cross-process access.
• Utilization: demand paging loads only used pages; copy-on-write lets forked processes share pages until modification; swapping and sparse mappings let the system run more processes than physical RAM.

As a systems administrator, understanding these lets you tune swappiness, monitor page faults, configure hugepages, and diagnose memory-related performance or OOM issues.