Linux Troubleshooting and Diagnostics Questions

In depth troubleshooting and diagnostic techniques for complex Linux issues at the system and kernel level. Includes advanced use of strace, ltrace, perf, ftrace, and reading proc and sys filesystems, root cause analysis of memory leaks and resource exhaustion, diagnosing intermittent failures and I O bottlenecks, log analysis, service debugging, containerized environment troubleshooting, and strategies for progressive isolation, replication, and remediation of production incidents. Senior level expectations include understanding kernel interactions, tracing user space to kernel transitions, and designing observability approaches to prevent recurrence.

HardTechnical

25 practiced

Case study: Your org runs a fleet where occasional kernel OOM events have caused data corruption in a critical service. As SRE lead, propose a remediation roadmap that covers immediate protective measures, medium-term fixes, and long-term policy and telemetry changes. Include stakeholder communication, testing plans, timelines, and trade-offs of each remediation step.

Sample Answer

Situation: Repeated kernel OOMs in our fleet have caused corruption in a critical service’s on-disk state. As SRE lead I must stop ongoing damage, restore safety, and prevent recurrence while coordinating stakeholders.

Remediation Roadmap

1) Immediate (0–7 days) — protect production and stop new corruption- Actions: - Put affected service into read-only or degraded mode via feature-flag/traffic-shifting; halt writes to vulnerable data paths. - Deploy node-level mitigations: enable early OOM notifications (oomd/earlyoom), tune vm.overcommit and swapiness, and configure user-space OOM killer (systemd’s MemoryMax/MemoryAccounting or cgroups v2) to kill least-critical processes instead of DB processes. - Increase replication/backup cadence and take immediate snapshot backups of critical data.- Testing & timeline: roll one mitigation in a canary pool (few nodes) within 24 hours, expand fleet in 48–72 hours.- Trade-offs: temporary reduced functionality/perf; quicker stability vs. user impact.

2) Medium-term (2–8 weeks) — eliminate root causes and harden services- Actions: - Run memory-profiling and heap dumps on suspect processes under controlled load; fix leaks or memory bloat in application code. - Re-architect critical write paths to be crash-safe: fsync, transactional writes, write-ahead logging, idempotent operations. - Adjust orchestrator pod limits/requests and QoS class to prevent eviction of critical pods; implement QoS-driven placement. - Automate graceful shutdown handlers to flush buffers on SIGTERM/oom-notify.- Testing & timeline: end-to-end tests in staging (2 weeks), gradual rollout with canary (2–4 weeks).- Trade-offs: engineering time, potential refactors; improves durability and reduces recurrence.

3) Long-term (2–6 months) — platform policy, telemetry, and culture- Actions: - Policy: enforce mem limits, MemoryQoS, and safe default cgroup settings via admission controllers/opa; require transactional durability review for critical services. - Telemetry: add high-cardinality OOM, memory pressure, pg_stat, fsync latency metrics; pipeline alerts for early memory rise and unsafe write patterns; retention for postmortem analysis. - Chaos & testing: add memory-pressure chaos tests in CI/CD and periodic disaster-recovery drills; include OOM scenarios in SLO error budget calculations. - Runbook & stakeholder comms: publish runbook for OOM incidents, designate RTO/RPO, set regular updates to Product, Engineering, Legal for compliance.- Timeline: policy rollout and telemetry in 1–3 months; culture and chaos testing ongoing.

Stakeholder Communication- Immediate: incident bulletin within 2 hours describing impact, mitigations, and expected user effect; daily updates until stable.- Medium: weekly technical stakeholder sync with remediation progress, risk assessment, and timelines.- Long-term: quarterly reliability review with metrics, SLOs, and mitigation maturity.

Testing Plan & Verification- Canary → canary-plus → phased rollout with monitoring thresholds gating each phase.- Automated regression tests for data integrity; runpoint-in-time restores from snapshots to validate backups.- Post-deployment verification: synthetic writes/reads, fsync latency checks, and data-consistency checksums.

Risk & Trade-offs Summary- Fast mitigations reduce immediate damage but may impair functionality.- Medium fixes require dev time and may introduce deployment risk—mitigate via canaries.- Long-term investments minimize recurrence and improve diagnosability but cost months of work and cultural change.

This roadmap balances urgent containment, durable fixes, and systemic improvements with clear timelines, testing gates, and stakeholder communication to restore and sustain data integrity.

MediumTechnical

25 practiced

Design a Go program (describe algorithm and sketch code) that samples /proc/<pid>/stat and /proc/vmstat every 10 seconds for a target PID, computes per-second major and minor page fault rates and RSS delta, and exposes these as Prometheus metrics on /metrics. Describe concurrency model, parsing pitfalls (comm field), and how to handle PID reuse.

Sample Answer

Approach (brief):- Poll /proc/<pid>/stat every 10s to read process minor/major faults and RSS, compute per-second rates by taking deltas / interval.- Also read /proc/vmstat if you want system-wide counters (optional). Expose metrics with prometheus/client_golang.- Concurrency: one sampling goroutine per monitored PID (ticker + context), Prometheus HTTP server in main goroutine. Use channels/context for shutdown; no heavy locking because prometheus metrics are concurrency-safe.

Key parsing pitfalls:- comm (program name) in /proc/<pid>/stat is in parentheses and may contain spaces or parentheses. Do NOT split the line naively on spaces. Locate first '(' and the matching ')' to extract comm, then split the remainder.- Field positions: after extracting comm, split remaining tokens; use documented field indices (minflt, majflt, rss).- RSS is in pages — multiply by page size (syscall.Getpagesize()) to get bytes.

PID reuse handling:- Read the process starttime field (stat field 22 after parsing) on first sample and keep it. On each sample, compare starttime; if it changes, the PID has been reused — reset previous baselines and metrics for the new process instance and optionally emit a metric/event.

Sketch Go code (core parts):

package main

import (
  "bufio"
  "context"
  "fmt"
  "net/http"
  "os"
  "strconv"
  "strings"
  "syscall"
  "time"

  "github.com/prometheus/client_golang/prometheus"
  "github.com/prometheus/client_golang/prometheus/promhttp"
)

var (
  minorRate = prometheus.NewGauge(prometheus.GaugeOpts{Name: "proc_minor_faults_per_second", Help: "Per-second minor page faults"})
  majorRate = prometheus.NewGauge(prometheus.GaugeOpts{Name: "proc_major_faults_per_second", Help: "Per-second major page faults"})
  rssDelta  = prometheus.NewGauge(prometheus.GaugeOpts{Name: "proc_rss_delta_bytes_per_second", Help: "RSS delta bytes per second"})
)

func init() {
  prometheus.MustRegister(minorRate, majorRate, rssDelta)
}

type sample struct {
  minflt, majflt uint64
  rssPages       int64
  startTime      uint64
  at             time.Time
}

func readProcStat(pid int) (sample, error) {
  path := fmt.Sprintf("/proc/%d/stat", pid)
  f, err := os.Open(path)
  if err != nil { return sample{}, err }
  defer f.Close()
  scanner := bufio.NewScanner(f)
  scanner.Scan()
  line := scanner.Text()

  // extract comm exactly: find first '(' and last ')'
  l := strings.Index(line, "(")
  r := strings.LastIndex(line, ")")
  if l < 0 || r < 0 || r <= l { return sample{}, fmt.Errorf("unexpected stat format") }
  // after ) there is a space then the rest fields
  rest := strings.TrimSpace(line[r+1:])
  parts := strings.Fields(rest)
  // fields after comm: state (1), ppid(2), ... minflt is field 9 after comm? We map positions per proc man:
  // According to proc(5): after comm: state(1), ppid(2), pgrp(3), session(4), tty_nr(5), tpgid(6), flags(7),
  // minflt(8), cminflt(9), majflt(10), cmajflt(11), utime(12) ... rss is field 20 after comm.
  // So indices (0-based) in parts: minflt -> 7, majflt -> 9, rss -> 19, starttime -> 21
  if len(parts) < 22 { return sample{}, fmt.Errorf("stat fields too short") }

  minflt, _ := strconv.ParseUint(parts[7], 10, 64)
  majflt, _ := strconv.ParseUint(parts[9], 10, 64)
  rssPages, _ := strconv.ParseInt(parts[19], 10, 64)
  startt, _ := strconv.ParseUint(parts[21], 10, 64)

  return sample{minflt: minflt, majflt: majflt, rssPages: rssPages, startTime: startt, at: time.Now()}, nil
}

func monitor(ctx context.Context, pid int) {
  var prev *sample
  pageSize := int64(syscall.Getpagesize())
  ticker := time.NewTicker(10 * time.Second)
  defer ticker.Stop()

  // initial sample
  s, err := readProcStat(pid)
  if err != nil { return }
  prev = &s

  for {
    select {
    case <-ctx.Done():
      return
    case <-ticker.C:
      cur, err := readProcStat(pid)
      if err != nil {
        // process may have exited; reset metrics to 0 and continue waiting for revive
        minorRate.Set(0); majorRate.Set(0); rssDelta.Set(0)
        // attempt to re-open next tick
        prev = nil
        continue
      }
      // check starttime for PID reuse
      if prev != nil && cur.startTime != prev.startTime {
        // PID reused: reset baseline
        prev = &cur
        minorRate.Set(0); majorRate.Set(0); rssDelta.Set(0)
        continue
      }
      if prev == nil {
        prev = &cur
        continue
      }

      dt := cur.at.Sub(prev.at).Seconds()
      if dt <= 0 { prev = &cur; continue }
      minorRate.Set(float64(cur.minflt - prev.minflt) / dt)
      majorRate.Set(float64(cur.majflt - prev.majflt) / dt)
      rssBytesDelta := float64((cur.rssPages - prev.rssPages) * pageSize)
      rssDelta.Set(rssBytesDelta / dt)

      prev = &cur
    }
  }
}

func main() {
  pid := 12345 // replace or pass as arg
  ctx, cancel := context.WithCancel(context.Background())
  defer cancel()
  go monitor(ctx, pid)

  http.Handle("/metrics", promhttp.Handler())
  http.ListenAndServe(":9000", nil)
}

Complexity and notes:- I/O: O(1) per sample (reading small files).- Prometheus library is goroutine-safe.- Be defensive: handle parsing errors, process exits, PID reuse via starttime comparison.- Consider adding labels (e.g., pid, cmd) and robust logging/alerts on PID churn.

HardTechnical

30 practiced

Design a continuous, low-overhead profiling strategy for a production service using perf and eBPF that detects CPU and allocation regressions over time. Include sample rates, which events to collect, how to aggregate stack samples, storage considerations for profiles, and how to detect regressions automatically.

Sample Answer

Situation: We're an SRE team needing continuous, low-overhead production profiling to catch CPU and allocation regressions early without impacting SLAs.

Design summary (high level)- Use perf + eBPF-based samplers to collect occasional stack samples (CPU) and sampled allocation events (heap), aggregate into folded stacks, store compressed profiles (pprof or collapsed stacks) in object storage, and run automated change-detection (statistical + anomaly models) against rolling baselines to open alerts or tickets.

Sampling rates & events (practical, low-overhead)- CPU sampling: 100 Hz per process/cgroup (sample_freq=100) — balances signal and overhead; on very hot hosts use 50 Hz or statistical adaptive sampling. Collect PERF_COUNT_HW_CPU_CYCLES and PERF_SAMPLE_CALLCHAIN / PERF_SAMPLE_STACK_USER.- Wall-clock / latency: occasional timer-based samples (1–5 Hz) for wall time if needed.- Allocation sampling: sample allocations probabilistically rather than every malloc. Use jemalloc/tcmalloc sampled allocator (e.g., 1 sample per 16KB allocated) or use eBPF uprobes/tracepoints to sample malloc/free at a rate ~1/512–1/4096 calls depending on allocation churn. Record size buckets.- Other perf events: context-switches, major/minor page faults (low rate) to correlate.- Metadata: pid, tid, comm, cgroup/container id, host, git sha, build-id, timestamp, sample weight.

Collection implementation notes- Prefer kernel perf_event_open for CPU stack sampling plus BPF for flexible filtering and allocation tracepoints. Use BPF to: - Constrain sampling to cgroups or containers to avoid system-wide noise. - Capture only user stacks or kernel+user optionally. - Sample allocations by probabilistic hashing to avoid hot-path overhead.- Ensure user-space unwinding (frame pointers or DWARF-based unwinding) enabled in build. Use build-id and debuginfo for symbolization. Use BTF/kallsyms for kernel frames.

Aggregation of stack samples- Normalize samples to "folded stack" strings: "funcA;funcB;funcC count".- At ingest, map IPs to symbol names and annotate with binary/build id.- Aggregate by (service, version, cgroup, folded_stack) summing counts and total allocation bytes (for allocation samples).- Store per-minute aggregates and roll up to 1h/day windows with retention tiers.- Produce pprof-compatible profiles periodically (e.g., 1m, 5m, 1h).

Storage considerations- On-agent: keep short buffer (in-memory + local small disk) and push compressed profiles every 1–5 minutes.- Object storage: store aggregated profiles as: - pprof protobufs (gzipped) for direct tooling support - collapsed-stack text for quick diffs - metadata JSON for easy indexing- Retention: High-resolution (1–5m) for 7 days, 1h rollups for 30–90 days, aggregated monthly for 1 year. Deduplicate identical folded stacks and compress with gzip/snappy.- Index metadata in time-series DB or search index (Elasticsearch/ClickHouse/TSDB) for fast queries.

Regression detection (automatic)- Define signals to monitor per-symbol / per-stack: - CPU% (samples normalized to CPU-seconds) - Alloc rate (bytes/sec and samples/sec) - Hot-spot latency proxies (wall-time samples)- Baselines & windows: - Rolling baseline window (e.g., last 7 days, excluding outliers/time-of-day seasonality) and short-term window (last 1h or 3h). - Use normalization by traffic (requests/sec) or CPU capacity to handle load changes.- Detection algorithms: - Statistical test + magnitude threshold: flag if short-term mean > baseline mean * 1.3 (30% relative increase) AND two-sided t-test or bootstrap gives p < 0.01, sustained for N consecutive windows (e.g., 3 windows). - CUSUM or EWMA for faster detection of small shifts. - Use robust measures (median + MAD) to avoid sensitivity to outliers. - Change point detection (Pruned Exact Linear Time / Bayesian) for larger shifts correlated with deploys.- Correlation rules: - Require signal persistence AND deployment correlation: if spike coincides with a deploy, create a high-priority alert and attach profiles pre/post deploy for triage. - Cross-validate with orthogonal metrics (increased latency, error rate, CPU usage) before paging.- False-positive control: - Use minimum absolute threshold (e.g., function must contribute >=0.5% CPU baseline or >=1MB/s alloc change). - Require multiple hosts or replicas affected (>=2) to avoid per-host noise.

Example workflow- Every minute agent pushes gzipped pprof and folded stacks for a service.- Aggregator merges per-host into service-level rollups.- Detection job computes per-symbol CPU% and alloc bytes rate: baseline = median(last 7d same time-of-day); current = last 15m; run EWMA + t-test.- On detection, create an incident with top-diff flamegraphs (before/after), allocation diffs, raw pprof links, and suggested suspect list.

Operational best practices- Start conservative (50–100Hz CPU; coarse allocation sampling), measure overhead (latency, CPU), tune sampling rates.- Ensure symbolization pipeline and SLOs for profile ingest to avoid backlogs.- Integrate with CI/CD to label profiles by release to narrow regressions to commits.- Run occasional higher-sample snapshots (e.g., 400Hz for 30s) on canaries for forensic detail.

This strategy gives continuous, low-cost observability into CPU and allocation regressions, fast automated detection with statistical rigor, and a practical storage/aggregation model for SRE troubleshooting.

HardTechnical

33 practiced

You notice a process consumes a lot of CPU in kernel mode (high 'system' CPU). Provide a diagnostic checklist and concrete commands (perf, ftrace, /proc, sysctl) to determine whether the kernel work is due to intensive syscalls, network stack, filesystem contention, or kernel bug. Include steps to gather evidence for a kernel bug report.

Sample Answer

Start with a reproducible, minimal checklist, then run targeted commands and collect artifacts for a kernel bug report.

Checklist (high level)- Confirm system vs user CPU: is system CPU consistently high or spikey?- Correlate with process PIDs, cgroups, containers.- Narrow to subsystem: syscalls, network, fs, interrupts, or kernel bug.- Collect lightweight evidence first, then deeper tracing if needed.- Preserve logs, perf.data, ftrace, dmesg, /proc snapshots.

Quick triage (lightweight)- Observe top-like metrics:

bash

# show system vs user
top -b -n1 -o %CPU
# per-cpu breakdown (system time)
mpstat -P ALL 1 3
pidstat -ut 1 1   # user/system per pid

Identify offending PID / cgroup

bash

ps -eo pid,ppid,cmd,%cpu,%mem,stime,etime --sort=-%cpu | head
# or for containers
cat /proc/<pid>/cgroup

Check syscall intensity- Count syscalls per process:

bash

# strace short sample (low overhead if short)
strace -c -p <pid> -t -f -qq -o /tmp/strace.out sleep 5
# or use perf syscall counts
sudo perf stat -e syscalls:sys_enter_* -p <pid> sleep 5

Trace kernel hot paths with perf

bash

# system-wide sample profile (symbols)
sudo perf record -F 99 -a -g -- sleep 10
sudo perf report --stdio --call-graph
# per-pid
sudo perf record -F 99 -p <pid> -g -- sleep 10
sudo perf report --stdio

Look for kernel functions: e.g. do_sys_open, vfs_read, tcp_recvmsg, skb_*.

Check interrupts and softirqs

bash

# interrupts
cat /proc/interrupts
# softirq activity
watch -n1 cat /proc/softirqs

Network stack focused

bash

# sockets of process
ss -pan | grep <pid>
# netstat counters
sudo ethtool -S eth0
# per-protocol perf events
sudo perf record -e net:netif_receive_skb -a -- sleep 10

Filesystem contention

bash

# vfs and IO stats
iostat -xz 1 3
sudo blktrace -d /dev/sdX -o - | blkparse -i -
# dentry/inode stats
cat /proc/sys/vm/vfs_cache_pressure
# see blocked tasks
sudo perf probe -x /boot/vmlinux --list  # set probes if needed

detailed ftrace (low-level)

bash

# enable function_graph for pid
sudo bash -c 'echo function_graph > /sys/kernel/debug/tracing/current_tracer'
sudo bash -c "echo <pid> > /sys/kernel/debug/tracing/set_ftrace_pid"
sudo bash -c 'echo 1 > /sys/kernel/debug/tracing/tracing_on'
# run workload briefly, then dump
sudo cat /sys/kernel/debug/tracing/trace > /tmp/trace.txt
# reset
sudo bash -c 'echo 0 > /sys/kernel/debug/tracing/tracing_on'

Use trace-cmd/KernelShark for nicer views.

Collect /proc and sysctl state

bash

# kernel and config
uname -a
cat /proc/version
sysctl -a > /tmp/sysctl.txt
# process specific
cat /proc/<pid>/stack
cat /proc/<pid>/status
# open files
sudo ls -l /proc/<pid>/fd

When suspecting a kernel bug — evidence to gather1. Repro recipe: minimal steps, inputs, exact kernel build, modules loaded, containerization details, and timings.2. Artifacts: - perf.data (sudo perf record ...; sudo perf script > perf.script) - ftrace trace file (/sys/kernel/debug/tracing/trace) - dmesg output (dmesg -T) - /proc/kallsyms or System.map matching kernel - /proc/<pid>/stack and kernel stack traces from perf script - full uname -a, /boot/config-$(uname -r), lsmod - core dump if crashable (enable coredump & reproducible) - steps to reproduce plus any userland programs and inputs3. Map addresses to symbols:

bash

sudo perf script > /tmp/perf.script
# if addresses appear, use addr2line or /proc/kallsyms
addr2line -e /boot/vmlinux-$(uname -r) <hex-address>

4. Try latest stable kernel or upstream patch to see if issue persists; note results.

Filing the report- Provide concise title, exact kernel version and config, hardware, workload, reproduction steps, attach perf.data, ftrace trace, dmesg, System.map, and explain why you believe it's a kernel issue (hot kernel functions, missing syscall/userland cause, reproducible in kernel-only traces). Include regression window if known.

Notes and cautions- Trace for short intervals to limit overhead.- Use perf/ftrace as root.- For production, replicate on staging if possible.- Use tcpdump/pcap if network packets might be involved.

MediumTechnical

33 practiced

Explain a methodology to find a memory leak in a long-running C++ service running on Linux. Include tools and steps using /proc/<pid>/smaps, pmap, valgrind (massif), heaptrack, sampling profilers, and how to interpret differences between RSS and VIRT growth. Also cover when to use heavy tools versus lightweight sampling in production.

Sample Answer

Start by establishing reproducible symptoms and baseline metrics:- Confirm leak by monitoring RSS and VIRT over time (Prometheus/Grafana). RSS steadily rising means resident pages increasing; VIRT rising with stable RSS often indicates address space growth (mmaped allocations, shared libraries, reserved arenas) not necessarily actively resident memory.

Stepwise methodology:1. Lightweight live inspection- pmap -x <pid> for summary, /proc/<pid>/smaps for detailed per-segment RSS/Anon/Private/Shared and swap. Use smaps_rollup for an aggregate. Look for growing Anonymous or Private_Clean/Dirty regions and growing heap/anonymous mappings.- grep VmRSS/VmSize in /proc/<pid>/status for quick checks.

2. Low-overhead sampling in production- Use a sampling profiler (e.g., perf, eBPF tools like bcc/pyperf, or jemalloc's stats if using jemalloc) to attribute allocation hot paths without stopping the process. Heap sampling has low overhead and can point to code paths allocating most memory.

3. Targeted tracing- If using glibc/malloc hooks or tcmalloc/jemalloc, enable their profiling (jeprof/pprof) to get alloc profiles.

4. Offline/heavy-weight analysis (use when you can reproduce in staging or accept downtime)- Valgrind massif for heap snapshots and dominance trees to find long-lived allocations (heavy, slow, high overhead—don't run in production).- heaptrack (lower overhead than valgrind) records allocations with stack traces and lets you diff snapshots over time.- Run in a controlled environment with representative load, capture snapshots at multiple times, and diff to find growing allocation stacks.

Interpreting RSS vs VIRT:- RSS growth = more physical pages resident -> likely real leak.- VIRT growth w/o RSS growth = address space reserved (mmap, large stacks, memory-mapped files) or fragmentation; inspect smaps to find mappings.- Swap growth indicates pages being evicted; check OOM/killing risk.

When to use which:- Production: start with low-overhead metrics, smaps/pmap, sampling profilers, jemalloc/tcmalloc stats, and eBPF. Only use heavy tools (valgrind/heaptrack) in staging or on a reproduceable dump because they change timing and memory layout and are CPU/memory intensive.

Final steps:- Once suspect stack traces identified, fix (free, use pool, reduce retention), add tests and malloc hooks for regression detection, and instrument alerts for sustained RSS growth.

Unlock Full Question Bank

Get access to hundreds of Linux Troubleshooting and Diagnostics interview questions and detailed answers.

Join thousands of developers preparing for their dream job.