Advanced Linux Performance and Services Questions

Advanced administration focused on service lifecycle, process management, and system performance. Topics include deep systemd service management and unit file authoring, dependency ordering and service recovery, process lifecycle and signal handling, cgroups and resource controls, tuning kernel parameters, diagnosing CPU and memory pressure, understanding page cache and swap behavior, out of memory scenarios, I O performance analysis, interpreting load average, and using performance and sampling tools such as top, htop, pidstat, iostat, vmstat, sar, and perf for identifying bottlenecks and implementing mitigations.

EasyTechnical

24 practiced

Given /proc/meminfo shows low free memory but high cached memory, explain whether this necessarily means the system is under memory pressure. What command(s) would you run to validate and what result would confirm healthy caching versus dangerous memory pressure?

HardSystem Design

30 practiced

Hard: Design a capacity planning exercise for a new service that will run on VMs: list required measurements to collect during benchmarking and in production, explain how you model headroom for spikes, and how to convert observed metrics (CPU steal, iowait, context switches) into provisioning decisions.

Sample Answer

Requirements & goals:- Support target SLOs (latency p95/p99, availability), expected QPS, peak concurrency, and cost constraints.- Workload shape: steady vs bursty, stateful vs stateless, IO vs CPU bound.

Measurements to collect during benchmarking (synthetic and realistic):- Throughput (requests/sec), latency distribution (p50/p95/p99), error rates under controlled load curves.- Resource usage per request: CPU user/sys (%) and cores used, memory RSS & working set, disk IOPS and throughput, network bytes/sec and packets/sec.- Kernel metrics: context switches/sec, interrupts/sec.- Storage latency (read/write ms), queue depth.- VM-specific: CPU steal%, iowait%, CPU ready (hypervisor), and per-vCPU utilization.- Startup/shutdown times, connection churn, GC pauses, thread count.- Load profile traces (traffic replay) and garbage collection/RT logs.

Measurements to collect in production (continuous):- Same as above plus service-level: concurrent sessions, queue lengths, backlog, time-to-first-byte, SLO burn rate, incident markers.- Burst indicators: spikes in requests/sec and queue length, tail latencies.- Aggregate histograms and percentiles, plus per-instance metrics for noisy-neighbor detection.

Modeling headroom for spikes:- Define two headroom components: safety (planned cushion) and burst (absorbs tail traffic).- Safety: set margin M% = max(10%, historical 99th-percentile utilization delta). Example: if typical CPU at p95 is 60% and 99th is 78%, choose safety 25%.- Burst: use queuing model (M/M/c or measured queueing) to map arrival-rate spikes to needed service capacity. For short spikes (seconds-minutes), rely on autoscaling with cooldowns; for longer spikes, provision capacity such that p99 latency stays < SLO. Compute required instances N = ceil((baseline QPS * (1 + burst_factor)) / per-VM sustainable QPS_at_target_latency).- Use multi-dimensional headroom: CPU, memory, IOPS, network. Provision according to the tightest resource (bottleneck).

Converting observed metrics to provisioning decisions:- CPU steal%: indicates host oversubscription or noisy neighbors. If steady steal > 2-5% or spikes correlate with latency, migrate VM, choose less packed hosts, or increase instance count. Provision decision: reduce target per-VM CPU utilization so expected CPU_util + steal_margin < target (e.g., target <= 70%).- iowait% & storage latency: when iowait rises and disk latency or queue depth increases above thresholds (e.g., read latency > 20ms or queue depth > baseline*3), scale vertically (bigger disk/IOPS) or horizontally if workload is parallelizable. Map increased iowait to effective throughput loss: measure QPS per iowait bucket and choose instances to restore throughput.- Context switches/sec: high rates + increased sys CPU imply contention or excessive threading. If context switches exceed baseline by X and p95 latency degrades, consider increasing CPU cores per VM (to reduce contention) or reducing thread concurrency. Provisioning rule: if context_switches/sec per core > threshold and sys_cpu% > 15%, add cores or instances.- Composite decision process: 1. Detect which resource is saturated via utilization + latency (e.g., CPU % high, CPU ready/steal high). 2. Translate target headroom into capacity increment: delta_N = ceil(current_demand / per-VM_capacity * (1 + required_headroom)) - current_N. 3. Prefer remediation with lowest cost and fastest effect: autoscale horizontally first for stateless, vertical scale for single-threaded CPU-bound, storage resizing for iops-bound. 4. After change, validate by observing that targeted metrics (latency, steal, iowait) return within target ranges within a defined window.

Example calculation:- Observed p95 latency SLO breached at 1200 QPS with current N=10 VMs => per-VM sustainable QPS = 120. Measured CPU p95 = 75%. Desired CPU headroom = 25% => target per-VM CPU p95 = 50% → expected per-VM QPS at 50% ≈ 80. Required N = ceil(1200 / 80) = 15.

Best practices:- Automate benchmarking with traffic replay and chaos tests (noisy neighbor, disk saturation).- Maintain per-VM capacity profiles (QPS vs latency vs resource usage).- Use correlated alerts (latency + steal or iowait) to trigger scaling policies and runbooks.- Revisit models with periodic load tests and post-incident analysis.

EasyTechnical

36 practiced

Define cgroups (control groups) at a high level and explain the difference between cgroups v1 and cgroups v2. As an SRE, why would you prefer cgroups v2 for new deployments?

HardSystem Design

31 practiced

Hard: Design a robust automated remediation system for a fleet that detects when a service exceeds CPU quota in its cgroup for sustained periods and automatically moves it to a lower priority slice or scales out replicas. Define detection logic, safe remediation steps, and how to avoid cascading remediation causing more instability.

Sample Answer

Requirements & constraints:- Detect sustained cgroup CPU quota breach per service instance (not transient spikes).- Remediate safely: reduce priority (move to lower slice) or scale out replicas.- Avoid cascading remediation (throttling, oscillation, resource starvation).

High-level design:- Metrics pipeline: node agent (e.g., systemd/cgroup reader or cadvisor) -> metrics store (Prometheus) -> alerting & decision engine (remediator service).- Remediator has a state store for actions, cooldowns, and global view (service-level replicas, node capacity).

Detection logic:- Sample cgroup cpu.stat and cpu.max (or cpu.cfs_quota* & cpu.cfs_period*).- Compute utilization = used_quota / allowed_quota over sliding windows.- Trigger if utilization > 95% averaged over 3 consecutive windows (e.g., 30s windows => 90s sustained) AND tail-latency or error-rate SLOs degrade.- Cross-check with node-level metrics (overall load, other pods) to avoid misattribution.

Safe remediation steps (ordered, incremental, idempotent):1. Mark instance as “candidate” and annotate event store.2. Soft remediation: throttle affinity/CPU shares — move to lower systemd slice (nice/CPUshares) for 30s.3. If still high after cooldown, request orchestration action: - If stateless and autoscalable: scale out one replica (k8s HPA/CustomController). - If stateful: reduce CPU limit only after validation/coordination with owner.4. If scaling out, wait for new pod readiness and re-evaluate load distribution before further action.5. If remediation fails or causes errors, rollback by restoring original slice/limits.

Avoid cascading & instability:- Global rate limit: X remediations per minute per cluster/team.- Circuit breaker per service: if >N remediations in T minutes, pause automated actions and notify on-call.- Dependency-aware checks: don’t move many replicas of same service on same node—spread decisions across nodes.- Cooldown windows and exponential backoff per instance.- Simulation/impact check: before moving slices, ensure node has spare CPU shares; before scaling, ensure cluster has capacity (or schedule to other nodes).- Safe defaults and human-in-loop for high-risk services (stateful DBs).

Observability & safety:- Auditable events, playbooks triggered, runbook links in alerts.- Canary: start with low-risk namespaces/services, collect metrics and tweak thresholds.- Unit + integration tests: simulate cgroup metrics and validate remediator behavior.

Trade-offs:- Conservative thresholds reduce false positives but lengthen remediation. Aggressive actions reduce outage time but risk cascading; use multi-step incremental approach to balance.

This design balances automated corrective action with safeguards to prevent bulk destabilization while remaining observable and reversible.

HardTechnical

33 practiced

Hard: Provide a checklist and sequence of commands to produce a comprehensive performance packet for a single host after a reproducible 2-minute performance degradation window: include logs, perf data, iostat, sar, vmstat, top, and network stats. Explain how you ensure privacy/security when collecting traces from production.

Sample Answer

Situation: You have a reproducible 2-minute degradation window on one host and must produce a comprehensive performance packet (logs + perf + iostat + sar + vmstat + top + network) suitable for postmortem and safe for transport.

Checklist (before capture)- Ensure disk space for captures (df -h /var/tmp)- Notify stakeholders; get approval if sensitive data- Create output dir with restrictive perms: /var/tmp/perf-packet-$(date +%s)- Set UTC timestamp baseline

Capture sequence (start ~30s before window, cover 30s after; adjust intervals to 1s-5s):

1) Create folder and lock permissionssudo mkdir -p /var/tmp/perf-packet-$(date -u +%Y%m%dT%H%M%SZ) && sudo chmod 700 $_cd $_

2) System metadatauname -a > host.infolsb_release -a >> host.info 2>/dev/nullcat /proc/cmdline > proc.cmdlineip addr show > ip.addr

3) Centralized logs (journalctl / app logs)sudo journalctl --since "-1 minute" --until "+3 minute" --output=short-precise > journal.window.log# For file logs (nginx, app)sudo grep -a -n "" /var/log/myapp/*.log --since="$(date -u -d '1 minute ago' +%Y-%m-%dT%H:%M:%S)" > app.window.log || true(If system lacks --since support, use tail -n with known line counts or collect full file and slice by timestamps.)

4) perf (CPU hotspots)sudo timeout 180 perf record -F 99 -a -g -- sleep 180sudo perf report --stdio > perf.report.txtsudo perf script > perf.raw.script

5) iostat, sar, vmstat (sampling every 1s)sudo timeout 180 sh -c 'iostat -xz 1 > iostat.window.log' &sudo timeout 180 sh -c 'vmstat 1 > vmstat.window.log' &sudo timeout 180 sh -c 'sar -n DEV 1 180 > sar.net.dev.log; sar -u 1 180 > sar.cpu.log; sar -b 1 180 > sar.io.log' &

6) top (batch)sudo timeout 180 top -b -d 1 -n 180 > top.window.log &

7) Network stats (ss, netstat, tcpdump short capture)sudo ss -tunap > ss.before.logsudo timeout 120 tcpdump -n -s 0 -w tcpdump.window.pcap 'port 80 or port 443' &# Alternatively limit capture to relevant iface and ring buffer: sudo timeout 120 tcpdump -i eth0 -W 10 -C 10 -w tcpdump.window.pcap ...

8) dmesgsudo dmesg --ctime > dmesg.window.log

9) Filesystem and process list snapshotssudo lsblk -f > lsblk.logsudo ps auxwww --sort=-%cpu | head -n 200 > ps.top200.log

10) Post-capturewait# Compress and hashsudo tar czf perf-packet.tgz . && sha256sum perf-packet.tgz > perf-packet.tgz.sha256# Restrict accesssudo chown root:ops perf-packet.tgz && sudo chmod 640 perf-packet.tgz

Privacy & Security Controls- Principle of least privilege: run captures with sudo minimal where needed and store in dir with 700 perms.- Avoid PII: redact or exclude application logs where PII likely lives. Use scripts to scan and mask emails, IPs, or tokens (e.g., sed -E to replace emails with [redacted]).- Minimization: capture only necessary time window and required files; do not collect entire home directories or databases.- Hash and audit: produce SHA256 and note who created packet (audit.log entry).- Secure transfer: use scp/sftp to a secured collector using key-based auth to an approved server; or upload to secure artifact store (S3 with SSE, restricted IAM). Example:scp perf-packet.tgz user@collector.example.com:/secure-path/- Retention & deletion: document retention period; delete local copy after verified ingest (shred -u perf-packet.tgz).- Legal/compliance: follow company policy, get approvals for collecting user data, and involve privacy/infosec if necessary.

Notes and best practices- If perf not allowed in prod, use eBPF tools (bcc, perf-map-agent) or flamegraphs off-host.- Use timestamps in filenames and include UTC everywhere.- Automate this sequence as a script that takes start/duration and includes redaction flags.

Unlock Full Question Bank

Get access to hundreds of Advanced Linux Performance and Services interview questions and detailed answers.

Join thousands of developers preparing for their dream job.