Production Incident Response and Diagnostics Questions
Covers structured practices, techniques, tooling, and decision making for detecting, triaging, mitigating, and learning from failures in live systems. Core skills include rapid incident triage, establishing normal baselines, gathering telemetry from logs, metrics, traces, and profilers, forming and testing hypotheses, reproducing or simulating failures, isolating root causes, and validating fixes. Candidates should know how to choose appropriate mitigations such as rolling back, applying patches, throttling traffic, or scaling resources and when to pursue each option. The topic also includes coordination and communication during incidents, including incident command, stakeholder updates, escalation, handoffs, and blameless postmortems. Emphasis is also placed on building institutional knowledge through runbooks, automated diagnostics, improved monitoring and alerting, capacity planning, and systemic fixes to prevent recurrence. Familiarity with common infrastructure failure modes and complex multi system interactions is expected, for example cascading failures, resource exhaustion, networking and deployment issues, and configuration drift. Tooling and methods include log analysis, distributed tracing, profiling and debugging tools, cross system correlation, and practices to reduce mean time to detection and mean time to resolution.
HardSystem Design
64 practiced
Propose a chaos engineering program to validate incident readiness for network partitions and persistent storage faults. Define experiment goals and hypotheses, steady-state determination, safe blast radius, automation tooling, required observability to measure impact, rollback strategies, and how to incorporate observed failures into runbooks and capacity plans.
Sample Answer
**Clarify scope & objectives**- Goal: Prove our teams and systems can detect, mitigate, and recover from network partitions and persistent storage faults with RTO/RPO targets met.- Hypotheses: (1) Services degrade gracefully under partial network split; (2) Storage tier fails to meet durability but system falls back to replicas/readonly modes without data loss.**Steady-state definition**- Baseline: 99.95% API success rate, <100ms p95 latency, replication lag <5s, no error budget consumed in last 24h.**Safe blast radius**- Start in staging → canary cluster → single AZ production (1 app shard, 5% traffic) → expand.- Use service-level traffic steering and circuit-breakers; kill-switch to abort experiment.**Experiment design & automation**- Tools: Chaos Mesh/Gremlin/Netflix Chaos Monkey + custom scripts for persistent-storage faults (fs-sync pause, disk latency injection, IO errors via dmsetup).- Network partition: split nodes via iptables/tc to simulate asymmetric loss and latency.- Persisted faults: inject IO errors, simulate full disk, corrupt LVM snapshot.**Observability**- Metrics: success rate, p95/p99 latency, replication lag, queue depth, disk IO/util, fs error rates.- Traces: distributed tracing spans for cross-service calls.- Logs: structured error logs with correlation IDs.- Alerting: experiment-specific alarms and experiment dashboard.**Rollback & safety**- Immediate controls: automated rollback playbook: remove iptables rules, revert dmsetup, switch traffic back to healthy pool.- Automated health checks post-rollback with staged traffic ramp.**Postmortem & operationalization**- Capture failures into runbooks: reproducible steps, detection thresholds, mitigation commands, ownership.- Update capacity plans: quantify additional replication throughput, write IOPS, buffer sizes needed; run load tests to validate.- Continuous improvement: schedule periodic re-runs, track mean time to detect/mitigate, train on runbooks during game days.This program emphasizes incremental risk, automated tooling, robust observability, and closing the loop into runbooks and capacity planning.
MediumTechnical
67 practiced
Describe a safe rollback procedure for a microservices deployment when a recent release causes failures. Include steps for canary vs full rollback, data compatibility and schema checks, dependency mapping, the sequencing order to avoid cascading regressions, and verification checks before declaring recovery.
Sample Answer
**Situation & goal**As a systems engineer I’d execute a safe, reversible rollback minimizing user impact and data risk when a release causes failures.**High-level steps**- Pause new traffic (feature flags/load balancer) and stop further rollouts.- Triage quickly to decide canary vs full rollback based on blast radius and metrics.**Canary rollback**- Isolate failing canary instances; revert only that subset to previous image/version.- Re-route traffic back to healthy stable instances via LB weights.- Monitor key metrics (error rate, latency, SLOs) for 15–30m before expanding.**Full rollback**- If failures are systemic, coordinate rolling rollback service-by-service using blue/green or rolling update to previous stable tag.**Data compatibility & schema**- Verify migration rollbacks are safe: if schema was forward-only, application rollback must be accompanied by compatible reads/writes.- Prefer backward-compatible DB migrations; if irreversible migrations ran, halt rollback and execute compensating migration scripts or read-only maintenance mode.**Dependency mapping & sequencing**- Use dependency graph: rollback downstream consumers first, then core shared services, to avoid cascading regressions.- Example order: edge/API gateways → auth/session → service A (consumer) → service B (producer) → shared DB.- Lock CI/CD pipelines and notify dependent teams.**Verification before declaring recovery**- Smoke tests, business-critical end-to-end flows, synthetic transactions, and canary user checks.- Metrics: error rate back to baseline, latency within threshold, no new 5xx, DB inconsistency checks.- Post-mortem ticket with root cause, lessons, and plan to harden CI/CD/migrations.
MediumTechnical
90 practiced
During an incident you observe unusual outbound traffic and unexpected process activity on several hosts. Describe how you would differentiate a security breach from an operational failure, the immediate containment steps, evidence collection best practices for forensic analysis, and when and how to escalate to the security/forensics team.
Sample Answer
**Situation assessment — differentiate breach vs operational failure**- I first compare indicators: alert sources (IDS, EDR, network), process hashes, unusual outbound destinations, timing, and configuration changes. - Breach indicators: unknown binaries, signed-but-unexpected processes, C2 domains/IPs, lateral movement signs, credential anomalies. - Operational failure signs: recent patching, misconfigured jobs, legitimate backups/replication spikes, scheduled cron tasks. I query process owners, change logs, and deployment pipelines to rule out benign causes.**Immediate containment steps**- Isolate affected hosts from network (apply host-based firewall rules or move to quarantined VLAN) while preserving access for response.- Block suspicious external IPs/DNS at perimeter and endpoint level.- Disable compromised accounts/keys and rotate affected credentials.- Suspend suspicious processes only if it won’t destroy volatile evidence; prefer snapshotting first.**Evidence collection best practices**- Preserve volatile data: collect memory dumps, running process lists, open network connections, and active user sessions.- Acquire disk images or snapshots using trusted tools; record hashes and timestamps.- Gather logs: system, application, cloud audit, firewall, and EDR telemetry. Export EDR/IDS data centrally.- Maintain chain of custody: who, when, tools used; writeable logs stored off-host.- Use read-only mounts and avoid rebooting hosts unless necessary.**Escalation criteria and process**- Escalate immediately to security/forensics if you see confirmed C2, data exfiltration, lateral movement, or multiple host compromise. - Provide a concise briefing: timeline, affected assets, evidence collected, containment actions, and recommended next steps. - Hand off artifacts (images, memory, logs) with chain-of-custody notes and remain available to support forensics and remediation.
HardTechnical
69 practiced
You observe p99 latency spikes in a critical distributed transaction path. Describe a detailed methodology using traces to quantify each span's contribution, detect queuing versus compute time, visualize with heatmaps and waterfall charts, correct for sampling bias, and propose targeted fixes with confidence levels based on data.
Sample Answer
**Situation & goal**I need to diagnose p99 latency spikes on a distributed transaction path using traces, quantify each span’s contribution, separate queuing vs compute time, visualize results, correct sampling bias, then propose targeted fixes with confidence scores.**Methodology**- Instrumentation & data collection - Ensure spans include timestamp, duration, service/node, host, queue_start/queue_end (or enqueued_at/dequeued_at) tags; enable full traces for p99 windows (temporarily raise sampling).- Quantify span contribution - For each trace compute total trace_time and per-span contribution = span_duration / trace_time. - Aggregate contributions across p99 traces (top 1%) and report median/90/99 contribution per span.- Detect queuing vs compute - Define compute_time = span_duration - (queue_end - queue_start) if queue timestamps exist. - If not available, infer queueing via concurrency correlation: high host cpu idle + many inflight requests => queuing; use service-level backlog metrics and span wait events.- Visualization - Waterfall charts: sort p99 traces by total time; show stacked spans with color split for queue vs compute. - Heatmaps: matrix of span_name vs time-of-day showing p99 frequency/intensity; alternate heatmap with contribution percentiles. - Include distributions (CDFs) for compute vs queue times.- Correct sampling bias - Reweight sampled traces by inverse sampling probability per service and time bucket; bootstrap to get confidence intervals. - Cross-validate with aggregate metrics (latency histograms, percentiles from metrics pipeline).- Propose fixes & confidence - For each top contributor produce action, e.g. increase worker threads (low risk, medium confidence), optimize DB query with index (high impact, medium-high confidence), add backpressure/load-shedding (high confidence for queuing), cache hotspots (medium confidence). - Assign confidence by evidence: direct span dominance + low variance = high; inferred queuing with indirect signals = medium; speculative = low.- Outcome tracking - Deploy targeted change behind feature flag/traffic percentage, monitor p99, span contributions, and validate with A/B.Example: if DB-span accounts for 60% median of p99 and has negligible queue tags, prioritize query plan and index; confidence high if flamegraphs + DB slow query logs match.This gives a measurable, visual, bias-corrected path from diagnosis to action with evidence-based confidence.
MediumTechnical
80 practiced
You receive alerts of a sudden CPU usage spike across a pool of application servers behind a load balancer. Describe a structured triage plan to identify whether this is caused by sudden traffic increase, a recent deploy, runaway threads, garbage collection issues, kernel-level factors, or a third-party dependency. Specify commands, metrics, and a timeframe for each step.
Sample Answer
**Quick confirmation (0–5 min)** - Goal: verify alert and blast radius. - Commands/metrics: check monitoring dashboard (Prometheus/Grafana) for CPU% across LB pool, request rate (RPS), error rate, latency. For AWS/ELB: view CloudWatch CPUUtilization, RequestCount. For Kubernetes: `kubectl top pods --all-namespaces`. - Action: mark affected hosts, mute noisy alerts if needed.**Traffic vs load (0–10 min)** - Goal: detect sudden traffic surge or traffic anomaly. - Commands/metrics: RPS, 95/99th latency, connection count. Nginx/HAProxy stats, `curl` health endpoints, `aws cloudwatch get-metric-statistics` or Grafana panels. - If RPS spike correlates with CPU, scale out or enable autoscaling.**Recent deploy check (0–15 min)** - Goal: rule out bad release. - Commands/metrics: deployment timeline from CI/CD (Jenkins/GitLab), `kubectl rollout history`, Git commit hashes, correlate deploy timestamps with CPU spike. - Action: if correlated, roll back or deploy fix.**Per-host process triage (5–20 min)** - Goal: identify runaway threads/processes. - Commands: `top -b -n1 | head`, `ps -eo pid,ppid,cmd,%cpu --sort=-%cpu | head`, `pidstat -u 1 5`, `htop`. - Note PID(s) with high CPU.**JVM/Garbage collection (10–30 min, if JVM app)** - Goal: GC causing CPU. - Commands/metrics: check GC logs (`gc.log`), `jstat -gcutil <pid> 1000 5`, `jstack <pid>` for stuck threads, `jmap -heap <pid>`. Monitor GC pause times, young/old gen usage. - Action: adjust GC settings, restart instance if needed.**Kernel/OS level (10–30 min)** - Goal: detect kernel preemption, interrupts, I/O wait. - Commands/metrics: `vmstat 1 5`, `iostat -x 1 5`, `sar -q`, `mpstat -P ALL 1 3`, `dmesg | tail -n 200` (looking for OOM, hardware errors), `perf top` if available. - Look for high %wa, IRQ storms, OOM killer.**Network / third-party dependency (10–30 min)** - Goal: external calls causing CPU (timeouts, retries). - Commands/metrics: trace outbound request rates, `ss -tunp`, `netstat -anp`, `tcpdump -nn -s0 -c 100 host <third-party-ip> and port <port>`, application logs showing slow dependency. Correlate increased latency/errors with CPU. - Action: rate-limit/backoff or circuit-breaker.**Mitigation & next steps (0–60 min)** - Short-term: scale out, put servers in drain, restart offending process, roll back deploy. - Long-term: add better metrics (per-thread CPU, GC metrics, tracing), automated runbooks, capacity tests. Throughout: document timestamps, commands run, and revert actions. Prioritize safety: if unsure, drain nodes before restarting.
Unlock Full Question Bank
Get access to hundreds of Production Incident Response and Diagnostics interview questions and detailed answers.
