Production Incident Response and Diagnostics Questions
Covers structured practices, techniques, tooling, and decision making for detecting, triaging, mitigating, and learning from failures in live systems. Core skills include rapid incident triage, establishing normal baselines, gathering telemetry from logs, metrics, traces, and profilers, forming and testing hypotheses, reproducing or simulating failures, isolating root causes, and validating fixes. Candidates should know how to choose appropriate mitigations such as rolling back, applying patches, throttling traffic, or scaling resources and when to pursue each option. The topic also includes coordination and communication during incidents, including incident command, stakeholder updates, escalation, handoffs, and blameless postmortems. Emphasis is also placed on building institutional knowledge through runbooks, automated diagnostics, improved monitoring and alerting, capacity planning, and systemic fixes to prevent recurrence. Familiarity with common infrastructure failure modes and complex multi system interactions is expected, for example cascading failures, resource exhaustion, networking and deployment issues, and configuration drift. Tooling and methods include log analysis, distributed tracing, profiling and debugging tools, cross system correlation, and practices to reduce mean time to detection and mean time to resolution.
HardTechnical
82 practiced
A primary database node failed and triggered an automatic failover to a secondary. After failover, cache eviction storms and retry amplification caused multiple services to experience cascading failures and partial outages. Describe an RCA approach (what logs/metrics/traces to examine), how to reproduce safely, and concrete design changes to prevent recurrence (both infra and application-level).
Sample Answer
RCA approach — what to collect and examine- Timeline & correlation: collect cluster events, leader-election logs, and orchestration events (K8s events, etcd/consul logs) with precise timestamps.- Metrics (pre/post): cache hit/miss rates, eviction rates, TTL expirations, connection counts, RPS per service, error rates, latency P50/P95/P99, downstream queue depths, DB replication lag.- Traces: distributed traces for requests that failed or retried (span trees showing retries -> cascading calls).- Application logs: retry logic, backoff parameters, circuit-breaker events, connection errors.- Cache internals: eviction logs, memory pressure, eviction algorithms (LFU/LRU) metrics, client-side keyspace hotness.- Infrastructure: network errors, DNS resolution, load-balancer health checks, instance scale events.- Retry amplification signals: request fan-out graphs, simultaneous retry storm timestamps, backoff jitter absence.- Alert/incident history: prior similar events or config changes (deploys, failover tests).How to reproduce safely- Create an isolated staging environment mirrored to production traffic patterns (traffic shadowing or synthetic traffic generator).- Simulate primary failure: trigger controlled failover (shutdown or network partition) during representative load.- Inject cache eviction: artificially reduce cache capacity or throttle cache nodes to cause evictions; monitor miss spike.- Introduce retry conditions: temporarily make DB calls return 5xx to test client retry behavior.- Run canary tests first, apply chaos experiments incrementally, and have kill-switches and circuit-breaker toggles available.Concrete design changes to prevent recurrenceInfrastructure-level- Harden failover: ensure shorter, deterministic failover with warm standbys and pre-warmed connections; automate connection drain and gradual traffic shift.- Cache resilience: add multi-tier caching (local + shared), consistent hashing to reduce key re-sharding, larger headroom for memory, and eviction prioritization for critical keys.- Rate-limited reconnection: client libraries should stagger reconnects to cache on failover (exponential backoff + jitter).- Capacity & autoscaling: scale cache and downstream services proactively during failover windows; autoscale on miss-rate and error-rate triggers.Application-level- Defensive retries: implement idempotency, bounded retries, exponential backoff with jitter, and capped concurrent retries per caller.- Circuit breakers and bulkheads: per-downstream circuit breakers, request quotas and thread/pool isolation to prevent resource exhaustion.- Cache warming and grace: on failover, serve stale-but-valid data (stale-while-revalidate) and implement cache population strategies for hot keys.- Observability improvements: add SLO-driven alerts for miss-rate surge, retry amplification, and eviction storms; add dashboards cornering cache key hotness and retry fan-out.- Chaos + runbooks: schedule regular chaos testing of failover scenarios and maintain clear operational playbooks and automations to throttle traffic or flip circuit breakers quickly.Key rationale: reduce synchronized behavior (stagger/backoff/jitter), preserve critical cached state across failover, limit blast radius via isolation and quotas, and improve visibility so operators can act before cascades become outages.
EasyTechnical
84 practiced
Explain the difference between a root cause, contributing factor, and symptom. Provide a concise example from an incident where addressing a symptom made the problem reappear later, and explain how you would dig deeper to find the true root cause.
Sample Answer
Root cause, contributing factor, and symptom are different layers in incident analysis:- Symptom: an observable effect (what you see). Example: high HTTP 503 rate.- Contributing factor: something that increased likelihood or severity but didn’t by itself cause the incident (what made it worse). Example: absence of circuit-breaker, outdated client retries, or degraded cache hit-rate.- Root cause: the underlying defect that, if fixed, prevents recurrence (the true origin). Example: a memory leak in a newly deployed service causing thread pool exhaustion.Example incident (concise): We rolled back a service after an alert for high 503s. The 503s stopped (symptom resolved) but reappeared days later. Quick rollback hid the immediate failure but didn’t address the memory leak that only surfaced under specific load patterns—so the problem returned.How I’d dig deeper to find root cause:- Capture timeline and correlate metrics (CPU, memory, GC, thread count, queue lengths), logs, traces around failures.- Reproduce in staging with similar load, using load tests and traffic replay.- Use "Five Whys" and fault-tree analysis to separate symptoms vs. causes.- Inspect recent deployments/config changes, dependency versions, and resource limits.- Identify contributing factors (e.g., missing circuit-breaker, aggressive retries) and remediate them as mitigations.- Implement permanent fix (patch leak, add limits), add targeted monitoring/alerting for leading indicators, and document RCA and runbook.
EasyTechnical
62 practiced
Describe 'headroom' and 'safety margin' in capacity planning. Given 95th percentile request rate of 20k RPS and a business rule to maintain at least 30% headroom, outline how you would compute required cluster resources and what monitoring you would use to ensure the headroom remains available during spikes.
Sample Answer
Headroom vs safety margin (brief):- Headroom: the operational spare capacity you keep available for traffic spikes so you don’t run at saturated capacity (expressed as % of capacity kept free). - Safety margin: extra reserved capacity beyond normal headroom to cover failures, degradation, or measurement error (often a small fixed number of nodes or %).Compute required cluster resources:1. Translate business rule into target usable capacity: - 95th percentile steady load = 20,000 RPS. - Required usable capacity = 20,000 / (1 - 0.30) = 28,571 RPS.2. Determine per-node (or per-pod) sustainable throughput (measured from benchmarks — e.g., a pod handles 5,000 RPS at p95 latency SLO). - Nodes needed = ceil(28,571 / 5,000) = 6 nodes/pods.3. Apply safety margin: reserve e.g. +1 node or +10% to tolerate a node failure or perf regression → provision 7 nodes (or 6 + 10% = 7).4. Ensure redundancy zones: spread across AZs and keep capacity to tolerate an AZ loss if required (may increase node count).Monitoring and controls to maintain headroom:- Real-time metrics: - Request rate (RPS) at service and per-node level, plus 95th/99th latency and error rates. - CPU, memory, thread/connections, GC pause times, and queue/backlog depth. - Pod/node ready count and pod restart rate.- Derived headroom metric: headroom_pct = 1 - (current_RPS / provisioned_capacity). Alert when headroom_pct < 30% (and a warning at 40%).- Autoscaling: - Horizontal Pod Autoscaler (or cluster autoscaler) driven by RPS or custom metrics (e.g., requests-per-pod) with conservative cooldowns and predicted scale-up steps to avoid flapping. - Use target-based scaling that ensures scaling completes before headroom breach (scale-up threshold lower than 30%).- Synthetic and chaos tests: - Regular synthetic load tests that validate capacity and SLOs. - Periodic failure drills (simulate node loss) to verify safety margin.- Dashboards & alerts: - Dashboard showing provisioned vs used capacity, headroom_pct, latency SLOs. - Alerts: headroom breach, scaling failures, increased error budget burn.- Continuous validation: - Rebenchmark per-pod throughput after releases. - Adjust per-node numbers when performance changes.This approach ties the business headroom requirement to measurable capacity, adds a safety margin for failures, and uses monitoring + autoscaling + testing to keep headroom available during spikes.
HardSystem Design
129 practiced
Design a multi-region disaster recovery plan for a write-heavy service requiring RPO ≤ 5 minutes and RTO ≤ 15 minutes. Discuss replication strategy (sync vs async), consistency model, failover orchestration, DNS and traffic cutover, data reconciliation after failback, and trade-offs between latency and data safety.
Sample Answer
Requirements (clarify): write-heavy service, RPO ≤ 5 min, RTO ≤ 15 min, multi-region active–passive (or active–active?) — given write-heavy and tight RPO, I propose active–passive with a warm standby in secondary region.High-level architecture:- Primary region handles reads/writes. Secondary region runs hot standby application nodes and read replicas; accepts no writes until failover.- Use a durable transactional store (e.g., primary DB cluster + WAL shipping), distributed queue for critical write events, and change-data-capture (CDC).Replication strategy:- Hybrid: synchronous intra-region (for durability/consistency) and near-synchronous async cross-region using streaming replication with a max lag SLA. Use WAL shipping + CDC with a target lag < 5 minutes; implement write-ahead log archival to object storage as insurance.- Pure sync cross-region would guarantee RPO=0 but increases write latency—unacceptable for write-heavy workloads. Async with tight monitoring meets RPO.Consistency model:- Primary offers strong consistency (linearizable) within region. Cross-region reads from standby are read-your-writes only after failover. During failover, promote secondary to primary and apply pending WALs—some recent writes (≤5 min) might be delayed if replication lag exceeded.Failover orchestration:- Automated runbook via orchestration system (e.g., Terraform + orchestration scripts, or Kubernetes operators + vaulted credentials) triggered by health checks and on-call confirmation.- Steps: detect outage → stop writes at edge (global traffic manager) → ensure WALs/CDC caught up to target point → promote replica → run schema checks/migrations → smoke tests → open writes.DNS and traffic cutover:- Use Global Traffic Manager (GTM) / DNS with low TTL (60s) + health-based routing, or preferably global load balancer with anycast. Cutover: update GTM to send traffic to secondary, or flip BGP/anycast announcements. Use connection draining, retry logic at clients, and short TTL to meet RTO. Ensure TLS certs and secrets available.Data reconciliation after failback:- After primary recovers, treat it as standby: replicate from current primary (formerly secondary) using base backup + WAL to avoid split-brain. Reconcile divergent writes via CDC stream reconciliation, application-level idempotency, or conflict resolution policies (last-write-wins only with care). Run consistency checks and backfill missing events from archival logs.Monitoring, safety and trade-offs:- Monitor replication lag, commit latency, queue depth, and end-to-end RPO estimators. Set automated rollback thresholds.- Trade-offs: synchronous cross-region replication gives zero data loss but high write latency. Async lowers latency but requires careful RPO monitoring and increases complexity for reconciliation. Active–active reduces RTO but needs conflict resolution and stronger coordination (consensus protocols) which add latency.- Operational controls: runbooks, rehearsals (chaos tests), periodic failover drills, and alerts tied to SLOs.This design balances write performance and RPO/RTO by using near-synchronous async replication, automated orchestration, tight monitoring, and robust reconciliation to minimize data loss while keeping latency acceptable.
EasyTechnical
70 practiced
A page fires: "service-x 5xx error rate = 15% over 2 minutes". Walk me through the first 10 minutes of triage as an on-call SRE. What commands and telemetry do you immediately gather, what hypotheses do you form, and what quick mitigations might you take while investigating?
Sample Answer
Immediate objective: contain user impact, gather facts, and decide next action.Minute 0–2 — Triage kickoff & context- Acknowledge alert, open incident channel (Slack/PagerDuty), set severity, notify stakeholders.- Quick facts to collect: alert window, total requests, affected endpoints, regions, deploys in last 30–60m, error types (5xx subcodes).- Commands/queries: - kubectl get pods -n svc-namespace -o wide (or docker/nomad commands) - kubectl logs <pod> --tail=200 | grep -i "ERROR" - prometheus: rate(http_requests_total{service="service-x",status=~"5.."}[2m]) - Grafana dashboard: error rate, latency, throughput, CPU/memory, GC - traceroute / curl to specific endpoints to reproduce (curl -v -I https://service-x/health) - Check recent deploys: gitops/CI pipeline, kube events: kubectl get events -n svc-namespace --sort-by=.metadata.creationTimestamp - Check upstream/downstream: dependency error metrics, DB errorsMinute 2–6 — Form hypotheses (prioritized)- Recent deploy introduced a bug causing 5xx — correlate deploy time with spike.- Resource exhaustion (OOM, file descriptors, thread pool) causing failures under load.- Upstream dependency failing (db, auth, payment) returning 5xx.- Traffic spike or malformed requests (DDOS or bad client).- Config change (feature flag, env var) causing misbehavior.Minute 6–10 — Quick mitigations & next steps- If deploy suspected: roll back or scale down new revision (kubectl rollout undo deployment/service-x).- If resource exhaustion: scale replicas temporarily (kubectl scale deployment/service-x --replicas=N) or increase CPU/memory; restart unhealthy pods.- If upstream failing: enable graceful degradation / circuit breaker (route traffic to fallback) and throttle outgoing calls.- If traffic spike or bad clients: apply rate-limiting, WAF rules, or block offending IPs at edge.- If isolate: take subset of traffic to a stable canary or disable feature flag.- Continue collecting logs/traces for root cause and update incident channel every 5 minutes.Key notes:- Prioritize user impact mitigation over deep debugging.- Keep runbook, SLO/error budget visible; document actions for postmortem.
Unlock Full Question Bank
Get access to hundreds of Production Incident Response and Diagnostics interview questions and detailed answers.
