Root Cause Analysis and Diagnostics Questions

Systematic methods, mindset, and techniques for moving beyond surface symptoms to identify and validate the underlying causes of business, product, operational, or support problems. Candidates should demonstrate structured diagnostic thinking including hypothesis generation, forming mutually exclusive and collectively exhaustive hypothesis sets, prioritizing and sequencing investigative steps, and avoiding premature solutions. Common techniques and analyses include the five whys, fishbone diagramming, fault tree analysis, cohort slicing, funnel and customer journey analysis, time series decomposition, and other data driven slicing strategies. Emphasize distinguishing correlation from causation, identifying confounders and selection bias, instrumenting and selecting appropriate cohorts and metrics, and designing analyses or experiments to test and validate root cause hypotheses. Candidates should be able to translate observed metric changes into testable hypotheses, propose prioritized and actionable remediation steps with tradeoff considerations, and define how to measure remediation impact. At senior levels, expect mentoring others on rigorous diagnostic workflows and helping to establish organizational processes and guardrails to avoid common analytic mistakes and ensure reproducible investigations.

HardTechnical

22 practiced

You are asked to implement organization-level gating of rollouts based on SLOs and RCA outcomes. Propose a governance model that includes automated enforcement (CI/CD gates, feature-flag kill-switches), roles and responsibilities (who can override), escalation paths, and metrics that measure policy effectiveness. Explain how to balance agility with safety.

Sample Answer

Situation: We need organization-level gating of rollouts so SLO breaches and RCA findings prevent unsafe changes while preserving developer velocity.

Governance model (high level):- Policy: “If service SLO burn-rate > X or recent RCA severity ≥ P1 within Y days, block incremental rollouts beyond canary.” Policy encoded in policy-as-code (Rego/OPA) and stored in git.

Automated enforcement:- CI/CD gates: pipeline step queries policy engine + SLO datastore (Prometheus/Thanos) and RCA ledger. If policy denies, pipeline fails with actionable reason and remediation link.- Feature-flag kill-switches: every rollout must attach a kill-switch that supports instant global disable via API + signed audit. Orchestration (LaunchDarkly/FF system) integrated with incident playbooks to auto-disable when alert threshold crossed.- Auto-rollback: Canary monitors metrics; on threshold breach, CI triggers automated rollback and notifies on-call.

Roles & responsibilities:- SREs: define SLOs, run policy engine, maintain monitoring and automation, author remediation playbooks.- Service Owners/Engineers: implement flags, respond to CI denials, own fixes and RCA.- Release Manager: approves risk exceptions during low-risk windows.- Product/Business Lead: approves high-risk overrides for critical business needs; only allowed with documented mitigation and timebox.- Overrides: allowed via gated “expedite” flow requiring two approvers (Release Manager + Business Lead) and automatic expiration + audit trail.

Escalation paths:1. CI gate fail → engineer notified, immediate triage.2. If can't remediate within T1, escalate to SRE on-call for mitigation (feature disable or partial rollback).3. If business-critical and override requested, use expedite flow; SRE must implement compensating controls and time-limited scope.4. Post-action: mandatory RCA within 72 hours; policy updated if required.

Metrics to measure policy effectiveness:- Reduction in incidents correlated with rollouts (incidents/100 deploys)- SLO breach frequency post-rollout vs baseline- False positive rate of gates (denials later judged unnecessary)- Time-to-restore (MTTR) when gate triggers vs manual- Time-to-deploy (median pipeline runtime) — track impact on velocity- Override frequency & justifications; % overrides that led to incidents- RCA completion rate and time-to-RCA

Balancing agility with safety:- Start with soft gates (warnings) and telemetry dashboards, then harden to blocking gates as confidence grows.- Use progressive exposure (canary + staged rollout) to minimize blast radius.- Make policies configurable per service risk tier: low-risk teams get lighter gates, high-risk systems stricter controls.- Invest in fast feedback (short telemetry windows, synthetic checks) and high-quality runbooks so remediations are quick.- Continuously measure gate false positives and developer friction; iterate policy thresholds and automation to preserve velocity.

This model uses policy-as-code, RBAC, automation, and data-driven iteration so safety scales without becoming a bottleneck.

MediumTechnical

21 practiced

Scenario: A user complains of being charged twice for the same purchase. Walk through your RCA approach: generate the full hypothesis set (e.g., client retried, network retry, gateway duplicate, idempotency key missing, webhook replay), list the logs/traces/DB records you would examine, define cohorts to select (payment method, region, time window), and propose short-term and long-term remediations (reconciliation job, idempotent APIs).

Sample Answer

Hypothesis set (complete, prioritized)- Client retried: user tapped “Pay” / double-clicked before UI updated.- Network retry: flaky connection caused client to resend request.- Gateway duplicate: PSP accepted and forwarded duplicate authorizations.- Idempotency key missing/incorrect: server didn’t dedupe retries.- Race in server: parallel requests created two charge records before unique constraint.- Webhook replay/duplicate webhook delivery from gateway.- Reconciliation delay: charge posted twice due to async job retrying after uncertain state.- Manual ops or support tool accidentally triggered a second capture/refund flow.

Logs / traces / DB records to examine- Client-side logs / session timestamps (if available) and device id.- Ingress API logs: request timestamps, headers, client IP, user-agent.- Application traces (distributed tracing): trace-id, span timings for payment flow.- Payment request payloads: idempotency-key header, order_id, amount, currency.- DB: orders/payments table rows, unique constraints, created_at, status transitions, transaction_ids.- External gateway responses and webhook delivery logs: PSP transaction_id, response codes.- Retry/queue logs for background workers (e.g., job id, attempts).- Support tooling/console audit logs for manual actions.- Network/load balancer logs to spot duplicate requests.

Cohorts to select- By payment method (card, Apple Pay, ACH) — PSP-specific behaviors.- By region/ISP/time window around complaint — network issues.- By idempotency-key presence vs missing.- By client app version / mobile OS.- By high-throughput windows (deployment / outage times).- By PSP / gateway endpoints and webhook receiver endpoints.

Short-term remediations (quick customer-impact reduction)- Manual reconciliation & refund: prioritize affected user, issue refund, communicate.- Stop-gap dedupe: add middleware to detect duplicates by (order_id, amount) within short window.- Disable aggressive retry behavior in client remotely if possible (feature flag).- Alerting: create ad-hoc alerts on duplicate successful captures within X minutes.- Throttle webhook retries or set webhook receiver to immediately ack duplicates.

Long-term remediations (robust fixes)- Enforce idempotent payment API: require idempotency-key and persist key->outcome atomically.- Database-level dedupe: strong unique constraints and transactional creation pattern (INSERT ... ON CONFLICT).- Improve tracing: propagate idempotency-key and trace-id through entire stack and to PSPs.- Reliable webhook handling: dedupe by PSP transaction_id and idempotency key.- Reconciliation job: nightly/near-real-time job to reconcile PSP transactions with DB, detect anomalies and auto-remediate or flag.- Client SDK changes: exponential backoff + disable duplicate UI actions; documented retry semantics.- SLA with PSP: guarantee idempotent behaviors or contract for webhook guarantees.- Post-incident: run RCA, update runbooks, add e2e tests simulating retries and webhook replays.

Key metrics to track- Duplicate-charge rate per million transactions- Fraction of payments with missing idempotency-key- Mean time to detect & remediate duplicates- False-positive rate from reconciliation jobs

This approach gives a forensic checklist to identify root cause, immediate steps to protect users, and engineering changes to prevent recurrence.

EasyTechnical

26 practiced

Explain the fishbone (Ishikawa) diagram technique and construct one for a high 95th-percentile latency spike in a public API. Include at least these categories: People, Process, Platform, Code, Network, Data. For each category, list three plausible causes that could produce the latency spike and state one concrete data source or test you would use to validate each cause.

Sample Answer

The fishbone (Ishikawa) diagram is a root-cause analysis tool that organizes possible causes of a problem into categories (the "bones") so teams can systematically hypothesize and test sources. For a high 95th‑percentile latency spike in a public API, here's a fishbone organized by People, Process, Platform, Code, Network, Data — each with three plausible causes and one concrete data source/test to validate.

People- On-call inexperience leading to slow remediation — validate: incident timeline from pager/ops playbook and mean time to acknowledge (MTTA) in PagerDuty.- Misconfiguration by engineer (wrong feature flag) — validate: deployment/git diff and feature-flag audit logs.- Insufficient capacity planning decisions — validate: capacity planning docs vs current traffic; run headroom/capacity report.

Process- Rolling deployment without canary checks — validate: CI/CD pipeline logs and canary metrics during deployment window.- Missing SLAs for downstream services — validate: SLO/SLA docs and alert history for downstream calls.- Inefficient incident runbooks — validate: postmortem notes and simulated runbook tabletop exercise timing.

Platform- Resource exhaustion on hosts (CPU/IO) — validate: host metrics (CPU, iowait) from Prometheus during spike.- Noisy neighbor in multi-tenant cluster — validate: container metrics and kubelet/containerd logs; correlate pod co-scheduling.- Control-plane throttling (API server) — validate: kube-apiserver metrics and error/latency logs.

Code- Slow algorithmic change in new release — validate: APM traces (span durations) comparing versions.- Blocking synchronous call to slow downstream — validate: distributed traces showing long external call spans.- Memory leak causing GC pauses — validate: heap/GC metrics and JVM/Go runtime pause logs.

Network- Packet loss or increased retransmits — validate: network interface counters, tcp retransmits, and MTR during window.- Load balancer misrouting or health-check flapping — validate: LB access logs and health-check status timeline.- DNS resolution delays — validate: DNS query latency metrics and dig/ldns query tests from clients.

Data- Database slow queries / locks — validate: DB slow query log and time-to-first-row, plus SHOW PROCESSLIST or pg_stat_activity.- Cache stampede or eviction causing DB thundering herd — validate: cache hit/miss rates and TTL patterns during spike.- Large queries returning excessive payloads — validate: response size metrics and query plans; reproduce with EXPLAIN.

This structured list gives testable hypotheses; prioritize by impact and likelihood, then run the corresponding metric/log/tracing checks to converge on root cause.

EasyTechnical

21 practiced

Explain the difference between correlation and causation in the context of incident diagnostics. Provide a concrete example where a metric (e.g., increased error rate) correlates with a recent deployment but is actually caused by a third variable (for example, a traffic reroute). Describe at least three methods you would use to test true causation versus coincidental correlation.

MediumTechnical

19 practiced

You receive an alert: a 20% drop in active sessions for users aged 18-24 on mobile in the last 2 hours. Draft a prioritized diagnostic plan: list top 6 hypotheses, which logs/metrics/traces to inspect first, how to slice cohorts, what short mitigations you might apply in the first 15 minutes, and which experiments or controlled rollbacks you might run if initial checks are inconclusive.

Sample Answer

High-level goal: rapidly determine whether drop is user-side, product-change, infra, or data/analytics issue and apply quick mitigations to restore sessions while minimizing blast radius.

Top 6 hypotheses (prioritized):1. Client-side regression in mobile app release (SDK, UI crash, auth) 2. Backend API degradation (increased latency/errors) affecting mobile endpoints 3. Feature flag/config rollout targeting 18–24 mobile cohort 4. Third-party dependency outage (CDN, auth provider, analytics) 5. Mobile OS / carrier-specific network issue or store-side change 6. Analytics/ETL bug causing false drop in reported active sessions

Logs / metrics / traces to inspect first (priority order):- Real user monitoring (RUM) and mobile crash reports (stack traces, crash-free %) - Mobile app release/version adoption timeline and recent deployments / feature flags - API error rate (4xx/5xx), p95/p99 latency, request volume for mobile-specific endpoints (ingress LB, app servers) - CDN logs, auth provider status, third-party service health pages - Analytics pipeline lag / ingestion failure metrics, data freshness alerts - Distributed traces for representative mobile sessions (trace latency, failed spans)

How to slice cohorts:- By app version, device OS and version (iOS/Android), carrier, region, app language, feature-flag bucket, time-of-day, acquisition channel (organic/paid), and age subgroup (18,19–20,21–24) to localize cause.

First 15-minute mitigations:- Roll back or pause recent client-side config/feature-flag rollout to default for 18–24 mobile cohort. - If backend shows elevated errors/latency, scale up instances / increase concurrency limits or revert recent deploy. - Enable a lightweight client-side fallback (e.g., serve cached content) via remote config if possible. - Route traffic away from suspect CDN/region with traffic steering. - Notify product/analytics to mark metrics as potentially impacted while investigating.

Experiments / controlled rollbacks if inconclusive:- Canary rollback: revert backend or config for a small % of traffic (start 5%, escalate to 50%) and measure session recovery in that slice. - Client A/B: force 10% of 18–24 mobile users to previous app config/version and compare session metrics vs control. - Traffic shadowing: duplicate requests to a fixed-stable backend to validate backend behavior without impacting users. - Feature-flagged progressive disable: disable suspect features progressively by dimension (app version, region) to isolate cause.

What success looks like:- One slice shows session recovery after rollback/disable -> root cause identified. - If analytics pipeline is root, restore ETL and backfill; otherwise, run postmortem and implement guardrails (release gating, better RUM alerts, automated rollbacks).

Unlock Full Question Bank

Get access to hundreds of Root Cause Analysis and Diagnostics interview questions and detailed answers.

Join thousands of developers preparing for their dream job.