Root Cause Analysis and Diagnostics Questions

Systematic methods, mindset, and techniques for moving beyond surface symptoms to identify and validate the underlying causes of business, product, operational, or support problems. Candidates should demonstrate structured diagnostic thinking including hypothesis generation, forming mutually exclusive and collectively exhaustive hypothesis sets, prioritizing and sequencing investigative steps, and avoiding premature solutions. Common techniques and analyses include the five whys, fishbone diagramming, fault tree analysis, cohort slicing, funnel and customer journey analysis, time series decomposition, and other data driven slicing strategies. Emphasize distinguishing correlation from causation, identifying confounders and selection bias, instrumenting and selecting appropriate cohorts and metrics, and designing analyses or experiments to test and validate root cause hypotheses. Candidates should be able to translate observed metric changes into testable hypotheses, propose prioritized and actionable remediation steps with tradeoff considerations, and define how to measure remediation impact. At senior levels, expect mentoring others on rigorous diagnostic workflows and helping to establish organizational processes and guardrails to avoid common analytic mistakes and ensure reproducible investigations.

HardTechnical

25 practiced

A security incident exfiltrated sensitive PII. Propose a hybrid investigation that combines fishbone diagramming and fault-tree analysis to identify root causes, preserve evidence, and propose prioritized remediation steps. Include what logs and artifacts to collect first and how to avoid contaminating forensic evidence.

Sample Answer

Approach (hybrid): use a fishbone diagram to quickly enumerate contributing factors (broad, qualitative), then translate the most likely branches into a fault-tree (logical, boolean paths) to identify minimal cut-sets (root causes) and drive prioritized remediation.

1) Fishbone (brainstorm categories)- People: compromised credentials, phishing, privileged misuse- Process: missing code review, bad deploy process, weak change controls- Technology: vulnerable library, misconfigured S3/DB, lack of encryption- Network/Environment: open ports, weak perimeter, VPN misconfig- Data: PII discovery/labeling gaps, backups exposed

2) Convert to Fault-Tree- Top event: PII exfiltration- Decompose into OR of paths (ex: compromised account OR app vuln exploited OR misconfigured storage)- For each path, decompose into AND/OR gates (e.g., compromised account = credential leak AND valid session OR MFA bypass)- Compute minimal cut-sets to prioritize fixes with highest reduction in risk.

Evidence collection & order (preserve volatile first)1. Volatile (collect immediately, from most to less volatile) - Memory dump (RAM) of affected hosts - Active process list, network connections, open ports (lsof/ss/netstat) - Live network capture (pcap) if possible - STIX/TAXII/SIEM alerts snapshot2. Host & application - Full disk images (bit-for-bit) using write-blockers or trusted imaging tools - System logs (/var/log, Windows Event Logs) - Application logs (web server, app, auth service) - DB audit logs, query logs, backups metadata3. Network & cloud - Firewall, IDS/IPS alerts and logs - Proxy, load balancer logs - Cloud provider audit logs (CloudTrail, GCP audit logs, Azure Activity) - VPN/auth logs, SSO/OAuth provider logs4. Identity & access - IAM changes, role assignments, recent token issuances - Rotation timestamps for keys/secrets5. Artifacts - Staged files, exfil tools, scheduled tasks, containers/images - Hashes of binaries, suspicious scripts, registry keys

How to avoid contaminating evidence- Triage on isolated network segment; avoid logging into compromised accounts from investigator machines.- If possible, take snapshots/images; do not run commands that modify timestamps or files. Use read-only mounts and write-blockers.- Capture volatile data before rebooting or shutting down hosts.- Use known-forensic tools (FTK Imager, dd with hashes, Volatility for memory) and document exact commands, times, and operator.- Maintain chain-of-custody: who collected what, when, hashes (MD5/SHA256), storage location.- Limit number of people with access; use immutable storage for evidence.- When live interaction is required, record session, minimize changes, and log every action.

Prioritized remediation (based on minimal cut-sets)1. Immediate containment (hours) - Revoke compromised credentials, rotate keys, block exfil endpoints, isolate affected hosts - Apply temporary firewall rules, revoke tokens/sessions2. Short-term fixes (days) - Patch exploited vulnerabilities, close misconfigurations (S3, DB), enforce MFA - Rebuild compromised hosts from known-good images; preserve images for forensics3. Mid-term (weeks) - Deploy DLP for PII discovery/monitoring, refine SIEM rules and alerting, enable structured DB auditing - Harden CI/CD: signed artifacts, review gates, secret scanning4. Long-term (months) - Least-privilege IAM, periodic access reviews, automated rotation of secrets, secure logging pipeline with tamper-evidence - Run tabletop exercises, update incident response playbook, add forensic capture playbooks and retention policies

Developer-focused actions- Identify vulnerable code/deps; create PRs to fix and add tests- Add structured logging and correlation IDs to trace user actions- Instrument endpoints with app-level audit events to speed future investigations

Why this works- Fishbone ensures coverage of human/process/tech vectors. Fault-tree forces logical rigor and identifies minimal sets to fix to prevent recurrence. Prioritizing volatile evidence preserves high-value artifacts; strict forensic hygiene prevents evidence tampering and supports legal/regulatory needs.

HardSystem Design

20 practiced

Distributed transactions exhibit an increase in partial commits and rollbacks. Design tracing and instrumentation to surface where partial failures occur (which service, which stage), and propose architectural or compensating changes to improve end-to-end reliability while minimizing user-visible impact.

Sample Answer

Requirements & constraints:- Surface exactly where partial commits/rollbacks happen (service, stage, data shard).- Low overhead on latency and throughput.- Actionable signals for automated compensation or retry with minimal user-visible impact.- Support heterogeneous services (microservices, DBs, queues).

High-level approach:1) Distributed tracing + structured instrumentation (per-request causal id + fine-grained spans).2) Transaction metadata propagation (tx-id, intent, stage, policy) via headers.3) Transaction coordinator components: short-lived coordinator for orchestration + durable transaction log for recovery.4) Compensating patterns (Saga with idempotent steps, Outbox, CQRS for eventual consistency).

Instrumentation design:- Generate a global transaction-id at the entry (API gateway). Propagate via W3C Trace Context + "X-Tx-Id".- For each logical transaction step emit a span with attributes: tx_id, step_name, step_role (try/confirm/compensate), resource (DB/queue), version, shard, attempt, deadline.- Record explicit step outcome: START, SUCCESS, FAILURE, COMPENSATED, TIMEOUT, RETRIED.- Persist critical step transitions to a durable transaction log (append-only) to enable offline analysis and recovery.- Correlate traces with metrics: per-step latencies, error rates, partial-commit ratio.- Use OpenTelemetry + Jaeger/Elastic/Datadog APM; export events to a time-series DB and an observability pipeline.

Detecting partial failures:- Define invariants per tx (e.g., steps A,B,C must all be SUCCESS). Run continuous queries on transaction log to find incomplete tx older than threshold.- Alert on patterns: step X success but subsequent step Y failure; repeated compensations; high variance in step latency indicating timeouts.

Architectural/compensating changes:- Prefer Saga (choreography or orchestrator) with idempotent, reversible operations. Ensure compensating actions are well-tested and transactional on their own.- Use Outbox pattern for reliable messaging from local DB to event bus to avoid lost messages on commit.- Design long-running transactions as state machines persisted in the durable log; coordinator retries or triggers compensations asynchronously.- Implement bounded retries with exponential backoff, circuit breakers and escalation to human workflows if automatic compensation fails.- Use write-intent flags (soft locks) and tombstones to make partial results detectable and prevent duplicate application.- Offer user-facing eventual-consistency UX: optimistic UI with status, retry button, and background reconciliation; guarantee no double-charges via idempotency keys.

Trade-offs:- Sagas + Outbox increase complexity and eventual consistency — but avoid 2PC blocking and scale better.- Durable transaction logs add storage/latency but are essential for observability and recovery.- Instrumentation adds overhead; mitigate with sampling policies and critical-step persistence.

Example flow:- Client -> API assigns tx:tx123. Call Service A (Reserve), span A:START -> SUCCESS (persist). Publish event via Outbox. Service B (Charge) span B:START -> TIMEOUT -> RETRY -> FAILURE (persist). Coordinator sees A success, B failure -> triggers compensate: Service A:COMPENSATE. UI shows "Processing — refund pending". Alert created for manual review if compensation fails twice.

Operational playbook:- Dashboards: partial-commit heatmap by service/step, oldest open txs, compensation latency histogram.- Automated responders: retry policies, auto-compensate thresholds.- Postmortem: increase instrumentation granularity on hot spots; add idempotency or shorten timeouts.

This design surfaces where partial failures occur, gives automated recovery paths, and limits user impact through idempotency, outbox reliability, and clear UX for eventual consistency.

EasyTechnical

25 practiced

Explain the roles of logs, metrics, and distributed traces in diagnosing production problems. For a user-reported error that lacks a clear stack trace, describe how you would use each signal (metrics, logs, traces) in sequence to narrow down the root cause and why.

EasyTechnical

26 practiced

Given a Postgres table definition:

events(
  id UUID,
  service_name TEXT,
  status_code INT,
  occurred_at TIMESTAMPTZ
)

Write a SQL query to compute the daily error rate (status_code >= 500) per service for the last 7 days. Return columns: service_name, day, error_count, total_count, error_rate_percent.

Sample Answer

Approach: aggregate events by service and day for the last 7 days, counting total events and errors (status_code >= 500). Use generate_series to ensure days with zero events appear, and compute percentage safely with NULLIF.

sql

WITH days AS (
  SELECT generate_series(
    date_trunc('day', now()) - interval '6 days',
    date_trunc('day', now()),
    interval '1 day'
  )::date AS day
),
services AS (
  SELECT DISTINCT service_name FROM events
)
SELECT
  s.service_name,
  d.day,
  COALESCE(e.error_count, 0)      AS error_count,
  COALESCE(e.total_count, 0)      AS total_count,
  CASE WHEN COALESCE(e.total_count, 0) = 0 THEN 0
       ELSE round(100.0 * e.error_count / e.total_count, 2)
  END                              AS error_rate_percent
FROM services s
CROSS JOIN days d
LEFT JOIN (
  SELECT
    service_name,
    (occurred_at AT TIME ZONE 'UTC')::date AS day,
    COUNT(*) FILTER (WHERE status_code >= 500) AS error_count,
    COUNT(*)                                AS total_count
  FROM events
  WHERE occurred_at >= date_trunc('day', now()) - interval '6 days'
    AND occurred_at <  date_trunc('day', now()) + interval '1 day'
  GROUP BY service_name, day
) e
  ON e.service_name = s.service_name AND e.day = d.day
ORDER BY s.service_name, d.day;

Key points:- date_trunc + generate_series produces the 7-day window and guarantees rows for zero-count days.- FILTER clause simplifies conditional aggregation.- NULL/zero-safe percentage via CASE/NULLIF and rounding to 2 decimals.Edge cases:- No events for a service/day handled by COALESCE to zero.- Timezone: cast occurred_at to date in UTC (adjust if you need a different timezone).

EasyTechnical

21 practiced

What does MECE (mutually exclusive, collectively exhaustive) mean for hypothesis generation in RCA? Provide an example MECE hypothesis set for investigating a 15% drop in purchase conversions over the past 48 hours.

Unlock Full Question Bank

Get access to hundreds of Root Cause Analysis and Diagnostics interview questions and detailed answers.

Join thousands of developers preparing for their dream job.