Root Cause Analysis and Diagnostics Questions

Systematic methods, mindset, and techniques for moving beyond surface symptoms to identify and validate the underlying causes of business, product, operational, or support problems. Candidates should demonstrate structured diagnostic thinking including hypothesis generation, forming mutually exclusive and collectively exhaustive hypothesis sets, prioritizing and sequencing investigative steps, and avoiding premature solutions. Common techniques and analyses include the five whys, fishbone diagramming, fault tree analysis, cohort slicing, funnel and customer journey analysis, time series decomposition, and other data driven slicing strategies. Emphasize distinguishing correlation from causation, identifying confounders and selection bias, instrumenting and selecting appropriate cohorts and metrics, and designing analyses or experiments to test and validate root cause hypotheses. Candidates should be able to translate observed metric changes into testable hypotheses, propose prioritized and actionable remediation steps with tradeoff considerations, and define how to measure remediation impact. At senior levels, expect mentoring others on rigorous diagnostic workflows and helping to establish organizational processes and guardrails to avoid common analytic mistakes and ensure reproducible investigations.

HardTechnical

25 practiced

An ML model in production relied on a daily feature that was missing for 4 months. Propose strategies to reconstruct that feature for retraining (imputation using similar users, model-based reconstruction, using proxy features), describe validation approaches, and explain how you'd measure downstream model performance impact after retraining.

Sample Answer

Situation: A daily feature was missing for 4 months, and we need to reconstruct it to retrain the ML model without harming product metrics.

Strategy (reconstruction options and trade-offs):- Imputation using similar users: cluster users by stable attributes (demographics, long-term behavior) and impute the missing feature as the cluster mean/median or conditional distribution. Low engineering cost, but risks bias if clusters shifted.- Model-based reconstruction: train a supervised model (e.g., gradient-boosted trees or time-series model) using historical periods when the feature existed to predict the daily feature from other reliable inputs. Higher accuracy, needs labeled history and validation.- Proxy features: identify correlated signals that continued during the outage (event logs, coarse engagement metrics) and transform/aggregate them to approximate the missing feature. Fast to deploy and robust if proxies are strongly correlated.- Hybrid: use model-based predictions but blend with cluster-based priors and uncertainty estimates (e.g., Bayesian shrinkage) to avoid overconfidence.

Validation approaches:- Backtest on historical windows: hide the feature for some past 4-month-equivalent windows, reconstruct using each method, compare reconstructed vs actual with MAE/RMSE, distributional similarity (KS test), and calibration.- Holdout by cohort/time: reserve a recent holdout cohort for final validation to check generalization.- Uncertainty and drift checks: compare reconstructed feature distributions to historical; run data-drift tests and monitor covariate shifts.- Shadow evaluation: run the downstream model in shadow using reconstructed feature and compare predictions vs live predictions when possible.

Measuring downstream impact after retraining:- Offline: measure standard model metrics (AUC, log loss, calibration) on a held-out set and compute predicted outcome distribution differences. Use causal uplift or counterfactual-simulated KPIs if available.- Online: run an A/B experiment (or staged rollout) comparing retrained model vs current baseline. Track primary business KPIs (conversion, revenue per user, retention) and safety metrics (false positives, latency). Ensure experiment is powered to detect meaningful differences and run long enough to cover weekly cycles.- Monitoring post-launch: set up real-time dashboards for feature distributions, model score drift, and KPI deltas; implement alerting for regressions and rollback criteria.

Execution plan (PM perspective):- Week 1: scope, prioritize reconstruction approach with engineering and data science, pick metrics and validation plan.- Week 2–3: implement candidate reconstructions and backtests.- Week 4: select best method, retrain model, run offline validation and pre-launch shadow run.- Week 5–8: A/B rollout, monitor, and iterate.

This approach balances speed, accuracy, risk control, and measurable business outcomes while coordinating engineering, data science, and stakeholders.

MediumTechnical

18 practiced

You notice your team frequently jumps to implementation before validating hypotheses. As the PM, design meeting cadences, required artifacts (hypothesis templates, gating criteria), and coaching exercises to shift the team toward hypothesis-driven diagnostics. Include measurable indicators you would track to show cultural change.

Sample Answer

Start by clarifying the goal: move decisions from opinion-driven to evidence-driven, reducing rework and improving impact.

Meeting cadences- Weekly Discovery Sync (30–45m): review active hypotheses, experiments, data, blockers. Decisions: continue/stop/iterate.- Bi-weekly Outcome Review (60m): product + eng + analytics review recent launches vs hypotheses; surface learnings.- Monthly Strategy Checkpoint (90m): prioritization resets based on validated opportunities.- Ad-hoc Gating Review: quick 30m when a feature is ready to move from experiment to build.

Required artifacts- Hypothesis template (mandatory before design): Problem statement; Hypothesis: “If [change], then [measurable outcome] by X% within Y weeks”; Key assumptions; Primary metric & guardrail metrics; Experiment design; Success criteria; Risks & rollback plan.- Gating checklist for implementation: hypothesis exists, experiment run or replicated signal, primary metric meets threshold OR qualitative validation (N user interviews) + analytics review, engineering estimate & rollback plan.- Experiment log (shared board): treatment, sample, duration, results, learnings.

Coaching exercises- Hypothesis-writing workshop (90m): small groups convert backlog items into testable hypotheses; peer feedback.- Pre-mortem sessions: before experiments/design, list failure modes and what validation would catch them.- Role-play user interviews + analytics walkthroughs: PMs practice eliciting assumptions and defining metrics.- Weekly “learning retro” (15m): share one surprising finding and what changed.

Measurable indicators (targets within 3–6 months)- % of backlog items with hypothesis before design: baseline → target 80%- % of major features released with prior experiment/validation: target 70%- Time from hypothesis to validation (median days): reduce by 30%- Rework rate (features rolled back or hotfixed): reduce by 50%- Number of validated learnings logged per month: increase by 3x- Stakeholder trust score (survey): +15 points

Why this works- Cadences create predictable feedback loops; artifacts make expectations explicit; coaching builds skills. Together they shift incentives from “ship fast” to “ship informed,” measurable through the indicators above.

EasyTechnical

38 practiced

You have two tables:

users(user_id int, signup_at timestamp)
events(event_id int, user_id int, event_name text, occurred_at timestamp, properties json)

Write a Postgres SQL query that computes the 7-day conversion rate (signup -> first 'purchase') for weekly signup cohorts. Only count each user once and include cohorts with zero conversions. Explain your main SQL choices briefly.

Sample Answer

To compute weekly signup cohorts and the 7-day conversion rate (signup -> first 'purchase'), we:- bucket users by signup week,- find each user's first purchase (if any),- count cohort sizes and cohort conversions (first purchase within 7 days),- LEFT JOIN so cohorts with zero conversions are included.

sql

WITH cohorts AS (
  SELECT
    user_id,
    date_trunc('week', signup_at)::date AS cohort_week,
    signup_at
  FROM users
),
first_purchase AS (
  -- one row per user with their first purchase timestamp (if any)
  SELECT
    user_id,
    MIN(occurred_at) AS first_purchase_at
  FROM events
  WHERE event_name = 'purchase'
  GROUP BY user_id
),
cohort_metrics AS (
  SELECT
    c.cohort_week,
    COUNT(*) AS cohort_size,
    COUNT(fp.user_id) FILTER (
      WHERE fp.first_purchase_at IS NOT NULL
        AND fp.first_purchase_at <= c.signup_at + INTERVAL '7 days'
    ) AS conversions_7d
  FROM cohorts c
  LEFT JOIN first_purchase fp ON c.user_id = fp.user_id
  GROUP BY c.cohort_week
)
SELECT
  cohort_week,
  cohort_size,
  conversions_7d,
  CASE WHEN cohort_size = 0 THEN 0.0
       ELSE ROUND(conversions_7d::numeric / cohort_size * 100, 2)
  END AS conversion_rate_pct_7d
FROM cohort_metrics
ORDER BY cohort_week;

Why these choices:- date_trunc('week') groups by calendar week for cohorts.- MIN(...) per user ensures we count only the user's first purchase.- LEFT JOIN + COUNT FILTER preserves cohorts with zero conversions.- The time comparison uses signup_at + INTERVAL '7 days' to enforce the 7-day window.

MediumTechnical

24 practiced

Design a diagnostic funnel to investigate a sudden drop in purchases originating from product detail pages. List the specific events for each funnel step, recommended diagnostic metrics (step conversion, time between steps, error rates), and the queries or slices you would run to identify the step with the largest leak. Include monitoring thresholds you might configure.

Sample Answer

Situation: We need a rapid, actionable diagnostic funnel to locate where purchases from Product Detail Pages (PDP) are leaking.

Funnel steps & specific events1. PDP View — event: product_view (product_id, user_id, session_id, timestamp)2. Add to Cart — event: add_to_cart (product_id, user_id, session_id, timestamp, qty)3. Cart View / Mini-cart Open — event: cart_view (session_id, timestamp)4. Begin Checkout — event: checkout_start (session_id, user_id, timestamp)5. Payment Initiated — event: payment_init (order_id, session_id, timestamp, payment_method)6. Purchase Confirmed — event: purchase (order_id, session_id, user_id, total, timestamp)

Recommended diagnostic metrics (for each adjacent step)- Step conversion rate = users who completed step N+1 / users who completed step N- Absolute leakage = count(N) - count(N+1)- Relative drop = % change vs baseline period (week-over-week, 28d)- Median & 90th percentile time_between_steps (timestamp difference)- Error / failure rates: payment failures, API error codes, validation errors- Device/session-quality signals: bounce rate, page load time, JS errors

Queries / slices to identify largest leak1. Compute funnel conversion and drop by step (weekly):

sql

WITH step_counts AS (
  SELECT
    date_trunc('day', timestamp) as day,
    COUNT(DISTINCT CASE WHEN event='product_view' THEN session_id END) AS views,
    COUNT(DISTINCT CASE WHEN event='add_to_cart' THEN session_id END) AS adds,
    COUNT(DISTINCT CASE WHEN event='cart_view' THEN session_id END) AS cart_views,
    COUNT(DISTINCT CASE WHEN event='checkout_start' THEN session_id END) AS checkouts,
    COUNT(DISTINCT CASE WHEN event='payment_init' THEN session_id END) AS payments,
    COUNT(DISTINCT CASE WHEN event='purchase' THEN order_id END) AS purchases
  FROM events
  WHERE product_id = :product_id AND timestamp BETWEEN :start AND :end
  GROUP BY day
)
SELECT day,
  views,
  adds, adds::float/views as conv_view_add,
  cart_views, cart_views::float/adds as conv_add_cart,
  checkouts, checkouts::float/cart_views as conv_cart_checkout,
  payments, payments::float/checkouts as conv_checkout_payment,
  purchases, purchases::float/payments as conv_payment_purchase
FROM step_counts
ORDER BY day;

2. Rank steps by leakage:

sql

SELECT step,
  count_before, count_after,
  (count_before - count_after) AS absolute_leak,
  (1 - count_after::float/count_before) AS leak_pct
FROM (
  VALUES
  ('view->add', views, adds),
  ('add->cart', adds, cart_views),
  ('cart->checkout', cart_views, checkouts),
  ('checkout->payment', checkouts, payments),
  ('payment->purchase', payments, purchases)
) t(step, count_before, count_after)
ORDER BY absolute_leak DESC;

3. Slice the worst step by dimensions:- Device type, browser, OS_version, country, channel, product_variant, A/B cohort:

sql

SELECT device_type, browser, count_before, count_after,
  (1 - count_after::float/count_before) as leak_pct
FROM per_device_counts
WHERE step = 'view->add'
ORDER BY leak_pct DESC LIMIT 20;

4. Time and error diagnostics on that step:- Histogram of time_between_steps; join to logs for JS errors, network failures, 5xx responses, payment decline codes.

Suggested monitoring thresholds (alerts)- Step conversion drops >20% week-over-week OR >10 percentage points absolute vs baseline — Page-level alert- Absolute purchases drop >15% vs 7-day moving average — Business alert- Payment failure rate >1% (or 2x baseline) — Critical alert- Median time_between_steps increases >100% (or >30s) — Performance alert- JS error rate >0.5% of PDP sessions — Engineering alert- Browser-specific conversion drop >30% for >1% of traffic — Investigate regressions

Investigation priority1. Identify step with largest absolute leak and steepest relative drop.2. Slice by device/browser/channel and surface common denominators.3. Correlate with deploy/release timestamps, CDN errors, API error spikes, frontend JS error spikes, and backend latency.4. Reproduce: simulate user flow on worst device/browser, inspect console/network, and check A/B flags.

This approach yields a fast path to isolate the leak, prioritize fixes (frontend issue, backend/API, payment gateway, or UX), and set thresholds to detect regressions proactively.

MediumTechnical

21 practiced

Design the instrumentation schema required to diagnose checkout failures across web and mobile. Specify event names, required attributes (for example payment_type, error_code, cart_value, device_os, sdk_version, session_id), sampling decisions, and how you would ensure idempotency and support for replay/backfill when downstream schemas change.

Sample Answer

Situation: We need an instrumentation schema to quickly diagnose checkout failures across web and mobile, enable root-cause analysis, and support reliable analytics over time.

Suggested event model (core events):- checkout_start- cart_update (item_add/item_remove)- checkout_submit- payment_attempt- payment_result (success/failure)- checkout_complete- checkout_error (generic client-side blocking error)

Required attributes (attach to every checkout-related event):- event_id (UUID) — globally unique per event- session_id — deterministic per user session- user_id (nullable) — if available- timestamp_utc (ISO8601 ms)- platform (web/mobile)- device_os, device_model, browser, sdk_version- app_version- locale- env (prod/stage)- page_or_screen- cart_value (decimal), cart_items_count- payment_type (card/wallet/ACH)- payment_method_id (tokenized, hashed)- error_code (string), error_message (nullable)- http_status (if applicable)- network_type (wifi/cellular)- latency_ms (network/API where relevant)- upstream_service (payment_gateway/checkout_api)- attempt_number (int) — retry count for idempotency tracking- event_version (int)

Sampling strategy:- Always ingest 100% of failure events (payment_result.failure, checkout_error).- Deterministic session-based sampling for success/completion: hash(session_id) mod 100 <= N to get a stable percent (e.g., 5–10%) to control volume while preserving reproducible cohorts.- High-value baskets (cart_value > threshold) always sampled.- Adjustable dynamic sampling flags pushed from server-side config for incidents.

Idempotency & dedup:- Require event_id and attempt_number. Downstream consumers dedupe by event_id with a sliding window (e.g., 7 days).- For replayed events, include original_timestamp and origin (device/server/replay_job).- For payment_attempt/payment_result, include payment_gateway_transaction_id to reconcile with external provider.

Supporting replay/backfill & schema evolution:- Use a central schema registry with strict semantic versioning (major: incompatible, minor: additive). Each event carries event_version.- Persist raw, append-only events to a durable event lake (compressed JSON/Avro/Parquet) as the source of truth.- Consumers read from event lake or change-data-capture topic; backfill/reprocess by applying transformation jobs that understand event_version and map to the current analytic model.- Keep nullable extension fields rather than removing fields; when removing or renaming, add mapping rules with deprecation windows.- Store enrichment metadata separately (e.g., resolved_user_id) to allow safe rejoining during backfills.- Provide a replay pipeline that tags replayed events (origin=replay, replay_id) and prevents double-counting by honoring event_id dedupe.

Operational notes:- Track SLAs: time-to-first-failure-alert, percent of failures with full context (stack, gateway id).- Define dashboards: failure rate by error_code, payment_type, cart_value, device_os, sdk_version; funnel abandonment by step.- Run periodic audits to ensure schema coverage and migration plans communicated 2–4 weeks before breaking changes.

This balances diagnostic depth, cost control via sampling, clear idempotency, and safe reprocessing for analytics and incident investigation.

Unlock Full Question Bank

Get access to hundreds of Root Cause Analysis and Diagnostics interview questions and detailed answers.

Join thousands of developers preparing for their dream job.