Root Cause Analysis and Diagnostics Questions

Systematic methods, mindset, and techniques for moving beyond surface symptoms to identify and validate the underlying causes of business, product, operational, or support problems. Candidates should demonstrate structured diagnostic thinking including hypothesis generation, forming mutually exclusive and collectively exhaustive hypothesis sets, prioritizing and sequencing investigative steps, and avoiding premature solutions. Common techniques and analyses include the five whys, fishbone diagramming, fault tree analysis, cohort slicing, funnel and customer journey analysis, time series decomposition, and other data driven slicing strategies. Emphasize distinguishing correlation from causation, identifying confounders and selection bias, instrumenting and selecting appropriate cohorts and metrics, and designing analyses or experiments to test and validate root cause hypotheses. Candidates should be able to translate observed metric changes into testable hypotheses, propose prioritized and actionable remediation steps with tradeoff considerations, and define how to measure remediation impact. At senior levels, expect mentoring others on rigorous diagnostic workflows and helping to establish organizational processes and guardrails to avoid common analytic mistakes and ensure reproducible investigations.

HardTechnical

24 practiced

Describe how you would evaluate whether a proposed remediation (e.g., retraining with new data) actually fixed the root cause. Specify the metrics, experiment design (control/treatment), monitoring windows, and rollback criteria you'd use to validate remediation in production.

Sample Answer

Goal: Validate that retraining with new data fixed the root cause (vs. masking symptoms). I’d treat this as an A/B experiment with pre-defined metrics, monitoring windows, and strict rollback rules.

1) Hypothesis & scope- Hypothesis: Retrained model reduces error caused by X (e.g., class imbalance → false negatives on class A) without degrading other behaviors.- Define affected slices (e.g., class A, region, device).

2) Metrics- Primary metric: the bug-specific objective (e.g., F1 score on class A; reduction in false negative rate by ≥ absolute 5%).- Secondary (safety) metrics: overall accuracy, macro F1, calibration/Brier score, latency, resource use, and business KPIs (conversion, revenue, user complaints).- Slice metrics: per-cohort precision/recall, fairness metrics (demographic parity), and model confidence distribution.

3) Experiment design- Randomized traffic split: control = current model, treatment = retrained model. Use stratified sampling to ensure enough traffic for affected slices.- Traffic allocation: start 5% treatment, 95% control. If safe after checks, stepwise increase (5→20→50) with prespecified gates.- Statistical plan: precompute required sample sizes for primary metric delta with 80–95% power, adjust for multiple comparisons (Bonferroni or FDR for many slices).

4) Monitoring windows- Short-term (first 0–24 hours): real-time alerts on error spikes, latency, and crash rates; monitor primary metric on streaming labeled feedback if available.- Medium-term (1–7 days): accumulate labeled examples to evaluate statistically significant changes; monitor user-facing KPIs and drift detectors.- Long-term (2–8 weeks): stability of improvements, concept/data drift, and downstream impact.

5) Rollback criteria- Hard stops (immediate rollback): any critical outage, SLA breach, or >X% increase in severe error rate (e.g., false negatives on safety-critical class increases >1% absolute) or latency >2x.- Metric-based rollback: if treatment fails to meet minimum uplift on primary metric after N samples (predefined) or causes degradation in any safety metric beyond tolerated thresholds (e.g., overall accuracy drop >1.5% or statistically significant degradation in fairness).- Gradual rollback: if stepwise ramp shows deteriorations at a later ramp stage, revert to last safe allocation.

6) Logging, observability & audit- Log inputs, predictions, model versions, and downstream labels. Maintain sample reservoir for human review of failures.- Run shadow evaluation on historical and fresh labeled data; use explainability tools to confirm model behavior change aligns with root cause fix.- Post-deployment causal checks: counterfactual tests or feature-ablation to ensure the retrained model uses corrected signal rather than new spurious correlations.

7) Postmortem & permanent release- If experiment meets uplift with no safety regressions across windows, promote to full rollout with continued monitoring and a rollback window (e.g., 30 days).- Document lessons, dataset provenance, and mitigation to prevent recurrence.

Concrete thresholds and timelines should be calibrated to business risk: for low-risk features allow looser thresholds and faster ramps; for safety-critical systems use conservative thresholds, larger sample sizes, and human-in-the-loop validation.

HardTechnical

25 practiced

You need to teach a junior data scientist how to form mutually exclusive and collectively exhaustive (MECE) hypothesis sets during RCA. Create a short training exercise with an example production issue, the MECE hypothesis set you expect them to produce, and grading criteria to evaluate correctness and thoroughness.

Sample Answer

Exercise brief (15–20 min prep, 10–15 min debrief):Situation: Overnight our text-generation API for customer support shows a 5× increase in error rate and 10× latency for inference. Users see timeouts; SLOs breached. Ask the junior data scientist to produce a MECE hypothesis set explaining root causes and to propose the first 2 investigative steps for the top 2 hypotheses.

Expected MECE hypothesis set (categories — each with 1–2 example hypotheses):1. Input / Data issues - Upstream changed request format (longer prompts) causing OOM/timeouts - Malicious or malformed inputs triggering expensive decoding paths2. Model issues - Recent model rollout introduced heavier decoding (beam size, sampling) - Model weights corrupted or incompatible with serving code3. Serving code / inference pipeline - Regression in batching logic causing single-item batches - Tokenizer/version mismatch causing infinite loops4. Infrastructure / resource constraints - GPU memory pressure from other jobs; autoscaling misconfigured - Network/storage IOPS throttling model load time5. External dependencies - Third-party embedding service degraded increasing end-to-end latency6. Monitoring/observability faults (meta) - Alerting false-positive due to metric tag changes

Why MECE: categories are disjoint (data vs model vs infra vs external vs observability) and together cover the full pipeline from request to response.

First investigative steps (example):- Check recent deploys, commit history, and canary rollout logs (targets model & serving code).- Aggregate input length distribution and error traces in last 2 hours (targets input/data).

Grading criteria (total 100):- Correctness & MECE structure (40): categories mutually exclusive, cover all pipeline areas, no major missing area (40–25).- Specificity & plausibility (20): hypotheses are actionable and reflect AI system failure modes (20–10).- Prioritization & triage plan (20): selects highest-impact hypotheses and gives immediate, safe checks (20–10).- Communication & conciseness (10): clear, structured presentation suitable for incident war room (10–5).- Learning mindset (10): proposes follow-up to close monitoring gaps and prevent recurrence (10–0).

Scoring guidance: 85–100 = strong; 70–84 = acceptable with gaps; <70 = needs coaching (missed major categories or non-MECE).

EasyTechnical

22 practiced

Describe the difference between correlation and causation in the context of model diagnostics. Give a concise example where a metric change is correlated with a deployment event but is not caused by it, and explain how you would test whether the relationship is causal.

EasyTechnical

24 practiced

A newly deployed image-classification model occasionally returns NaN losses during training. Describe a concise step-by-step diagnostic checklist to find the root cause, including data, preprocessing, model, and infrastructure checks. Mention specific commands or checks you would run.

Sample Answer

Situation: You're seeing intermittent NaN losses during training. Use this concise diagnostic checklist (data → preprocessing → model → infra) with concrete checks/commands.

1) Data sanity- Quick stats: file counts and sizes - ls -l data/ | wc -l ; du -sh data/- Inspect examples and labels: - head -n 5 data/labels.csv - python -c "import pandas as pd; print(pd.read_csv('data/labels.csv').describe())"- Check image integrity & shape: - python -c "from PIL import Image; import os; print([ (f,Image.open(f).size) for f in os.listdir('data')[:5] ])"- Detect corrupted pixels/Nans: - python -c "import numpy as np, imageio; a=imageio.imread('img.png'); print(np.isnan(a).any(), np.isinf(a).any())"

2) Preprocessing pipeline- Log transforms/normalization: verify mean/std not zero or NaN. - python snippet:

python

import numpy as np
    imgs = np.stack([load(f) for f in batch])
    print(np.nanmean(imgs), np.nanstd(imgs))

- Check division by zero in scaling, clipping ranges, label encoding (no NaN labels).

3) Model & training loop- Detect NaNs in forward/backward: - In PyTorch:

python

for name,param in model.named_parameters():
        if torch.isnan(param).any(): print('NaN in',name)

- After loss backward:

python

print('loss:', loss.item(), torch.isnan(loss).any())
    torch.autograd.set_detect_anomaly(True)

- Check gradients: - After loss.backward():

python

for p in model.parameters():
        if p.grad is not None and torch.isnan(p.grad).any(): print('NaN grad', p.shape)

- Reduce learning rate / disable warmup to see if issue disappears.- Add gradient clipping: - torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)- Check unstable ops (log/exp/div): add eps (e.g., log(x+1e-8)).

4) Optimizer & loss numerical issues- Inspect optimizer states for NaNs: - for k,v in optimizer.state.items(): print(k,[torch.isnan(x).any() for x in v.values()])- Switch optimizer (Adam→SGD) to test sensitivity.

5) Mixed precision / GPU- If using AMP/fp16, try full fp32 run. FP16 often causes overflow → NaN.- Disable AMP or set torch.cuda.amp.autocast(enabled=False).

6) Repro & isolation- Run single-batch deterministic run: - torch.manual_seed(0); cudnn.deterministic=True- Use small reproducible script training on one batch to reproduce.

7) Infrastructure & hardware- Check GPU health and OOM: - nvidia-smi - dmesg | grep -i -E 'cuda|oom|gpu'- Set CUDA_LAUNCH_BLOCKING=1 to get accurate stack traces: - export CUDA_LAUNCH_BLOCKING=1; python train.py

8) Logging & monitoring- Add per-batch logging of loss, max/min activations: - print(torch.nanmax(x), torch.nanmin(x))- Use asserts to fail fast: - assert not torch.isnan(loss).any()

9) If still unresolved- Bisect changes (git bisect) to find commit introducing NaNs.- Run with smaller LR, fp32, simpler model to isolate.

Result: Follow order above to quickly localize whether NaNs originate from bad data, preprocessing (division/normalization), unstable ops in model, optimizer state, mixed precision, or hardware—then apply targeted fixes (clip grads, add eps, lower LR, disable AMP, repair data).

EasyTechnical

26 practiced

Briefly explain training-serving skew. Provide two concrete examples of issues that arise from skew and describe one automatic test you would add to CI to catch each example before deployment.

Unlock Full Question Bank

Get access to hundreds of Root Cause Analysis and Diagnostics interview questions and detailed answers.

Join thousands of developers preparing for their dream job.