Learning from Incidents and Post Incident Review Questions
Responding to incidents with curiosity rather than blame. Asking 'why' questions to understand root causes, proposing systemic improvements, and sharing knowledge from incidents with the team. Showing humility and demonstrating growth from past mistakes.
MediumTechnical
38 practiced
Compare and contrast two root-cause analysis techniques (for example, '5 Whys' and the Fishbone/Ishikawa diagram). Explain how you would apply each method to an ML data-pipeline outage that caused model regressions, and describe strengths and weaknesses for ML incidents.
Sample Answer
5 Whys vs Fishbone (Ishikawa) — concise comparison and ML applicationDefinitions:- 5 Whys: iterative “why?” questioning to peel back causal layers until a root cause is reached.- Fishbone: structured diagram that maps categories (e.g., Data, Process, People, Tools, Environment, Model) to possible causes, encouraging parallel hypothesis generation.How I’d apply to an ML data-pipeline outage that caused model regressions:5 Whys (quick, sequential):- Why did model performance drop? Because input feature distributions changed.- Why did distributions change? Because new data source schema added NULLs.- Why were NULLs introduced? Upstream ETL job changed casting logic.- Why did ETL change without validation? CI checks lacked schema-contract tests.- Why lacked tests? No ownership for schema evolution and no pre-deploy gating.Action: fix ETL, add schema contracts and automated validation.Fishbone (broad, exploratory):- Create a fishbone with branches: Data, Feature Engineering, ETL, Monitoring, Model, Deployment, People.- Under Data: schema change, sampling bias, late-arriving data.- Under ETL: job schedule, dependency failures, silent retries.- Under Monitoring: alert thresholds, metric gaps.- Run a cross-functional session to populate, prioritize likely causes, run targeted checks (replay data, compare feature stats, inspect ETL logs).Action: triage multiple contributing causes (e.g., ETL bug + missing validation + incomplete alerts) and implement layered fixes.Strengths & weaknesses for ML incidents:- 5 Whys strengths: fast, lightweight, good when a single causal chain exists; helps assign a clear corrective action. Weaknesses: can stop at a superficial root cause, biased by facilitator, misses multiple concurrent contributors common in ML systems.- Fishbone strengths: systematic, surfaces multiple contributing factors across people/process/tech; great for complex pipelines and cross-team troubleshooting. Weaknesses: more time-consuming, can produce long lists of hypotheses requiring discipline to prioritize and test.Recommendation:Use both: start with 5 Whys to get an immediate remediation path; parallelly run a fishbone session to uncover systemic fixes (tests, contracts, monitoring, ownership) to prevent recurrence.
HardBehavioral
38 practiced
Describe a situation where you had to hold someone accountable after an incident while maintaining a blameless culture. How did you balance learning, accountability, and the individual's development? Explain the process, communications, and follow-up you used.
Sample Answer
Situation: In my previous role as an ML engineer, a recently deployed recommendation model began surfacing highly biased results for a key user segment two days after release. This caused customer complaints and a 12% drop in engagement for that segment.Task: As the team lead on the model, I needed to (1) stop customer harm, (2) find root cause, (3) hold the person responsible for the deployment decisions accountable, and (4) preserve a blameless culture so we could learn and prevent recurrence.Action:- Immediate mitigation: I rolled back the release and flipped the traffic to the previous model to stop the regression while we investigated.- Blameless postmortem: I facilitated a postmortem meeting with engineering, data science, and product—setting ground rules: no finger-pointing, focus on timelines, facts, and systems. We reconstructed events from CI/CD logs, model evaluation artifacts, and data pipelines.- Root cause and accountability: The investigation showed the engineer who ran last-minute feature normalization changes didn’t update the validation dataset, causing distribution shift. In the postmortem I explicitly stated facts (what changed, when, and its impact) without labeling intent. I asked the engineer to explain their decision; they acknowledged the oversight.- Individual coaching and development plan: In a private one-on-one I balanced accountability and support. I described the impact, reiterated expectations about end-to-end validation and peer review, and asked about blockers that led to the shortcut. Together we created a remediation plan: pair-programming sessions on CI validation, a checklist for pre-deploy ML sanity checks, and training on dataset versioning (DVC) and unit tests for data transformations.- Process-level fixes: We added mandatory automated distribution-shift checks in CI, required a second reviewer for model-formatting or feature-engineering changes, and added a canary release policy with segment-level telemetry thresholds.- Communication and follow-up: I documented the postmortem with action items, owners, and deadlines; shared it with the team and stakeholders. I scheduled weekly check-ins with the engineer for 8 weeks to track skill growth and adherence to the checklist. I also presented the learnings in our engineering forum to spread knowledge.Result: The immediate rollback restored engagement within 24 hours. The new CI checks and canary policy prevented similar regressions; no repeat incidents in the following 9 months. The engineer improved their deployment rigor, completed the training, and later led a successful model release. The team reported higher trust in our release process and appreciated the blameless tone that kept morale high while enforcing accountability.Learning: Accountability in a blameless culture means separating error from intent: make facts visible, require responsibility for remediation, and invest in the person's development and system improvements so the whole team benefits.
HardTechnical
70 practiced
Define a set of quantitative metrics to measure the effectiveness of post-incident reviews for ML systems. Include leading and lagging indicators, how to collect the data (tools/tickets), and target values you might set to track progress over 6–12 months.
Sample Answer
Framework: categorize metrics into Detection & Response (leading), Learning & Remediation (leading/lagging), and Business/Recurrence (lagging). For each metric I give definition, how to collect, and suggested 6‑/12‑month targets (assume current baseline unknown; targets assume a mature team moving from ad-hoc to reliable processes).1) Mean Time To Detect (MTTD) — time from user-impacting symptom to alerting/acknowledgement.- Why: faster detection reduces user impact and gives more time to remediate.- Collect: alert timestamps from monitoring (Prometheus/Sentry/Evidently) + PagerDuty/Jira ack times.- Targets: 6mo ≤ 30m, 12mo ≤ 15m.2) Mean Time To Mitigate (MTTM) — time from detection to temporary mitigation (rollback/kill switch).- Collect: incident/ticket lifecycle in PagerDuty/Jira; runbook action logs.- Targets: 6mo ≤ 2h, 12mo ≤ 30m.3) Mean Time To Resolve (MTTR) — time from detection to full resolution (root cause fixed or permanent mitigation).- Collect: Jira incident tickets, commit/PR timestamps referencing incident.- Targets: 6mo ≤ 24h, 12mo ≤ 8h.4) Postmortem Turnaround Time — time from incident close to completed post-incident review doc.- Why: timely reviews capture fresh details.- Collect: link creation/completion fields in Confluence/Jira postmortem template.- Targets: 6mo ≤ 5 working days, 12mo ≤ 48 hours.5) Postmortem Quality Score (leading) — rubric-scored (0–5) on hypothesis, evidence, RCA depth, action items, owner, timelines.- Collect: peer review score in postmortem template (mandatory reviewer + checklist).- Targets: 6mo avg ≥ 3.5, 12mo ≥ 4.5.6) Action Item Closure Rate & Time-to-Implement — % of postmortem action items closed and median time to close.- Collect: Jira tasks tagged to postmortems with due/completion dates.- Targets: 6mo closure ≥ 70% within SLA (90 days); 12mo ≥ 95% within SLA (30 days), median close ≤ 14 days.7) Recurrence Rate (per incident class) — % of incidents that repeat within 90 days with same root cause.- Collect: incident taxonomy in Jira; automated similarity matching on tags/labels.- Targets: 6mo ≤ 25% reduction, 12mo ≤ 60% reduction overall.8) Severity-Weighted Customer Impact (lagging) — aggregated user-hours affected weighted by severity.- Collect: incident impact estimates in postmortem and customer support logs.- Targets: 6mo drop 40%, 12mo drop 70%.9) Model-Specific Leading Indicators:- Drift detection lead time: time between drift signal and human review. - Collect: Evidently/EOS alerts, review tickets. - Targets: 6mo ≤ 24h, 12mo ≤ 6h.- Data lineage completeness: % of features with end-to-end lineage and tests. - Collect: metadata store (Data Catalog) reports. - Targets: 6mo ≥ 70%, 12mo ≥ 95%.10) Verification Rate — % of postmortem hypotheses validated with data experiments or tests.- Collect: links to experiment notebooks/metrics and sign-off in postmortem.- Targets: 6mo ≥ 60%, 12mo ≥ 90%.Implementation notes (how to instrument):- Standardize incident schema and tags in PagerDuty/Jira: severity, impacted service, RCA tag, postmortem link, action-item links.- Enforce postmortem template in Confluence with mandatory fields and rubric.- Automate metric extraction: ETL from monitoring, ticket system, and metadata store into a BI dashboard (Looker/Metabase).- Weekly dashboards and monthly review of metric trends; tie action items to engineering OKRs.Why these matter: leading metrics (MTTD, MTTM, postmortem quality, verification rate) drive behavioral change and enable faster detection/learning; lagging metrics (recurrence, customer impact) measure ultimate business impact. Use targets as stretch goals and iterate after baselineing in month 0–1.
MediumTechnical
48 practiced
During a postmortem, an engineer loudly blames the data team for supplying bad labels while the data team counters that feature engineering changed semantics. As the ML engineer facilitating the review, how would you mediate the discussion to keep it productive and blameless while ensuring the root cause is found and actions assigned?
Sample Answer
Situation: In a postmortem I was facilitating, an engineer loudly blamed the data team for bad labels while the data team countered that upstream feature engineering changed semantics. Tension was rising and the meeting risked becoming unproductive.Task: As the ML engineer facilitator, I needed to keep the discussion blameless, discover the root cause, and produce clear, actionable follow-ups.Action:- Re-establish norms quickly: I reminded everyone this was a blameless postmortem focused on learning, not finger-pointing, and asked for concise facts only.- Reframe to a factual timeline: I proposed we build a timeline of events (data collection → labeling → feature engineering changes → model training → deployment → observed regression) and capture exact timestamps, commits, and config changes.- Surface evidence before opinions: I asked each side to state the specific observations (example: label distribution shifted by X%, feature null rate increased from A→B, model drift metrics) and to point to commits, schema changes, or dataset snapshots.- Isolate variables & form hypotheses: We listed plausible root causes (labeling process change, feature transformation change, data pipeline bug) and prioritized which to test first based on likelihood and cost to verify.- Assign forensic tasks with owners and deadlines: e.g., Data team to provide pre/post label histograms and labeling instructions (owner: Sam, 48h); FE team to run a diff of transformation logic and sample outputs (owner: Priya, 48h); ML to retrain model on archived data slices (owner: me, 72h).- Document interim findings in the postmortem doc and schedule a short sync in 72 hours to review results.Result: The structured, evidence-first approach calmed the room, produced a prioritized test plan, and led to discovering a config change in the feature pipeline that changed normalization ordering—once fixed and the model retrained, metrics recovered. We closed the loop by adding preventative actions: automated dataset snapshots before deployment, a checklist for transformation changes, and a shared SLA for providing labeled data artifacts.This taught me that quick norm-setting, a factual timeline, isolating variables, and assigning small, time-boxed forensic tasks turn heated postmortems into efficient root-cause investigations.
EasyBehavioral
45 practiced
Tell me about a time when an incident exposed a knowledge gap on your team (e.g., unfamiliar failure mode or tool). What concrete steps did you take to help the team learn and prevent recurrence, and how did you measure that the fix worked?
Sample Answer
Situation: On a production rollout of a recommendation model, we had a partial outage caused by an unfamiliar failure mode in the model-serving platform: GPUs on the inference cluster would hang under a specific mixed-batch request pattern. The incident revealed the team lacked experience debugging low-level GPU runtime issues and no runbook existed.Task: As the ML engineer on-call, I needed to restore service, identify root cause, and ensure the team could prevent and quickly recover from recurrence.Action:- Immediate: I worked with SREs to roll traffic to CPU-based replicas to restore availability within 45 minutes, and captured traces, logs, and a failing request replay.- Diagnosis: Reproduced the hang in a staging environment using the replayed requests and narrowed it to a PyTorch + CUDA driver interaction with mixed-batch sizes.- Knowledge capture: Wrote a postmortem documenting root cause, triggers, and mitigation steps; created a concise runbook with detection signals and rollback playbook.- Team enablement: Ran a 90-minute knowledge-sharing session demonstrating the reproduction steps, how to read relevant GPU logs, and how to run stress tests. Paired two engineers through the reproduction and mitigation the following week.- Preventive changes: Added automated nightly stress tests for mixed-batch inference on staging and pinned a tested CUDA/PyTorch driver matrix in our CI image builds.Result / Measurement:- Recovery time for similar issues fell from ~45 minutes (incident) to under 10 minutes in a simulated failure drill we ran two weeks later.- Nightly stress tests caught two driver-related regressions before they reached staging in the next quarter.- Postmortem follow-up survey: 100% of participants reported confidence in reproducing and mitigating the issue.This approach closed the knowledge gap, reduced mean time to recover, and prevented recurrence through automation and shared expertise.
Unlock Full Question Bank
Get access to hundreds of Learning from Incidents and Post Incident Review interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.