Incident Investigation, Root Cause Analysis, and Postmortems Questions

Covers the discipline of investigating and learning from production and technical incidents: forming and testing hypotheses, gathering and validating evidence, applying short-term mitigations versus long-term fixes, coordinating across teams during the incident, and running the postmortem or root cause analysis afterward. Candidates should describe the troubleshooting or investigative approach used, obstacles encountered, how mitigation and long-term remediation were sequenced, and the concrete process or system changes that resulted. Applies to incidents in software systems, ML/AI models and pipelines, infrastructure, and security findings.

HardTechnical

26 practiced

Implement (or outline in Python/pseudocode) an algorithm to compare SHAP-like feature contribution summaries between two model versions and detect significant shifts in feature importance. Address sampling variability, categorical features, and multiple-testing correction for many features.

Sample Answer

Approach (brief):- For each feature, compare the distribution of absolute SHAP values (or signed if direction matters) between model A and model B using a robust, nonparametric test that accounts for sampling variability (bootstrap / permutation).- For categorical features, aggregate per-category contributions and test per category or test the feature by using weighted averages across categories.- Correct for multiple testing across many features with Benjamini–Hochberg (FDR). Report effect sizes and bootstrap CIs.

Python outline (implements bootstrap-based two-sample test + BH correction):

python

import numpy as np
from scipy.stats import mannwhitneyu
from statsmodels.stats.multitest import multipletests

def bootstrap_mean_diff(x, y, n_boot=1000, seed=None):
    rng = np.random.RandomState(seed)
    diffs = []
    nx, ny = len(x), len(y)
    for _ in range(n_boot):
        bx = rng.choice(x, size=nx, replace=True)
        by = rng.choice(y, size=ny, replace=True)
        diffs.append(np.mean(bx) - np.mean(by))
    diffs = np.array(diffs)
    return np.mean(diffs), np.percentile(diffs, [2.5, 97.5])

def compare_shap(shap_A, shap_B, feature_meta, n_boot=1000, alpha=0.05):
    # shap_A / shap_B: dict feature -> array of per-sample shap contributions (signed or abs as chosen)
    pvals, effects, cis, names = [], [], [], []
    for feat, x in shap_A.items():
        y = shap_B[feat]
        names.append(feat)
        # If categorical, feature_meta[feat]['type']=='cat' and meta provides categories array per sample
        if feature_meta.get(feat, {}).get('type') == 'cat':
            # test per-category, combine using max-statistic p-value (conservative) OR weighted mean
            cats = feature_meta[feat]['categories']  # list of categories
            cat_p = []
            cat_effects = []
            for c in cats:
                maskA = feature_meta[feat]['values_A'] == c
                maskB = feature_meta[feat]['values_B'] == c
                if maskA.sum() < 5 or maskB.sum() < 5:
                    continue
                _, ci = bootstrap_mean_diff(x[maskA], y[maskB], n_boot=n_boot)
                cat_effects.append((np.mean(x[maskA]) - np.mean(y[maskB]), ci))
                # nonparametric p-value
                p = mannwhitneyu(x[maskA], y[maskB], alternative='two-sided').pvalue
                cat_p.append(p)
            pval = max(cat_p) if cat_p else 1.0
            effect = np.mean([e[0] for e in cat_effects]) if cat_effects else 0.0
            ci = np.mean([e[1] for e in cat_effects], axis=0) if cat_effects else (0,0)
        else:
            # numeric feature: compare distributions robustly
            # use Mann-Whitney U for p-value and bootstrap for effect & CI
            pval = mannwhitneyu(x, y, alternative='two-sided').pvalue
            effect, ci = bootstrap_mean_diff(np.abs(x), np.abs(y), n_boot=n_boot)
        pvals.append(pval); effects.append(effect); cis.append(ci)
    # Multiple-testing correction (FDR)
    rej, pvals_corr, _, _ = multipletests(pvals, alpha=alpha, method='fdr_bh')
    results = []
    for i, name in enumerate(names):
        results.append({
            'feature': name,
            'pvalue': pvals[i],
            'pvalue_adj': pvals_corr[i],
            'significant': bool(rej[i]),
            'effect_mean_diff': effects[i],
            'ci_95': tuple(cis[i])
        })
    return results

Key concepts and reasoning:- Use nonparametric tests (Mann–Whitney) because SHAP distributions are often non-normal and heavy-tailed.- Bootstrap mean differences provide effect sizes and CIs capturing sampling variability.- For categorical features, compare per-category contributions (or compute per-category mean |SHAP|) because categories can have very different importances; combine conservatively.- Multiple-testing correction via Benjamini–Hochberg controls FDR when testing many features.

Complexity:- O(F * (n + n_boot*(n_sample))) roughly; bootstrap dominates. Can subsample or reduce n_boot for speed; parallelize per feature.

Edge cases and best practices:- Ensure paired alignment (same test set) if samples correspond between models; use paired test (e.g., Wilcoxon signed-rank) when appropriate.- If sample sizes per category are tiny, aggregate categories or flag as insufficient data.- Report both statistical significance and practical effect size; small p-values with tiny effect may be ignorable.- Consider permutation-test alternative for exact null when distributional assumptions are suspect.

MediumTechnical

31 practiced

You deploy a new NLP model and production accuracy (measured on delayed human labels) is 12% lower than the canary tests. Provide a structured, step-by-step investigation plan you would execute in the first 2 hours to determine whether this is caused by data drift, a model regression, a serving bug, or label-sampling bias. Include which metrics and sample checks you perform.

Sample Answer

Goal: Triage root cause within 2 hours and produce actionable next steps (data drift, model regression, serving bug, or label-sampling bias).

0–10 min — quick facts & snapshot- Confirm timeframe, model version, canary vs prod config, rolling percent, and traffic split.- Record baseline canary metrics: accuracy, F1, sample size, latency, input source IDs, and delayed-label pipeline details.

10–40 min — sanity checks for serving bugs- Compare request/response logs (canary vs prod) for same model binary/hash, model config, tokenizer, preprocessing, and ensemble ordering.- Metric: % of inference errors/exceptions, input truncation rates, tokenization length distributions.- Sample check: 100 recent prod inputs and outputs vs same inputs replayed to canary/model locally. Look for divergences.

40–80 min — data drift assessment- Compute distributional metrics for key features/text: - Unigram & bigram frequency KL-divergence, embedding centroid drift (cosine distance), input length, language detection. - Thresholds: KL > 0.1 or embedding cosine distance > 0.05 as triggers (tune to org).- Sample checks: 200 random prod inputs vs historical/canary sample; inspect top-50 differing tokens and classes.

80–100 min — label quality and sampling bias- Verify delayed-label pipeline: labeling coverage, sampling method, labeler pool, label latency.- Metrics: label completion rate, inter-annotator agreement (kappa), label distribution vs expected.- Sample check: 100 labeled prod samples; compare blind re-label by senior annotator or gold set.

100–120 min — model regression vs upstream change- Run A/B backtest: score a holdout of recent prod inputs with the canary and previous known-good model.- Metrics: delta accuracy/F1/confidence calibration, class-wise confusion matrices, softmax temperature shifts.- If canary matches but prod differs → serving or input pipeline issue. If canary worse on same inputs → model regression. If labels disagree or low agreement → label bias.

Deliverable after 2 hours:- One-page report: findings (evidence + metrics), priority (likely cause), immediate mitigations (rollback to previous version, stop labeling, replay pipeline), and next actions (deeper forensic, fix and redeploy).

EasyBehavioral

32 practiced

As an AI Engineer, explain what a blameless postmortem is for a production AI model incident. Describe the essential sections that should appear in the postmortem report (for example: incident timeline, symptoms, root cause analysis, immediate mitigation, long-term action items, owners, and measurable success criteria). Explain why each section is important and how you'd ensure the postmortem results in concrete process or system improvements.

HardSystem Design

27 practiced

Design an end-to-end incident response architecture for AI models serving 100M inferences/day. Requirements: fast detection of quality degradation, automated mitigations (rollbacks/fallbacks), forensic data capture (prompts, retrievals, model outputs), multi-region failover, low additional latency, and immutable audit trails for compliance. Provide high-level components, data flows, storage options, and key trade-offs.

Sample Answer

Requirements clarified:- Detect quality degradation quickly across 100M/day inferences (~1.2k rps avg, peak higher).- Automated mitigations: rollback, fallback routing.- Capture forensic data (prompts, retrievals, model outputs, metadata).- Multi-region failover, minimal added latency, immutable audit logs for compliance.

High-level architecture:- Inference Gateway (edge + regional): lightweight proxy that performs routing, sampling, schema validation, and synchronous low-latency checks.- Telemetry Pipeline: high-throughput streaming (Kafka/Cloud PubSub) for events: sampled inputs, retrieval traces, model outputs, latencies, confidence scores, and behavioral signals.- Real-time Detector Service: streaming analytics (Flink/Beam) and ML monitors (drift detectors, metric thresholds, anomaly detection) emitting alerts and automated actions.- Orchestrator / Remediation Engine: policy engine (rego) + workflow runner that can trigger A/B config flips, instantaneous fallback to safe models, or full rollback via CI/CD APIs.- Forensics Store: append-only object store for raw artifacts (S3 with versioning + WORM like S3 Object Lock / Azure Immutable Blob) plus a searchable index (Elasticsearch/Opensearch and BigQuery) for analysis.- Audit Trail Service: write-once logs (blockchain-backed or cloud WORM + WAF-signed events) storing decisions, signatures, and who/what triggered actions.- Multi-region Control Plane: active-active region-aware routing (DNS + Anycast + regional load balancers) with cross-region replication of telemetry and forensics (async, strong-consistency for control metadata).- Operator UI & Playbooks: incident dashboard, runbooks, and replay tooling.

Data flows (brief):1. Client -> Edge Gateway (light checks, sampling).2. Gateway routes to regional model endpoint; concurrently sends sampled event to Telemetry Pipeline.3. Telemetry -> Real-time Detector -> if anomaly, send to Orchestrator.4. Orchestrator executes mitigation (e.g., route to fallback model, increase sampling, rollback), logs immutable audit, and stores forensic snapshot.

Storage options:- Hot metrics: TSDB (Prometheus/Influx) and streaming store for detectors.- Searchable forensic: Opensearch + column store (BigQuery/Redshift) for analytics.- Immutable audit: S3 Object Lock / Azure Immutable Blob or ledger DB (AWS QLDB) + cryptographic signatures.

Key trade-offs:- Latency vs. observability: higher sampling rate improves detection but increases bandwidth/storage and slight added per-request overhead — mitigate by async sampling and token-based minimal synchronous capture.- Consistency vs. availability in multi-region: prefer eventual replication for forensics to keep low latency; keep control plane replicated with consensus for safety-critical ops.- Storage costs vs. retention: compress/anonymize PII, tier older data to cold storage; keep high-fidelity recent window.- Automation risk: automated rollback reduces MTTR but can oscillate — use conservative thresholds, cooldowns, and human-in-the-loop approvals for high-impact actions.- Immutable logs complexity: WORM/ledger adds cost and operational overhead but required for compliance.

Final notes:- Instrument confidence, calibration, backpressure signals, retrieval provenance (hashes), and semantic logging.- Build replayable traces to reproduce incidents offline.- Enforce RBAC, signed policies, and regular tabletop tests to validate automation safety.

HardTechnical

28 practiced

A production LLM is hallucinating personal data (names, addresses) in responses. Draft an investigative plan that includes immediate containment actions, hypotheses about failure modes (e.g., retrieval contamination, training data leakage, prompt templates), the forensic artifacts you need (prompt history, retrieval logs, datastore snapshots, model logits), and concrete prevention measures (retrieval augmentation, sanitization, red-teaming).

Sample Answer

Investigative plan — production LLM hallucinating personal data

1) Immediate containment (first 24 hours)- Roll back to a safe snapshot or disable the model endpoint for sensitive-use paths. Put service into “read-only / degraded” mode that routes queries to a stub response explaining temporary unavailability.- Block all downstream logging/export of generated outputs to external systems and stop any automated workflows that ingest LLM outputs.- Apply a runtime filter: deny any generation containing detected PII patterns (names, addresses, SSNs) using regex/NER and return safe fallback.- Notify stakeholders (privacy, legal, ops) and create an incident channel with access control.

2) Hypotheses (prioritized)- Retrieval contamination: vector DB contains user-submitted PII or poisoned documents; retrieval augmentation returns verbatim PII.- Training-data leakage: model memorized personal data from pretraining or fine-tuning corpora.- Prompt-template leakage: prompt or system messages include placeholders or example text with real PII that get surfaced.- Context-window bleed: concatenated user history contains earlier PII that model reuses.- Post-processing bug: hallucinated PII injected by enrichment code (entity resolver, knowledge augmentor).- Model hallucination mechanism: temperature or decoding settings causing confident fabrication of plausible PII.

3) Forensic artifacts to collect- Prompt history and full context for all requests (with access-controlled copy).- Retrieval logs: queries, returned document IDs, similarity scores, timestamps.- Datastore snapshots: current and historical dumps of vector DB, metadata, and raw documents.- Model artifacts: fine-tuning datasets, training manifests, data provenance records.- Model outputs and logits (or token probabilities) for representative queries to see token-level confidence.- Server logs, middleware code (post-processors, templates), deployment manifests and any recent config changes.- Access logs: who/what wrote to vector DB or uploaded documents.

4) Investigation steps- Reproduce with minimal input to isolate whether PII appears with blank or controlled prompts.- Correlate generated PII with retrieval hits — check if exact phrases exist in datastore snapshots.- Inspect prompt templates and recent changes for hard-coded examples containing real names/addresses.- Run membership tests on model using candidate PII from suspected training sources.- Examine logits to see if model is regurgitating high-probability memorized sequences vs. low-prob hallucinations.- Audit recent ingestion pipelines for user data leakage into public corpora or vector DB.

5) Concrete prevention measures- Retrieval augmentation: apply strict document filtering, provenance scoring, and redact PII in retrieved documents before concatenation.- Sanitization: enforce automatic PII detection and redaction on ingested data; normalize or anonymize examples in prompt templates and training corpora.- Access controls & provenance: enforce write-auth, signed ingestion, immutable change logs for datastore; quarantine new data until approved.- Runtime defenses: NER-based PII detectors, response sanitizers, and a “PII hallucination” circuit-breaker that blocks/scrubs outputs.- Model-level: reduce temperature / use constrained decoding and use token-level penalization for personal-name patterns when appropriate.- Training hygiene: remove or sanitize identified PII from training/fine-tuning datasets; use differential privacy for sensitive data.- Red-teaming & monitoring: scheduled adversarial tests (prompt injection, retrieval poisoning), continuous monitoring for PII triggers, alerting, and post-deploy audits.- Operational: maintain incident runbooks, periodic privacy reviews, and staff training on data handling.

6) Post-incident- Notify affected users/regulators per policy, preserve evidence, and implement prioritized fixes (retrieval sanitization, prompt template cleanup) before full restore. Track metrics: PII-rate per million responses, time-to-detect, time-to-mitigate.

This plan balances immediate risk reduction, precise forensics to identify root cause, and layered defenses to prevent recurrence.

Unlock Full Question Bank

Get access to hundreds of Incident Investigation, Root Cause Analysis, and Postmortems interview questions and detailed answers.

Join thousands of developers preparing for their dream job.