Evaluates how a candidate responds to urgent, high stakes, or time sensitive incidents such as production outages, security incidents, regulatory investigations, compliance failures, customer escalations, or other critical operational problems. Interviewers assess the candidate's ability to rapidly gather and prioritize incomplete or ambiguous information, perform quick diagnosis and root cause analysis, triage and prioritize multiple competing issues, and make pragmatic decisions under time pressure using clear decision criteria. The scope includes short term containment actions, trade offs between temporary workarounds and longer term fixes, risk identification and mitigation, escalation thresholds, and knowing when to pause for more information or to delegate and call for help. Candidates should demonstrate clear and concise stakeholder communication, documentation of rationale, attention to accuracy and quality under deadlines, stress and resilience strategies, and mechanisms to follow up and prevent recurrence by implementing safeguards and lessons learned. At senior levels this also includes leading teams through incidents, setting priorities under pressure, coordinating cross functional stakeholders, maintaining team morale, and measuring outcomes and impact. Strong answers use concrete examples of specific incidents, the decision criteria used, trade offs made when data was limited, how uncertainty and stress were managed, and what was learned and institutionalized afterward.
HardTechnical
51 practiced
How would you structure a forensic evidence collection process for a complex ML incident that may require legal review? Include steps for immutable logging, access control, timestamps, hashes, and chain-of-custody considerations so the collected evidence is defensible.
Sample Answer
Situation & scope: First define the incident boundaries (models, training data, feature store, serving infra, logs, alerting traces, personnel, timeframe) and whether evidence may be used in litigation or regulatory review. Notify legal/compliance and preserve volatile state immediately.Step-by-step defensible collection1. Triage & preserve- Isolate affected systems (quarantine containers/hosts) to prevent contamination.- Snapshot ephemeral state (VM snapshots, container images) and memory dumps via forensically-sound tools.2. Immutable logging & timestamps- Ensure all collected logs are stored in append-only WORM/immutable object storage (S3 Object Lock/legal hold or internal immutable filesystems).- Record system times and sync status (NTP/chrony) at collection time. Obtain RFC 3161/Timestamp Authority (TSA) timestamps for key artifacts to prove time of capture.3. Hashing & integrity- Compute strong cryptographic hashes (SHA-256 or SHA-3) of every artifact immediately after capture. Record hashes in write-once logs and sign them with a dedicated evidence key (HSM or KMS).- For large datasets or many files, produce Merkle trees so individual items can be later verified efficiently.4. Access control & audit- Apply least-privilege RBAC for who can access evidence buckets. Enforce MFA and ephemeral admin credentials.- Log all access to evidence storage in an immutable audit trail (SIEM) and ship audit logs to separate, append-only storage.5. Chain-of-custody (CoC) documentation- For every artifact record: collector identity, role, collection method, date/time (UTC + TSA), hash, storage location, and transfer history.- Use standardized CoC forms (digital) signed by each custodian; store signatures and CoC entries in immutable storage.6. Transport & storage- Transfer via encrypted channels (TLS 1.3) with endpoint verification; where possible, use physical transport with tamper-evident seals.- Store master copies in encrypted, access-restricted vaults; maintain read-only working copies for analysts.7. Evidence catalog & reproducibility- Capture reproducible environment artifacts: container images, model binaries, training code, dependency manifests, dataset snapshots (with sample indices), model configuration, hyperparameters, and serving config with timestamps.- Record provenance metadata linking model versions to datasets and code commits (commit hashes).8. Verification & validation- On any later access, re-hash artifacts and compare to recorded signed hashes/TSA timestamps. Keep HSM-stored keys to allow signature verification.9. Legal coordination & retention- Follow legal hold and retention policies; involve counsel before any deletion or normal retention expiration.- Prepare tamper-evidence and attestation packets for legal teams (signed hashes, TSA receipts, CoC log).Automation & playbook- Automate snapshot + hash + TSA timestamp + CoC entry pipeline via an incident response tool (playbooks) to reduce human error.- Regularly test the process with table-top exercises and maintain an evidence readiness kit (trusted keys, scripts, contacts).Tools & examples- Immutable object storage (S3 Object Lock), TSA services (RFC 3161), HSM/KMS for signing (AWS KMS / Cloud HSM), SIEM for audit, open-source forensic tools (Volatility for memory, plaso for log timelines), and Merkle-tree libraries for large datasets.Why this is defensible- Immutable storage, TSA timestamps, HSM-signed hashes, and strict CoC create provenance, integrity, and auditability. Combining automation with legal oversight minimizes human error and preserves admissibility.
HardTechnical
60 practiced
Explain how to instrument canary analysis for ML model quality using statistical hypothesis tests. Describe which test(s) you would use for metrics like CTR or precision, sample-size considerations, and how to interpret p-values vs practical significance under time pressure.
Sample Answer
Approach (summary)Instrument canary analysis by treating the canary as an experiment: collect metric observations for control (current model) and treatment (canary). Use appropriate statistical tests for metric types, plan sample sizes up-front, and combine statistical significance with practical ("minimum detectable effect") thresholds. Under time pressure prefer sequential-friendly methods.Which tests to use- Click-through rate (CTR): CTR is a binomial proportion. Use a two-proportion z-test (large samples) or Fisher’s exact test (small counts). For streaming/online checks, use a sequential test like the O’Brien–Fleming group sequential design or a Wald Sequential Probability Ratio Test (SPRT).- Precision (or other rates): Also a proportion — same two-proportion z-test or exact test. If metric is a continuous score (e.g., latency, probability calibration), use t-test or nonparametric (Mann–Whitney) depending on distribution.Sample-size considerations- Compute sample size from desired power (1−β, commonly 80–90%), significance α (commonly 0.01–0.05 for production), baseline rate p0, and minimum detectable effect (MDE). For proportions: n ≈ (Z1−α/2√(2p̄(1−p̄)) + Z1−β√(p1(1−p1)+p2(1−p2)))^2 / (p1−p2)^2 where p̄=(p1+p2)/2. In practice, pick MDE reflecting business impact (e.g., 1% relative CTR lift). Remember to inflate for multiple looks (alpha spending) or use sequential methods.Interpreting p-values vs practical significance- p-value answers: if there’s evidence against the null (no difference). Small p-value + tiny effect that’s business-irrelevant should not trigger rollout. Always report effect size and confidence interval (CI). Use pre-specified MDE: only act if CI excludes negligible effects.- Under time pressure: prefer sequential tests or alpha-spending so you can stop early without inflating false positives. Also consider Bayesian A/B (posterior probability treatment > control) which gives more intuitive probabilities and supports stopping rules.Operational considerations- Monitor metrics for data quality, sample ratio mismatches, and correlated users (cluster by user in variance estimation).- Correct for multiple metrics with hierarchical prioritization or false discovery control (Benjamini–Hochberg) if needed.- Automate alarms: require both statistical significance and minimum practical uplift, and always run a short canary window followed by a longer validation stage before full rollout.Example short rule: require (1) ≥95% posterior probability treatment CTR > control by at least MDE, (2) no degradation on safety metrics, (3) stable data mapping—then promote.
MediumTechnical
49 practiced
You are designing SLOs for an online ML recommendation service to drive incident severity decisions. Propose 3 SLOs (with error budget definitions) that are most relevant to ML incidents, and explain how breach of each SLO should map to incident priority and escalation paths.
Sample Answer
SLO 1 — Recommendation Quality (online relevance)- SLO: 95% of user sessions receive recommendations with normalized relevance score ≥ threshold T over a 7-day rolling window.- Error budget: 5% of sessions per 7 days where relevance < T.- Breach → Severity mapping: - Minor overshoot (error budget burn 0–25%): P3 — notify ML on-call; create ticket to investigate recent model changes or feature drift. - Moderate (25–75%): P2 — page ML on-call and SRE; revert most recent model/feature deploy, roll back feature flags, run quick validation job on recent training data. - Severe (>75% or absolute drop below 90%): P1 — page ML lead, SRE, product owner; emergency rollback, pause online experiments, engage data-team to check data pipeline and label skew.- Rationale: relevance directly impacts user experience and business metrics.SLO 2 — Model Prediction Latency- SLO: 99.9% of inference requests < 200ms p95 over 24 hours.- Error budget: 0.1% of requests per 24h exceed 200ms.- Breach → Severity mapping: - Minor: P3 — alert SRE; check autoscaling/queueing. - Moderate: P2 — page SRE and ML infra; temporarily scale replicas, enable degraded lightweight model. - Severe: P1 — page SRE, ML infra lead, product; failover to cached/popular recommendations, throttle incoming traffic.- Rationale: latency affects throughput and converts to cascading backend failures.SLO 3 — Data Pipeline Freshness / Completeness- SLO: 99% of required feature batches arrive and pass integrity checks within 1 hour of window close (daily), measured over 7 days.- Error budget: 1% of windows can be late/invalid per 7 days.- Breach → Severity mapping: - Minor: P3 — alert data engineering; run manual backfill. - Moderate: P2 — page data engineer and ML on-call; pause model retraining, run incremental backfill, evaluate model degradation risk. - Severe: P1 — page data, ML, SRE, product; stop automated deploys, switch to fallback features or last-known-good snapshot, initiate root cause postmortem.- Rationale: stale or missing features cause silent model degradation and bad behavior.Operational notes:- Track error budget burn rates and automate escalation: e.g., if burn >50% in 24h, auto-escalate one level.- Tie SLOs to runbooks that list quick mitigations (rollback, degrade, cache, backfill) and responsible roles.
MediumTechnical
61 practiced
You discover that a feature-store ingestion job silently dropped 10% of rows in the last 24 hours. What immediate containment, verification, and communication steps would you take as the ML engineer responsible for dependent models? Be explicit about what you would do in the first hour.
Sample Answer
Situation: At 09:00 I notice monitoring alerts or a report showing ~10% drop in rows ingested to the feature store over the last 24 hours. This affects downstream models that rely on those features.First-hour plan — containment, verification, communication (explicit minute-by-minute actions):0–10 min — Triage & contain- Immediately mark the ingestion job as “investigating” and, if possible, pause or stop further runs to prevent more inconsistent writes (run kubectl scale or stop scheduler job).- Disable downstream automated retraining/serving pipelines or set them to “read-only / hold” to avoid serving models trained or scoring on partial data (flip a feature-flag or pause CI/CD job).10–25 min — Quick verification & scope- Run quick checks to confirm the 10% drop: query row counts/timestamps per partition in the feature-store (example SQL):
sql
SELECT ingestion_date, COUNT(*) FROM feature_table
WHERE ingestion_date >= current_date - INTERVAL '2 day'
GROUP BY ingestion_date;
- Compare schema, null rates, row counts with last successful run; check logs for errors, timeouts, upstream source lags, or API rate-limit errors.- Identify which features and partitions are affected and list downstream models that consume those features (lookup model->feature mapping).25–40 min — Root-cause signals collection- Pull ingestion job logs, orchestrator logs (Airflow/Kubernetes), and upstream source health dashboards. Search for exceptions, retries, throttling, or schema mismatches.- Check recent deploys/PRs to ingestion code, config changes (credentials, retention, filters), and infra changes (IAM, network).40–55 min — Impact assessment & mitigation plan- Classify impact: models for decisioning (high risk, pause), reporting (medium), experiments (low). For high-risk models, set routing to fallback models or manual review.- If a clear reversible config caused the drop (e.g., a new filter), prepare a rollback plan and test in staging.55–60 min — Stakeholder communication- Send an initial incident message to relevant channels (Pager/Slack/Email) with: - What: 10% ingestion drop detected affecting X features and Y models - When: observed at 09:00, window last 24h - Immediate actions taken: ingestion paused, downstream serving paused for critical models - Next steps and ETA for deeper investigation (e.g., “investigating root cause, will update in 60 minutes”)- Notify engineering on-call, data engineering, product owner, and risk/compliance if decisions are impacted.Why these steps- Containment prevents further bad data propagation.- Quick verification avoids false alarms and narrows scope.- Pausing downstream systems prevents automated decisions based on incomplete data.- Immediate communication aligns stakeholders and reduces business risk.Follow-ups after the first hour: implement rollback or backfill, run data-consistency checks, perform full root-cause postmortem, restore normal ops only after validation and communicate remediation and monitoring improvements.
MediumTechnical
57 practiced
You find evidence that a recent model retrain used corrupted labels from a downstream data source. Describe a pragmatic rollback and remediation plan that minimizes business disruption, preserves auditability, and prevents labeled-corruption recurrence.
Sample Answer
Situation: During post-deploy validation I discover a recent retrain used labels from a downstream source that are corrupted (systematic bias/format bug). This threatens model quality and business decisions.Immediate rollback (minimize disruption, preserve audit trail)1. Hot rollback to the last known-good model version in serving (use model registry like MLflow): switch traffic to previous stable model with weighted canary or 100% if impact high.2. Freeze retrain/build pipelines and mark the offending training run as “invalid” in the experiment tracking system; capture run ID, git commit, data snapshot IDs, container image, and serving config for audit.3. Create an incident ticket and notify stakeholders (ML, data engineering, product, compliance).Remediation (repair data + retrain safely)1. Quarantine the corrupted label dataset: snapshot current raw inputs and labels to immutable storage, record checksums and dataset lineage in data catalog.2. Root-cause analysis: identify how corruption entered (ETL bug, schema mismatch, human labeling tool issue). Reproduce on a small sample.3. Repair labels: if possible, restore from backups or rerun deterministic labeling. Otherwise, sample + human review (HITL) to relabel a representative subset and estimate corruption rate.4. Rebuild training dataset excluding corrupted partitions or corrected labels. Validate with automated data quality tests (label distributions, label-feature correlation checks, statistical drift tests).5. Retrain in an isolated environment; run full validation (unit tests, CI, explainability checks, fairness metrics). Run shadow deployment to compare with production on real traffic without affecting decisions.6. Only promote new model after passing canary A/B tests and stakeholder sign-off.Prevent recurrence (process & tooling)1. Enforce data contracts and schema validation at ingestion; fail fast with alerts if constraints violated.2. Add label quality checks: labeler consistency metrics, inter-annotator agreement, and automated heuristics to detect outliers.3. Integrate lineage and immutable snapshots for every dataset used in training; require dataset IDs in model registry.4. Harden CI/CD: require retrain approvals, automated QA gates, and shadow testing before production rollout.5. Access controls and audit logs for label pipelines and manual labeling tools.6. Schedule periodic audits and monitoring: label drift alerts, model performance regression detection, and dashboards for stakeholders.Why this approach: immediate rollback stops bad decisions quickly; quarantining and snapshotting preserves forensic evidence; isolated retrain + shadowing prevents repeated harm; data contracts and automated checks reduce human/ETL error risk while keeping an auditable trail for compliance.
Unlock Full Question Bank
Get access to hundreds of Crisis Management and Decision Making interview questions and detailed answers.
