Model Degradation, Sunset & Stakeholder Alignment Questions

Lifecycle management of ML models in production, including monitoring for data drift and concept drift, detecting performance degradation, planning model updates or retirement (sunsetting), governance, risk management, and aligning stakeholders on decommissioning decisions.

EasyTechnical

56 practiced

List the key production monitoring metrics you would track for (A) binary classification models and (B) regression models. For each metric explain why it matters, how it can indicate degradation, and whether it is a technical metric (e.g., latency) or a business metric (e.g., revenue). Also discuss differences between online (real-time) and offline monitoring use-cases.

Sample Answer

For production monitoring, track metrics that detect model quality drift, data issues, and operational problems. Below are practical metrics and why they matter.

A) Binary classification- Accuracy / Balanced Accuracy — business/technical: simple health check; falling values indicate label drift or concept drift; sensitive to class imbalance.- Precision & Recall (and F1) — business: show false-positive/false-negative impact (e.g., fraud vs. missed fraud); shifts reveal degradation in particular error type.- ROC-AUC / PR-AUC — business: threshold-agnostic discrimination; dropping AUC signals loss of separability.- Confusion matrix counts — business: gives actionable error breakdown.- Calibration (e.g., Brier score, reliability plots) — business: miscalibration affects downstream decisions and risk scoring.- Population Stability Index (PSI) / feature drift metrics — technical/business: detect input distribution shifts causing performance loss.- Latency, throughput, error rates — technical: service health and SLAs.

B) Regression- RMSE / MAE / MAPE — business: measure prediction error; increasing error shows model becoming less accurate.- R² — business: proportion of variance explained; decline implies concept drift.- Residual distribution / bias over segments — business: reveals systematic under/over-prediction.- Prediction intervals coverage — business: uncertainty calibration; coverage drop signals misestimated uncertainty.- PSI / feature drift and input null rates — technical/business.- Latency and error rates — technical.

Online vs Offline use-cases- Online (real-time): monitor latency, throughput, per-request errors, recent prediction distributions, immediate business KPIs (conversion, fraud rate). Enables alerting and automatic rollback. Requires streaming, low-latency aggregation and short windows to catch sudden issues.- Offline (batch/backtest): compute holdout-label metrics (AUC, RMSE), longer-term drift analysis, retraining triggers, fairness and slice analyses. Useful for diagnosing root causes and scheduling retraining.

Practical notes- Combine technical and business metrics; set tiered alerts (e.g., latency first, then performance drops). Use statistical tests (KS/PSI) and labeling pipelines to tie offline ground-truth to online signals.

HardTechnical

69 practiced

During an audit you are asked to prove that a retired model did not use a banned data source. The model registry currently lacks explicit lineage linking. Describe the emergency steps you would take to gather evidence for the auditor now, and describe the longer-term engineering and governance changes (mandatory lineage capture, producer-side tagging, ingestion-time source IDs, immutable logs) you would implement to ensure full lineage for future models.

Sample Answer

Situation: An auditor demands proof that a retired model never used a banned data source, but the registry lacks lineage. This is urgent because noncompliance has legal and reputational risk.

Immediate (emergency) steps to gather evidence:1. Freeze artifacts — snapshot model binary, training code, config, and any stored feature/metadata; record timestamps and hashes for chain-of-custody.2. Query available backups and storage: search training job logs (Kubernetes/airflow/ML platform), data access logs (S3/GCS access logs, DB audit trails), and ETL job logs for any mentions of the banned source or its table/key patterns.3. Reconstruct training environment: obtain the exact training commit, dependency hashes, and run parameters; re-run a dry instrumentation of the data loading step against retained training inputs (if available) to enumerate input datasets.4. Inspect feature store and feature tables referenced by model code or serialized feature lineage files (feature IDs, lookup keys). Cross-check with data catalog entries and dataset owners.5. Collect system-level evidence: IAM access logs, network logs, and scheduler execution records showing which datasets were read during the training window.6. Produce a signed, time-stamped evidence package for the auditor summarizing findings, gaps, and confidence level; if gaps remain, explicitly document why and that remediation is underway.

Longer-term engineering & governance changes to guarantee lineage:1. Mandatory lineage capture in model registry: require recorded edges from model → features → datasets → ingestion jobs with immutable metadata (producer, version, timestamp, hash).2. Producer-side tagging: enforce producers to embed canonical source IDs and dataset-version metadata at ingestion time (e.g., file manifest with dataset_id:version and schema hash).3. Ingestion-time source IDs: assign deterministic dataset IDs and content hashes at ingest; log these to an append-only store (e.g., write-ahead log) and publish to the data catalog.4. Immutable audit logs: stream all data reads/writes, job starts/completions, and model training events into a tamper-evident ledger (WAL or cloud audit logs + signed entries). Retain for compliance windows.5. Feature store integration: require feature definitions to include upstream dataset links and transformations; store computed feature provenance and batch/online lineage.6. Automated lineage enforcement: block model registry promotion to production unless lineage is complete; add CI checks that validate that all features point to permitted sources.7. Tooling and governance: implement a data catalog with dataset ownership, allowed-use policies, and automated alerts on banned-source usage. Provide playbooks for auditors and run periodic lineage audits and red-team tests.8. Culture and process: mandate producer SLAs, training for data engineers on tagging, and incident playbooks for possible lineage gaps.

These steps provide immediate evidentiary reconstruction and a durable, automated lineage system that prevents future ambiguity and enables rapid, auditable proofs of provenance.

EasyTechnical

60 practiced

Define label drift and explain why it is often harder to detect than feature drift in production. Describe one pragmatic approach to detect label drift when labels arrive with significant latency (for example, weeks or months after prediction).

Sample Answer

Label drift (a.k.a. target drift) is a change over time in the distribution P(Y) or in P(Y|X) such that the relationship between inputs and the target, or the base-rate of the target, shifts relative to the training period. It differs from feature drift (changes in P(X)) because it directly affects the ground-truth you’re trying to predict.

Why it’s harder to detect than feature drift:- Features are observable immediately; you can run offline statistical tests on incoming X in real time.- Labels often arrive with latency (e.g., weeks/months), so you can’t directly compare predicted vs actual Y promptly.- Some performance proxies (score distributions, calibration) can remain stable even when true label rates change, masking impact.

Pragmatic approach for delayed labels — cohort-based delayed-evaluation with rolling baselines:1. Group predictions into time-cohorts (e.g., daily/weekly) and record prediction metadata (timestamp, score, cohort id).2. When labels eventually arrive for a cohort, compute cohort-level metrics: observed label rate, calibration error, and prediction distribution.3. Compare each cohort’s observed label rate to the expected rate (model’s mean predicted probability) using a statistical test (e.g., two-proportion z-test or bootstrap for small samples) or divergence metric (Jensen–Shannon).4. Maintain a rolling baseline (moving average of recent cohorts) and flag cohorts whose observed vs expected difference exceeds a threshold (statistically significant and business-relevant).5. Backfill alerts: because labels are delayed, raise an alert only when drift is confirmed, but use interim proxies (score distribution shifts, sudden calibration change) to prioritize investigations.

Why this works:- Respects label latency but still gives timely detection once labels arrive.- Cohort aggregation reduces noise and enables statistical testing.- Combining with score/covariate monitoring provides an early-warning triage before labels come in.

Practical tips:- Choose cohort size to balance statistical power vs detection delay.- Use bootstrap CIs if sample sizes are small or outcomes rare.- Track effect size and business impact (e.g., change in false positive cost) — not just p-values.

MediumTechnical

57 practiced

A fraud detection model's precision drops by 15% while recall remains roughly the same. Describe a systematic investigation plan: enumerate the data sources and logs you would examine, the statistical comparisons and tests you would run (feature distribution comparisons, threshold checks), sampling strategies for manual review of false positives, and how you would determine root cause and remediation steps.

Sample Answer

High-level goal: isolate why precision (FP rate up) fell while recall stayed stable. I would run a structured RCA in parallel tracks: data, model, threshold/serving, and labeling/feedback.

1) Data & logs to inspect- Ingestion logs: row counts, null rates, schema changes, timestamps, failed records.- Feature-store / transform logs: versions, code commits, missing feature imputation, batch vs. online consistency.- Model serving logs: score distributions, request metadata, model version, latency, feature availability per request.- Downstream transaction logs: merchant/category, device, geolocation, payment method.- Label pipeline logs: label arrival delay, labeling rules/SLAs, human review logs.

2) Statistical comparisons / tests- Compare score distributions (current vs baseline): KS test, Wasserstein distance, PSI per time window.- Feature distribution shifts: PSI for numeric/categorical, Chi-square for categories, t-test where appropriate.- Correlate feature importance / SHAP change: compare average SHAP per feature (bootstrap CI).- Confusion matrix over cohorts: by time, merchant, device, geography.- Calibration check: reliability diagram, Brier score.- Threshold sensitivity: precision-recall curve, check if threshold drifted or post-processing rules changed.

3) Sampling strategy for manual review of false positives- Stratified sampling by score band (high-confidence vs near-threshold), time window (before vs after drop), merchant, user-risk, and feature-change cohorts.- Oversample new/rare merchants and top contributors to PSI/feature drift.- For each sample: save raw features, transformed features, model score, SHAP explanation, downstream outcome, reviewer notes.

4) Determining root cause- Triangulate evidence: e.g., increased PSI in feature X + higher SHAP magnitude for X on FPs suggests feature shift causing misclassification.- If score distribution unchanged but FP up in specific merchants → upstream data change or business process change.- If labels delayed/changed → apparent precision drop due to label lag or policy change.- Use A/B (shadow) replay: rerun historical requests through current model and current requests through previous model; see which model/version causes FPs.

5) Remediation steps- Short-term: tighten threshold or add ensemble/rule-based blocker on high-FP cohorts; deploy human-in-loop for affected slices.- Medium-term: fix data pipeline (impute, backfill, schema), retrain with recent drifted data, update feature engineering, recalibrate probabilities.- Long-term: add monitoring (automated PSI, SHAP drift alerts), CI for feature code, automated shadow testing for new model releases, improved labeling latency analytics.

Metrics to validate fixes: precision by cohort, overall false positive rate, business cost (chargebacks, conversion), and monitor for unintended recall degradation.

MediumTechnical

50 practiced

Describe how you would detect and mitigate a feedback loop where model predictions influence future training labels (for example, a recommendation model using clicks as positive labels). Provide monitoring signals that indicate feedback bias (e.g., rising correlation between score and label), and list mitigation techniques such as randomized exploration, delayed labeling, counting unique exposures, or counterfactual estimation.

Sample Answer

Approach: Treat this as a data-generation bias problem — model scores change user behavior, which changes labels, which then retrains the model (positive feedback). Detection requires monitoring causal signals, then mitigate by changing data collection and training.

Monitoring signals (what to track and why):- Rising correlation corr(score, label) over time beyond population shifts — suggests score is driving labels.- Increasing score-conditioned label lift: P(label|score) rising while overall base rate stable.- Reduction in diversity of exposures (fewer unique items/users) and concentration metrics (Gini/HHI) — model narrowing.- Label latency distribution shortening (more immediate clicks after exposure) — indicates promoted items get clicks faster.- Exposure imbalance: items with high predicted score get disproportionately more impressions and labels.- A/B test drift: control vs treatment label divergence after serving model.

Mitigations (practical techniques and trade-offs):- Randomized exploration: allocate epsilon of traffic to random or uniform sampling to collect unbiased labels. Trade-off: short-term utility loss.- Delayed labeling / hold-out logging: log labels from a randomized holdout cohort not served model recommendations; use for evaluation and training to avoid contaminated labels.- Count unique exposures + propensity weighting: track exposure counts per (user,item) and weight labels by 1/propensity to correct for selection.- Counterfactual estimation / IPS/DR: use logged propensities to estimate unbiased reward; combine with doubly robust estimators.- Train on de-biased dataset: remove or downsample labels from high-exposure windows or use inverse propensity scoring.- Causal modeling: model effect of exposure on label via uplift models; include exposure as feature and regularize.- Periodic shadow models / offline evaluation: score but don’t serve for a cohort to get fresh labels.- Monitoring + automated alerts: set thresholds on corr(score,label), exposure concentration, and drift; trigger mitigation rollout.

Example: implement epsilon-greedy (5% random) + log propensities; daily compute corr(score,label) and HHI; if corr increases > X sigma, increase randomness or revert to holdout training data. This balances business metrics and long-term model health.

Unlock Full Question Bank

Get access to hundreds of Model Degradation, Sunset & Stakeholder Alignment interview questions and detailed answers.

Join thousands of developers preparing for their dream job.