Data Investigation and Root Cause Analysis Questions

Direct answer. A systematic root-cause plan for a production model's accuracy drop needs to check, in a deliberate order, whether the world the model sees at serving time still matches the world it was trained on: data drift and label shift first (since those are both common and relatively cheap to check), then feature-importance and upstream schema changes, then serving-versus-training skew, before assuming the model itself has somehow degraded.

Structured elaboration. Data drift check: compare the distribution of each input feature in recent production traffic against the training-time (or a recent stable baseline) distribution, using a distribution-comparison test appropriate to the feature type. Label shift check, where ground truth is available with some delay: has the base rate or distribution of the TARGET itself changed, independent of any feature drift, since a model calibrated for one label distribution can look worse purely because reality's label mix shifted. Upstream schema or transformation changes: did a feature-computation pipeline change (a unit conversion, a default-value change, a join that started producing nulls) in a way that alters what the model receives without anyone touching the model itself. Serving-versus-training skew: does the exact same input, computed through the serving pipeline, produce the same feature values as it would through the training pipeline, since a subtle mismatch between the two (different libraries, different rounding, different handling of missing values) is a surprisingly common, hard-to-spot cause. The decision criteria: retrain if the drift is broad, gradual, and reflects a genuine, likely-persistent shift in the population; roll back if the timing cleanly matches a specific deploy and reverting it restores prior behavior; collect more data first if the signal is still ambiguous and the cost of a wrong retrain or rollback is high.

Worked example (drift check executed in Python). Comparing a production feature (avg_order_value) between a recent sample and its training-time baseline using a Kolmogorov-Smirnov test on the numeric distribution and a chi-square test on a categorical region feature: the numeric feature shows a clear, statistically significant shift (Kolmogorov-Smirnov, KS, statistic 0.31, p near zero, consistent with the production mean rising from roughly 50 to 58), while the categorical feature shows no significant shift (chi-square p=0.07, mix essentially unchanged). That combination pinpoints the likely driver precisely: something changed the DISTRIBUTION of order values reaching the model (a pricing change, a shift in which customer segment is transacting), not the categorical mix of where those customers are from, which narrows the retrain-versus-investigate decision considerably.

Trade-offs and pitfalls. Running many per-feature drift tests at once raises a multiple-comparisons concern (some features will look 'significant' by chance alone); apply a correction (like Bonferroni) or treat a single flagged feature among dozens with appropriate skepticism, and prioritize investigating features with both a statistically significant AND a practically large shift, not just a technically significant one.

Direct answer. When a production ML feature's quality visibly degrades after a change (more hallucinations from a generative service, or a composite engagement metric dropping after a drift event), the investigation should decompose the headline symptom into measurable sub-signals and communicate both the diagnosis and the remediation plan in language that lands with both engineering and non-technical stakeholders.

Structured elaboration. Decomposition: break a composite or fuzzy symptom (like 'quality degraded' or 'engagement dropped') into concrete, trackable sub-metrics that can each be independently investigated, for example splitting a generative service's quality regression into a factuality-proxy score, a fallback or refusal rate, and rephrase/complaint signals in user logs, or splitting a composite engagement metric into its component behaviors (session frequency, session depth, a specific feature's usage) to see which sub-metric is actually driving the aggregate move. Investigate each sub-signal with the same root-cause discipline used elsewhere (timing correlation with a model or config change, segment concentration, data-versus-real-behavior checks). For communication, engineering needs the specific mechanism and evidence (which sub-metric moved, what changed, what the proposed fix targets); non-technical stakeholders need the plain-language translation of what happened, how it's being addressed, and what to expect, without requiring them to understand the underlying sub-metrics.

Worked example. A composite 'engagement' metric drops 6% after a recommendation-model update. Decomposing into sub-metrics shows session FREQUENCY is unchanged, but session DEPTH (actions per session) dropped meaningfully, concentrated in users who received recommendations from the updated model. Tracing further shows the updated model is surfacing a narrower range of content categories than before, causing users to run out of relevant items faster and end sessions earlier. The engineering-facing finding names the specific mechanism (recommendation diversity narrowed); the stakeholder-facing summary states plainly that a recent model update reduced how varied users' recommendations are, which is shortening sessions, with a fix (restoring diversity in the ranking) already in progress and an expected timeline.

Trade-offs and pitfalls. A common failure is investigating the composite metric directly without decomposing it first, which tends to produce vague, hard-to-act-on findings ('engagement is down') rather than a specific, fixable mechanism; the decomposition step is what turns a symptom into something engineering can actually target.

Direct answer. A consistent performance gap across demographic groups needs to be investigated as a three-way attribution problem, same as any other metric disparity: is it explained by genuine, legitimate product-usage differences, by a downstream decisioning model treating groups differently on inputs that shouldn't matter, or by a data issue that happens to correlate with the demographic split, and each points to a very different remediation.

Structured elaboration. Product-design check: do the groups differ in ways that would legitimately explain a gap through normal usage patterns (device type, feature adoption, session frequency) rather than anything about how they're evaluated, and does controlling for those usage differences narrow or eliminate the gap. Decisioning-model check: for a product involving an automated approval or scoring model, examine whether the gap persists after controlling for legitimate, policy-relevant inputs, which would point at the model encoding a disparate impact through a proxy variable correlated with the demographic group, even if the group attribute itself isn't a direct input. Data-issue check: does the gap trace to a measurement or instrumentation difference between groups (a lower-quality data pipeline for one segment, a form or field that different groups fill out differently), rather than anything about the actual product or decisioning at all. Fairness metrics worth tracking depend on what's fair FOR this specific product and regulatory context (equal approval rates across groups, or equal TRUE POSITIVE rates conditional on genuine qualification, are different and sometimes conflicting definitions of fairness, and the right one depends on the domain and legal requirements, not a one-size-fits-all default). Communicating to non-technical stakeholders means translating the finding to what it means for real people affected, not just the statistical mechanism.

Worked example. A loan product shows a consistent approval-rate gap across demographic groups. Controlling for legitimate underwriting inputs (income, credit history) narrows but doesn't eliminate the gap, ruling out simple product-usage differences as the full explanation. Checking the decisioning model's feature importance shows a geographic-region feature is contributing meaningfully to the model's score, and that region feature correlates strongly with the demographic split, a classic case of a facially-neutral proxy variable carrying disparate impact even without the protected attribute being a direct input. That finding, not 'the model is biased' in the abstract but 'this specific feature is acting as a proxy and needs review', is what actually gets remediated (removing or adjusting the feature's weight) rather than a vaguer, harder-to-act-on conclusion.

Trade-offs and pitfalls. Concluding the gap is 'just' explained by legitimate usage differences without rigorously checking the decisioning-model and data-issue hypotheses first is a serious risk, both ethically and often legally, in a domain like lending; this class of investigation deserves a higher evidentiary bar and more caution before accepting the most comfortable explanation than an ordinary product metric investigation would.

Direct answer. When a product or model-quality metric drops shortly after a feature or model deployment, the triage needs to separate four distinct classes of cause quickly, because each points to a different owner and a different fix: a bug in the model or feature logic itself, a data-pipeline issue feeding it, a rollout or deployment mechanics problem, or an external factor unrelated to the change at all.

Structured elaboration. Check logs and metrics in an order that eliminates the cheapest-to-rule-out classes first: input feature distributions (did the data the model or feature actually received look normal, ruling in or out a data-pipeline cause), serving logs (did requests complete successfully and with expected latency, ruling in or out a deployment/infra cause), experiment or rollout assignment logs (did the right population actually receive the change, and only that population, ruling out a mis-targeted rollout), and user funnel behavior downstream of the change (does the drop trace specifically to users who experienced the new logic, or is it broader, which would argue against the new feature being the cause at all). The evidence needed before rolling back versus shipping a fix differs: a rollback is justified once the drop is confirmed to concentrate specifically in the exposed population and the magnitude is large enough that waiting for a root cause carries real cost, even without knowing the exact bug yet; a targeted fix (rather than a full rollback) is justified once the specific mechanism is understood well enough to patch confidently.

Worked example. A product-quality metric drops right after a new ranking model deploys. Feature-distribution checks show inputs look normal. Serving logs show no elevated errors or latency. Experiment-assignment logs confirm only the intended rollout percentage received the new model. Funnel analysis shows the drop concentrates precisely in the exposed cohort and nowhere else. That combination rules out data-pipeline and deployment-mechanics causes, and localizes the problem to the model's actual behavior; digging into a sample of exposed-user sessions finds the new model is systematically ranking a specific low-quality content category too highly, a genuine model-logic bug, which justifies either a targeted fix (excluding that category from ranking) or a rollback if a fast fix isn't available, rather than continuing to investigate the pipeline or deployment layers further.

Trade-offs and pitfalls. Confirming the drop concentrates in the exposed population is the single most valuable piece of evidence in this whole process, since it's what separates 'the change caused this' from 'something unrelated happened at the same time'; without that check, every other finding is much weaker evidence on its own.