Curiosity and Investigative Problem Solving Questions

Assess intellectual curiosity and investigative habits: asking why, digging into root causes, exploring alternatives, and demonstrating enthusiasm for learning how systems work. Candidates should show evidence of proactively researching problems, asking good follow up questions, and a mindset of continuous improvement. This topic captures behavioral indicators that predict thoroughness and long term growth potential.

MediumTechnical

49 practiced

Monitoring alerts that a commonly used feature's distribution has shifted since training. Design a step-by-step investigation to determine if this drift causes model degradation, describing statistical tests, visualizations, sample sizes needed, and how you'd prioritize fixes.

Sample Answer

1) Confirm and scope the alert- Reproduce the detection (timestamp, feature name, cohort). Verify data pipeline integrity (schema, missing values, encoding).- Define baseline: training-period feature distribution and a recent production window (e.g., last 7/30 days).

2) Data sampling & size- Aim for statistical power 0.8 and alpha=0.05. For common continuous features, ~200–500 samples per window often suffices to detect moderate shifts (Cohen’s d ~0.5); run a power calculation for your expected effect size. For categorical features, ensure >=5 counts per bin or aggregate rare categories.

3) Statistical tests- Continuous: Kolmogorov–Smirnov (KS) or Anderson–Darling for distribution differences; two-sample t-test for mean shifts (if normal); Mann–Whitney U for median shifts.- Categorical: Chi-square or Fisher’s exact test.- Multivariate / feature set: Population Stability Index (PSI) for a simple drift score; Maximum Mean Discrepancy (MMD) or classifier-based drift test (train a domain classifier).- Correct for multiple tests (Benjamini–Hochberg) if checking many features.

4) Visualizations- Overlayed histograms and empirical CDFs (training vs prod).- Boxplots and violin plots to show spread and outliers.- Time-series of feature moments (mean, std, percentiles) and PSI over time.- PDP/ICE plots to check whether feature -> prediction relationship changed.- Confusion matrix, ROC/AUC, calibration plots for model performance pre/post-shift.

5) Link drift to model degradation- Compare model metrics on a matched holdout from production before and after the shift (accuracy/precision/recall, AUC, calibration).- Slice analysis: compute metrics conditioned on feature buckets showing drift.- Conduct uplift test: A/B or backtesting — score historical data with current model; if possible, label a sampled recent batch and compute delta in performance; use bootstrapping to get confidence intervals.- Train a simple model excluding the shifted feature and compare performance to see feature importance impact.

6) Root-cause and prioritization- If model metric degradation is statistically significant and impacts business KPIs, prioritize urgent fixes.- Prioritization rubric: impact on user/business, severity of metric drop, frequency of affected traffic, complexity/time to fix.- Quick mitigations: input validation/filters, feature transformations, fallback rules, model recalibration.- Medium-term: incremental retrain with recent data, reweighting, or domain adaptation. Long-term: feature redesign, monitoring enhancement, upstream fixes.

7) Deployment & monitoring- Validate fixes with canary rollout and A/B test. Add automated alerts for performance, slices, and PSI thresholds. Record lessons, update runbooks.

Key practical notes:- Always pair statistical tests with effect sizes and visual checks.- Use bootstrapping for confidence intervals when labels are scarce.- Maintain reproducible notebooks and logging to accelerate future investigations.

HardTechnical

36 practiced

Your model exposes feature attributions but leadership wants concise root-cause explanations for errors in several high-value transactions. Describe how you'd combine explainability tools (SHAP, LIME, counterfactuals) with investigative steps (request logs, feature snapshots, label audits) to produce an executive-facing, actionable report.

Sample Answer

Situation: Leadership asked for concise root-cause explanations for a set of high-value transaction errors flagged by our model—they want actionable insight, not raw attribution tables.

Approach (overview): I’d run a short investigative pipeline combining feature-attribution tools (SHAP, LIME), counterfactual analysis, and production evidence (logs, snapshots, label audits), then synthesize findings into a two-page executive report with clear remediation steps.

Investigation steps:1. Triage & grouping — cluster the error transactions by outcome, product, and error signature (e.g., high-confidence false positives vs. late false negatives) to prioritize.2. Data & provenance — pull feature snapshots, request/transaction logs, model version, and timestamps for each group to check drift, missing upstream transforms, or latency-related artifacts.3. Label audit — re-check ground-truth labels (sampling + blind re-labeling) to detect label noise or process errors.4. Attribution analysis — compute SHAP summaries for the cluster (global and per-transaction local SHAP) to identify consistently important features; use LIME on representative transactions for quick, human-readable local explanations.5. Counterfactuals — generate minimal counterfactual edits per failure (e.g., “if feature X were Y, model would predict correctly”) to test causal hypotheses or business-rule conflicts.6. Corroboration — cross-check attributions with logs and feature snapshots (e.g., SHAP shows feature A dominates; logs show A was imputed recently) and with counterfactual sensitivity (small change fixes prediction).

Synthesis & report:- One-line executive summary: magnitude, business impact, and recommended priority fix.- Visuals: 1) bar of root-cause categories (data drift, label error, feature bug, model boundary) with percent of cases; 2) 2–3 annotated examples showing SHAP waterfall + counterfactual.- Findings: concise bullets with evidence (e.g., “40% of errors due to missing merchant_category populated by fallback rule — corroborated by logs and SHAP dominance”).- Recommended actions (priority, owner, ETA): quick fixes (data pipeline patch, guardrails), medium (retrain with corrected labels), long-term (monitoring, counterfactual tests in CI/CD).- Confidence & next steps: uncertainty bounds (how many cases audited), suggested experiments (A/B or patch-and-monitor) and KPIs to track.

Why this works: SHAP gives stable global/local importance; LIME provides interpretable local surrogates for stakeholder-friendly narratives; counterfactuals test actionable changes and reveal model brittleness. Combining these with provenance and label audits ensures attributions are grounded in reality, enabling leadership to act with confidence.

MediumTechnical

40 practiced

Offline evaluation shows Model A has higher accuracy, but online A/B returns that Model B improves conversion. Build a checklist of possible causes for this offline/online discrepancy and outline a prioritized investigation plan to find the root cause.

Sample Answer

Checklist — possible causes of offline vs online discrepancy- Metric mismatch: offline accuracy vs online conversion measure differ (e.g., accuracy vs CTR/conversion).- Data drift / distribution shift: training/test data not representative of live traffic (feature or population shift).- Labeling mismatch or label leakage in offline data.- Feature-parity / preprocessing differences between training and serving (normalization, missing-value handling, categorical encoding).- Inference-time bugs: model versioning, serialization, rounding, thresholding, or wrong model deployed.- Feedback loop / user-interaction effects: model changes user behavior (novelty, priming), downstream causal effects change conversion.- Experimentation issues: sample ratio mismatch, flaky randomization, interference, instrumentation/metric logging bugs.- Exposure/selection bias: model B changes which users see offers, creating different user mix.- Latency / UI differences: slower responses change user behavior; caching/AB routing problems.- Confounders/time effects: seasonality, promotion overlap, platform differences (mobile vs web).- Statistical noise / insufficient power and multiple-testing mistakes.- Reward hacking / proxy optimization: model optimizes proxy that isn’t aligned with conversion.

Prioritized investigation plan1. Validate the experiment - Confirm randomization, sample ratios, confidence intervals, duration, and no overlapping experiments. - Recompute online metrics from raw logs; check instrumentation integrity.

2. Reproduce online scoring offline - Replay production request logs through both models (shadowing) to compare scores/decisions. - Ensure identical feature generation, preprocessing, thresholds; test byte-for-byte parity.

3. Compare distributions - Compare feature distributions and user segments between offline test set and experiment traffic; look for covariate and label shift. - Stratify A/B results by key cohorts (device, geography, new vs returning users).

4. Inspect model differences - Analyze prediction-level disagreements: confusion matrix, top-k feature importance, SHAP on samples where models differ and where conversions change. - Check decision thresholds and calibration differences.

5. Check downstream/UX effects - Observe user flows: does Model B change ordering, timings, or UI elements? Measure latency and rendering logs. - Run qualitative analysis (session replays, user feedback) for behavior changes.

6. A/B follow-ups and sensitivity tests - Run targeted experiments: (a) Shadow deploy Model A in production logging true conversions; (b) small holdout to rule out novelty; (c) swap thresholds or ablate features. - Instrument additional metrics (engagement, bounce, time-to-convert) to detect intermediate mechanisms.

7. Conclude and act - If online uplift reproducible and causal, prioritize Model B but fix offline evaluation metrics to reflect real objective. - If discrepancy due to bug or bias, rollback and address root cause: fix preprocessing, retrain with representative data, or change metric alignment.

Tools & checks: replay pipelines, data drift monitors, feature-store parity tests, logging/observability dashboards, permutation/SHAP analyses, power calculations.

EasyBehavioral

49 practiced

Tell me about a time when your curiosity led you to discover an issue or an improvement in an ML project. Describe the initial observation, the specific "why" question(s) you asked, the investigative steps you ran, and the concrete outcome or change that resulted.

HardTechnical

37 practiced

A mission-critical fraud detection model's recall falls sharply overnight and labeled data for new fraud patterns is limited. Provide a 30-60-90 day strategy covering immediate triage steps, short-term mitigations, accelerated labeling strategies, and long-term model/data improvements to restore and harden recall.

Sample Answer

30-day (Immediate triage & containment)- Goal: stop recall bleeding, understand root cause.- Actions: - Run fast root-cause checklist: compare recent input feature distributions (drift), upstream schema changes, feature pipeline failures, model serving version, latency/time-windowing, and label arrival lag. Use data drift tools (e.g., Evidently, Deequ) and frozen feature snapshots. - Rollback to last-known-good model if canary shows much better recall; enable shadow mode to compare. - Implement high-recall fallbacks: raise score threshold for manual review, route low-confidence but high-risk transactions to human reviewers, enable rule-based heuristics tuned to catch known patterns (e.g., velocity, device-change). - Instrument tight monitoring and alerting (per-class recall, per-segment recall, feature drift, label latency).- Metrics: target restore >80% of prior recall within 7 days or justify trade-offs.

60-day (Short-term mitigations & accelerated labeling)- Goal: stabilize recall and rapidly grow labeled examples of new fraud.- Actions: - Active learning loop: prioritize uncertain/high-impact samples for labeling (uncertainty sampling, query-by-committee). Integrate into annotation UI and SLA. - Weak supervision & heuristics: craft labeling functions (Snorkel) using domain rules, proxy signals (chargebacks, disputes, device fingerprint), and ensemble to produce probabilistic labels. - Synthetic augmentation: simulate new fraud patterns where plausible (feature perturbations, user-behavior sequences) and use for pretraining. - Expand human-in-loop: temporary label team + fraud SME rotations; provide tools with contextual logs, features, and counterfactuals to speed accurate labels. - Retrain on combined dataset (gold + weak + synthetic) and deploy via staged rollout with monitoring.- Metrics: label throughput, precision/recall on a validation slice of emergent patterns.

90-day (Long-term model & data improvements)- Goal: harden system against future pattern shifts.- Actions: - Build a continuous learning pipeline: automated drift detection triggers dataset expansion, retraining cadence, and model validation gates. - Use modular models: combine fast-updating lightweight detectors (online learners / streaming models) for new patterns with robust periodic batch models. Consider meta-ensemble with gating. - Invest in representation learning: sequence models or embeddings capturing behavior (time-series, user embedding) to generalize to unseen fraud. - Catalog and lineage: feature store with versioning, label lineage, and explainability logs to speed triage. - Improve feedback loops: faster ground-truth ingestion (integrate downstream dispute/chargeback feeds), SLAs with ops and fraud teams, and prioritized features for labeling. - Post-deployment: A/B tests, canarying, and automated rollback policies; run red-team fraud simulations quarterly.- Outcomes & trade-offs: accept short-term manual-review cost to protect business; long-term reduces label latency and increases model adaptability.

Key rationale: immediate containment prevents loss, active learning + weak supervision accelerates labeled examples, and structural pipeline changes ensure rapid detection and recovery from future drifts.

Unlock Full Question Bank

Get access to hundreds of Curiosity and Investigative Problem Solving interview questions and detailed answers.

Join thousands of developers preparing for their dream job.