Predictive Modeling and Machine Learning Fundamentals Questions

Start by treating feature distributions feeding dashboards as product metrics with SLIs (e.g., mean, variance, percentiles, proportion of missing). Monitor both distributional drift and population composition drift.

Statistical tests & metrics

• Continuous: Kolmogorov–Smirnov (KS) for distribution change, Population Stability Index (PSI) for business-friendly drift scoring, Wasserstein distance for magnitude, KL divergence if supports.
• Categorical: Chi-square test, JS divergence, or change in top-k category shares.
• Label/target shift: label-distribution comparison and confusion-matrix calibration drift.
• Significance + effect size: report p-values AND practical thresholds (e.g., PSI > 0.2 signals moderate/large drift).

Thresholding & alerting

• Use tiered thresholds: warning (small changes, e.g., PSI 0.1–0.2) and critical (>0.2). Combine statistical alarms with minimum sample-size guards to avoid noisy alerts.
• Add rolling baselines (7/30/90-day) and seasonality-aware baselines. Use control charts (CUSUM, EWMA) for persistent shifts.

Visualizations for dashboards

• Time-series of SLI metrics with thresholds annotated (PSI, mean, % missing).
• Overlaid density/CDF plots or histogram snapshots (current vs baseline) with delta annotations.
• Heatmaps/cohort views to localize drift by segment (region, device).
• Feature importance/SHAP change plots to show which inputs shifted and downstream metric impact.
• “What-if” simulated downstream KPI range using current vs baseline feature distributions.

Communicating uncertainty & degradation

• Bundle three views for stakeholders: (1) signal: What changed (stat/test + effect size); (2) impact: estimated KPI/model output change with confidence intervals; (3) recommended action: investigate data source, rollback, retrain, or annotate.
• Use plain-language summaries (“feature X increased mean by 25% since baseline; model score expected to drop ~3–5% (95% CI)”) and attach technical appendix with tests and sample sizes.
• Provide runbook and SLA: who owns investigation, triage steps, and expected timelines.
• Surface model uncertainty on dashboards: show model score distributions, calibration plots, and note when alerts are from small sample sizes.

Operationalize

• Automate checks in the ETL/feature pipeline, log drift events, and integrate with incident systems (PagerDuty/Slack). Store historical drift metrics for trend analysis and A/B checks post-retraining.

This approach balances rigorous statistical detection, practical thresholds to avoid alert fatigue, clear visual evidence, and stakeholder-facing impact statements with confidence intervals and remediation steps.

Overview: Build a near real-time analytics pipeline that combines rule-based triage for high-precision blocking and a lightweight ML scoring layer for risk prioritization. BI owns dashboards, alert metrics, KPI tracking, and feedback instrumentation to close the loop with ops.

Data sources:

• Transaction stream (payments, timestamps, amounts, merchant, channel, geo, device id, IP)
• User profile (KYC status, account age, historical velocity)
• External signals (blacklists, device reputation, BIN risk, geolocation / IP risk)
• Investigation outcomes and label store (fraud / false positive / not fraud)

Feature engineering (real-time + contextual):

• Immediate features: amount, merchant, channel, device_id, IP_country
• Behavioral features (last N minutes/hours/day): tx_count, tx_sum, avg_amount, ratio_new_mcc, failed_auth_count
• Velocity/Delta features: time_since_last_tx, unique_merchants_last_24h
• Device/IP enrichments: device_score, proxy_flags, ASN risk
• Aggregate and cohort features: account_age_days, pct_chargeback_cohort
  Implement streaming feature store (e.g., Redis, RocksDB, Feast) to serve features with <1s latency.

Model vs rules:

• Rules layer: deterministic, high-precision rules for immediate blocking (confirmed stolen card, explicit blacklist, velocity > extreme threshold). Maintain as readable rule set in config.
• ML layer: gradient-boosted tree or lightweight neural net (XGBoost/LightGBM) trained on labeled history for probability score. Retrain weekly; use online calibration if distribution shifts.
• Combined decision: if rule blocks -> block; else compute ML score -> tiered action (allow, challenge, manual review, block) using score thresholds and contextual overrides.

Thresholds & alerts:

• Use score buckets mapped to action: e.g., score > 0.95 block; 0.8–0.95 manual review; 0.5–0.8 challenge (2FA); <0.5 allow.
• Select thresholds to meet business constraints: target precision for blocks (e.g., 99% precision) and target recall for fraud capture. Calibrate with ROC/precision-recall and cost matrix (cost of false positive vs false negative).
• Implement dynamic thresholds per segment (high-value merchants, new accounts).

Presenting false positives to operations:

• Dashboards showing: FP rate by rule, by merchant, by time window, by feature contributors (SHAP or feature importance for model-based alerts).
• For each manual-review alert, show reconstructed context: timeline of user actions, device/IP enrichments, model score and top contributing features, rule triggers that applied.
• Provide sortable lists of recently overturned blocks with metadata, and aggregate metrics (FP rate, review throughput, average time to resolve).

Instrumenting feedback:

• Embed unique alert IDs and link to case management (investigation outcome writes back labels).
• Provide a simple ops form/buttons: mark fraud / false positive / unclear; capture reasons and notes.
• Automatic ingestion of investigation outcomes into label store; tag training dataset with investigation timestamps.
• Track latency from alert to resolution; include feedback accuracy metrics on BI dashboards.
• Use feedback for: (1) continuous model recalibration, (2) rule tuning (e.g., disable rule with high FP), (3) alert threshold adjustment.

Operational considerations:

• Monitor data drift, feature staleness, and model performance by cohort.
• Implement shadow testing for model changes and A/B test threshold adjustments.
• Define SLAs for alert review and daily automated reports to stakeholders with top FP drivers and recommended remediations.

This design balances speed, precision, transparency for ops, and a feedback loop to continuously reduce false positives while catching fraud.

Approach (high level)

• Start with cohort-based LTV as the baseline (acquisition week/day + first-order value), then layer probabilistic lifetime models to forecast future transactions and spend per customer.
• Use BG/NBD (or Pareto/NBD) to model repeat transaction process and a Gamma–Gamma model for monetary value. For alternative validation or time-to-churn concerns, fit survival models (Cox or parametric) to check consistency.

Dealing with right-censoring & heterogeneity

• Right-censoring: use likelihood-based BG/NBD which natively handles censored observation windows; for survival models use Kaplan–Meier for nonparametric estimates and include censoring indicators in Cox/parametric fits.
• Heterogeneity: capture unobserved heterogeneity via BG/NBD’s latent heterogeneity or with hierarchical Bayesian extensions (customer-level random effects). Segment cohorts by observed heterogeneity (promo vs organic, geography, device) before pooling.

Covariates to include

• Acquisition channel/campaign, promo type and discount, first-order value, days-to-second-order, geography, local restaurant density, device/OS, user demographics, time of year/seasonality, prior DoorDash usage (if reactivated users).

Model validation & uncertainty

• Backtest on historical cohorts (holdout periods), compare predicted vs realized cumulative revenue, use calibration plots and Brier/CRPS scores.
• Present uncertainty with prediction intervals (e.g., 95% predictive intervals for cohort LTV), fan charts for cumulative LTV over time, and scenario analysis (best/median/worst; varying retention or order frequency).
• Quantify uplift per dollar spent on acquisition by simulating marginal LTV under alternative assumptions.

Operationalization & stakeholder communication

• Build dashboards showing cohort curves, median and interval LTV, sensitivity knobs (window length, discounting rate). Provide clear assumptions: discount rate, modeling window, treatment of reactivations, and known biases (e.g., selection from high-value targeting).
• Recommend actionable KPIs: payback period, CAC vs median LTV, probability customer exceeds CAC. Offer a short executive summary + appendix with model metrics and validation results.

Implementation notes

• Pipeline: SQL for cohort extraction, pandas/statsmodels/BTYD/pyro or Stan for hierarchical fits, export results to Looker/Tableau. Automate weekly retraining and recalibration.

Permutation importance measures how much model performance drops when a feature’s values are randomly shuffled — it estimates a feature’s effect on predictions (model-agnostic and computed on held-out data). Mean-decrease-in-impurity (MDI) sums how much each feature reduces node impurity across trees during training — it is fast but biased toward features with many categories or high cardinality and reflects training-time splits rather than predictive impact on unseen data.

Two short sentences for executives:

• "Feature importance ranks which customer attributes most influence churn predictions, but MDI reflects how the model split data during training while permutation importance shows the actual drop in prediction accuracy when a feature is disrupted.
• "Use permutation importance to understand real predictive impact on holdout data; treat MDI as a quick internal signal and beware that correlated features can mask each other’s importance."

Framework: treat churn both as a time-to-event problem (when a customer will churn) and as a near-term classification problem (who’s likely to churn in the next 30/60/90 days). That lets you power long-term retention strategy and immediate intervention workflows.

Feature selection

• Start with domain-driven feature groups: product usage (DAU/WAU, feature depth, feature recency, session length, trend/slope), support logs (tickets count, sentiment score, time-to-first-response, escalation rate), billing (failed payments, retries, invoice disputes), account metadata (tenure, plan, seat count).
• Pre-filter using business rules (e.g., only subscribers) and correlations to remove duplicates.
• Use automated techniques: mutual information / chi-square for categorical, LASSO / ElasticNet for sparse selection, tree-based feature importance (RandomForest/XGBoost) and permutation importance.
• Validate with stability selection and check multicollinearity (VIF), plus create orthogonal derived features (ratios, recency-frequency trends).

Modeling approach

• Survival analysis for time-to-churn: Cox Proportional Hazards for interpretability; Random Survival Forests or Gradient-boosted survival models for non-linearities. These produce hazard rates and survival curves per customer.
• Classification for operational playbooks: train XGBoost/LightGBM on labeled windows (e.g., churn within 30/60/90 days). Use temporal cross-validation (rolling windows) to avoid leakage.
• Consider hybrid: use survival model to estimate risk horizon and classification for actionable short-term risk scoring.

Evaluation metrics (operational focus)

• Survival: Concordance index (C-index), time-dependent AUC, Brier score.
• Classification: ROC AUC, Precision@k and Recall@k (focus on top-decile outreach), PR-AUC for imbalanced data, calibration (reliability curves), and lift / cumulative gains.
• Business KPIs: expected retained revenue (ARR at risk), cost-per-intervention vs expected CLTV saved, conversion of interventions (measured via A/B tests).
• Use decision thresholds tuned to budgeted contact capacity (e.g., maximize expected revenue saved per outreach slot).

Operationalization & dashboard

• Key panels: risk funnel (cohort -> % high/medium/low risk), survival curves by segment (plan, tenure), top predictors (global SHAP summary and per-cohort partial dependence), churn heatmap (geography, plan), billing failure timeline, recent support interactions.
• Action widgets: ranked outreach list (top N customers with predicted time-to-churn, risk score, recommended action), segment-level recommendations (retain with discount, proactive onboarding, technical outreach), predicted ARR at risk.
• Enable filters (time window, product, account owner) and exportable lists for CS/PO teams.
• Show model health: calibration, top feature drift alerts, model performance over time, and A/B test results for interventions.

Recommended actions & experiment plan

• Map risk bands to playbooks: High-risk + recent payment failure -> pause auto-retry + immediate account outreach; High-risk with low usage -> targeted in-product onboarding + trial of premium feature; Medium-risk -> email nurture + NPS survey.
• Run randomized control trials: measure lift in retention and ROI per playbook; feed results back to model as new labels.
• Maintain a retraining cadence (weekly monthly depending on drift), monitor data pipelines, and document feature lineage for auditability.

This approach balances rigorous time-to-event modeling, operational scoring for immediate interventions, and a BI dashboard that surfaces explainable drivers and prioritized action lists tied to business impact.