Predictive Modeling and Machine Learning Fundamentals Questions

Approach — evaluate, mitigate, monitor; communicate trade-offs and legal/ethical risks.

Evaluate fairness

• Data checks: use SQL/Excel to summarize outcomes by race (selection bias, missingness, label quality). Example: GROUP BY race, COUNT(), mean(default_rate), coverage.
• Metrics: compute both group and individual metrics — demographic parity, equalized odds (TPR/FPR by group), disparate impact ratio (positive_rate_min / positive_rate_max), calibration within groups (predicted score vs actual). Visualize in Tableau/Power BI (score distributions, ROC by group).
• Statistical tests: confidence intervals, bootstrap to test whether differences are significant, and examine sample sizes.

Mitigation while preserving utility

• Pre-processing: reweighting or optimized sampling to correct label/representation bias (e.g., inverse probability weights).
• In-processing: include fairness-aware constraints or regularization (e.g., add equalized odds penalty) if you can collaborate with modeling team.
• Post-processing: threshold adjustment per group or calibrated equalized odds; use when model can't be retrained.
• Evaluate trade-offs: report AUC, precision/recall and fairness metrics together; use Pareto analyses to show utility vs fairness options.

Monitoring & deployment

• Set SLAs/alerts for drift in fairness metrics; automate weekly reports and dashboards for stakeholders.
• Maintain documentation: data lineage, experiments, rationale for chosen mitigation.

Legal & ethical considerations to surface

• Regulatory requirements (e.g., ECOA, fair lending): avoid using protected attributes directly for adverse decisions; however, document if race is used only for auditing/mitigation.
• Proxy risk: features correlated with race can produce disparate impact—explain risk and mitigation.
• Transparency & governance: recommend a review board, explainability summaries for decisions, and consent/PII handling.
• Stakeholder ask: propose options (no-change, mitigation method A/B tests, reject model) with quantified impacts and recommended path.

Outcome-focused ask: propose an experiment (A/B or shadow run) comparing baseline vs mitigation, plus dashboarding and periodic legal review before production.

L1 (Lasso) and L2 (Ridge) are penalties added to a regression loss to shrink coefficients and reduce overfitting. Mathematically:

• Ridge adds an L2 penalty: minimize RSS + λ Σ βj^2. This penalizes large coefficients smoothly; solution shrinks coefficients toward zero but rarely makes them exactly zero.
• Lasso adds an L1 penalty: minimize RSS + λ Σ |βj|. The L1 penalty has a “pointy” geometry that encourages exact zeros for some βj, performing implicit feature selection.

Practical consequences for feature selection and interpretability:

• Feature selection: Lasso can set irrelevant features’ coefficients to zero, producing a sparse model you can present as “these X features matter.” Ridge keeps all features with smaller weights—useful when many correlated predictors each carry signal.
• Correlated features: Ridge distributes weights among correlated variables; Lasso tends to pick one and zero others (possibly arbitrarily), which can be unstable.
• Interpretability: Lasso yields simpler, more interpretable models (fewer nonzero features). Ridge yields more stable coefficient estimates when multicollinearity exists but is less sparse.
• When to choose: use Lasso if you want automatic feature selection and a compact model to communicate to stakeholders; use Ridge when you care about prediction accuracy with correlated features and prefer stable coefficients. Consider Elastic Net (combines L1 and L2) when you need both sparsity and stability.

Approach — threshold selection

• Pick the score threshold that selects the top 5,000 users by predicted churn score (i.e., the 5,000 highest scores). Practically that means choosing the score quantile q = 5000 / N (N = current weekly eligible population). If your monitoring only reports metrics at fixed quantiles (e.g., precision@10%), map 5,000 to the nearest quantile (example below).

Estimate expected true positives

• If 5,000 corresponds to the top 10% (i.e., N = 50,000), you can use the provided precision@10% = 0.45.
• Expected true positives (TP) = precision * targets = 0.45 * 5,000 = 2,250 retained users.

Financials / ROI calculation

• Benefit per true positive = $120 NPV.
• Total benefit = TP * 120 = 2,250 * 120 = $270,000.
• Targeting cost = 5,000 * $5 = $25,000.
• Net value = 270,000 - 25,000 = $245,000.
• ROI = Net / Cost = 245,000 / 25,000 = 9.8 → 980% (or benefit/cost ratio = 270,000 / 25,000 = 10.8).
• Break-even check: minimum TP to break even = cost / 120 = 25,000 / 120 ≈ 208.3 → min precision = 208.3 / 5,000 ≈ 4.17%. The model’s precision@10% (45%) is well above break-even.

If 5,000 is not 10%:

• Compute quantile q = 5,000 / N. If q differs from 10%, interpolate precision using validation PR curve or compute precision at that q directly from holdout data. Use that precision in the same formula TP = precision * 5,000.

Model limitations and how they affect recommendations

• Precision@10% is an empirical estimate — it depends on the validation set; real-world precision can differ due to population shift or seasonality. Always validate on recent data.
• AUC=0.78 indicates good discriminative power but doesn’t tell you calibration — predicted probabilities may need recalibration for correct quantile cuts.
• Precision/recall are aggregate metrics: they don’t account for heterogeneous lift (some users are easier to retain). Consider uplift modelling (causal effect) because some targeted users might have stayed without an offer; standard churn prediction overestimates incremental impact.
• Operational constraints: offer capacity, overlap with other campaigns, and customer experience (over-targeting) affect realized ROI.
• Recommendation: run a small randomized pilot / holdout A/B test that targets the chosen top-5000 group and a control group to measure actual retention lift and realized NPV, monitor model drift, recalibrate thresholds periodically, and consider moving toward uplift models for more efficient spend.

Overview: build a time-aware pipeline that produces a per-user 30-day churn probability by combining user-level behavioral signals with cohort-level trends, trains robust models, and validates without time leakage.

Feature engineering

• User-level: recency (days since last activity), frequency (events per 7/30 days), monetary (LTV, avg order value), engagement depth (unique features used), session features (avg session length, sessions/week), trend features (delta frequency last 7 vs prior 7).
• Behavioral aggregates: counts, rates, rolling means, exponential decays (weight recent events more), time-since-first, percent-change features.
• Cohort-level: define cohorts by signup week/month or acquisition source. For each cohort compute retention curve and summary stats: 7/30-day retention, mean sessions/user, cohort size.
• Cohort retention slope: fit linear regression on log-retention over first N days to get early decay rate (slope = d(log(retention))/dt). Include cohort std dev and recent-week slope as features.
• Feature pipelines: standardize/clip outliers, encode categorical (target / frequency encoding with leakage-safe aggregation), impute missing.

Model choices

• Start with logistic regression (baseline, interpretable, calibrated probabilities with Platt/isotonic). Use L1/L2 regularization and interaction terms if needed.
• Tree-based (XGBoost/LightGBM/CatBoost) for nonlinearity and interactions; use monotonic constraints if business rules require. Calibrate outputs (isotonic/Platt) when using for probability thresholds.
• For production scoring, prefer tree models for performance, logistic for explainability and quick monitoring.

Evaluation metrics

• Discrimination: AUC-ROC and AUC-PR (use PR when churn is rare).
• Business-focused: precision@k (top-k intervention), recall@k, lift, and expected savings per user (combine predicted prob × cost).
• Calibration: Brier score, calibration plots; ensure predicted probabilities map to observed churn rates (critical for policy).
• Backtesting: rolling-window validation reporting metrics over time.

Handling class imbalance

• Use class weights in models or scale_pos_weight in XGBoost.
• Experiment with upsampling minority (SMOTE carefully for time-series) or downsampling majority in training only.
• Optimize business metric (precision@k) rather than accuracy; consider focal loss for extreme imbalance.

Preventing time leakage

• Temporal splits: train on time windows earlier than validation/test. Use forward chaining (rolling origin) cross-validation: train on T0..Tn, validate on Tn+1 window.
• Compute features using only data available at prediction time (cutoff timestamp). Use “as-of” joins to prevent future info bleeding into features.
• When aggregating cohort stats, compute cohort features based only on data up to cohort-age at scoring time.

Cohort-level feature validation

• Holdout entire cohorts: validate by withholding some cohorts (e.g., signup months) to ensure cohort features generalize to unseen cohorts.
• Hierarchical CV: nested CV that groups by cohort to avoid leakage from same cohort across folds.
• Backtest cohort drift: monitor cohort feature distributions and model performance over time; if cohort-level predictors degrade, retrain or add cohort age interactions.
• Causal check: ensure cohort features aren’t proxies for future actions (e.g., marketing campaigns launched after cohort formation). If correlated, remove or treat separately.

Operational considerations

• Monitor calibration and precision@k in production; set alert thresholds for drift.
• Feature store with versioned as-of computed features for reproducibility.
• Retrain cadence: weekly or monthly depending on drift; use incremental scoring for near-real-time if needed.

This approach balances interpretability, business utility (precision@k and savings), and rigorous temporal validation so predictions are actionable and reliable.

Situation: We model daily support tickets per user and see variance > mean — classic overdispersion.

Short comparison

• Poisson regression (GLM, log link) assumes mean = variance (Var(Y)=µ). Coefficients: log-rate changes; exp(beta) = multiplicative change in expected count per unit change in predictor.
• Negative Binomial (NB) relaxes variance: Var(Y)=µ + αµ^2 (or µ(1+µ/θ) depending on parametrization). Same interpretation for betas but adds dispersion parameter α (or θ) to model extra-Poisson variability.

Estimating and interpreting dispersion

• Fit Poisson first; compute Pearson chi-square/df or deviance/df; values >>1 indicate overdispersion.
• Fit NB via maximum likelihood to estimate α (or 1/θ). α>0 means variance > mean; larger α = more overdispersion. Report estimate with CI and p-value (likelihood ratio vs α=0).

Formal tests

• Overdispersion test: compare Poisson vs NB by likelihood ratio test (LRT) — NB should improve fit significantly. Also use score tests or Cameron & Trivedi test; check residual plots and Pearson chi-square/df.

When zero-inflated models

• If there are excess structural zeros beyond NB predictions (two processes: “always zero” vs count), consider zero-inflated Poisson (ZIP) or zero-inflated NB (ZINB). Use Vuong test or compare AIC/BIC and inspect predicted vs observed zero counts.

Practical guidance for stakeholders

• Present intuitive metrics: expected tickets per user, multiplicative effects (e.g., “feature X increases expected tickets by 1.4×”), and uncertainty (CIs).
• Explain why NB chosen: “Poisson underestimates variability, NB accounts for user-level heterogeneity; model fit improved (ΔAIC, LRT p-value, Pearson χ²/df)”.
• If zeros modeled: explain business meaning of structural zeros (e.g., users not eligible) and implications for interventions.
• Provide actionable recommendations, diagnostics plots, and a short appendix with model formulas and assumptions for technical audiences.