Marketing and Growth Analytics Questions

Definition & formula:
Customer retention rate (monthly) = (Number of users active in month M and still active in month M+1) / (Number of users active in month M) × 100%.
This measures month-to-month stickiness for an existing active population.

Assumptions (explicit):

• A user is "active in month X" if they have any active subscription-days in that calendar month (partial-month activity counts as active).
• cancel_date is the last active day (inclusive). start_date is first active day.
• Single subscription per user; reactivations are treated as new active periods but the same user can be counted as retained if active across months.

Sample users (2023):

• A: start 2023-01-10, cancel 2023-03-05 → active in Jan, Feb, Mar (partial)
• B: start 2023-02-01, active → active Feb, Mar...
• C: start 2023-01-15, cancel 2023-01-31 → active only Jan

Compute:

• Jan → Feb:
  • Active in Jan = {A, C} = 2
  • Of those, active in Feb = {A} = 1
  • Retention = 1/2 = 50%
• Feb → Mar:
  • Active in Feb = {A, B} = 2
  • Of those, active in Mar = {A, B} = 2
  • Retention = 2/2 = 100%

Edge cases & notes:

• Partial-months: alternate approach is prorated-retention (count by days active / days in month) — use when revenue-weighted retention is needed.
• Reactivations: decide whether to treat reactivated users as retained (continuous) or new (depends on business definition).
• Trials/comped periods: usually exclude or label separately.
• Timezones, missing dates, multiple subscriptions, and cancels effective at period boundary (midnight vs inclusive) need consistent handling.

Reporting to stakeholders:

• Provide cohort retention table (rows = cohort month, columns = month offset) + line charts for N-day/30-day rolling retention.
• Show counts alongside percentages, confidence intervals, and segment by acquisition channel, plan, and region.
• Annotate product/marketing events that could explain changes.
• Recommend KPIs: 30/60/90-day retention, LTV implications, and actionable segments (e.g., low-retention cohorts for targeted onboarding).

ARIMA

• When preferred: Short-term forecasting with fairly stable, linear patterns and enough historical data (monthly series with clear autocorrelation).
• Data requirements: Stationary or made stationary (differencing); at least 50–100+ time points improves parameter estimation.
• Interpretability: High — parameters (p,d,q) map to AR/M A components; residuals diagnostics straightforward.
• Seasonality & holidays: SARIMA handles fixed seasonal cycles; explicit holiday effects require exogenous regressors (SARIMAX).
• Pitfalls: Sensitive to nonstationarity, structural breaks, and multiple external drivers; poor at handling irregular seasonality and external regressors unless modeled explicitly.

Prophet

• When preferred: Business signals with multiple seasonalities, changepoints, and calendar effects; fast prototyping and stakeholder-facing forecasts.
• Data requirements: Works with messy data, missing points, and relatively fewer records (dozens to hundreds of months).
• Interpretability: Medium — additive components (trend, yearly/weekly seasonality, holidays) are intuitive; changepoints automated but configurable.
• Seasonality & holidays: Built-in flexible seasonality and holiday regressors with custom events.
• Pitfalls: Over-smoothing or overfitting changepoints if not tuned; assumes additive structure; less optimal for complex autocorrelation.

LSTM (Deep Learning)

• When preferred: Complex, nonlinear relationships, many external features (product usage, campaigns), or when sequence patterns are long and subtle.
• Data requirements: Large labeled history and features; months-only frequency may be limiting unless many series (panel/transfer learning).
• Interpretability: Low — black-box; requires SHAP, attention, or surrogate models for explanations.
• Seasonality & holidays: Can learn seasonality from data or ingest explicit calendar features; needs careful feature engineering.
• Pitfalls: Overfitting, data-hungry, long training/tuning, unstable forecasts under regime change, poor uncertainty quantification unless Bayesian or ensemble approaches used.

Practical guidance

• Start simple: benchmark with ARIMA/Prophet; add LSTM when nonlinearity and features justify complexity.
• Always include business covariates (pricing, campaign flags), backtest with rolling windows, and forecast intervals. Validate for structural breaks and align model choice with explainability needs for stakeholders.

Clarify goals & constraints:

• Detect anomalies in multivariate campaign metrics across regions, account for spatial correlation and shared (global) causes, flag whether anomaly is global vs localized, and provide timely, interpretable alerts.

Approach (pipeline):

1. Preprocess: align timestamps, impute missing values, de-seasonalize & detrend per metric-region (remove daily/weekly patterns) so residuals represent anomalous deviations.
2. Model background covariance with both spatial and cross-metric structure:
   • Fit a spatio-temporal covariance model (e.g., separable Cov_time ⊗ Cov_space) or graph-based covariance where regions are nodes (weights from geography or historical correlation).
   • Alternatively, learn covariance via shrinkage (Ledoit–Wolf) to be robust for high-dim data.
3. Dimensionality reduction for signal separation:
   • Apply PCA / Robust PCA on residual matrix (metrics × regions). Leading components capture common-cause/global modes; sparse components capture localized anomalies (RPCA: X = L + S).
   • Interpret loadings: a component with broad nonzero loadings across regions indicates global incident.
4. Online multivariate change detection:
   • Multivariate EWMA on principal-component scores: z_t = λ x_t + (1−λ) z_{t−1}; monitor Hotelling’s T² on the low-dim subspace using estimated covariance, or use MEWMA chart thresholds (accounting for correlated PCs).
   • Simultaneously, run univariate EWMA/Shewhart on region-specific residuals to catch strong local spikes.
5. Decision logic to separate incidents:
   • If anomaly score is high on first k PCA components (explained variance > threshold) and many regions contribute -> classify as global/common-cause.
   • If MEWMA/T² triggers but only a small subset of regions has high residuals or the sparse S from RPCA has support limited to few regions/metrics -> localized incident.
   • Use change-point localization: compute contribution scores per region = squared Mahalanobis distance of region residuals; rank to surface root regions.

Practical details & validation:

• Choose k via scree or cumulative variance (e.g., 80%). Use bootstrapped thresholds or FDR-controlled p-values for alarms to control false positives.
• Implement streaming via incremental PCA or EWMA on PC scores for low latency.
• Evaluate with injected synthetic incidents (global vs local), ROC/AUC, precision@k for localization, and time-to-detection.
• Provide explainability: show PC loadings, contribution scores, and time series overlays to stakeholders.

Trade-offs:

• RPCA + MEWMA separates global vs local well but costs compute; shrinkage PCA + MEWMA is faster. Graph-based covariance improves spatial sensitivity if region adjacency/traffic is meaningful.

Situation: During an active campaign a new marketing channel (Channel X) is launched and we need attribution to reflect its contribution quickly without retraining a full model from scratch.

Approach:

1. Feature updates

• Add lightweight features immediately: a binary flag for exposure/click on Channel X, recency and frequency counts, and simple interactions with key existing channels.
• If using channel embeddings (for high-cardinality sources), compute an embedding for Channel X from clickstream/cohort behavior and append it.
• Normalize any new features to match training-time distributions.

2. Incremental retraining strategies

• Warm-start / fine-tune: initialize model from current weights and fine-tune on recent data that includes Channel X. Use a smaller learning rate and early stopping to avoid catastrophic forgetting.
• Online/streaming updates: for models supporting online learning (e.g., FTRL, SGD, some tree boosters with incremental updates), feed new events in mini-batches to update weights.
• Two-model ensemble: keep the original stable model and train a lightweight adapter model that predicts adjustments attributable to Channel X; combine predictions (stacking) to avoid full retrain.
• Use importance sampling or reweighting if the new channel is underrepresented.

3. Monitoring & detecting degradation

• Deploy in shadow mode first: score live traffic with both original and updated models, compare attribution distributions.
• Track key metrics: conversion lift by channel, calibration (Brier/log-loss), AUC/PR, prediction distribution shifts.
• Data drift detection: monitor feature distributions (population stability index, KL divergence) for new-channel features and covariates.
• Set alert thresholds and perform periodic backtests (rolling window) to detect systematic bias.
• If degradation persists, schedule full retrain with an expanded training set.

Outcome/Why: This plan lets us incorporate Channel X quickly, minimize risk of destabilizing production models, and ensures we have safety checks to trigger full retraining only when necessary.

Requirements & constraints:

• Weekly incremental revenue estimates per paid channel with 95% CI
• Scale to ~100M monthly impressions (~8–9M/week)
• Use experiments/holdouts where feasible; econometrics where not
• Low latency ETL and automated reporting

High-level design:

1. Experimentation layer

• Run randomized A/B tests or geo-based experiments where possible (user-level or geo-level depending on engineering constraints).
• Power analysis automated: pre-compute required sample/impression sizes for target MDE and weekly cadence; if infeasible, fall back to holdout or modeling.
• Implement asymmetric allocation (smaller holdouts) to reduce revenue loss but keep statistical power (use sequential analysis with alpha spending to allow weekly looks).

2. Holdout strategy

• For channels where randomization at user-level is impossible (e.g., platform targeting constraints), use persistent holdouts (e.g., 1–5% of users or ZIP3 regions) rotated across time to reduce bias.
• Ensure no contamination via cookie/user id hashing and monitor overlap.

3. Econometric layer

• For infeasible experiments, deploy hybrid models:
  • Bayesian structural time series (BSTS)/CausalImpact for channel-level weekly incrementality using pre-periods and contemporaneous controls.
  • Hierarchical Bayesian model combining experimental priors (from channels with experiments) and observational signals (spend, impressions, seasonality, promo indicators). Model controls: price, seasonality, other media spend, macro covariates.
  • Use synthetic control for region-level inference when a natural donor pool exists.

4. Data platform & pipeline

• Ingest impression, click, spend, conversion, revenue, and user identifiers into a data warehouse (event-level). Aggregate into weekly windows; maintain rolling daily updates for freshness.
• Feature store: lagged spends, moving averages, seasonality, holidays, product launches.
• Orchestration: Airflow to run ETL, model training, posterior updates, and report generation nightly.

5. Statistical details & CI

• For experiments/holdouts: use difference-in-means with bootstrap and sequential-correction for weekly reporting; report Bayesian credible intervals if Bayesian tests used.
• For econometric models: sample from posterior to produce weekly credible intervals; propagate uncertainty from model parameters and measurement noise.
• Combine estimates: when both experiment and model exist, use inverse-variance weighting (or hierarchical pooling) to produce a single channel estimate.

6. Validation & monitoring

• Backtest with historical holdouts and known campaigns.
• Monitor model drift, overlap between channels, and falsification tests (placebo periods).
• Alert on divergence between experiment and model > threshold.

Trade-offs & considerations:

• Persistent holdouts cost revenue but reduce bias—minimize size but meet power.
• Bayesian pooling leverages cross-channel info to improve estimates for low-volume channels.
• Weekly cadence favors simpler models with fast updates; heavier retraining monthly.

Outputs:

• Weekly dashboard: per-channel incremental revenue, ROI, 95% CI, power, sample sizes, and alerts.
• API for downstream reporting and budget optimization.

This design balances randomized evidence where possible, principled econometrics where not, and scalable engineering to provide reliable weekly incrementality with quantified uncertainty.