Experimentation and Validation Questions

Clarify requirements and risk tolerance first: what minimum detectable effect (MDE) matters at 30+ days, business cost of a false positive vs false negative, and sample throughput. Then set evaluation timeframes and interim rules.

Timeframes

• Short horizon: 1, 7, 14 days — track early signals and sample/power accumulation.
• Primary evaluation: 30 days (when retention effects are expected).
• Long horizon: 60–90 days to confirm persistence.

Interim analyses

• Pre-specify 2–3 interim looks (e.g., day 7, day 14, day 30) with alpha-spending (O’Brien–Fleming or Pocock) to control Type I error.
• Use group-sequential testing or Bayesian sequential updating. For Bayesian: compute posterior probability that lift > business-relevant threshold; stop if prob > 0.95 (efficacy) or < 0.05 (futility).
• Require minimum sample size/enrollment at each look to avoid noisy decisions.

Decision rules: wait for full 30-day primary if the posterior/adjusted p-value is inconclusive or near threshold. Allow early stop only if effect is strong and robust across segments, or clearly futile to save cost.

Metrics

• Short-term: Day N retention rates (D1, D7, D14), activation funnel conversion, engagement depth (sessions/day), early churn hazard.
• Long-term: 30-day retention (binary and time-to-event), cohort survival curves, repeat engagement frequency, LTV proxies.

Visualizations

• Cumulative enrollment and statistical power over time.
• Lift-over-control with 95% CI at each look (line chart).
• Kaplan–Meier retention curves by cohort/treatment with log-rank p-values.
• Rolling window retention (e.g., week-over-week) heatmaps.
• Bayesian posterior probability trace over time.
• Heterogeneity charts: segmented lifts with forest plot.

Explainability: report sample sizes, MDE, alpha spending plan, and sensitivity analyses (instrumental/robustness checks). Recommend waiting for 30-day primary unless interim rules trigger decisive early action.

Cohort analysis groups users by a shared start event (e.g., sign-up week) and tracks their behavior over time to measure retention or engagement differences across these groups.

Example — measure onboarding retention for users who signed up in January:

Definition

• Cohort: users with first_sign_up_date between 2025-01-01 and 2025-01-31. Optionally break into weekly cohorts (Jan week1, week2...) to surface intra-month differences.
• Event: “active” during onboarding (e.g., completed onboarding step, first session, or returned to app).
• Metric: percent of cohort active in each subsequent week/day since sign-up.

Windows & rationale

• Use day-0 through day-30 for tight onboarding (daily view first 7 days, then weekly up to 30 days). Daily windows capture quick drop-offs; weekly smooths noise and is easier for PM storytelling.
• Align windows relative to each user’s signup date (retention by day N since signup).

SQL sketch to produce weekly retention matrix

WITH signups AS (
  SELECT user_id, DATE(first_sign_up_at) AS signup_date
  FROM users
  WHERE first_sign_up_at BETWEEN '2025-01-01' AND '2025-01-31'
),
events AS (
  SELECT user_id, DATE(event_at) AS evt_date
  FROM user_events
  WHERE event_name = 'completed_onboarding' OR event_name = 'session_start'
)
SELECT
  DATE_PART('week', events.evt_date - signups.signup_date) AS week_since_signup,
  COUNT(DISTINCT events.user_id) AS active_users,
  COUNT(DISTINCT signups.user_id) AS cohort_size,
  ROUND(100.0 * COUNT(DISTINCT events.user_id) / COUNT(DISTINCT signups.user_id), 2) AS pct_retained
FROM signups
LEFT JOIN events ON events.user_id = signups.user_id
GROUP BY week_since_signup
ORDER BY week_since_signup;

Visualization & storytelling

• Retention table/heatmap: rows = cohorts (or single Jan cohort), columns = days/weeks since signup; cell = % retained. Heatmap quickly shows where drop-offs occur.
• Retention curve: line plot of % retained vs. days/weeks for the January cohort; annotate key drops (e.g., day 1, day 7) and sample sizes.
• Funnel / cumulative chart: show progression through onboarding steps for the January cohort.

What to tell PMs

• Start with headline: “January cohort retains X% at day1, Y% at day7, Z% at day30.”
• Highlight actionable insights: e.g., a big drop at day1 suggests improving first-run experience or communication; a later drop suggests follow-up messaging or feature value.
• Provide confidence: show cohort size, statistical significance if comparing cohorts, and suggested experiments (A/B test onboarding flow, push reminders) with expected lift.

Edge considerations

• Use user-level deduplication, timezone alignment, and handle users who signed before January but reactivated.
• If cohorts are small, aggregate weeks or combine months to reduce noise.

Start by clarifying objectives and constraints: business KPI (CTR, revenue, retention), available logs (impressions, clicks, conversions, timestamps, exposures, position), and treatment (personalized ranking algorithm vs baseline).

Offline pipeline (stepwise):

1. Data preparation & logging hygiene

• Reconstruct sessions with impression lists, item positions, features at decision time, and any contextual signals.
• Keep raw exposure and action timestamps to model temporal dependencies.

2. Dataset splitting strategies

• Time-based holdout: train on t0..tN, validate on tN+1..tN+k, test on later period to mimic deployment shift.
• User-level split for generalization: hold out a set of users entirely for final test (prevents leakage).
• Counterfactual replay split: keep impression histories and treat later interactions as evaluation targets.
• K-fold with temporal blocking for robust variance estimates.

3. Offline metrics mapping to online KPIs

• Ranking metrics: NDCG@k, MAP — proxy for relevance and downstream CTR but insensitive to position / exposure.
• Calibration / prop score: log-loss or Brier for click probability quality (important for revenue/auction).
• Business proxies: Weighted-CTR (weight by revenue per click), Expected Revenue per Impression (sum p(click|item)*revenue(item)).
• Use AUC for binary click prediction but combine with rank metrics for ranking tasks.

4. Simulating online effects

• Position bias: apply IPS (Inverse Propensity Scoring) using propensities estimated from randomized buckets or click models (e.g., examination models). Use IPW/Narator correction when estimating metrics: weighted NDCG or weighted CTR.
• Feedback loops & exposure dynamics: run counterfactual policy evaluation (Off-Policy Evaluation) with Doubly Robust estimators to reduce variance/bias.
• Session simulation: build a simple user simulator (parametric or learned from data) that consumes ranked lists, simulates clicks conditioned on position and relevance, and updates user state to capture long-term effects (diversity, novelty, satiation).
• Explore-offline: simulate A/B variants by injecting exploration (ϵ-greedy or Thompson sampling) in simulator and measure longer-horizon retention/revenue.

5. Evaluation protocol & robustness

• Use IPS / DR-corrected estimates where logged policy differs from evaluated policy; when propensities are unknown, instrument a small randomized bucket to estimate them.
• Report confidence intervals via bootstrap over users or sessions.
• Sanity checks: calibration plots, log-rank concordance, and counterfactual lift vs naive metrics.
• Backtest over multiple time windows and user cohorts; track distributional shifts.

6. Decision criteria before deployment

• Statistically significant improvement on IPS/DR-adjusted business metric in holdout users + no degradation in cold-start cohorts.
• Robustness under simulator stress-tests and bounded risk in randomized rollout (ramp with monitoring).

Concrete example:

• Train ranking model to predict p(click). Evaluate predicted p with log-loss and rank with NDCG@10. Estimate position propensities from a 5% randomized exposure bucket, compute IPS-weighted CTR and DR estimate of revenue per 1k impressions. Use a learned user simulator to project 7- and 30-day retention and revenue to compare policies before a staged online rollout.

This pipeline balances realistic bias correction, business-aligned metrics, and caution via counterfactual methods and simulation to reduce surprise when moving to online personalization.

Novelty effect — a short-term spike in user behavior driven by users’ curiosity or attention to a new feature rather than a lasting change in underlying preferences or utility. It appears as a large lift immediately after launch that decays over time.

Analysis plan to detect and mitigate novelty vs sustained impact

1. Data checks (sanity + diagnostics)

• Verify timestamp alignment, consistent cohort definitions, and no concurrent launches or marketing campaigns.
• Plot daily/weekly treatment vs control metrics (mean + CI) for 4–8 weeks; inspect decay shape.
• Check engagement distribution (quantiles) to see if lift concentrated in a small subgroup (power users).
• Use event-level logs to validate feature exposure: ensure exposure timing matches metric spikes.

2. Segmentation

• New vs returning users, frequency buckets (low/med/high activity), geography, device, and attribution channel.
• Compare persistence: compute survival curves (time-to-first-reversion) for users who tried the feature vs controls.
• Cohort-by-week analysis: follow cohorts exposed in different weeks to detect novelty fading consistently.

3. Statistical tests & models

• Time-series mixed-effects model: metric ~ treatment * week + (user random effect) to estimate interaction and decay slope.
• Exponential/linear decay fit to treatment lift; test whether asymptote > 0 (sustained effect).
• Use permutation tests on post-week-1 period to check significance after initial spike.
• Pre-register primary evaluation window (e.g., weeks 2–4) to avoid cherry-picking.

4. Experimental design options

• Staggered rollout (step-wedge): roll out to random groups at different times to separate time/time-varying novelty from sustained effect.
• Holdout ramp: after initial ramp, keep a permanent holdout group to measure long-run lift.
• Re-exposure test: re-introduce feature to a subset after washout to see if lift reappears (indicative of novelty).
• Factorial test: add reminders/education vs passive exposure to see whether initial education sustains behavior.

5. Mitigation & decision rules

• If decay fits novelty and asymptote ≈ 0: deprioritize or redesign (increase utility, habituation hooks).
• If partial sustained lift: quantify LTV using area under lift curve; require threshold for productization.
• Report uncertainties (CIs, model assumptions) and recommend further experiments (longer holdouts, UX changes).

This plan distinguishes transient attention-driven boosts from durable behavioral change using diagnostics, segmentation, time-aware modeling, and robust experimental designs (staggered rollouts, holdouts, re-exposure).

Diagnosis (is attrition informative)

• Compare baseline covariates: test balance of pre-randomization variables between attriters and non‑attriters within each arm (t-tests / standardized mean differences). Large imbalances suggest attrition related to prognosis.
• Model dropout: fit logistic regression (or gradient boosted tree) predicting dropout using baseline covariates and treatment indicator and their interactions. Significant covariates or a treatment×covariate interaction imply informative attrition.
• Examine timing & reasons: timestamped logs and survey reasons; if treatment causes earlier exits tied to likely outcomes, attrition is likely informative.
• Check post-baseline proxies: if you have intermediate outcomes (e.g., early engagement), test whether early measures predict later dropout and final outcome—evidence of MNAR.
• Simple placebo: compare pre-treatment outcome proxies across groups conditional on staying; imbalance suggests bias.

If attrition is non‑random — two approaches (with diagnostics & tradeoffs)

1. Inverse Probability Weighting (IPW)

• Estimate propensity to remain P(R=1 | X, T) via logistic/ML using baseline X and treatment T.
• Compute stabilized weights w = P(R=1)/P(R=1 | X, T) and weight observed outcomes; estimate weighted difference in means or regression.
• Diagnostics: check weight distribution (no extreme weights), covariate balance after weighting, and perform weighted bootstrap for CIs.
• Assumption: Missing At Random (MAR) conditional on X,T. If unobserved predictors of dropout exist, IPW is biased.

2. Multiple Imputation / Selection Models + Sensitivity Analysis

• Multiple imputation under MAR: impute missing outcomes conditional on X,T using MICE or joint models; combine estimates with Rubin’s rules.
• For MNAR, implement sensitivity analyses:
  • Pattern‑mixture with delta adjustments: shift imputed outcomes for attriters by δ and report effect as function of δ (range guided by domain knowledge).
  • Selection model (Heckman-style) jointly model outcome and selection with exclusion restriction if available.
• Report how effect changes across plausible δ; present conclusions robust only within credible δ range.

Bonus / nonparametric bounds

• Lee bounds (trimming): if attrition is monotone (dropout only reduces probability of observed outcome), compute worst/best-case bounds without MAR assumptions.
• Useful when attrition differs by arm and you want assumption-light inference (but bounds may be wide).

Practical workflow

1. Run diagnosis and document likely mechanism.
2. If MAR plausible, produce IPW and MI estimates and show concordance.
3. If MNAR plausible, report MNAR sensitivity (delta / selection model) and Lee bounds; present ranges to stakeholders and recommend data collection changes to reduce attrition in future experiments.