Experimentation and Validation Questions

Approach summary: define leading and lagging metrics, use cohort-based attribution with a long window, run experiment + observational checks, and analyze causal impact on long-term retention and diversity trends (not just short-term clicks).

Metrics

• Short-term: Click-through rate (CTR), immediate watch/click depth.
• Long-term retention (primary): 7/28/90-day active retention rate (percentage of users who return and engage).
• Engagement quality: session length per user, content completion rate, return frequency.
• Diversity: entropy of content categories per user, Gini index of content exposure, fraction of unique creators consumed.
• Business: revenue per user (if applicable), churn rate.

Attribution window & cohorts

• Use cohort by exposure date (day of first exposure to the new algo) and track each cohort for 90 days (primary), with 28-day as intermediate check.
• For experiments, expose randomized treatment/control; measure retention at 7/28/90 days.

Analysis methods

• A/B test with pre-specified primary metric: 28- or 90-day retention. Use intention-to-treat analysis.
• Survival analysis (Kaplan–Meier) to compare time-to-churn between groups.
• Difference-in-differences on historical rollout to control for trends.
• Mediation analysis to see if reduced diversity mediates retention loss.
• Segment analysis: new vs power users, content verticals.

Avoiding being misled by short-term gains

• Pre-register primary metric = long-term retention; treat CTR as secondary.
• Run uplift analysis: check if short-term CTR lifts decay over time (plot retention curves).
• Monitor diversity metrics alongside CTR—if CTR rises and diversity drops, flag risk.
• Use interim checks but avoid early stopping for short-term CTR wins.
• Sensitivity checks: simulate how much short-term gain would need to persist to offset long-term retention loss.

Example SQL (cohort retention at 28 days)

WITH exposures AS (
  SELECT user_id, MIN(exposure_date) AS cohort_date, treatment
  FROM user_exposures
  WHERE experiment = 'rec_algo_v2'
  GROUP BY user_id, treatment
),
activity AS (
  SELECT user_id, activity_date
  FROM user_activity
)
SELECT e.treatment,
       COUNT(DISTINCT e.user_id) AS cohort_size,
       COUNT(DISTINCT CASE WHEN a.activity_date BETWEEN e.cohort_date AND e.cohort_date + INTERVAL '28 day' THEN e.user_id END) AS retained_28d,
       COUNT(DISTINCT CASE WHEN a.activity_date BETWEEN e.cohort_date AND e.cohort_date + INTERVAL '90 day' THEN e.user_id END) AS retained_90d
FROM exposures e
LEFT JOIN activity a ON a.user_id = e.user_id
GROUP BY e.treatment;

Communicate results with retention curves, diversity-trend charts, and clear decision rule: accept new algo only if long-term retention (primary) is non-inferior and diversity loss is within tolerable bounds.

Business hypothesis:

• If we add an in-onboarding progress bar + contextual tips (treatment), then new users will be more likely to complete onboarding vs current flow (control) because perceived progress and timely help reduce friction.

Primary metric:

• Onboarding completion rate within first 7 days (binary per user). This directly measures the business goal.

Secondary metrics:

• Time-to-complete onboarding (median minutes/hours)
• Drop-off rate at each onboarding step (step-level funnel)
• Short-term engagement: Day-7 active rate and Day-14 retention
• Support contacts or NPS during onboarding (quality signal)

Sample size approach (qualitative):

• Use baseline completion rate from historical data (e.g., 20%). Choose minimum detectable effect (MDE) meaningful to growth (e.g., +10% relative → 2 percentage points absolute). Power at 80% and alpha 0.05. Compute users needed per arm using standard proportion power formulas; if traffic is low, run test longer until sample achieved. Ensure experiment runs multiple full weeks to cover day-of-week effects.

Guardrails:

• Pre-register primary metric and analysis plan before launch.
• Minimum sample per arm and minimum duration (e.g., 2 weeks or until sample) to avoid early peeking.
• Monitor safety metrics: crash rate, error logs, support volume — pause if adverse impact > X%.
• Segment checks: ensure randomization balance by platform, geography, and acquisition source.
• Use intent-to-treat analysis; exclude bots and internal traffic.

Reporting to product & growth:

• Deliver a one-pager: hypothesis, primary result (lift, 95% CI, p-value), absolute and relative change, sample sizes, run duration, and power/MDE used.
• Visualize: cohort funnel (control vs treatment), step-level drop-offs, time-to-complete CDF, and segmented effects (by platform/acquisition).
• Interpret practical impact: estimate incremental completed onboardings and projected revenue/LTV impact.
• Recommendation: adopt, iterate, or reject with proposed next steps (e.g., A/B/n for variations or rollout plan with monitoring). Include appendix with raw numbers and statistical assumptions for reproducibility.

Clarify objective and interference: state the primary metric (e.g., DAU, messages per user) and that SUTVA is violated because one user’s treatment can change others’ outcomes. Goal: estimate total effect (direct + spillovers) and, if possible, decompose direct vs. network spillover.

Design overview — cluster randomization:

• Randomize treatment at cluster level so interference is mostly inside clusters, not across them.
• Clusters can be user communities (graph partitions), households, or geographic / org units depending on product.

Practical cluster choices:

• Social-graph clusters: run community detection (Louvain, METIS) on interaction graph and cut into many balanced clusters; pros: aligns with network ties; cons: computation, potential boundary edges causing spillover.
• Households/orgs: simple, low cross-cluster interference when relevant.
• Time-cohorts (for some messaging features): weaker for durable connections.

Handling boundary spillovers:

• Use buffer zones (exclude border nodes from analysis) or augment clustering to minimize cross-cluster edges.
• Measure cross-cluster edge density to assess contamination.

Statistical power and sample-size implications:

• Use design effect: DE = 1 + (m-1)*ICC, where m = avg cluster size, ICC = intracluster correlation. Effective sample size shrinks by DE.
• Power depends more on number of clusters than cluster size. So prefer many small clusters over few large ones.
• Compute power using cluster-level variance estimates (or pilot data): simulate outcomes with assumed direct and spillover effects to get realistic power.

Advanced designs and estimation:

• Two-stage/randomized saturation (partial population) design: Randomize clusters to different treatment saturation levels (e.g., 0%, 25%, 75%) and within clusters randomize individuals. This lets you estimate direct and spillover effects.
• Use Horvitz-Thompson or inverse-prob weighting to account for unequal assignment probabilities.
• Regression with cluster-robust SEs, and randomization-based permutation tests to account for complex dependence.
• Consider network-aware estimators (exposure mapping) that define each user’s treatment exposure as function of neighbors’ statuses.

Practical steps for a data analyst:

1. Pre-experiment: quantify between/within-cluster edge density, estimate ICC from historical metric.
2. Choose clustering method balancing number of clusters, computational cost, and contamination risk.
3. Power simulations using realistic network structure and assumed effects.
4. Run experiment, monitor balance, check spillover diagnostics (outcomes vs. neighbor treatment rate).
5. Analysis: pre-specify estimands, use cluster-level aggregations, two-stage estimators for spillovers, adjust for covariates to increase precision.

Trade-offs:

• More clusters → power but higher deployment complexity.
• Larger clusters reduce cross-cluster contamination but increase DE and hurt power.
• Two-stage designs estimate spillovers but require larger sample and more complex analysis.

This approach balances causal validity with operational practicality and gives targetable estimands (total, direct, spillover) useful for product decisions.

Confounding factors:

• Increased traffic volume and different user segments (new vs. returning) from the marketing campaign.
• Changes in user intent and conversion propensity (promotion-driven behavior).
• Temporal effects (weekend/holiday timing) and interaction with variant exposure.
• Channel mix shift (paid, email, social) altering attribution and funnel behavior.
• Differential exposure if promotion targets subset of users overlapping unevenly with A/B buckets.

How to detect them:

• Compare traffic and user composition vs. baseline (daily users, new/returning, geography, device, acquisition channel).
• Plot time-series by cohort and channel for key metrics (CTR, conversion, revenue) split by variant.
• Run balance checks: randomization proportions before/after promotion; chi-square tests on covariates.
• Check interaction terms: include “promotion_period” and channel indicators in regression to test heterogeneous effects.

Mitigation strategies:

• Pre-specify and exclude promotion-impacted windows or run sensitivity analyses with and without the week.
• Use covariate-adjusted analysis (ANCOVA / regression) including acquisition channel, new/returning, and promo flag.
• Stratify or run subgroup analyses (only organic users) to estimate unbiased treatment effect.
• Instrumental approach: if possible, use users randomized before promotion or use difference-in-differences comparing pre/post lift across variants.
• Report results transparently: show adjusted and unadjusted estimates, confidence intervals, and limitations; recommend rerunning the experiment if uncertainty remains high.

This approach detects bias, quantifies its impact, and provides robust adjusted estimates or a clear rationale to delay decisions.

Approach: assign each user a signup cohort week (ISO year-week), convert event timestamps to their event week starts, compute week_offset = number of weeks since signup (0 = signup week), then group by cohort_week and week_offset counting unique active users.

import pandas as pd

def weekly_cohort_retention(df):
    """
    Input df: columns ['user_id','signup_at','event_at','event_name']
    Returns DataFrame with columns: cohort_week (YYYY-WW), week_offset (int), active_users (unique count)
    """
    # ensure datetimes
    df = df.copy()
    df['signup_at'] = pd.to_datetime(df['signup_at'])
    df['event_at']  = pd.to_datetime(df['event_at'])
    
    # week start (Monday) for ISO week alignment
    df['cohort_week_start'] = df['signup_at'].dt.to_period('W').apply(lambda p: p.start_time)
    df['event_week_start']  = df['event_at'].dt.to_period('W').apply(lambda p: p.start_time)
    
    # week_offset in weeks (could be negative if event before signup; filter if undesired)
    df['week_offset'] = ((df['event_week_start'] - df['cohort_week_start']).dt.days // 7).astype(int)
    
    # optional: ignore events before signup
    df = df[df['week_offset'] >= 0]
    
    # cohort label as ISO year-week string
    df['cohort_week'] = df['cohort_week_start'].dt.strftime('%G-%V')  # %G ISO year, %V ISO week number
    
    # unique active users per cohort/week
    out = (df.groupby(['cohort_week','week_offset'])['user_id']
             .nunique()
             .reset_index(name='active_users')
             .sort_values(['cohort_week','week_offset']))
    return out

Key points & complexity:

• Uses to_period('W') to align weeks; grouping is O(n) average.
• Counts unique users (retention measure).

Edge cases handled:

• Non-datetime inputs cast to datetime.
• Events before signup are excluded (filter); adjust if you want to include negative offsets.
• Weeks use ISO-year/week to avoid mixing Dec/Jan.
• Duplicate events per user-week are deduped by nunique.
• Timezones: convert to a common timezone before calling if needed.