HardTechnical

63 practiced

Propose a statistical and machine learning approach to estimate heterogeneous treatment effects (HTE) across user segments for an A/B test. Discuss methods like causal forests or uplift models, how you'd validate results, control for overfitting and multiple comparisons, and make the outputs interpretable for product managers.

Sample Answer

Goal: estimate how treatment effect varies across user segments so product can target features confidently.

Approach overview:1) Data prep: merge experiment assignment, outcomes, covariates (pre-period behavior, demographics, device, acquisition channel). Ensure randomization integrity (balance tests) and drop post-randomization covariates.

2) Modeling options (practical, role-appropriate):- Causal Forests / Generalized Random Forests: nonparametric HTE estimates with honest splitting to avoid bias. Good default for continuous or binary outcomes.- Uplift models (two-model or single-model with treatment indicator and interaction terms): simpler and interpretable for business rules.- X-learner or T-learner ensemble: useful when treatment/control sizes differ.Choose Causal Forest for discovery; complement with simpler uplift/logistic regression with treatment*feature interactions for explainability.

3) Prevent overfitting & multiple comparisons:- Use honest estimation (sample-splitting): train trees on one sample, estimate effects on held-out sample.- Cross-fitting / cross-validation for hyperparameters.- Regularization for parametric models and limit depth/features in trees.- For many segments, control false discovery: use Benjamini–Hochberg FDR on segment-level effect estimates or use hierarchical shrinkage (Bayesian multilevel) to pull noisy estimates toward global mean.

4) Validation:- Calibration: check predicted CATE vs. actual ATE in bins (decile uplift calibration).- Policy risk / Qini / AUUC and uplift curves to evaluate ranking quality.- Backtesting: simulate targeting top-k users and measure lift in holdout.- Sensitivity: test robustness to covariate shifts and alternative outcome definitions.- Placebo checks: run the pipeline on pre-experiment periods or on randomized "fake" treatments.

5) Interpretability for PMs:- Produce ranked segments (top-N lift) with effect size and confidence intervals.- Use simple decision rules derived from model (e.g., "users with >3 purchases and mobile = +6pp lift").- Feature importance and partial dependence / SHAP for CATE drivers, translated into plain language.- Visuals: uplift curve, calibration plot, and a small table: segment definition, estimated incremental metric, expected incremental revenue, CI, sample size.- Recommend experiments to validate top actionable segments (targeted rollout A/B) before full rollout.

Result: combine robust causal ML (causal forest) for discovery with parsimonious, interpretable models and conservative statistical control (honest estimation + FDR/shrinkage) to provide PMs prioritized, reliable, and explainable targeting recommendations.

EasyTechnical

61 practiced

Explain the difference between primary metrics, secondary metrics, and guardrail metrics. For an e-commerce checkout improvement (faster checkout flow), name one metric for each category, justify your choices, and describe how you would monitor them during an experiment.

EasyTechnical

65 practiced

You have two tables:

sql

users(user_id bigint, signup_date date)
orders(order_id bigint, user_id bigint, amount numeric, occurred_at timestamp)

Write a PostgreSQL query to compute weekly conversion rate (users with an order that week / new users that week) for the last 12 weeks and a 95% confidence interval for each week.

Sample Answer

Approach: build weekly cohorts by signup week (date_trunc('week', signup_date)), count new users per week, count how many of those placed >=1 order within the same week, compute conversion = p = x/n and a 95% Wilson score confidence interval (better for small n than normal approx). Limit to last 12 weeks.

sql

WITH cohorts AS (
  SELECT
    date_trunc('week', signup_date)::date AS week_start,
    user_id
  FROM users
  WHERE signup_date >= date_trunc('week', current_date) - interval '11 weeks'
),
orders_in_week AS (
  SELECT
    c.week_start,
    c.user_id
  FROM cohorts c
  JOIN orders o
    ON o.user_id = c.user_id
    AND date_trunc('week', o.occurred_at)::date = c.week_start
  GROUP BY c.week_start, c.user_id
),
agg AS (
  SELECT
    c.week_start,
    COUNT(DISTINCT c.user_id) AS n_new_users,
    COUNT(DISTINCT o.user_id) AS n_converted
  FROM cohorts c
  LEFT JOIN orders_in_week o
    ON c.week_start = o.week_start AND c.user_id = o.user_id
  GROUP BY c.week_start
)
SELECT
  week_start,
  n_new_users,
  n_converted,
  ROUND(n_converted::numeric / NULLIF(n_new_users,0), 4) AS conversion_rate,
  -- Wilson score 95% CI
  ROUND((
    (phat + z*z/(2*n) - z * sqrt((phat*(1-phat) + z*z/(4*n))/n))
  ), 4) AS ci_lower,
  ROUND((
    (phat + z*z/(2*n) + z * sqrt((phat*(1-phat) + z*z/(4*n))/n))
  ), 4) AS ci_upper
FROM (
  SELECT
    week_start,
    n_new_users,
    n_converted,
    n_new_users::numeric AS n,
    COALESCE(n_converted::numeric / NULLIF(n_new_users,0), 0) AS phat,
    1.96 AS z
  FROM agg
) t
ORDER BY week_start DESC;

Key points:- Uses weekly cohort = signup week and counts orders in same week.- Wilson CI is robust for small samples.Edge cases: weeks with zero new users (NULLIF prevents div-by-zero), duplicated orders per user handled by DISTINCT. You can extend window or compute conversion in later windows (e.g., 7-day from signup) by adjusting date range check on occurred_at.

MediumTechnical

63 practiced

Given tables:

sql

users(user_id, country)
orders(order_id, user_id, amount, occurred_at)

Write SQL (Postgres) to compute incremental revenue per user between treatment and control, grouped by country, with a 95% CI using bootstrapping or the delta method. Explain your approach in comments.

Sample Answer

-- Approach:-- 1) Aggregate total revenue per user in the analysis window.-- 2) For each country x treatment group compute: count (n), mean revenue per user, sample variance.-- 3) Use the delta method / CLT: difference in means ~ Normal with variance var_t/n_t + var_c/n_c.-- 4) Report incremental revenue per user (treatment - control) with 95% CI per country.

WITH user_revenue AS ( -- sum revenue per user in the period of interest (adjust occurred_at filter as needed) SELECT u.user_id, u.country, SUM(o.amount) AS revenue FROM users u LEFT JOIN orders o ON o.user_id = u.user_id AND o.occurred_at >= '2025-01-01' -- example start AND o.occurred_at < '2025-02-01' -- example end GROUP BY u.user_id, u.country),group_stats AS ( -- assume there's a way to know treatment vs control; here we assume users table has a boolean "is_treatment" SELECT country, is_treatment, COUNT(*)::numeric AS n, AVG(revenue)::numeric AS mean_rev, CASE WHEN COUNT(*)>1 THEN VAR_SAMP(revenue)::numeric ELSE 0 END AS var_rev FROM user_revenue ur JOIN users u USING (user_id) -- to access is_treatment flag if not present in user_revenue GROUP BY country, is_treatment),pivot AS ( -- pivot treatment and control per country SELECT COALESCE(t.country, c.country) AS country, t.n AS n_t, t.mean_rev AS mean_t, t.var_rev AS var_t, c.n AS n_c, c.mean_rev AS mean_c, c.var_rev AS var_c FROM group_stats t FULL OUTER JOIN group_stats c ON t.country = c.country AND t.is_treatment = true AND c.is_treatment = false WHERE (t.is_treatment = true OR c.is_treatment = false))SELECT country, (mean_t - mean_c) AS incremental_rev_per_user, -- delta method standard error CASE WHEN n_t IS NULL OR n_c IS NULL OR (n_t=0 OR n_c=0) THEN NULL ELSE sqrt( COALESCE(var_t,0)/n_t + COALESCE(var_c,0)/n_c ) END AS se_diff, -- 95% CI CASE WHEN n_t IS NULL OR n_c IS NULL OR (n_t=0 OR n_c=0) THEN NULL ELSE (mean_t - mean_c) - 1.96 * sqrt( COALESCE(var_t,0)/n_t + COALESCE(var_c,0)/n_c ) END AS ci_lower, CASE WHEN n_t IS NULL OR n_c IS NULL OR (n_t=0 OR n_c=0) THEN NULL ELSE (mean_t - mean_c) + 1.96 * sqrt( COALESCE(var_t,0)/n_t + COALESCE(var_c,0)/n_c ) END AS ci_upperFROM pivotORDER BY country;

-- Notes:-- - If users table lacks is_treatment flag, substitute join to an experiment assignment table.-- - Bootstrap alternative: export per-user revenues and run bootstrap in Python/R for non-parametric CIs.

HardTechnical

55 practiced

You must pre-register an analysis plan for an experiment tracking 30 metrics across 5 segments. You want to optimize power for the primary metric while controlling the False Discovery Rate (FDR) at 5% across secondary tests. Propose a statistical strategy, including adjustments, prioritization, and practical considerations.

Sample Answer

Plan summary: define one clearly prioritized primary metric (and whether it’s pooled or a specific segment) to be tested at two-sided alpha = 0.05 with power optimization; treat the remaining 29 metrics × 5 segments as a secondary family for FDR control at 5%.

Pre-registered items (must specify):- Exact primary hypothesis (metric, segment(s), population, time window).- Statistical model (e.g., linear regression on metric with treatment indicator + pre-period covariates; link for non-normal metrics).- Estimand (ATE, lift %) and handling of missing/outliers.- Test statistic, transformation, and variance estimator (robust/clustered if needed).- Multiple-testing procedure and decision rules.

Strategy details1) Power optimization for primary:- Run sample-size / MDE calculations using realistic baseline variance and expected ICC by segment. If segmented, choose whether to power the pooled primary or a pre-specified focal segment.- Use covariate adjustment (pre-period metric, demographic strata) to reduce variance and increase power.- Consider blocking/stratified randomization and analyze accordingly.

2) Multiplicity control for secondaries:- Define the secondary family: all non-primary (e.g., 149 tests). Pre-specify grouping (e.g., 29 metrics × pooled + 5 segment tests).- Use Benjamini–Hochberg (BH) to control FDR at 5% on secondary family. If test statistics are dependent and dependency is complex, use Benjamini–Yekutieli or Storey’s q-values (or run a dependency-aware permutation to get empirical null p-values).- Implement hierarchical/gatekeeping: test primary first at full alpha; only if primary is significant proceed to BH on secondaries. This preserves primary power and provides structured inference.

3) Practical adjustments & robustness:- Pre-specify effect-size thresholds for “practically significant” (not just p-value).- Use bootstrap/permutation p-values if assumptions are doubtful.- Pre-specify handling of multiple looks (stop/go rules) — if interim analyses are planned, adjust via alpha-spending or use combination tests; otherwise, disallow peeking.- Simulate entire analysis pipeline before running to estimate realized FDR and power under realistic correlations among metrics/segments.

Reporting & operational notes:- Publish the exact code, seed, and dataset definitions in the pre-registration.- Report unadjusted p-values, BH-adjusted q-values, effect sizes with CIs, and decision rules outcome.- Emphasize business-action thresholds and avoid binary claims when effects are small but statistically significant.

This approach preserves power for the primary objective while controlling false discoveries across many secondary tests and is robust to metric correlation when dependency-aware methods or resampling are used.

Metrics, Guardrails, and Evaluation Criteria Questions

Sample Answer

Sample Answer

Sample Answer

Sample Answer

Sample Answer

Unlock Full Question Bank