Experimentation Metrics and Strategy Questions

Designing experiments and selecting appropriate primary, secondary, and guardrail metrics to evaluate hypotheses while protecting long term user value. This includes choosing metrics that reflect both short term signal and long term outcomes, reasoning about metric interactions and potential unintended consequences, and applying statistical considerations such as minimum detectable effect, sample size and power analysis, test duration, and external validity across segments and platforms. Candidates should also discuss experiment risk mitigation, stopping rules, and how to operationalize experiment results into product decisions.

HardTechnical

75 practiced

Design a composite 'user-satisfaction' metric combining NPS score, 30-day retention rate, and support-ticket volume per user that can be used as a primary experiment metric. Specify normalization, weighting rationale, handling opposite directions (higher better vs lower better), and how you'd perform sensitivity analysis to understand the impact of weight choices on decisions.

Sample Answer

Situation: We need a single experiment-primary metric that combines NPS (higher = better), 30-day retention rate (higher = better), and support-ticket volume per user (lower = better).

Design (normalization + direction):1. Normalize each component to a 0–1 scale using robust min-max with winsorization at 1st/99th percentiles to limit outliers: - norm_X = (X - p1(X)) / (p99(X) - p1(X)), clipped to [0,1].2. Align directions: for tickets (lower is better) use norm_tickets_inv = 1 - norm_tickets.3. Optionally convert to z-scores if you prefer mean-centered weighting (useful when components have different distributions).

Composite formula: US = w1 * norm_NPS + w2 * norm_Retention + w3 * norm_tickets_inv then scale US to 0–100: US_score = 100 * US / (w1+w2+w3).

Weighting rationale:- Business-driven base: choose weights reflecting business value and sensitivity. Example: retention drives revenue so w2=0.5, NPS indicates word-of-mouth w1=0.3, tickets reflect friction w3=0.2.- Statistical reliability adjustment: downweight metrics with higher measurement noise (use inverse-variance weighting: w_i ∝ value_weight_i / var_i).

Sensitivity analysis (to understand impact on experiment decisions):1. Grid sweep: evaluate resulting treatment vs. control delta_US across plausible weight combinations (e.g., w1∈[0,0.6], w2∈[0.2,0.8], w3=1-w1-w2). Record sign and significance of treatment effect.2. Monte Carlo: sample weights from a prior distribution (e.g., Dirichlet around chosen weights), recompute metric using bootstrapped metric estimates; compute proportion of samples where decision (significant lift) changes.3. Tornado/One-way sensitivity: vary one weight at a time to show which weight drives decision thresholds.4. Robustness checks: repeat using z-score normalization and alternative winsorization thresholds.5. Report: decision stability (% of weight-space that leads to same experiment conclusion), and worst-case estimate (min lift across plausible weights).

Example actionable rule: require treatment to show a significant positive delta_US across >80% of sampled weight configurations to be declared a win.

MediumTechnical

59 practiced

You observe that treatment effect varies across countries and device types. How would you test for heterogeneous treatment effects robustly? Describe statistical tests, use of interaction terms, correction for multiple comparisons, and how you'd decide whether to roll out globally or regionally.

Sample Answer

First, clarify definition and pre-specify hypotheses (countries, device types, or their cross-tabs) to avoid p-hacking.

Testing strategy:- Start with regression with interaction terms: Y = α + τ·T + β·X + γ1·(T×Country) + γ2·(T×Device) + ε. Test joint significance of γ’s (Wald or F-test / likelihood ratio) to detect heterogeneity.- Use clustered, heteroskedasticity-robust SEs by randomization unit (user/account/country) and, for many groups, use wild cluster bootstrap for small cluster counts.- For many segments, run model-based adjustments: hierarchical/multilevel model (random slopes for T by country/device) to borrow strength and get shrinkage estimates; this reduces false positives.- Nonparametric / ML alternatives: causal forests / generalized random forests to estimate conditional average treatment effects (CATEs) and test heterogeneity via permutation tests or split-sample inference.

Multiple comparisons:- If reporting many subgroup estimates, control false discovery: Benjamini–Hochberg (FDR) preferred for power; Bonferroni if you want strict family-wise error control. Use pre-registration to define families.- Report adjusted p-values and confidence intervals; consider simultaneous CIs (e.g., Bonferroni or Scheffé) when making multiple claims.

Robustness checks:- Balance tests within subgroups, sensitivity to covariate adjustment, placebo outcomes, and randomization inference / permutation tests.- Check consistency across specifications (fixed effects, excluding small-n groups).

Decision rule for rollout:- Combine statistical evidence (significant heterogeneity after corrections, sufficient power) with business criteria: minimal detectable effect vs. cost, operational constraints, and risk tolerance.- If effect is positive and homogeneous → global rollout. If heterogeneous and beneficial in some regions/devices but null/harmful elsewhere → regional rollout limited to segments with robust, adjusted positive CATEs; consider phased rollout with monitoring and adaptive allocation.- Document uncertainty, thresholds for expansion/stop, and run follow-up experiments if needed.

HardSystem Design

107 practiced

Propose a design for a central metric registry and metric contract system to prevent metric drift and misinterpretation across analytics consumers. Include a schema for metric definitions (ID, name, version, canonical SQL, owner, tags), CI checks, deprecation policy, automated lineage, and how to handle metric backfills and migrations safely.

Sample Answer

Requirements:- Single source of truth for metric definitions accessible to analytics, product, ML, and dashboarding consumers.- Strong semantic versioning, validation, lineage, and auditability.- Safe backfills/migrations with reproducible re-computation.- Automated CI checks and deprecation lifecycle.

High-level architecture:- Central Metric Registry service (Git-backed + API + UI) + metadata DB (Postgres) + artifact store for canonical SQL (repo) + lineage engine (Airflow/DBT) + CI pipeline (GitHub Actions) + query/execution engine (Presto/Spark/warehouse).- Consumers read metrics via Registry API or fetch compiled SQL/templates for execution.

Metric schema (example fields):- id: uuid- name: string (stable slug)- version: semver (major.minor.patch)- canonical_sql: string (parameterized SQL in repo)- owner: {team, contact}- description: text- tags: list[string]- inputs: list[{table, column, source_type}]- aggregation: {type, window}- freshness: policy- created_at, updated_at- status: active|deprecated|removed- backfill_policy: immediate|scheduled|manual- test_cases: list[{input_snapshot, expected_value}]- lineage_hash: fingerprint of inputs + SQL

Workflow & CI checks:- All changes via PR to metrics repo. CI runs: - SQL lint & style (sqlfluff) - Static type checks & parameter validation - Unit tests: run canonical SQL against a small test dataset or mocked tables; validate outputs against test_cases - Lineage fingerprinting: compute and store lineage_hash; fail if inputs changed without version bump - Semantic version enforcement: major bump required for breaking contract changes (aggregation, filters, semantics); minor/patch for performance or non-breaking changes - Backwards-compatibility checker: simulate existing dashboards/queries against new version- Merge gated by owner approval.

Versioning & contract rules:- Semver: major = breaking semantic change; minor = additive non-breaking change; patch = bugfix/perf.- Registry API serves metrics by name+version or name@latest-stable.- Consumers default to pinned versions; platform recommends auto-upgrade to minor/patch, manual for major.- Contract test suite ensures published metrics expose documented columns and semantics.

Deprecation policy:- Mark deprecated with timeline (e.g., 90 days) and replacement pointer.- Announce via dashboards/channels; automatically flag dependent dashboards and scheduled jobs.- After deprecation window, metric moved to removed only after confirming migrations completed.

Automated lineage:- Parse canonical_sql to extract input tables/columns; store in lineage graph (Neo4j or metadata DB).- Trigger dependency scans: identify dashboards/ML features depending on metric.- On input change, notify owners and require new metric version if semantics change.

Backfills & migrations:- Backfill plan recorded in metric metadata with deterministic backfill SQL and idempotent job (airflow/dag).- Backfill executed on separate staging partition; results compared with previous values via reconciliation job (sampled diffs, KS tests).- For migrations requiring historical recompute (major change), registry creates a migration bundle: - new metric version + canonical backfill SQL + reconciliation job + rollout window - Run backfill in shadow mode; produce diff reports and anomaly thresholds. - Only after automated checks and owner sign-off swap consumer aliases (name@latest-stable) to new version.- Keep frozen historical metric versions for reproducibility; queries stored with version tag.

Observability & governance:- Metric usage telemetry, SLA alerts on freshness, drift detection (statistical), and a catalog UI for exploration.- Quarterly audits and monthly automated drift scans.

Why this works:- Git + CI enforces reproducibility and code review; semver + tests protect consumers; lineage + automated notifications reduce surprise breakages; controlled backfill/migration with shadowing ensures safe history recomputation.

HardTechnical

62 practiced

Design an experiment to estimate the long-term causal effect of a personalization algorithm on customer lifetime value (LTV). Address randomization unit, blocking, measurement window (≥12 months), surrogate short-term metrics, handling censoring and churn, how to attribute incremental revenue amid pricing changes, sample-size implications over months, and threats to validity with mitigation strategies.

Sample Answer

Framework: treat this as a longitudinal randomized controlled trial with survival/causal estimand (incremental discounted LTV at 12+ months).

Design summary:- Randomization unit: randomize at customer level (user/account) to avoid interference; if strong interference (household/shared accounts) or algorithm uses network effects, randomize at cluster (household/region) and account for cluster in analysis.- Blocking/stratification: stratify by pre-experiment covariates predictive of LTV and treatment uptake — e.g., cohort tenure, past 6‑12 month spend quantiles, product mix, and device. Within strata, do 1:1 random assignment to balance baseline risk.- Measurement window: follow users for at least 12 months; compute cumulative revenue with appropriate discounting (e.g., monthly discount factor). Pre-specify primary endpoint at 12 months and secondary endpoints at 3, 6, 18 months.- Surrogate short-term metrics: optimize power and interim signals using validated surrogates that correlate with long-term LTV (e.g., retention rate at 30/90 days, incremental purchase frequency, average order value). Use surrogate validation (Prentice-like checks or meta-analytic correlation) on historical data to justify.- Handling censoring and churn: treat churn as absorbing state. Use survival analysis (Cox or parametric AFT) and Kaplan–Meier for retention curves; use inverse probability censoring weights (IPCW) to adjust when customers are right-censored due to study end or attrition unrelated to treatment. For revenue streams, use recurrent-event models or joint models for purchase timing and amount; consider Tobit or zero-inflated models for sparse spend.- Attribution amid pricing changes: pre-specify an attribution model: adjust observed revenue for exogenous price changes by normalizing to a constant price index (convert quantities × baseline prices). If discounts/promotions differ, instrument for price using exogenous promotion assignment or include time × treatment interaction fixed effects. Use difference-in-differences with time fixed effects to net out economy-wide price movements. For catalog/pricing experiments, track product-level unit sales and compute counterfactual revenue by re-pricing units at baseline prices.- Sample size and power over months: compute sample to detect a minimum detectable effect (MDE) in discounted 12-month LTV using historical variance of customer-level 12‑month revenue. Because variance and event rates evolve, plan adaptive sample-size re-estimation after 3–6 months using blinded pooled variance. Account for loss-to-follow-up and clustering (design effect). If using interim checks, correct alpha with group sequential methods or use pre-specified Bayesian stopping rules.- Analysis plan: intent-to-treat (ITT) primary; complier-average causal effect (CACE) as secondary if noncompliance is material. Estimate incremental LTV with regression adjusting for blocking covariates and using robust clustered SEs. For heterogeneity, pre-specify subgroups.- Threats to validity and mitigations: - Spillovers/interference: use cluster randomization or measure cross-group exposures; sensitivity analysis assuming partial interference. - Non-random attrition: use IPCW and bounding (worst/best-case) analyses. - Seasonality/time trends: ensure randomization is rolling and balanced across calendar; include calendar fixed effects. - Policy/pricing confounds: coordinate experiment with pricing/Product teams; log all concurrent experiments and adjust with multivariate regression or restrict analysis window. - Multiple experiments and peeking: register analysis plan, limit peeks or adjust for multiple looks. - Surrogate failure: validate surrogate keys on holdout historical cohorts; keep LTV primary.Implementation notes:- Logging: capture impression/exposure, recommended content, actions, price and discount metadata, product-level unit sales, and customer identifiers.- Monitoring: weekly dashboards for balance, treatment exposure, and early safety signals; blinded variance checks for sample re-estimation.Outcome: yields unbiased estimate of long-term incremental LTV (and time-profile), with principled handling of censoring, pricing shifts, and threats to validity.

HardTechnical

58 practiced

An overall conversion rate increased from 2.0% (control) to 2.8% (treatment), but when stratified by device (mobile and desktop), conversion decreased in both segments. Explain how Simpson's paradox can cause this pattern, describe diagnostics to detect it, and propose how to resolve and present the correct interpretation to stakeholders.

Sample Answer

Situation: You run an A/B test and see overall conversion go from 2.0% → 2.8% (positive), but conversion falls on both mobile and desktop. This is a classic Simpson’s paradox: a lurking confounder (device mix shift between groups) reverses the direction of effects when you aggregate.

Why it happens (concise):- The treatment and control have different device distributions. Suppose treatment assigns many more users from a high-converting segment (say desktop) or fewer from a low-converting one (mobile); the weighted average can increase even though within each device stratum the treatment performs worse.- Aggregation hides the differing weights; per-stratum effects and weights produce the paradox.

Diagnostics to run:- Compute conversion rates and absolute/relative lifts per stratum and overall; show counts (n) and conversion counts.- Compare device distributions between groups (contingency table + chi-square or standardized difference) to detect imbalance.- Calculate weighted average of stratum-specific rates using the overall population weights to verify algebraic consistency.- Fit a regression (logistic/linear) with device indicator, treatment, and interaction term; check significance and sign of treatment and treatment×device.- Visualize: stacked bar of group composition, stratified rate plots, and a table of conditional vs marginal effects.

How to resolve and present:- If imbalance is due to randomization failure, re-run or exclude post-randomization assignment errors; document and, if possible, re-randomize.- Use adjustment: report adjusted treatment effect from a regression controlling for device (or inverse-probability weighting) to estimate average treatment effect for the target population.- Present both: (a) stratified results (mobile, desktop) with CIs and sample sizes, (b) adjusted estimate with method and assumptions, and (c) the raw aggregated result with a clear note that aggregation masks confounding.- Use a DAG to explain causality: show device → conversion and treatment ↔ device (imbalance). State which estimand stakeholders care about (ATE over current traffic mix vs. per-device effects) and recommend action accordingly.- Recommendation: prefer adjusted estimate or stratified decisions (e.g., hold or rollback) depending on business goal; be transparent about uncertainty and the root cause (imbalance vs true heterogeneity).

Unlock Full Question Bank

Get access to hundreds of Experimentation Metrics and Strategy interview questions and detailed answers.

Join thousands of developers preparing for their dream job.