Experiment Design, Analysis, and Causal Methods Questions

Design and analysis of experiments and causal inference methods for when randomization is not possible. Candidates should know strategies to ensure randomization and evaluate experiment quality compute sample size and minimum detectable effect select and interpret primary and guardrail metrics and design appropriate test duration. Analysis skills include hypothesis testing p values confidence intervals effect size estimation variance estimation and variance reduction segmentation and interaction analysis and robust reporting of uncertainty. This topic covers observational and quasi experimental approaches such as propensity score matching difference in differences and regression discontinuity how to reason about confounding and selection bias and when to prefer a quasi experimental approach over a randomized test. Candidates should be able to translate causal conclusions into actionable guidance recommend follow up analyses and triangulate evidence across methods.

HardSystem Design

25 practiced

A/B testing must comply with privacy constraints: no PII in logs and additional differential privacy (DP) noise is required for some metrics. How would privacy constraints change experiment design and analysis? Propose practical DP and aggregation approaches, discuss trade-offs between epsilon (privacy budget) and statistical power, and describe how to validate that results remain useful.

Sample Answer

Requirements & constraints:- No PII in logs (identifiers removed/hashed + strict retention).- Some metrics require differential privacy (DP) before export/analysis.- Maintain useful statistical power for A/B decision-making and explainability for stakeholders.

High-level approach:- Push DP and aggregation close to data sources (trusted ETL/service layer) so downstream BI sees only aggregated, privatized metrics.- Keep raw, PII-containing data in secure, access-controlled environments for auditing only (not for reporting).

Practical DP & aggregation approaches:1. Aggregation-first pipeline: - Compute cohort-level counts/sums in a secure environment. - Drop or bucket any quasi-identifiers (age groups, region). - Apply DP mechanisms to aggregated metrics: - Counts: Laplace or discrete Laplace; use thresholding/clip counts to bound sensitivity. - Sums/means: Clip individual contribution, then add Gaussian noise (for (ε,δ)-DP) or Laplace for pure ε-DP. - Publish only post-noise aggregates to BI tools.

2. Hierarchical / grouped DP: - Use hierarchical aggregation (daily → weekly → monthly) with budget splitting to support multiple dashboards while reusing privacy budget efficiently. - Use privacy accounting (advanced composition/Rényi) to track cumulative ε across reports.

3. Synthetic or bootstrap-enabled summaries (if needed): - Generate DP synthetic datasets for exploratory analysis; restrict confidence reporting. - Alternatively, release multiple noisy bootstrap estimates to approximate variance.

Trade-offs: epsilon vs statistical power- Lower ε (stronger privacy) → larger noise → reduced ability to detect small effect sizes; increases required sample size.- Mitigations: - Increase experiment sample size or effect size thresholds. - Pre-define primary metrics and minimize number of releases to conserve budget. - Use sensitivity reductions: clip user contributions, bucket continuous values, use per-user aggregation to bound sensitivity. - Choose (ε,δ)-DP with Gaussian mechanism when you need better utility for same ε budget.

Validation that results remain useful:- Power analysis under DP: simulate A/B with added DP noise to estimate detectable minimum effect size and required sample size; report these alongside results.- Calibration experiments: run parallel internal (non-DP) analyses on secure data to compare estimates vs DP outputs for a short period to quantify bias/noise.- Holdout & sanity checks: include non-private control metrics (if allowed) or noise-free aggregate checks at very high level to ensure no systemic bias.- Report uncertainty: always show confidence intervals that incorporate DP-added noise and document the ε used.- Monitoring: track false positive rate via repeated synthetic experiments; adjust thresholds and budget accordingly.

Operational considerations:- Centralize privacy policy and privacy budget ledger; require analysts to request budget and register experiments.- Automate DP transforms in ETL with reproducible parameters (ε, clipping thresholds, composition used).- Educate stakeholders: include trade-offs (e.g., longer ramp, larger sample) in experiment plans.

Result: A BI-friendly system that enforces no-PII, applies DP at aggregation, preserves interpretability by quantifying noise impact, and uses budget/accounting and validation to keep results actionable.

HardTechnical

31 practiced

Explain issues that arise with staggered adoption in difference-in-differences (DiD) designs (treatment timing heterogeneity) and describe modern estimators (e.g., Callaway & Sant'Anna, Sun & Abraham). How would you implement these on a panel dataset and interpret cohort-specific ATT estimates?

Sample Answer

Staggered adoption (units receive treatment at different times) breaks the classic two-way fixed effects (TWFE) DiD because TWFE implicitly averages many 2x2 comparisons — including "bad" comparisons where already-treated units serve as controls for later-treated units. That creates bias when treatment effects are heterogeneous over time or across cohorts (treatment timing heterogeneity) and can produce negative or misleading average estimates.

Modern estimators:- Callaway & Sant'Anna (2021): estimates group (cohort) and time-specific ATT_g,t, aggregates with user-chosen weights, and allows conditioning on covariates and pre-trends. Robust to heterogeneous dynamic effects.- Sun & Abraham (2021): constructs event-study coefficients using never-treated (or last-treated) as clean controls and provides cohort-specific event-study estimates, then aggregates carefully.

Implementation (panel; show both R and Python packages):- R: use did (Callaway & Sant'Anna) and eventstudyinteract / fixest + sunab (Sun & Abraham via estimatr/fixest).

library(did)
# data: id, time, Y, G (first treated period, Inf for never-treated), Xs
cs <- att_gt(yname="Y", tname="time", idname="id", gname="G", xformla= ~ X1+X2, data=df)
agg_cs <- aggte(cs, type="simple")  # overall ATT
summary(agg_cs)
# cohort-specific:
cs$att.gates  # ATT_g,t matrix accessible via summary(cs)

- Python: use linearmodels for TWFE (not recommended) and the econometrics package didkit or use R via rpy2; also use statsmodels to implement manual event-study with sunab-style dummies.

python

# using rpy2 is recommended; otherwise use custom stacking approach or didkit

How to implement practically as a BI analyst:1. Inspect treatment timing distribution; create G (cohort) variable.2. Plot pre-trends by cohort (placebo) to check parallel trends.3. Estimate cohort-time ATT_g,t (Callaway & Sant'Anna) and plot event-study for each cohort and aggregated ATT.4. If covariates matter, include them or use doubly-robust options in att_gt.

Interpreting cohort-specific ATT:- ATT_g,t is the average treatment effect for units first treated in period g, measured at time t (relative to treatment). Event-time k = t - g interpretation: at k=0 it's the immediate effect; k>0 are dynamic post-treatment effects.- Use cohort plots to see heterogeneity: positive early effects but fade later means average may under/overstate impact.- Aggregate ATT should be interpreted with awareness of weighting: report cohort-specific estimates + weighted average, and show how weights distributed across cohorts.

Practical notes:- Always show cohort-level event-study plots, pre-trend tests, and sensitivity (exclude late cohorts, use never-treated only).- Document which estimator used, control group definition, and weighting when presenting results to stakeholders.

HardTechnical

32 practiced

You have multiple randomized experiments across segments and supporting observational analyses. Outline an approach to combine evidence using meta-analysis or hierarchical Bayesian modelling to produce a single decision metric with credible intervals. Explain how you would weight studies based on quality, incorporate study-level covariates, and communicate resulting uncertainty.

Sample Answer

Start by treating each experiment/observational analysis as a “study” and model the true effect sizes with a hierarchical (random-effects) Bayesian model so you get a single pooled decision metric and full credible intervals that respect between-study heterogeneity and study-level covariates.

Model outline- Likelihood: observed effect y_i ~ Normal(theta_i, sigma_i^2) where sigma_i is the study SE (from experiment) or estimated from observational analysis.- Hierarchy: theta_i ~ Normal(mu + X_i beta, tau^2) where mu is overall mean effect, X_i are study-level covariates (segment, design type, bias score), beta are coefficients, tau captures between-study heterogeneity.- Priors: weakly informative for mu, beta; half-normal or half-Cauchy for tau.

Weighting studies by quality- Use inverse-variance naturally via sigma_i: smaller SE => more weight.- Encode study quality as either: - A precision multiplier: sigma_i_eff = sigma_i / sqrt(w_i) with w_i in (0,∞) from quality score; higher w increases weight. - A covariate in X_i so beta adjusts for systematic bias (e.g., observational vs randomized). - Informative priors on theta_i for low-quality studies (shrink more toward mu) by inflating their sigma_i or using hierarchical variance that depends on quality.

Incorporate study-level covariates- Include categorical/continuous covariates in X_i to explain heterogeneity (segment, sample size, measurement differences).- Allow interactions (e.g., segment × treatment) if you expect effect modification.- Consider meta-regression: posterior for beta tells how effect varies by covariates.

Practical Bayesian implementation (PyMC pseudocode)

python

import pymc as pm
with pm.Model():
    mu = pm.Normal('mu', 0, 1)
    beta = pm.Normal('beta', 0, 1, shape=K)  # K covariates
    tau = pm.HalfNormal('tau', 1)
    # quality weight w_i precomputed between 0.5-2.0
    sigma_eff = sigma_i / pm.math.sqrt(w_i)
    theta = pm.Normal('theta', mu + X.dot(beta), tau, shape=N)
    y = pm.Normal('y', theta, sigma_eff, observed=y_i)
    trace = pm.sample()

Diagnostics & robustness- Posterior predictive checks, R-hat, effective sample sizes.- Sensitivity analyses: vary quality weighting, priors, exclude observational studies, trim influential studies (leave-one-out).- Publication bias checks: funnel plots, Egger-style regression (as covariate).

Decision metric & communication- Report pooled posterior mean (mu) and 90–95% credible interval; also present segment-specific posterior estimates (mu + X_segment beta).- Visuals: forest plot of study estimates with posterior pooled estimate; density plots of posterior for decision metric; decision-threshold probability P(effect > threshold).- Translate uncertainty: e.g., “There is a 78% probability revenue uplift > $0.50 per user; expected uplift = $0.72 (90% CI: 0.10–1.35).”- Provide recommendations with scenario analysis: conservative (use lower CI bound), expected-value (posterior mean), or risk-tolerant (upper tail).- Document assumptions (quality scoring, priors), show sensitivity results, and supply interactive dashboard filters so stakeholders can view segment-level posteriors and probability metrics.

Why this approach- Hierarchical Bayes pools strength across studies while respecting heterogeneity and allowing covariate adjustment.- Quality-weighting and sensitivity checks ensure observational biases and study design differences are explicitly handled, producing defensible, communicable decision metrics with credible intervals.

EasyTechnical

27 practiced

Explain what an A/B test is and describe the end-to-end process you would follow to run a simple A/B test on a website checkout button. In your answer include: how you'd assign users (randomization method), choose treatment/control proportions, pick a primary metric and guardrail metrics, pre-launch sanity checks (instrumentation, logging, balance), how you'd monitor the test while it runs, and key post-hoc analysis steps.

Sample Answer

An A/B test compares a treatment (B) to a control (A) by randomly exposing users to one of the variants and measuring differences in predefined metrics to infer causal impact.

Setup and randomization- Randomization: deterministically hash a stable user identifier (user_id or cookie_id) modulo 10000 to assign buckets. This avoids drift and keeps assignment consistent across sessions.- Proportions: use 50/50 for maximum power for a single variant. If riskier, use 70/30 (control heavier) or ramping (e.g., 10%→50%) with corresponding power adjustments.

Metrics- Primary metric: checkout conversion rate (users who complete purchase / users who reach checkout).- Guardrails: revenue per user, average order value, cart abandonment rate, page load time, and error rate to catch negative side effects.

Pre-launch sanity checks- Instrumentation: validate events (view, add_to_cart, begin_checkout, purchase) in staging and prod; ensure unique event IDs and timestamps.- Logging: capture assignment bucket, user_id, session_id, timestamp, device, and page version.- Balance checks: compare demographics and key pre-experiment metrics across groups (traffic source, device, geography) to confirm randomization.- Power/sample-size: compute required sample and min detectable effect given baseline conversion, alpha (0.05), beta (0.8).- QA: visual check of button variant, smoke tests, and test users flagged to see both variants.

Monitoring during run- Real-time dashboard with daily closure (avoid peeking at many p-values). Monitor primary and guardrails, traffic split, instrumentation errors, and sample accumulation.- Alerts for sudden drops in conversion, revenue, or error spikes.- Respect pre-specified stopping rules (run until minimum sample and duration—usually >1 business cycle—are reached).

Post-hoc analysis- Verify randomization and no differential attrition.- Compute lift, confidence intervals, and p-value for the primary metric; report absolute and relative lift.- Check guardrails and secondary metrics; if multiple comparisons exist, adjust (Bonferroni or control FDR).- Segment analysis (device, channel) to find heterogeneous effects—treat as exploratory unless pre-specified.- Practical significance: compare expected revenue impact and rollout cost.- Sanity checks: time-based effects, novelty, and invariant metrics (e.g., user sign-ups that should be unaffected).- Decision: if statistically and practically significant with no adverse guardrails, rollout; otherwise iterate or roll back.

This process ensures rigor (randomization, power), safety (guardrails, monitoring), and actionable conclusions (statistical + business significance) aligned to BI reporting needs and downstream dashboards.

MediumTechnical

42 practiced

You have a dataset of users with covariates and a binary treatment column. Outline code-level steps (Python or R) to compute propensity scores, perform nearest-neighbor matching, and evaluate balance diagnostics. List specific diagnostics (e.g., standardized mean differences, love plots) you would produce and thresholds you'd consider acceptable.

Sample Answer

Approach (brief): estimate propensity scores with logistic regression, perform 1:1 nearest-neighbor matching (optionally caliper), then evaluate balance using standardized mean differences (SMD), variance ratios, distributional checks and visualizations (love plot, histograms, QQ/KS). Aim for SMD < 0.1 (good) or < 0.2 (acceptable); variance ratio between 0.5–2.

Python (pandas + sklearn) example:

python

import pandas as pd
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import NearestNeighbors

# X: covariates DataFrame, T: binary treatment Series
model = LogisticRegression(max_iter=1000)
model.fit(X, T)
ps = model.predict_proba(X)[:,1]
df = X.copy()
df['T'] = T
df['ps'] = ps

# Nearest-neighbor 1:1 matching (treated -> control)
treated = df[df['T']==1].reset_index(drop=True)
control = df[df['T']==0].reset_index(drop=True)
nn = NearestNeighbors(n_neighbors=1).fit(control[['ps']])
dist, idx = nn.kneighbors(treated[['ps']])
caliper = 0.05
mask = dist.flatten() <= caliper
matched_t = treated[mask]
matched_c = control.iloc[idx.flatten()[mask]].reset_index(drop=True)
matched = pd.concat([matched_t.reset_index(drop=True), matched_c])

SMD function and diagnostics:

python

def smd(var, t, weights=None):
    g1 = var[t==1]; g0 = var[t==0]
    return (g1.mean() - g0.mean()) / np.sqrt((g1.var()+g0.var())/2)

# compute SMDs before and after
smds_before = {col: smd(df[col], df['T']) for col in X.columns}
smds_after = {col: smd(matched[col], matched['T']) for col in X.columns}

Diagnostics to produce:- Table of SMDs before/after (target: <0.1)- Love plot: dot plot of SMDs before vs after- Histograms/density plots of propensity scores by group (common support)- Variance ratios (treated_var / control_var) — acceptable 0.5–2- KS test or QQ plots for covariate distributions- Number matched, unmatched, and caliper rejection rate

R (tidyverse + MatchIt) quick commands:

library(MatchIt)
m.out <- matchit(T ~ x1 + x2 + x3, data=df, method="nearest", caliper=0.05)
matched_df <- match.data(m.out)
summary(m.out)        # shows SMDs before/after
plot(m.out, type="jitter")   # propensity score overlap
love.plot <- love.plot::love.plot(m.out) # love plot

Notes & best practices:- Check overlap/common support; trim outside region if needed.- Prefer SMDs over p-values (sample-size dependent).- If imbalance persists, consider adding interactions, non-linear terms to propensity model or use caliper/optimal matching.

Unlock Full Question Bank

Get access to hundreds of Experiment Design, Analysis, and Causal Methods interview questions and detailed answers.

Join thousands of developers preparing for their dream job.