Experiment Design, Analysis, and Causal Methods Questions

Design and analysis of experiments and causal inference methods for when randomization is not possible. Candidates should know strategies to ensure randomization and evaluate experiment quality compute sample size and minimum detectable effect select and interpret primary and guardrail metrics and design appropriate test duration. Analysis skills include hypothesis testing p values confidence intervals effect size estimation variance estimation and variance reduction segmentation and interaction analysis and robust reporting of uncertainty. This topic covers observational and quasi experimental approaches such as propensity score matching difference in differences and regression discontinuity how to reason about confounding and selection bias and when to prefer a quasi experimental approach over a randomized test. Candidates should be able to translate causal conclusions into actionable guidance recommend follow up analyses and triangulate evidence across methods.

MediumTechnical

32 practiced

List and describe the checks (and associated SQL or Python steps) you would run to validate randomization quality in an A/B test. Include balance tests for baseline covariates, time-based assignment drift checks, visualizations, and threshold criteria for when to stop, adjust, or re-randomize.

Sample Answer

Start with a checklist approach: (1) verify assignment randomness overall, (2) balance baseline covariates, (3) test time-based assignment/drift, (4) visualize, (5) decide thresholds & actions.

1) Basic assignment sanity (counts)SQL:

sql

SELECT variant, COUNT(*) AS n
FROM ab_assignments
WHERE assigned_at BETWEEN '2025-01-01' AND '2025-02-01'
GROUP BY variant;

Python:

python

df.groupby('variant').size()

2) Balance tests for baseline covariates- Categorical: chi-square or proportion testSQL (contingency):

sql

SELECT variant, country, COUNT(*) FROM users JOIN ab_assignments USING(user_id)
GROUP BY variant, country;

Python (chi2 / proportions):

python

from scipy.stats import chi2_contingency, ttest_ind
ct = pd.crosstab(df['variant'], df['country'])
chi2_contingency(ct)

- Continuous: t-test + standardized mean difference (SMD)

python

smd = (df[df.variant=='A'][col].mean() - df[df.variant=='B'][col].mean()) / df[col].std()
ttest_ind(...)

Thresholds: SMD > 0.1 or p < 0.01 after correction → flagged.

3) Time-based assignment / drift- Cumulative assignment over time (should be flat ratio)SQL (time series):

sql

SELECT date(assigned_at) AS day, variant, COUNT(*) FROM ab_assignments GROUP BY day, variant;

Python: compute cumulative proportion and run linear regression of assignment indicator vs time to detect slope ≠ 0. KS test for distribution of assignment times between variants.

4) Visualizations (build in Tableau/Matplotlib)- Bar chart of counts by variant- SMD plot for covariates (love plot)- Cumulative assignment ratio over time- Histograms / ECDFs for continuous covariates by variant

5) Multiple testing & correction- Use BH/FDR for many covariates; report both p-values and SMDs.

6) Decision thresholds & actions- If small imbalances (SMD 0.1–0.25 or p between 0.01–0.05): document covariate, adjust analysis with covariate controls (ANCOVA, regression/stratification), continue test.- If large imbalance (SMD > 0.25 or many covariates fail, or assignment proportions drift >2–5% from target): pause data collection, investigate implementation (cookie logic, join key issues), fix bug, and either (a) re-run experiment with fresh randomization or (b) continue but analyze only post-fix data and treat pre-fix as corrupted.- If time-dependent assignment bias detected: consider blocking/stratified randomization going forward or restrict analysis window.

Edge cases & practices:- Always pre-register checks and thresholds.- Run checks on holdout/QA samples before public rollout.- Log assignment events and use deterministic hashing where possible to avoid drift.- Report diagnostics in the experiment dashboard and include code snippets and decisions for auditability.

EasyBehavioral

24 practiced

You're presenting experiment results to a non-technical product manager. Outline exactly how you'd communicate the effect estimate and uncertainty (e.g., p-values, confidence intervals, expected revenue impact). Provide a short slide structure (3-5 bullets per slide) you would use to make an actionable recommendation.

MediumTechnical

26 practiced

Describe the role of pre-registration and stopping rules in experiments. Explain how sequential testing or group-sequential alpha-spending methods can be used safely, and list practical implementation steps (documentation, monitoring, adjusted thresholds) for an experimentation platform.

EasyTechnical

32 practiced

Define the 'unit of randomization' and explain why selecting the correct unit matters for experiment validity. Provide three examples of units (user, session, account) and describe consequences (bias, underpowered tests, interference) of choosing a unit that is too coarse or too fine.

EasyTechnical

32 practiced

Explain statistical power in the context of A/B testing. Why is power important when interpreting non-significant results, and how would you report power considerations (planned vs achieved) in your experiment summary?

Unlock Full Question Bank

Get access to hundreds of Experiment Design, Analysis, and Causal Methods interview questions and detailed answers.

Join thousands of developers preparing for their dream job.