Experimental Design and Analysis Pitfalls Questions

Covers the principles of designing credible experiments and the common errors that invalidate results. Topics include defining clear hypotheses and control and treatment groups, randomization strategies, blocking and stratification, sample size and power calculations, valid run length and avoiding early stopping, and handling unequal or missing samples. It also covers analysis level pitfalls such as multiple comparisons and appropriate corrections, selection bias and nonrandom assignment, data quality issues, seasonal and temporal confounds, network effects and interference, and paradoxes such as Simpson paradox. Candidates should be able to critique flawed experiment designs, identify specific failure modes, quantify their impact, and propose concrete mitigations such as pre registration, A and B testing best practices, adjustment methods, intention to treat analysis, A over A checks, cluster randomization, and robustness checks.

MediumSystem Design

29 practiced

Design an experiment logging and instrumentation plan that minimizes biases from missing events, deduplication errors, and late-arriving events during a streaming ML model rollout. Include schema-level requirements, monitoring checks, and automated alerts to detect problems early.

Sample Answer

**Clarify goals & constraints**I’d ensure end-to-end observability so model metrics during a streaming rollout are trustworthy despite missing, duplicate, or late events. Constraints: low added latency, bounded storage, and backward-compatible schemas.

**High-level design**- Insert lightweight event-id, source, and ingestion_ts at producer; immutable append-only stream (Kafka with exactly-once where possible).- Sidecar real-time dedup service using event-id + content-hash and tombstone TTL.- Buffering window for late events with watermarking; late-arrival pipeline writes to corrective topic that triggers recomputation for affected windows.

**Schema-level requirements**- event_id: UUID (producer-assigned)- record_ts (producer), ingestion_ts (broker), model_infer_ts- source_id, partition_key- payload_hash: SHA256 of predictable fields- optional original_offset and lineage metadata

**Monitoring checks**- Missing events: compare expected throughput (source heartbeat) vs observed; delta per source/partition.- Deduplication errors: track dedupe_ratio = 1 - unique_count/total_count; alert if > threshold.- Late arrivals: percent_late = late_count/total_count and tail-latency percentiles (P50/P95/P99) of event lag.- Schema drift: unknown fields, field-type changes, null-rate per field.- Consistency checks: payload_hash mismatches for same event_id.

**Automated alerts & actions**- Alert if missing > X% for 5 min → investigate producer/backpressure.- Alert if dedupe_ratio > Y% or rising → possible producer replay or mis-assigned event_id.- Alert if percent_late > Z% or P99 lag > SLA → extend watermark or pause rollout.- Auto-triggers: when late-arrival or dedupe anomalies occur, rerun affected model metrics for last N windows and tag rollout as degraded; open ticket with diagnostics bundle (sample records, hashes, offsets).

**Why this works**Producer-assigned IDs + content hashes reduce false dedupe. Watermark + corrective topic bounds late-event impact. Compact monitoring (rates, ratios, percentiles) surfaces root causes quickly while automated corrective recomputations preserve evaluation validity during rollout.

MediumTechnical

39 practiced

You must choose a unit of randomization for a feature that is cached at CDN edge nodes, making per-request randomization infeasible. Propose practical randomization units (e.g., user-id, session cookie, edge node) and discuss bias/variance trade-offs, contamination risks, and required sample adjustments.

Sample Answer

**Brief answer / recommendation**I would choose among three pragmatic units: user-id (or account-id), session cookie (short-lived), or CDN edge node (cache shard). Each trades bias, variance, and contamination differently — choose based on TTL, cache key granularity, and spillover risk.

**Units & when to pick**- User-id (best when cache key includes user identity): minimizes contamination across users; low bias if users are independent. Higher variance reduction (good) because many independent units.- Session cookie (when per-user identity unavailable): reduces longitudinal contamination within a session but risks cross-session leakage; useful if cookies are sticky and TTL aligns with feature rollout.- Edge node / cache shard (when cache is keyed by edge or global object): simplest to implement (set assignment per shard), but high variance and strong risk of bias because assignments correlate with geography, client mixes, traffic patterns.

**Bias / variance trade-offs**- User-id: low bias, low intra-cluster correlation so lower variance inflation.- Session cookie: moderate bias if users have multiple sessions; variance depends on average sessions per user.- Edge node: high potential bias (systematic differences across nodes) and large variance inflation because effective sample size = number of nodes.

Estimate variance inflation via design effect:

text

design_effect = 1 + (m - 1) * rho

Where m = average cluster size, rho = intra-cluster correlation. Measure rho from historical metrics.

**Contamination risks**- Cross-request caching: assignments can leak when cached object serves multiple users (edge node risk).- User churn: same user moving across edges -> contamination if randomization tied to edge.- Cookie deletion or identity merging -> treatment switches; treat as non-compliance.

Mitigations: align cache key with randomization unit (e.g., include user-id or experiment bucket in cache key), use short TTLs, serve randomized variant via client-side logic when feasible, sticky cookies plus fallback.

**Sample adjustments & analysis**- Randomize at chosen unit and analyze at that unit (cluster-level inference).- Use cluster-robust SEs or hierarchical models (mixed effects with random intercept per cluster).- Adjust sample size by design_effect; compute required N_cluster = desired_N_independent * design_effect.- Post-stratify or include node-level covariates (region, traffic volume) to reduce bias when using edge nodes.- For hypothesis tests with few clusters (edges), use permutation/bootstrap at cluster level or wild cluster bootstrap.

Example: if 100 edge nodes, average 10k users/node, and rho=0.02 -> design_effect ≈ 1 + (10000-1)*0.02 ≈ 201 → effective sample size shrinks dramatically; prefer user-id randomization or increase number of randomized clusters.

**Conclusion**Prefer user-id when possible (lowest bias, manageable variance). If constrained to cache shard, randomize and analyze at shard level, measure ICC, inflate sample size, adjust for node covariates, and use cluster-robust inference.

HardTechnical

35 practiced

You run an experiment where treatment assignment is deterministic via hashing, but you later discover a hash collision bug that mis-assigned 2% of users. Describe how to quantify the impact of this misassignment on estimated treatment effects and propose analysis corrections or remedial experiments to recover unbiased estimates.

Sample Answer

**Situation & goal**You discovered a bug causing 2% of users to be assigned to the wrong arm by deterministic hashing. Goal: quantify how that misassignment biases the ATE and propose corrections or remedial plans to recover unbiased estimates.

**1) Quantify bias from misassignment**- Model observed assignment A_obs vs true assignment A_true. Let p = P(misassigned) = 0.02 and assume misassignment flips treatment/control symmetrically (estimate from logs).- If true ATE = τ, observed (naive) ATE_hat = (1 − 2p) τ when misassignments are random flips. So bias ≈ −2p τ. Example: p=0.02 ⇒ attenuation ≈ 4% of τ.- More generally, represent misclassification matrix: - P(A_obs=1 | A_true=1) = 1 − p1 - P(A_obs=1 | A_true=0) = p0 Use correction: τ = (ATE_obs − p0 μ1 + p1 μ0) / (1 − p0 − p1), solving for true effect where μ1, μ0 are outcome means conditional on observed A (requires estimating p0,p1).

**2) Estimation & uncertainty**- Estimate p0,p1 from hashed logs or a gold-standard sample (recompute hash for a random subset).- Use bootstrap or delta method to propagate uncertainty in p estimates into τ variance.- If misassignment non-random (correlated with covariates), fit logistic model of misassignment on covariates and adjust via inverse-prob weighting (IPW) or include misassignment model in G-computation.

**3) Robust alternatives**- Instrumental-variable (IV) approach: use intended deterministic assignment (hash before bug) as instrument for received treatment. ITT estimate remains unbiased; use two-stage least squares to recover complier average causal effect.- Per-protocol corrected estimate: restrict to users with verified correct assignment (gold sample) and reweight to population.

**4) Remedial experiments**- Re-run the experiment with corrected hashing on a fresh, random sample; compare with the original via meta-analysis.- If feasible, perform targeted re-randomization on the 2% exposed cohort (or a fresh replicate with same covariate distribution) to directly measure τ.- Backfill: compute corrected assignment for past users if possible; reanalyze using true assignment.

**5) Practical steps**- Audit logs to estimate p0,p1 and check correlation with covariates (SRM).- Report both ITT and corrected estimates, with sensitivity bounds (e.g., assume worst/best-case p in plausible range).- Document and fix hashing bug, rerun replication prioritized by business impact.

This combination (analytic correction using misclassification matrix, IV/ITT reporting, bootstrap uncertainty, and a replication experiment) recovers unbiased inference and quantifies residual uncertainty for stakeholders.

HardTechnical

35 practiced

You evaluate 10 correlated engagement metrics. Propose a statistical testing strategy that accounts for correlation between metrics to control false discovery while maintaining power. Discuss multivariate testing (e.g., MANOVA), hierarchical models, clustering metrics and applying FDR within clusters, and pros/cons of each.

MediumTechnical

28 practiced

After randomization, you find a statistically significant imbalance in a strong pre-treatment covariate (user tenure) between control and treatment. Outline steps to adjust your analysis, including when to use regression adjustment, post-stratification, or re-randomization. Discuss how adjustments affect Type I error and interpretability.

Sample Answer

**Quick framing**You discovered a significant imbalance in a strong pre-treatment covariate (user tenure). Because tenure is pre-treatment, adjustment is valid and often beneficial (reduces variance, improves power). Outline the pragmatic steps and the trade-offs.

**Step-by-step plan**1. Verify and document - Re-check randomization code/logs and sample exclusions. - Quantify imbalance (mean/median, standardized difference, plots). - Pre-specification: note whether adjustment was pre-specified; document now.

2. Primary analysis choices (use depending on context) - Regression adjustment (ANCOVA) - When: default choice for continuous outcome or when you want a model-based, efficient estimate. - How: fit outcome ~ treatment + f(tenure) (+ other covariates); use linear model or generalized linear model as appropriate. - Benefits: unbiased for randomized experiments, typically reduces variance and tightens CIs. - Post-stratification / stratified analysis - When: tenure has natural buckets (new/medium/long) or nonlinearity; good for interpretability and when model assumptions worry you. - How: estimate treatment effect within strata then weight by population or sample strata sizes. - Benefits: transparent, robust to model misspecification; can increase precision. - Re-randomization - When: only feasible prospectively (design stage) — not recommended after seeing outcomes; if you must, only re-randomize based on covariates before data collection and report procedure. - Note: post-hoc re-randomization invalidates p-values if not accounted for.

3. Inference & robustness - Use robust SEs or randomization-based permutation tests that condition on observed covariates. - If using ANCOVA, verify model fit (nonlinear terms, interactions); consider adding tenure×treatment interaction if treatment effect may vary by tenure. - Report both unadjusted and adjusted estimates and CIs; pre-specify adjusted estimate as primary if possible.

**Type I error and interpretability** - Properly applied ANCOVA or post-stratification preserves nominal Type I error in randomized experiments; adjustments typically reduce variance but do not induce bias. - Post-hoc re-randomization or selecting adjustments after seeing outcome can inflate Type I error unless you correct for selection. - Adjusted estimates are conditional (average treatment effect conditional on model/strata); show marginal/unadjusted results too for interpretability. Always transparently report methods and sensitivity analyses.

**Practical recommendation**Run ANCOVA with flexible tenure terms (splines or bins), present stratified estimates, run permutation tests for p-values, and document everything. If imbalance was anticipated, adopt rerandomization in future experiments.

Unlock Full Question Bank

Get access to hundreds of Experimental Design and Analysis Pitfalls interview questions and detailed answers.

Join thousands of developers preparing for their dream job.