A and B Test Design Questions

Designing and running A and B tests and split tests to evaluate product and feature changes. Candidates should be able to form clear null and alternative hypotheses, select appropriate primary metrics and guardrail metrics that reflect both product goals and user safety, choose randomization and assignment strategies, and calculate sample size and test duration using power analysis and minimum detectable effect reasoning. They should understand applied statistical analysis concepts including p values confidence intervals one tailed and two tailed tests sequential monitoring and stopping rules and corrections for multiple comparisons. Practical abilities include diagnosing inconclusive or noisy experiments detecting and mitigating common biases such as peeking selection bias novelty effects seasonality instrumentation errors and network interference and deciding when experiments are appropriate versus alternative evaluation methods. Senior candidates should reason about trade offs between speed and statistical rigor plan safe rollouts and ramping define rollback plans and communicate uncertainty and business implications to technical and non technical stakeholders. For developer facing products candidates should also consider constraints such as small populations cross team effects ethical concerns and special instrumentation needs.

EasyTechnical

62 practiced

You're asked to design an experiment to change the checkout flow with the business goal of increasing completed purchases (revenue). As a BI analyst identify and justify an appropriate primary metric, list at least four guardrail metrics (technical and business), and propose concrete threshold values or decision rules for those guardrails that would trigger pausing or rolling back the experiment.

Sample Answer

Primary metric (primary KPI)- Checkout conversion rate (completed purchases / checkout sessions). Justification: directly measures the experiment’s goal (completed purchases) while controlling for traffic to checkout; easy to compute in BI tools and can be tied to revenue via AOV.

Guardrail metrics (technical + business) and decision rules1) Payment failure rate (failed payment attempts / payment attempts) — Technical/business - Trigger: absolute increase ≥0.5 percentage points AND relative increase ≥20% vs. baseline, statistically significant (p < 0.05) sustained for 24 hours → pause and investigate; if persists 72 hours → rollback.

2) Page load time on checkout (median page load) — Technical - Trigger: increase >300 ms OR relative +25% vs. baseline, sustained for 24 hours → pause; if increase causes CRC (conversion) drop concurrently → rollback.

3) 500-level error rate on checkout APIs (errors / API calls) — Technical - Trigger: error rate >0.5% OR >3x baseline in any 1-hour window → immediate pause and automatic rollback until fixed.

4) Cart abandonment rate (abandoned carts / carts created) — Business - Trigger: absolute increase ≥3 percentage points AND statistically significant over rolling 48h → pause; if increase persists beyond 7 days → rollback.

5) Revenue per visitor (RPV) or Average Order Value (AOV) — Business (monitor alongside primary) - Trigger: RPV or AOV decrease ≥5% and statistically significant (p < 0.05) over full experiment sample → pause and investigate impact on monetization.

6) Refund/chargeback rate and support tickets about checkout — Business/ops - Trigger: refund or chargeback rate increase ≥1 percentage point OR +50% relative; support tickets about checkout up >30% week-over-week → pause and investigate; rollback if fraud/UX regressions confirmed.

Operational rules- Use statistical significance (p < 0.05) and pre-specified minimum sample size per variant before making final decisions.- For “pause” conditions, stop new user assignments but keep logging to diagnose; for “rollback”, revert immediately to control if thresholds indicating user harm or revenue damage are met.- Report automated alerts to engineering, product, and analytics with dashboard links and leading indicators for fast remediation.

HardTechnical

47 practiced

Compare Bayesian sequential testing and frequentist alpha-spending approaches for A/B tests. Discuss interpretability, the role and choice of priors, pre-specification requirements, stopping rules, computational cost, and how each approach maps to business decision thresholds. Recommend an approach for a high-velocity growth team that runs many small experiments daily, and justify your choice.

Sample Answer

Situation: You're deciding between Bayesian sequential testing and frequentist alpha‑spending for many daily A/B tests. Below is a role‑appropriate comparison and recommendation focused on interpretability, priors, pre‑specification, stopping rules, compute, and business thresholds.

Comparison:- Interpretability - Bayesian: Provides direct probability statements (e.g., “P(treatment > control) = 92%”), which map naturally to business language and dashboards. - Frequentist alpha‑spending: Controls Type I error over multiple looks, but p-values and confidence intervals are less intuitive for non‑statisticians.- Role and choice of priors - Bayesian: Priors matter. Weakly informative or historical hierarchical priors stabilize estimates for small daily experiments; prior choice should be documented and tested via sensitivity analysis. - Frequentist: No explicit priors; historical information can’t be formally incorporated without complex adjustments.- Pre‑specification requirements - Bayesian: More flexible — optional fixed horizon; you can stop anytime with valid posterior updates if your decision rule is pre‑defined. - Frequentist: Requires pre-specification of alpha spending function (e.g., O’Brien‑Fleming) and interim look schedule to guarantee Type I control.- Stopping rules - Bayesian: Flexible stopping based on posterior probability or expected loss; stopping does not invalidate inference if decision thresholds are pre-specified. - Frequentist: Stopping based on interim looks must follow alpha spending; ad‑hoc stopping inflates Type I error.- Computational cost - Bayesian: Potentially higher if using MCMC, but most A/B tests (binomial/normal) can use conjugate updates or fast approximate posteriors (Laplace, variational), so overhead is small. - Frequentist: Very cheap computationally.- Mapping to business thresholds - Bayesian: Easily translate to business rules (e.g., ship if P(lift > 1% ) > 95% or expected revenue gain > threshold). - Frequentist: Map decisions from p-value or adjusted CI, but decision‑oriented metrics (expected regret) are less direct.

Recommendation for a high‑velocity growth team:- Use a Bayesian sequential framework with weakly informative or hierarchical priors and standardized decision rules (e.g., stop and ship if P(lift > δ) > 0.95; stop and reject if P(lift < 0) > 0.95). Justification: - Interpretability: Business stakeholders understand probabilities and expected impact. - Flexibility: Can stop anytime and run many small tests without complex alpha allocation. - Efficiency: Priors borrow strength across experiments, improving power for small daily tests. - Operational fit: Integrates into dashboards (posterior probability, expected lift, credible intervals) and automatable pipelines; use fast conjugate updates to keep compute minimal.Implementation notes:- Pre‑register priors and decision thresholds; perform offline simulations (A/B simulations) to calibrate false‑positive/negative rates and expected regret.- Monitor prior drift and periodically re‑fit hierarchical priors from accumulated experiment data.- Provide clear dashboards showing posterior probabilities, expected revenue impact, and sample sizes to aid business decisions.

This approach balances statistical rigor, operational speed, and business clarity for a high‑velocity BI environment.

MediumTechnical

56 practiced

Write Python pseudocode or a short script that computes a 95% bootstrap confidence interval for median revenue per user given an array of per-user revenues. Use 10,000 bootstrap samples. Explain how you would handle users with zero revenue and heavy-tailed distributions in your implementation.

Sample Answer

Approach: use non-parametric bootstrap — resample users with replacement 10,000 times, compute median each sample, and report the 2.5 and 97.5 percentiles as a 95% CI. Handle zero-revenue users by keeping them in the sample (they are valid observations). For heavy-tailed revenue, use robust summary (median) and consider log-transform (with shift) as an alternative to stabilize variance; when logging, transform back with bias correction if needed or report CI on log-scale.

python

import numpy as np

def bootstrap_median_ci(revenues, n_boot=10000, ci=0.95, seed=42, log_transform=False, eps=1e-6):
    rng = np.random.default_rng(seed)
    x = np.asarray(revenues)
    # Keep zeros: they are real users. Optionally drop NaNs.
    x = x[~np.isnan(x)]
    if x.size == 0:
        raise ValueError("No data")
    if log_transform:
        # shift to allow zero values
        x = np.log(x + eps)
    medians = np.empty(n_boot)
    n = x.size
    for i in range(n_boot):
        sample = rng.choice(x, size=n, replace=True)
        medians[i] = np.median(sample)
    lower, upper = np.percentile(medians, [(1-ci)/2*100, (1+(ci))/2*100])
    if log_transform:
        # exponentiate back
        lower, upper = np.exp(lower) - eps, np.exp(upper) - eps
    return np.median(revenues), (lower, upper)

# Example:
# median, (low, high) = bootstrap_median_ci(revenue_array)

Key points:- Zeros retained since they reflect real users; removing them biases per-user metrics.- For heavy tails, median is robust; log-transform reduces skew and may tighten CI but changes interpretation.- 10k samples is typical; increase for more precision. Complexity: O(n * n_boot) time, O(n_boot) memory for medians. Edge cases: all zeros, very small sample sizes — report warnings.

MediumTechnical

79 practiced

You are running an experiment on a social feed where treatment can affect friends' engagement (network interference). Describe at least three randomization strategies to mitigate interference (e.g., cluster randomization, ego-network randomization, graph-clustering). For each strategy, explain pros/cons and implications for power and sample-size calculation.

Sample Answer

When interference is possible, randomizing at the user level can bias estimates. Here are three practical strategies, with pros/cons and sample-size implications, framed for BI work (metrics, dashboards, and reporting).

1) Cluster randomization (e.g., randomize connected components, communities)- Description: Partition the graph into clusters (communities) and randomize treatment at cluster level so entire clusters are all treated or all control.- Pros: Eliminates within-cluster spillover, simple analysis (compare cluster-level averages), maps well to rollout.- Cons: Loss of effective sample size (clusters << users); clusters may be heterogeneous; residual spillover between clusters if clustering imperfect.- Power/sample-size: Use cluster-level variance (ICC). Effective sample size ≈ number of clusters. Sample-size calc needs cluster count, average cluster size, and intra-cluster correlation (design effect = 1 + (m-1)*ICC).

2) Ego-network / egocentric randomization (treatment assigned to focal users and exclude their friends)- Description: Select disjoint egos and randomize treatment on egos; measure direct and spillover by comparing friends-of-treated vs friends-of-control in separate analyses.- Pros: Controls local interference, allows estimation of direct and immediate spillover; keeps many untreated units.- Cons: Requires excluding overlapping egos (limits coverage); selection can reduce generalizability.- Power/sample-size: Power driven by number of independent egos. Need to account for dependency among friends; effective sample size ≈ number of disjoint ego groups.

3) Graph-clustered/blocking with buffer zones (cluster + holdout neighbors)- Description: Create clusters but leave a buffer ring of users between clusters assigned to neither treatment nor measurement (or assigned to baseline) to reduce cross-cluster contamination.- Pros: Reduces spillover across cluster boundaries; more accurate causal estimate.- Cons: Sacrifices sample because buffer users removed from measurement; complex implementation and rollout.- Power/sample-size: Further reduces effective sample; compute based on number of measured clusters (excluding buffers). Must adjust for reduced measurement proportion.

Practical BI implications:- Estimate ICCs or local spillover magnitude from historical data to inform design and sample-size calculators.- Precompute cluster assignments and track cluster-level metrics in dashboards (cluster-level conversion, reach).- Run simulation-based power analysis on your network to quantify trade-offs, then present expected detectable effect and required users/clusters in executive reports.

HardTechnical

59 practiced

You detect a sample ratio mismatch in a live experiment and suspect instrumentation error. Provide a forensic checklist with concrete SQL queries and log checks: assignment-service logs, hashing-salt changes, duplicate events, time-zone issues, platform rollout discrepancies. Explain remediation strategies: reprocessing events, excluding corrupted data, or re-running experiments.

Sample Answer

Situation: You observe a Sample Ratio Mismatch (SRM) in a live A/B test and suspect instrumentation error. Below is a forensic checklist (ordered), concrete SQL checks and log checks, and remediation options.

Forensic checklist & concrete checks1) Verify raw assignment counts (by day, platform, region)

sql

-- assignments table: user_id, experiment_id, variant, assigned_at, platform, country
SELECT variant, platform, COUNT(DISTINCT user_id) AS users, DATE(assigned_at) AS day
FROM assignments
WHERE experiment_id = 'exp_123'
GROUP BY 1,2,3
ORDER BY day, platform, variant;

2) Compare assignment vs exposure/event payloads

sql

-- exposures/events table: user_id, experiment_id, event_type, event_time
SELECT a.variant, COUNT(DISTINCT e.user_id) AS exposures
FROM assignments a
LEFT JOIN exposures e
  ON a.user_id = e.user_id AND a.experiment_id = e.experiment_id
WHERE a.experiment_id='exp_123'
GROUP BY a.variant;

3) Hashing / bucketing salt changes (detect sudden reassignments)

sql

-- check last assignment per user to see flips
SELECT user_id, MIN(assigned_at) AS first_assigned, MAX(assigned_at) AS last_assigned,
       COUNT(DISTINCT variant) AS distinct_variants
FROM assignments
WHERE experiment_id='exp_123'
GROUP BY user_id
HAVING COUNT(DISTINCT variant) > 1
LIMIT 100;

4) Duplicate events / double-counting

sql

-- same event emitted multiple times
SELECT user_id, event_type, DATE(event_time) AS day, COUNT(*) AS cnt
FROM exposures
WHERE experiment_id='exp_123'
GROUP BY 1,2,3
HAVING cnt > 1
ORDER BY cnt DESC
LIMIT 50;

5) Timezone and timestamp skew

sql

-- check distribution of assigned_at hours
SELECT DATE(assigned_at) AS day, EXTRACT(HOUR FROM assigned_at AT TIME ZONE 'UTC') AS hour_utc, COUNT(*) 
FROM assignments
WHERE experiment_id='exp_123'
GROUP BY 1,2
ORDER BY day, hour_utc;

6) Platform rollout / SDK versions

sql

-- platform and SDK version distribution
SELECT platform, sdk_version, COUNT(DISTINCT user_id) AS users
FROM assignments
WHERE experiment_id='exp_123'
GROUP BY platform, sdk_version;

7) Assignment-service logs (examples to search)- Search for salt/seed changes: "hash_salt updated" or "experiment exp_123 seed"- Search for deployment timestamps around SRM start: "deploy", "release", "assignment-service started"- Search for error patterns: "failed to compute bucket", "null user_id", "exception"- Trace a sample user id to see assignment decisions: - grep logs for user_id=abc123 → confirm consistent variant and salt used

8) Cross-source reconciliation- Compare streaming ingestion counts (Kafka topic) vs DB loads- Check lag/consumer errors: look for offsets, consumer group rebalances.

Remediation strategies- If instrumentation bug identified and reversible: - Reprocess events from raw source (Kafka/S3) with corrected mapping; re-run analysis on reprocessed dataset. Communicate expected timeline and reproducibility.- If portion of data corrupted and reprocessing impossible: - Exclude affected time windows/platforms/SDK versions from analysis (document criteria). Recompute metrics and rerun SRM test.- If assignment salt changed mid-test and caused reassignment: - Prefer re-run experiment (new randomization) unless reprocessing can deterministically map old→new buckets.- Short-term mitigation: - Pause experiment or stop decisioning on affected cohorts to prevent bad exposures. - Notify stakeholders with clear impact assessment: affected segments, time window, expected bias direction.- Postmortem: root cause, checklist updates (monitoring + alerting for SRM, logging of seed changes, automated assignment vs exposures parity checks).

Key communication: provide reproducible SQL queries, list of affected user_ids and time windows, remediation plan and ETA, and follow-up verification queries to confirm fix.

Unlock Full Question Bank

Get access to hundreds of A and B Test Design interview questions and detailed answers.

Join thousands of developers preparing for their dream job.