A/B Test Design & Statistical Rigor Questions

Designing and statistically defending a controlled online experiment: framing a testable hypothesis, defining control and treatment variants, choosing the randomization unit, setting the primary success metric, and computing sample size, power, and minimum detectable effect. Covers the statistical foundations that make a readout trustworthy, including hypothesis testing, p-values, confidence intervals, statistical vs practical significance, and Type I/II error. Emphasizes avoiding the common pitfalls that invalidate a test, such as peeking, multiple-comparison inflation, underpowered designs, and how test duration and stopping rules affect the validity of conclusions.

MediumTechnical

42 practiced

Design an ethical review checklist for experiments that might materially impact privacy, pricing, or fairness. Include required approvals, data minimization principles, logging/audit requirements, and how to evaluate potential disparate impacts across protected groups.

Sample Answer

Situation: As a data analyst running experiments or analyses that can affect pricing, user privacy, or fairness, you need a practical, auditable checklist to ensure responsible decisions.

Ethical Review Checklist (for any analysis/experiment with material privacy/pricing/fairness impact)

Required Approvals (must be obtained before launch)- Business sponsor sign-off (scope, KPIs, rollback plan)- Data owner approval (source, permitted uses)- Privacy officer / DPO approval (consent, legal bases)- Legal/compliance review (regulatory risk)- Ethics or review board (for high-impact cases)- Engineering/Platform (implementation & logging capability)

Data Minimization & Handling- Purpose-limiting statement: list exact questions the dataset will answer- Minimize fields: include only attributes required for analysis/modeling- Use pseudonymization/anonymization when possible; store identifiers separately- Apply aggregation or differential privacy for published outputs- Retention policy: define TTL and safe deletion procedures- Use sampled subsets or synthetic data for development/testing where feasible

Logging & Audit Requirements- Maintain experiment registry: ID, owner, purpose, approvals, start/end dates- Data lineage logs: source, ingestion time, transformations, joins- Access logs: who accessed data, when, purpose (automated)- Model/analysis versioning: code, parameters, queries, and dataset snapshot hashes- Decision audit: record recommendations, approvers, and deployment actions- Retention of logs per compliance policy and easy retrieval for audits

Evaluating Disparate Impact Across Protected Groups- Define protected attributes relevant to context (race, gender, age, location). If not collected, document reasons and legal/ethical considerations.- Use group-aware metrics: compare primary KPI and harm metrics by group (conversion, price charged, error rates, false positives/negatives).- Statistical tests: - Disparate impact ratio (selection rate ratio) with thresholds (e.g., 0.8 rule) and confidence intervals - Absolute/relative differences and standardized effect sizes - Significance testing + power analysis to ensure sample sizes suffice - Use bootstrap or permutation tests for nonparametric confidence- Intersectional checks: evaluate combinations of attributes (e.g., older women in a region) where sample size allows- Fairness thresholds & tolerances: predefine acceptable deltas and escalation paths if exceeded

Mitigation & Decision Gates- If impact exceeds thresholds: require mitigation plan (feature removal, repricing, calibrated thresholds, post-processing corrections)- Run A/B tests with fairness-constrained objectives or reweighting techniques- Staging: internal-only rollout → monitored pilot → full rollout- Rollback criteria and automated safeguards (e.g., pause if group metric drops by X%)

Monitoring & Post-Deployment- Continuous monitoring dashboard per group (hourly/daily depending on impact)- Alerting for KPI or fairness breaches- Periodic re-review cadence (30/90/365 days depending on risk)- Maintain a public/internal summary of audit outcomes and lessons learned

Documentation & Transparency- Store: review artifacts, rationale, tests run, mitigation decisions- Communicate to stakeholders: data use, expected benefits, known risks, and user recourse- When appropriate, publish high-level fairness/privacy statements for affected users

Example (short): Before changing dynamic pricing, obtain legal + privacy approvals, restrict dataset to price-relevant features, run group-level revenue/price-distribution comparisons, apply disparate-impact ratio test (threshold 0.8), and require a pilot with monitoring and an automatic rollback if any protected group’s effective price increases beyond the allowed delta.

This checklist creates clear approvals, minimizes data exposure, provides auditable logs, and sets measurable routines to detect and mitigate disparate impact—scalable for analyses you routinely run as a data analyst.

MediumTechnical

39 practiced

Describe a robust event instrumentation scheme to support A/B testing: what core events and attributes you would require (exposure event, assignment id, variant name, user_id, timestamps, client/server), how to ensure idempotence and deduplication across devices, and how to design the schema to support later joins and metric computation.

Sample Answer

Core events & minimal attributes (SQL-friendly rows)- exposure (user saw experiment): exposure_id (UUID deterministic), experiment_id, assignment_id, variant_name, user_id (nullable device_id for guests), session_id, timestamp_utc, client_type (web/mobile/server), sdk_version, context (json/meta)- conversion/event (e.g., purchase, click): event_id (UUID), event_name, user_id, timestamp_utc, value/amount, experiment_id (nullable), assignment_id (nullable), variant_name (nullable), client_type, context- assignment_change: records forced reassignments, start_ts, end_ts, reason- heartbeat/ping: optional for dedupe/ordering

Why these fields- assignment_id + experiment_id + variant_name tie exposure to behavior.- exposure_id (deterministic by user_id+experiment_id+day) enables idempotent ingestion and dedup.- timestamps normalized to UTC and client/server origin allow correct attribution windows.

Idempotence & dedup across devices- Deterministic IDs: generate exposure_id = hash(experiment_id, user_id OR device_id, salt, date) so multiple devices produce same id and ingestion is idempotent.- Server-side canonicalization: when server receives client events include assignment_id/ exposure_id; server validates and enriches missing fields.- Ingestion dedup rules: dedupe by event_id for conversions; dedupe exposures by exposure_id keeping earliest timestamp per user+experiment.- De-dup across devices: prefer persistent user_id (auth) as canonical; on login, backfill device events to user by mapping device_id->user_id and merge exposures using exposure_id.- Windowing: apply attribution windows (e.g., 7 days) when joining exposures→conversions.

Schema design for joins & metrics- Raw_events table (append-only) with columns above; partition by date, cluster by experiment_id, user_id for fast range scans.- Materialized/derived tables: - assignments_by_user(experiment_id, user_id, assignment_id, first_exposure_ts, last_exposure_ts) - conversions_by_assignment(experiment_id, assignment_id, event_name, count, sum_value, window)- Precompute flagging columns: is_exposed boolean, exposure_ts to simplify SQL.- Use surrogate keys and indexes on (experiment_id, assignment_id, user_id, timestamp) for easy joins.- Store context as structured columns when frequently queried (country, platform) and JSON for low-use fields.

Metric computation notes- Use first-exposure-per-user for exposed cohort; define intent-to-treat vs per-protocol cohorts.- Compute uplift with cohort joins: exposures → conversions within attribution window, dedup by conversion event_id.- Always surface both user-level and device-level counts; document which users are deduplicated and how.- Keep raw events immutable to enable re-processing with corrected logic.

Operational & privacy- Log SDK version, sampling flag, and experiment bucketing algorithm version to allow reprocessing.- Hash pii (user_id) as necessary, store mapping tables separately and access-controlled.

MediumTechnical

37 practiced

Design an experiment to test a pricing change: state hypotheses, choose a primary metric (or metrics), list guardrails, propose an allocation and sample-size considerations, and discuss ethical or customer-experience considerations and regulatory constraints you might need to consider.

Sample Answer

Hypotheses- H0 (null): The pricing change does not change net revenue per user (NRPU) or churn.- H1 (alt): The pricing change increases NRPU without increasing churn beyond an acceptable threshold.

Primary metric(s)- Primary: Net Revenue Per User (NRPU) = average revenue per active user over 30 days (captures price × quantity and churn).- Secondary / guardrail metrics: 30-day churn rate, conversion rate (for trials/free→paid), Customer Support Volume per 1k users, NPS/CSAT.

Experiment design & allocation- Randomized controlled experiment with users randomly assigned to Control (current price) and Treatment (new price). Use stratified randomization by key covariates (region, user tenure, plan type) to ensure balance.- Allocation: start with 50/50 for power efficiency; if riskier, use smaller initial treatment (e.g., 90/10 ramp) with planned increases after safety checks.

Sample-size and duration considerations- Compute sample size via power analysis: assume baseline NRPU μ0 and standard deviation σ (estimate from historical data). For desired minimum detectable relative lift δ (e.g., 3–5%), power 80%, α=0.05: n_per_group = 2 * (Z_{1-α/2}+Z_{power})^2 * σ^2 / (δ·μ0)^2.- If metric is binary (churn), use standard proportion sample-size formulas.- Account for expected attrition, multiple segments, and seasonality; run at least one full business cycle (e.g., 4–8 weeks) plus pre-check period.

Guardrails & monitoring- Pre-specified stop rules: if churn increases > X% absolute (e.g., 1pp) or support volume spikes > Y%, pause experiment.- Real-time dashboards for primary and guardrail metrics, with automated anomaly detection and daily checks.- Statistical plan: pre-register primary metric, hypothesis, one-sided vs two-sided test, and multiple-testing correction for secondary metrics.

Analysis approach- Use intent-to-treat (ITT) for unbiased effect on NRPU and per-protocol checks.- Adjust for covariates with regression (ANCOVA) to improve power.- Run subgroup analyses (plan type, region) but treat as exploratory.

Ethical, customer-experience, and regulatory considerations- Transparency: ensure pricing change complies with displayed terms; avoid deceptive practices.- Anti-discrimination and price-gouging laws: verify differential pricing by protected attributes is not used; consult legal for jurisdictional constraints.- Consent/data privacy: all analytics must comply with GDPR/CCPA when using personal data; avoid linking pricing to sensitive attributes.- Billing accuracy and rollback: ensure systems can revert charges and honor refunds promptly.- Customer communication: plan messaging for rollout to avoid surprise; provide clear FAQs and support paths.

Practical rollout- Pilot in low-risk segments or internal beta; ramp using predefined gates.- Post-experiment: conduct revenue-lifecycle modeling to estimate long-term effects (LTV), not just short-term NRPU.

HardTechnical

45 practiced

Discuss advantages and disadvantages of adopting a Bayesian framework for A/B testing in a fast-paced growth environment. Include how you would specify priors for conversion rates, interpret posterior probabilities (e.g., probability treatment > control), handle multiple looks, and present Bayesian results to non-technical stakeholders.

Sample Answer

Advantages: Bayesian A/B testing gives direct probabilistic statements stakeholders prefer (e.g., “there’s a 92% probability treatment beats control”), naturally handles small samples via priors, and supports truly sequential monitoring without complex alpha-spending rules if decisions are based on posterior + loss. It also easily extends to hierarchical models (multi-armed tests, segments).

Disadvantages: Results depend on priors (must justify them), pure Bayesian stopping rules may not satisfy teams expecting frequentist Type I control, and communicating subjective priors and credible intervals can confuse non-technical audiences.

Specifying priors for conversion rates:- Use Beta-Binomial conjugacy. For an uninformative prior use Beta(1,1). - If historical baseline exists (e.g., baseline p0 = 0.05 and you want prior weight equivalent to 200 observations), set alpha = p0*N = 10, beta = (1-p0)*N = 190 → Beta(10,190). - Prefer weakly informative priors; run sensitivity checks with alternative priors (flat, skeptical, enthusiastic) and report robustness.

Interpreting posteriors:- Compute posterior for p_control ~ Beta(a_c + successes_c, b_c + failures_c) and p_treatment similarly.- Report P(p_treatment > p_control) = ∫∫ I(p_t>p_c) f(p_t) f(p_c) dp_t dp_c (Monte Carlo estimate). - Report posterior mean uplift, 95% credible interval for uplift, and probability uplift > MDE (minimum detectable/economic effect). Frame as business risk: e.g., “90% chance of >0%, 65% chance of >0.5% (our MDE).”

Handling multiple looks:- Bayesian updating is coherent under optional stopping for the posterior, but if stakeholders care about frequentist error rates, pre-specify a decision rule or loss function (e.g., stop when P(treatment>control) > 0.95 and expected loss of wrong decision < threshold). Alternatively calibrate stopping rule via simulation to estimate frequentist Type I/II properties and share those results.

Presenting to non-technical stakeholders:- Lead with clear business statements: probability treatment outperforms, expected absolute revenue impact, recommended action. - Use intuitive visuals: overlay posterior density curves, histogram of simulated uplift, and a simple gauge showing P(treatment>control) and P(uplift> MDE). - Include short sensitivity summary (what changes if prior differs) and recommended next steps (rollout, more data, segment follow-up). - Always pair probability statements with practical consequences (e.g., expected incremental conversions and revenue) so decisions map to business impact.

EasyTechnical

42 practiced

Implement a Python function that computes a two-proportion z-test p-value and absolute lift. Function signature: compute_z_test(convert_a, n_a, convert_b, n_b, alternative='two-sided'). Return z-statistic, p-value, and percent-lift (b over a). Assume inputs are integers and use standard normal approximation.

Sample Answer

Approach: compute pooled standard error under null (pooled proportion), compute z = (p_b - p_a) / se, get p-value from standard normal depending on alternative, and percent lift = (p_b - p_a) / p_a * 100 (handle p_a = 0).

python

import math
from math import sqrt
from scipy.stats import norm

def compute_z_test(convert_a, n_a, convert_b, n_b, alternative='two-sided'):
    """
    Returns (z_stat, p_value, percent_lift)
    Inputs:
      convert_a, n_a, convert_b, n_b: integers
      alternative: 'two-sided', 'larger' (b>a), or 'smaller' (b<a)
    """
    # sample proportions
    p_a = convert_a / n_a
    p_b = convert_b / n_b

    # pooled proportion under H0: p_a == p_b
    pooled = (convert_a + convert_b) / (n_a + n_b)
    se = sqrt(pooled * (1 - pooled) * (1/n_a + 1/n_b))

    # avoid division by zero
    if se == 0:
        z = 0.0
    else:
        z = (p_b - p_a) / se

    # p-value based on alternative
    if alternative == 'two-sided':
        p_value = 2 * (1 - norm.cdf(abs(z)))
    elif alternative == 'larger':
        p_value = 1 - norm.cdf(z)
    elif alternative == 'smaller':
        p_value = norm.cdf(z)
    else:
        raise ValueError("alternative must be 'two-sided', 'larger', or 'smaller'")

    # percent lift (absolute lift of B over A)
    if p_a == 0:
        percent_lift = float('inf') if p_b > 0 else 0.0
    else:
        percent_lift = (p_b - p_a) / p_a * 100.0

    return z, p_value, percent_lift

Key points:- Uses pooled SE appropriate for testing equality of proportions.- Percent lift is relative change; handle p_a=0 to avoid division error.- Complexity O(1); numeric stability depends on large-sample normal approximation.- Edge cases: very small n (normal approx invalid), zero variance (all successes or failures). Alternative: use Fisher's exact test for small counts or bootstrap for lift CI.

Unlock Full Question Bank

Get access to hundreds of A/B Test Design & Statistical Rigor interview questions and detailed answers.

Join thousands of developers preparing for their dream job.