A/B Test Design & Statistical Rigor Questions

Direct answer

SUTVA, the Stable Unit Treatment Value Assumption, is the assumption underlying a standard A/B test that one unit's observed outcome depends only on which arm that unit itself was assigned to, and not on which arms other units received. It has two components: no interference between units (your outcome is not affected by someone else's assignment) and no hidden variation of treatment (everyone labeled "treatment" received the same treatment). When either fails, the simple difference-in-means between arms is no longer an unbiased estimate of the causal effect you think you are measuring.

Structured elaboration

The two components

1. No interference between units. Unit $i$'s potential outcome under any assignment vector depends only on $i$'s own treatment, not on the treatments assigned to units $j \neq i$. This is the component that breaks in networked or shared-resource products.
2. No hidden variation of treatment (consistency). There is exactly one version of "treatment" and one version of "control"; a unit's potential outcome is well defined given only its treatment label. This breaks when the same nominal arm is implemented differently for different users (different rollout timing, different creative, a bug that only affects some treatment users).

Two real violations and why each biases interpretation

Social feed, no-interference violation. A treated user gets a new sharing feature and posts more; their connections, who are in control, now see more of that content in their own feed even though they were never assigned to treatment. The control group's outcome (engagement) is contaminated by treatment spillover, which pulls control's measured engagement up and understates the true treatment effect: you are comparing "treatment" against "control that is partially treated," not against a clean counterfactual.

Shared inventory or capacity constraint. A promotion arm drives more purchases, which draws down a shared inventory pool or a fixed daily capacity (delivery slots, support-queue capacity) that both arms draw from. Control users now see stockouts or longer wait times caused by treatment's demand, not by anything intrinsic to being in control. This inflates the apparent treatment effect (control looks artificially worse) and, separately, means the effect you measured at the tested traffic share will not hold at 100% rollout, because the resource contention itself scales with the treatment allocation percentage.

In both cases the estimate is not merely noisy, it is biased in a specific, name-able direction, and the bias would not shrink with more sample size because it comes from the assignment mechanism interacting with the product, not from sampling error.

Designing around a suspected violation

Once you suspect interference, the standard fix is to change the unit of randomization to one large enough to contain the spillover, i.e., cluster randomization: randomize by friend-group, geographic market, or server shard instead of by individual, so that most of the interference happens within a cluster (which is internally consistent, either all-treated or all-control) rather than across the treatment/control boundary.

Cluster randomization has its own cost, though: fewer independent units means higher variance for the same total traffic, since the effective sample size is closer to the number of clusters than the number of users. It also requires the outcome to be measurable and meaningful at the cluster level, and it does not eliminate the shared-capacity case unless the constrained resource is itself scoped per cluster.

Detecting a violation you did not design around

If the violation is discovered mid-experiment rather than anticipated, e.g., a caching-configuration bug causes some control users to intermittently render treatment-arm content, the diagnostic sequence is: quantify the leak rate first (what fraction of control exposures actually rendered the treatment experience, pulled directly from logs, not estimated), then decide whether the leak is small enough to bound the bias and proceed with a documented caveat, or large enough that the read is unusable and the fix is to patch the bug and rerun rather than to try to model the contamination away after the fact.

Worked example

A marketplace runs a two-armed test on a checkout redesign, expecting independent per-user outcomes. Mid-experiment, a shared caching layer bug is found: a same-session user occasionally gets served a stale cached page from the opposite arm. Pulling exposure logs, engineering finds this affected roughly 4% of control-arm page views (a number read directly from the cache-hit logs, not assumed). That is a version-of-treatment violation, not an interference violation: some "control" users received a materially different experience than the rest of control, so the control arm is not internally consistent. The team's next step is not to reweight or model this away, because the mechanism (a caching bug) has no principled correction; it is to fix the bug and rerun the experiment cleanly, treating the contaminated run as informative only about the presence of the bug.

Trade-offs and pitfalls

• Do not assume interference is symmetric or negligible just because the product does not look "social." Shared backend resources (queues, inventory, ranking models retrained on pooled data) create interference in products with no visible network feature.
• Cluster randomization trades bias for variance; do not adopt it reflexively for every experiment on a networked product when the actual interference is small relative to the direct effect, since you would be paying a real power cost for a small bias fix.
• A violation discovered after the fact is a data-quality incident, not a modeling problem to be adjusted away; resist the temptation to "correct" biased data with a post hoc statistical patch when the honest fix is to rerun cleanly.

Direct answer

An A/A test randomly splits traffic into two groups that receive the identical experience and compares their metrics as if they were a real experiment. You run one to validate the assignment and measurement pipeline before trusting a real A/B result on the same platform: since both groups get the same product, any statistically significant difference between them signals a problem in the pipeline (randomization, instrumentation, or analysis) rather than a real effect, because by construction there is no effect to detect.

Structured elaboration

Two valid use cases

• Validating the assignment and instrumentation pipeline. Confirms that the bucketing hash actually produces the intended split ratio, that each unit sees a stable, single experience, and that event logging correctly attributes actions to the assigned arm end to end (client instrumentation through to the analysis table).
• Establishing a baseline-variance estimate. Because there is no true effect, the spread of the A/A metric difference across many runs (or across a well-chosen resampling of the same data) tells you what "just noise" looks like for this metric on this population, which is useful input for planning: it is a sanity check on your variance assumptions, not a substitute for a proper power calculation.

What to check while it runs

• The realized split ratio against the intended one (a sample-ratio check): meaningfully off the intended ratio (say, 50/50 skewing to 49/51 in a way that recurs, not a single noisy day) points at a bucketing bug before you even look at outcome metrics.
• Core funnel and event counts by arm (sessions, page views, primary conversion event) to confirm the two arms are tracked with equal fidelity, not just equal traffic.
• Whether the metric of interest for the upcoming real experiment behaves as expected in the A/A read, since that is the metric whose baseline variance you actually need.
• Run it for at least one full natural cycle of the traffic (typically a full week, to span weekday/weekend mix) rather than a single day, since a one-day A/A window can look clean by luck or flagged by a day-specific anomaly that has nothing to do with the platform.

An A/A test is, at its core, a targeted way to surface three distinct failure classes: instrumentation errors (events not logged or misattributed), non-random assignment (the bucketing hash is not producing a genuinely random, independent split), and sampling biases (the two arms end up systematically different in composition despite a technically-random split, e.g., a bot-filtering rule that behaves differently by arm). Each class points at a different fix, which is why segmenting the flagged difference (below) matters more than the raw significance flag itself.

Two limitations or common misinterpretations

• A clean A/A result is not proof the pipeline is bug-free. With enough traffic, small true differences in a specific test can still slip through if the bug is intermittent (e.g., only affects a rare browser) or if the metric checked in the A/A test is not the one that will matter in the real experiment. Absence of a flagged difference is reassurance, not a guarantee.
• A single significant A/A result does not, by itself, mean the pipeline is broken. At a conventional significance threshold, some fraction of A/A tests will show a "significant" difference purely by chance even with a perfectly correct pipeline; treat one flagged metric as a prompt to investigate, not as an automatic verdict, especially if you are checking many metrics at once and did not correct for that.

Root-causing a significant A/A result

1. Recheck the sample ratio first. A skewed split is the fastest, most common finding and points straight at a bucketing bug rather than a downstream measurement issue.
2. Segment the difference. Break the flagged metric down by platform, geography, and new-vs-returning user; a difference concentrated in one segment (e.g., one app version) points at an instrumentation bug specific to that segment rather than a global randomization failure.
3. Check for a known confound in how the two arms are served, such as one arm being disproportionately served through a code path with different latency or caching behavior, which is functionally a version-of-treatment bug even though no real treatment was intended.
4. Re-run before escalating, if the first read used a short window: a single noisy day is a weaker signal than a difference that persists across multiple independent A/A windows.
5. If it persists and is not explained by a segment or a known bug, treat the underlying real-experiment platform as unvalidated until the discrepancy is resolved; shipping A/B decisions on top of an unexplained A/A anomaly defeats the purpose of running the check at all.

Worked example

A team runs an A/A test ahead of a planned homepage experiment and flags a significant difference in click-through rate. Step 1, sample ratio: 50.1% vs 49.9%, within normal noise, so not a bucketing problem. Step 2, segmentation: the CTR gap is near zero on Android and web but noticeably present on iOS. Step 3: engineering finds one arm's iOS client is on an older app version with a slightly different default tab order, an artifact of how the A/A test's client-side flag was staged rather than anything about the experiment platform itself. The root cause is a version-of-treatment bug traced to a real, checkable fact (the iOS staging config), not a p-value alone; the fix is correcting the staged rollout, not adjusting the metric.

Trade-offs and pitfalls

• Running A/A tests constantly, on every metric, invites exactly the false-alarm problem described above; use them at meaningful checkpoints (new platform, new metric pipeline, post-incident) rather than as a standing tax on every experiment.
• Do not use a single A/A run's variance estimate as your only power-planning input if you have a more direct historical baseline available; treat it as a cross-check.
• A quiet A/A test on a low-traffic metric provides much weaker reassurance than the same result on a high-traffic metric, because a real problem of a given size is harder to detect with less data; do not treat "clean" as equally strong evidence across metrics of very different volume.

Direct answer

Convert a sample-size target into a run duration in two steps: compute the required sample size per variant with the standard two-proportion test formula, then divide that by how much daily traffic actually lands in each variant (daily uniques times allocation), rounding up. The formula gives you a headcount; the traffic split is what turns a headcount into a calendar.

Structured elaboration

Step 1: the sample-size formula

For a two-sided test comparing baseline conversion $p_1$ against a target conversion $p_2$, at significance $\alpha$ and power $1-\beta$:

$n = \frac{\left(z_{1-\alpha/2}\sqrt{2\bar p(1-\bar p)} + z_{1-\beta}\sqrt{p_1(1-p_1)+p_2(1-p_2)}\right)^2}{(p_2-p_1)^2}, \quad \bar p = \frac{p_1+p_2}{2}$

This is per-variant sample size; assumes a two-sided test with no sequential peeking (one look at the end) and independent, one-conversion-per-user data. Sequential monitoring and multiple-comparisons corrections change this number and are a separate design decision, not part of the raw duration estimate.

Step 2: convert daily traffic into per-variant daily volume

$\text{daily per variant} = \text{daily unique visitors} \times \text{allocation share}$

Step 3: convert sample size into days

$\text{days} = \frac{n}{\text{daily per variant}}$

Round the sample size up (never down, since undershooting trades away the power you asked for) and round the resulting day count up to a whole day. If the metric has meaningful weekday/weekend variation, round further up to a whole number of weeks so every arm sees the same mix of weekdays and weekends; that decision is a separate seasonality question with its own reasoning (see the seasonality-planning answer), not something the raw formula above accounts for.

Assumptions and rounding choices worth stating out loud

• Two-sided test with a single, pre-planned final look; no interim peeking.
• Users are independent and contribute one conversion event each; no repeated exposure double counting.
• All intermediate values kept to about five significant figures before rounding the final answer, and the final day count always rounds up, not to the nearest day.
• A buffer of roughly 10-20% is a common practical addition on top of the raw day count to absorb data loss (bot filtering, QA holds, instrumentation gaps); state it as a buffer, not as part of the statistical requirement.
• If daily traffic is volatile rather than a single stable number, use a conservative (lower) daily estimate for the duration calculation rather than the average, since under-running the test is far more costly than over-running it by a day or two.

Worked example

Inputs: 60,000 daily unique eligible visitors, 50/50 allocation, baseline conversion $p_1 = 3.0\%$, desired minimum detectable effect of 8% relative, $\alpha=0.05$ two-sided ($z_{1-\alpha/2}=1.9600$), power 80% ($z_{1-\beta}=0.8416$).

Absolute MDE: $\Delta = 0.03 \times 0.08 = 0.0024$, so $p_2 = 0.0324$, $\bar p = 0.0312$.

$z_{1-\alpha/2}\sqrt{2\bar p(1-\bar p)} = 1.9600 \times \sqrt{2 \times 0.0312 \times 0.9688} = 1.9600 \times 0.2459 = 0.4819$

$z_{1-\beta}\sqrt{p_1(1-p_1)+p_2(1-p_2)} = 0.8416 \times \sqrt{0.03\times0.97 + 0.0324\times0.9676} = 0.8416 \times 0.2459 = 0.2069$

$n = \frac{(0.4819+0.2069)^2}{(0.0024)^2} = \frac{0.4744}{0.00000576} \approx 82{,}376 \text{ users per variant}$

Daily per variant: $60{,}000 \times 0.5 = 30{,}000$.

$\text{days} = \frac{82{,}376}{30{,}000} \approx 2.75 \rightarrow \text{round up to } 3 \text{ days minimum}$

Because this crosses a weekday/weekend boundary either way, the practical duration recommendation would be at least 7 days (one full week) rather than the bare 3-day statistical minimum, so weekday and weekend behavior are represented in the same proportion in both arms.

Trade-offs & pitfalls

• MDE sensitivity. The required sample size scales with the inverse square of the MDE, so halving the effect you want to detect roughly quadruples the required sample and the resulting duration; a stakeholder asking for a smaller MDE "just to be safe" is asking for a much longer test, not a marginally longer one.
• Binary-outcome formula does not transfer to continuous metrics. Revenue-per-user or time-on-task outcomes use the outcome's variance, not $p(1-p)$, in the same general formula shape; plugging a conversion-rate formula into a continuous metric silently understates or overstates the required sample.
• Shortcutting the calendar rounding. A raw day count under 7 does not mean the test is safe to run for that literal number of days; the weekly-cycle rounding matters as much as the raw arithmetic and is a common place teams cut a real corner under launch pressure.
• Traffic volatility. A single "daily uniques" number hides day-to-day swings; a duration plan built on a lucky high-traffic day will run short in practice.

Direct answer

Run for a whole number of full weekly cycles, decide before looking at any data how a holiday inside that window will be handled, and hold traffic allocation balanced by day-of-week (and by region and time zone if the test spans them) rather than trusting that a single aggregate 50/50 split will average out. When a weekday-positive, weekend-negative pattern shows up later, treat it as a hypothesis to falsify with three specific checks, real heterogeneity, a traffic-composition artifact, or noise, rather than reading the raw split at face value.

Structured elaboration

Duration and analysis window

Run for at least two, ideally three or more, full 7-day cycles. A partial week biases the pooled result toward whichever days happen to be over-represented, and a single week does not let you separate a real weekday/weekend pattern from that week's idiosyncrasies. If a holiday falls inside the planned window, decide up front, before seeing any results, between two options: exclude the holiday period from the primary analysis window and report a "typical week" estimate, or explicitly include it and report a distinct holiday-period estimate. Choosing between those two after looking at which one produces a better-looking result is a form of after-the-fact window selection and should be avoided; pre-register the choice in the analysis plan.

Traffic allocation and balance across time and geography

Stratify random assignment by day-of-week, and by region or time zone if the rollout spans them, so the same proportion of each arm is exposed every day and in every zone rather than relying on an aggregate split that could hide a skew. For a multi-region or multi-time-zone test, anchor "day" and "week" boundaries to each user's local time rather than a single server or UTC clock; otherwise one region's weekend gets miscounted against another region's weekday, and verify the treatment-to-control ratio stays constant across regions and hour-of-day buckets individually, not just in the combined total. Aggregate balance can look fine while a specific region or time window is quietly imbalanced, and that imbalance is exactly what later gets mistaken for a day-of-week effect.

Modeling the temporal structure instead of ignoring it

Rather than computing one pooled treatment effect and hoping seasonality washes out, fit day-of-week (and holiday, and region, if relevant) as explicit terms: outcome ~ treatment + day_of_week + treatment:day_of_week + region. This is standard regression-formula shorthand: ~ means "model the left-hand outcome using the terms on the right," so this line reads as "predict the outcome from the treatment, the day type, and the region," and treatment:day_of_week is an interaction term, a piece that lets the treatment's effect itself differ by day type rather than assuming it is the same on weekdays and weekends. The interaction term is what actually tells you whether the treatment effect differs by day type, instead of a single pooled number that could be hiding it.

Investigating a weekday-positive, weekend-negative split

Three checks, run in this order:

1. Is it real? Fit the treatment-by-day-type interaction term from the model above and check whether it is distinguishable from a null effect. This is one specific comparison, not a license to slice every available dimension until something looks significant; keep the interaction pre-specified as part of the analysis plan for exactly this reason.
2. Is it a traffic-composition artifact? Check whether the user mix itself differs by day type: a different device split, acquisition channel, or new-versus-returning ratio on weekends than weekdays. Re-run the interaction model with that covariate added and interacted; if the day-type interaction shrinks toward zero once the segment mix is controlled for, the apparent weekday/weekend split was really a segment-level pattern wearing a calendar label. Also check whether the rollout itself was staggered mid-week (a ramp that reached full exposure partway through the window) or whether an assignment-pipeline issue caused the treatment:control ratio to drift on certain days; both produce a day-type-looking artifact that has nothing to do with actual weekday or weekend behavior.
3. Is it noise? Compare the confidence interval on each day-type's estimate rather than the point estimates alone. Weekend traffic is frequently a fraction of weekday traffic, so a "negative" weekend estimate often carries a wide interval that comfortably contains the weekday estimate.

Worked example

Suppose the weekday arm has 8,000 users per group with control conversion 10.0% and treatment conversion 10.6% (a +0.6 percentage point delta), and the weekend arm has 2,000 users per group (lower weekend traffic) with control conversion 10.0% and treatment conversion 9.4% (a -0.6 percentage point delta). This is exactly the pattern in the question: positive on weekdays, negative on weekends.

Standard error of each delta, using $SE = \sqrt{\frac{p_c(1-p_c)}{n} + \frac{p_t(1-p_t)}{n}}$:

Weekday: $SE_{wd} = \sqrt{\frac{0.10\times0.90}{8000} + \frac{0.106\times0.894}{8000}} = 0.00481$, so the weekday delta's 95% interval is roughly $-0.34\text{pp}$ to $+1.54\text{pp}$, which already crosses zero.

Weekend: $SE_{we} = \sqrt{\frac{0.10\times0.90}{2000} + \frac{0.094\times0.906}{2000}} = 0.00936$, so the weekend delta's 95% interval is roughly $-2.43\text{pp}$ to $+1.23\text{pp}$, also crossing zero.

Testing whether the two deltas actually differ from each other: $z = \frac{0.006-(-0.006)}{\sqrt{0.00481^2+0.00936^2}} = \frac{0.012}{0.01052} \approx 1.14$, well under the 1.96 threshold for a two-sided 5% test. Both individual intervals already contain zero, and the two deltas are not statistically distinguishable from each other. With these particular sample sizes, the weekday-positive-weekend-negative pattern is fully consistent with noise, before ever needing to invoke a real behavioral difference or an artifact.

Trade-offs & pitfalls

• Trusting the point estimate over the interval. A sign flip between two point estimates feels meaningful; whether it survives a formal comparison of the two deltas, as above, is what actually determines whether there is anything to explain.
• Deciding the holiday treatment after seeing results. Choosing whether to include or exclude a holiday period based on which choice produces the preferred outcome is a subtle form of p-hacking through window selection, even when no single test is repeated.
• Assuming aggregate balance implies balance everywhere. A day-of-week or region-level imbalance can hide inside an aggregate 50/50 split and later masquerade as a real seasonal effect.
• Over-correcting into paralysis. Not every day-type split needs a full forensic investigation; reserve the three-check process for patterns that would actually change a rollout decision, and size the investigation to the stakes.

Direct answer

Beyond CUPED (using a pre-experiment covariate to residualize the outcome), the two other standard variance-reduction levers are stratified (blocked) randomization, which forces balance on a known factor at assignment time instead of hoping random chance balances it, and covariate or regression adjustment, which is the general case of "adjust for a predictive covariate" that CUPED is one specific, pre-experiment-only instance of. Both work by removing a source of outcome variance that is not related to treatment, so the same true effect becomes easier to distinguish from noise; both reduce required sample size roughly in proportion to how much outcome variance the factor explains, and neither invents a new number, they trade a known, explainable source of variance for a smaller residual.

Structured elaboration

Stratified (blocked) randomization

Instead of randomizing the whole population as one pool, split the population into strata on a factor known before assignment (country, device type, new vs. returning user), then randomize independently within each stratum so each arm gets a matched share of every stratum. This removes between-stratum variance from the treatment-effect estimator's variance, because the strata are balanced by design rather than by luck: with plain randomization on a highly imbalanced population, an unlucky split (e.g., treatment skewing toward the low-baseline country) inflates the observed variance of the effect estimate even though the true effect is unaffected.

It is applicable whenever you have a discrete, pre-assignment factor that is known to correlate with the outcome and is stable at randomization time. It differs from covariate adjustment in when the correction happens: stratification acts at assignment time (balance is enforced), while regression adjustment acts at analysis time (balance is estimated and subtracted after the fact). The two are complementary, not substitutes: stratify at assignment for the factors you can, and adjust for continuous covariates at analysis.

Covariate / regression adjustment

This is the general technique of fitting a model for the outcome on one or more covariates (not restricted to pre-experiment-only, unlike CUPED) and using the model to remove predictable variance from the outcome before comparing arms, most simply via ANCOVA (analysis of covariance), a linear regression of $Y$ on the treatment indicator and covariates that removes the variance those covariates explain from the comparison, the same variance-reduction logic as CUPED and stratification, just carried out as a regression rather than a pre-experiment covariate or a balanced split. It is applicable whenever you have covariates, pre-experiment or otherwise as long as they cannot themselves have been affected by treatment, that are predictive of the outcome. CUPED is the special case where the covariate is restricted to a pre-experiment value of the outcome metric itself; regression adjustment generalizes this to any number of eligible covariates and lets you combine several weak predictors into one stronger adjustment.

Effect on sample size and power

For both techniques, if the factor being controlled for explains a fraction $R^2$ of the outcome's variance, the variance of the treatment-effect estimator shrinks by roughly that same factor, and required sample size for a fixed target precision shrinks proportionally, since sample size for a fixed effect and power scales with the variance of the metric. A factor that explains little of the outcome variance buys little; a strong, well-chosen factor can meaningfully shorten the required test duration for the same statistical bar.

Worked example: stratifying a multi-country test

A test is planned across three countries with very different baseline conversion rates: Country A at 4%, Country B at 12%, Country C at 22%, in roughly equal traffic shares (each about one third of total users). Without stratification, plain randomization can by chance send more of one country's traffic to one arm, and even without that bad luck, the pooled outcome variance includes the between-country spread of baseline rates as extra noise the estimator has to average out.

Using the law of total variance, the overall variance of the outcome decomposes as:

$\operatorname{Var}(Y) = \underbrace{\mathbb{E}\big[\operatorname{Var}(Y \mid \text{country})\big]}_{\text{within-country variance}} + \underbrace{\operatorname{Var}\big(\mathbb{E}[Y \mid \text{country}]\big)}_{\text{between-country variance}}$

Stratifying by country and analyzing as a weighted average of within-country treatment effects removes the second (between-country) term from the treatment-effect estimator's variance, since each stratum is separately balanced and the between-stratum spread no longer contributes noise to the comparison. Concretely: with baseline rates of 4%, 12%, 22% and equal stratum weights, the between-country component of variance is

$\bar p = \frac{0.04+0.12+0.22}{3} = 0.1267$

$\operatorname{Var}(\bar p) = \frac{1}{3}\Big[(0.04-\bar p)^2 + (0.12-\bar p)^2 + (0.22-\bar p)^2\Big] = \frac{1}{3}(0.00751 + 0.0000445 + 0.00871) = 0.00542$

That 0.00542 is exactly the between-country variance component the stratified analysis removes from the pooled estimator's variance, computed directly from the three stated baseline rates, not asserted; how large a share of total variance that is depends additionally on the within-country binomial variance at each rate, which you would combine with this term using the same decomposition to get the full picture before quoting an overall percentage reduction.

Implementation for the multi-country case

• Assign the stratum at randomization time using the same deterministic hash-bucketing approach as the overall unit assignment, but nest it: hash within each country separately (or include country in the hash key) so each country independently hits its target split ratio.
• At analysis time, estimate the treatment effect within each country and combine as a weighted average (weighted by stratum size or by inverse variance), rather than pooling raw counts across countries, which is what actually realizes the variance reduction shown above.

Trade-offs and pitfalls

• Stratifying on too many dimensions at once shrinks individual strata until some contain too few units to balance meaningfully, and can create empty or near-empty cells, especially when crossing multiple categorical factors (country times device times cohort).
• A stratification factor chosen because it is convenient rather than because it is predictive buys little variance reduction while adding real implementation complexity; check the factor's explanatory power on historical data before committing the assignment pipeline to it.
• Regression adjustment on covariates measured close to, but not strictly after, the treatment start needs the same scrutiny as CUPED's pre-experiment-only requirement: any covariate that could plausibly be influenced by treatment invalidates the adjustment's unbiasedness, not just its efficiency.