A/B Test Design & Statistical Rigor Questions

Direct answer

When assignment happens per-device (a cookie on web, a device id on the app) but the real unit of interest is the person, users who touch the product on multiple devices get assigned independently on each device, so some of them land in both control and treatment at once. That breaks the assumption that each experimental unit receives exactly one arm: it dilutes the measured treatment effect, since a "contaminated" user's behavior is influenced by both arms, and it can double count outcomes if the same person's actions are logged and analyzed once per device-identity rather than once per person. The fix is to randomize and log at the most persistent identity you actually have, then dedupe and correct for the identities you had to resolve after the fact rather than at assignment time.

Structured elaboration

Where the bias enters

• Assignment stage: a user with a web cookie and a mobile device id gets two independent coin flips. If both land the same way, no harm; if they split, that person is exposed to treatment and control simultaneously, a violation of SUTVA (the assumption that one unit's outcome does not depend on another instance of its own assignment).
• Analysis stage: if the analysis unit is "device" rather than "resolved person," a contaminated person's activity appears once in each arm's numerator, and the two rows are not independent observations even though the analysis code treats them as such, which understates the true variance.
• Coverage bias: identity resolution itself makes mistakes (false merges linking two different people, false splits treating one person as two); if those errors are not random with respect to treatment, they introduce their own bias on top of the contamination.

Practical strategies

1. Randomize at the most stable identity you have. Prefer a logged-in account id over a device id or cookie whenever the user is authenticated; fall back to a deterministic device hash only for logged-out traffic, and treat that population as a separate, lower-confidence stratum in reporting.
2. Log everything needed to resolve identity after the fact. Persist device id, cookie id, and account id (hashed, respecting privacy) on every assignment and every outcome event, even when the assignment itself was made at device level, so contamination can be measured and corrected during analysis rather than discovered too late.
3. Define the primary analysis on the resolved (canonical) identity, not the raw assignment record: after backend identity merge, collapse a contaminated user into a single row and apply an explicit, pre-registered rule for what arm they count as, for example "any-device-treatment counts as treated," reported alongside a stricter "single-device-only" rule as a sensitivity check.
4. Quantify and bound the bias rather than ignore it. Report the primary result plus at least two sensitivity analyses: one restricted to users seen on exactly one device (removes contamination but shrinks the sample), and one using the any-device-treatment rule (keeps the full sample but is a diluted estimate of the true per-exposure effect).
5. Use cluster-robust standard errors at the resolved-identity level so that outcomes from the same person are not treated as independent observations even after collapsing to one row per person, since a person can still contribute multiple sessions or events.

Worked example

Assume 20% of users are active on exactly two devices (web and app) and 80% are single-device (an illustrative, stated split). Assignment happens independently per device with probability 0.5 to treatment. For a two-device user, the four equally likely device-pair outcomes are (control, control), (control, treatment), (treatment, control), (treatment, treatment), each with probability 0.25:
$P(\text{both control}) = P(\text{both treatment}) = 0.25, \qquad P(\text{split, i.e. contaminated}) = 0.5$
So among the 20% of users who are two-device, half get split across arms, which is 10% of the total user base. If a contaminated user's outcome is counted in both the treatment and control totals rather than resolved to one arm, then 10% of the treatment-arm numerator and 10% of the control-arm numerator are contributed by the exact same set of people, which both understates the between-arm difference and violates the independence assumption behind the standard error calculation. Restricting the primary analysis to the 90% of users who are single-device or resolve cleanly to one arm removes the contamination at the cost of 10% of the sample, which should be reflected directly in the power calculation for the corrected analysis.

Trade-offs and pitfalls

• The "any-device-treatment" rule is conservative and interpretable but structurally dilutes the estimated effect toward zero for contaminated users, since they experienced a mix of both arms; don't present it as an unbiased estimate of the pure per-exposure effect.
• Restricting to single-device users is cleaner statistically but changes who the estimate applies to: if multi-device users differ systematically (often more engaged, higher-value), the single-device estimate may not generalize to the full user base.
• Identity resolution is itself a model with false-merge and false-split error rates; a resolution pipeline retrained or changed mid-experiment can shift the contamination rate over time and should be monitored, not assumed constant.
• Logging every identifier needed for later resolution has real privacy and storage cost; agree on hashing and retention policy with the privacy function before instrumenting, not after a contamination investigation is already underway.

Direct answer

Run for a whole number of full weekly cycles, decide before looking at any data how a holiday inside that window will be handled, and hold traffic allocation balanced by day-of-week (and by region and time zone if the test spans them) rather than trusting that a single aggregate 50/50 split will average out. When a weekday-positive, weekend-negative pattern shows up later, treat it as a hypothesis to falsify with three specific checks, real heterogeneity, a traffic-composition artifact, or noise, rather than reading the raw split at face value.

Structured elaboration

Duration and analysis window

Run for at least two, ideally three or more, full 7-day cycles. A partial week biases the pooled result toward whichever days happen to be over-represented, and a single week does not let you separate a real weekday/weekend pattern from that week's idiosyncrasies. If a holiday falls inside the planned window, decide up front, before seeing any results, between two options: exclude the holiday period from the primary analysis window and report a "typical week" estimate, or explicitly include it and report a distinct holiday-period estimate. Choosing between those two after looking at which one produces a better-looking result is a form of after-the-fact window selection and should be avoided; pre-register the choice in the analysis plan.

Traffic allocation and balance across time and geography

Stratify random assignment by day-of-week, and by region or time zone if the rollout spans them, so the same proportion of each arm is exposed every day and in every zone rather than relying on an aggregate split that could hide a skew. For a multi-region or multi-time-zone test, anchor "day" and "week" boundaries to each user's local time rather than a single server or UTC clock; otherwise one region's weekend gets miscounted against another region's weekday, and verify the treatment-to-control ratio stays constant across regions and hour-of-day buckets individually, not just in the combined total. Aggregate balance can look fine while a specific region or time window is quietly imbalanced, and that imbalance is exactly what later gets mistaken for a day-of-week effect.

Modeling the temporal structure instead of ignoring it

Rather than computing one pooled treatment effect and hoping seasonality washes out, fit day-of-week (and holiday, and region, if relevant) as explicit terms: outcome ~ treatment + day_of_week + treatment:day_of_week + region. This is standard regression-formula shorthand: ~ means "model the left-hand outcome using the terms on the right," so this line reads as "predict the outcome from the treatment, the day type, and the region," and treatment:day_of_week is an interaction term, a piece that lets the treatment's effect itself differ by day type rather than assuming it is the same on weekdays and weekends. The interaction term is what actually tells you whether the treatment effect differs by day type, instead of a single pooled number that could be hiding it.

Investigating a weekday-positive, weekend-negative split

Three checks, run in this order:

1. Is it real? Fit the treatment-by-day-type interaction term from the model above and check whether it is distinguishable from a null effect. This is one specific comparison, not a license to slice every available dimension until something looks significant; keep the interaction pre-specified as part of the analysis plan for exactly this reason.
2. Is it a traffic-composition artifact? Check whether the user mix itself differs by day type: a different device split, acquisition channel, or new-versus-returning ratio on weekends than weekdays. Re-run the interaction model with that covariate added and interacted; if the day-type interaction shrinks toward zero once the segment mix is controlled for, the apparent weekday/weekend split was really a segment-level pattern wearing a calendar label. Also check whether the rollout itself was staggered mid-week (a ramp that reached full exposure partway through the window) or whether an assignment-pipeline issue caused the treatment:control ratio to drift on certain days; both produce a day-type-looking artifact that has nothing to do with actual weekday or weekend behavior.
3. Is it noise? Compare the confidence interval on each day-type's estimate rather than the point estimates alone. Weekend traffic is frequently a fraction of weekday traffic, so a "negative" weekend estimate often carries a wide interval that comfortably contains the weekday estimate.

Worked example

Suppose the weekday arm has 8,000 users per group with control conversion 10.0% and treatment conversion 10.6% (a +0.6 percentage point delta), and the weekend arm has 2,000 users per group (lower weekend traffic) with control conversion 10.0% and treatment conversion 9.4% (a -0.6 percentage point delta). This is exactly the pattern in the question: positive on weekdays, negative on weekends.

Standard error of each delta, using $SE = \sqrt{\frac{p_c(1-p_c)}{n} + \frac{p_t(1-p_t)}{n}}$:

Weekday: $SE_{wd} = \sqrt{\frac{0.10\times0.90}{8000} + \frac{0.106\times0.894}{8000}} = 0.00481$, so the weekday delta's 95% interval is roughly $-0.34\text{pp}$ to $+1.54\text{pp}$, which already crosses zero.

Weekend: $SE_{we} = \sqrt{\frac{0.10\times0.90}{2000} + \frac{0.094\times0.906}{2000}} = 0.00936$, so the weekend delta's 95% interval is roughly $-2.43\text{pp}$ to $+1.23\text{pp}$, also crossing zero.

Testing whether the two deltas actually differ from each other: $z = \frac{0.006-(-0.006)}{\sqrt{0.00481^2+0.00936^2}} = \frac{0.012}{0.01052} \approx 1.14$, well under the 1.96 threshold for a two-sided 5% test. Both individual intervals already contain zero, and the two deltas are not statistically distinguishable from each other. With these particular sample sizes, the weekday-positive-weekend-negative pattern is fully consistent with noise, before ever needing to invoke a real behavioral difference or an artifact.

Trade-offs & pitfalls

• Trusting the point estimate over the interval. A sign flip between two point estimates feels meaningful; whether it survives a formal comparison of the two deltas, as above, is what actually determines whether there is anything to explain.
• Deciding the holiday treatment after seeing results. Choosing whether to include or exclude a holiday period based on which choice produces the preferred outcome is a subtle form of p-hacking through window selection, even when no single test is repeated.
• Assuming aggregate balance implies balance everywhere. A day-of-week or region-level imbalance can hide inside an aggregate 50/50 split and later masquerade as a real seasonal effect.
• Over-correcting into paralysis. Not every day-type split needs a full forensic investigation; reserve the three-check process for patterns that would actually change a rollout decision, and size the investigation to the stakes.

Direct answer

CUPED (Controlled-experiment Using Pre-Experiment Data) is a variance-reduction technique, published by Microsoft researchers (Deng, Xu, Kohavi and Walker, 2013), that uses a metric measured before the experiment started to strip out some of the between-user noise in the outcome metric, without touching the unbiasedness of the treatment-effect estimate. It requires one thing: a pre-experiment value of a covariate, ideally the same metric you are about to test, measured for the same units, from a period before assignment happened (so it cannot itself have been affected by treatment). The payoff is a tighter estimate of the same treatment effect from the same traffic, which functionally means you can detect the same effect size with less traffic or in less time.

Structured elaboration

What it requires and assumes

• A pre-experiment covariate $X$ for every unit, measured strictly before random assignment (so treatment cannot have influenced it) and ideally correlated with the outcome $Y$ you're testing.
• The scaling coefficient must be estimated without using the treatment label, typically pooled across both arms or from control alone. This is what keeps the adjustment from injecting bias: you are not letting "knowledge of which arm someone is in" leak into how you weight their pre-period value.

How the adjustment is computed, in plain terms

For each unit, adjust its outcome by subtracting a scaled version of how far its pre-experiment covariate was from the population average:

$Y_i^{\text{CUPED}} = Y_i - \theta \left(X_i - \bar{X}\right)$

where the scaling coefficient is the population covariance of $X$ and $Y$ divided by the variance of $X$:

$\theta = \frac{\operatorname{Cov}(X, Y)}{\operatorname{Var}(X)}$

Intuitively: if a user's pre-period activity was above average, some of their post-period activity being above average too is "expected" and not informative about treatment, so it gets subtracted out. The treatment effect is then estimated as the difference in adjusted outcome means between arms, $\bar{Y}^{\text{CUPED}}_{\text{treat}} - \bar{Y}^{\text{CUPED}}_{\text{control}}$, which is algebraically the same as the raw mean difference minus $\theta$ times the pre-period mean difference between arms; since randomization makes that pre-period difference expected to be zero, the adjustment does not shift the expected value of the estimate, it only removes the part of each unit's variance that $X$ already explained.

The resulting variance reduction has a clean form: if $\rho$ is the correlation between the pre-experiment covariate and the outcome, the adjusted estimator's variance is $(1-\rho^2)$ times the unadjusted variance. A stronger pre-period predictor buys a bigger reduction; a weak one buys almost nothing.

What makes a covariate good or poor

What goes wrong with a poor choice

• Using a post-treatment or treatment-adjacent value as $X$ breaks the "cannot be affected by treatment" requirement and reintroduces bias: you would be conditioning on something treatment itself moved, which is a different and more serious problem than losing precision.
• Using a weakly correlated covariate costs almost nothing in bias but also buys almost nothing in variance reduction; the $(1-\rho^2)$ factor is close to 1.
• Using a covariate that is missing for a large share of units (commonly new users with no pre-period history) forces a choice: drop those units (losing sample and possibly biasing toward existing users) or impute a value (introducing a modeling assumption you did not need before).

Worked example

Suppose, for illustration, we assume a pre-experiment 7-day conversion rate correlates with the in-experiment conversion rate at $\rho = 0.6$ for a given product (a hypothetical, stated input, not an empirical claim about any real product). The variance reduction from CUPED is then derived directly:

$\frac{\operatorname{Var}(Y^{\text{CUPED}})}{\operatorname{Var}(Y)} = 1 - \rho^2 = 1 - 0.6^2 = 1 - 0.36 = 0.64$

So the adjusted estimator's variance is 64% of the unadjusted one, a 36% reduction, given only the stated assumption $\rho = 0.6$. If $\rho$ had instead been a weak 0.2, the reduction would be $1 - 0.04 = 0.96$, i.e., only a 4% cut, illustrating why the strength of the pre-period correlation, not just its existence, determines whether CUPED is worth implementing for a given metric.

Trade-offs and pitfalls

• CUPED does not help metrics with no meaningful pre-experiment history, most obviously new-user-only experiments; for those, look at stratification or other covariate-adjustment techniques instead.
• Estimating $\theta$ using data that has already been split by arm (rather than pooled or control-only) risks contaminating the coefficient with treatment information; keep the estimation step blind to arm assignment.
• CUPED reduces variance, it does not change what the true effect is; do not present the tighter confidence interval as evidence the effect itself grew, only that you can now see it more precisely.
• CUPED composes with stratified randomization (apply the adjustment within strata), which is a reason to know both rather than treat them as competing choices.

Direct answer

The primary metric is the single metric that answers "did this change achieve its intended goal," and it is what the ship decision is nominally based on. Guardrail metrics are metrics you are not trying to improve, but are watching to make sure the change does not cause unacceptable harm elsewhere; a guardrail regressing can override a primary metric win. A secondary metric is different from both: it is additional signal you are curious about or want to understand mechanism through, but a secondary metric moving in a bad direction does not, by itself, block a ship decision the way a guardrail breach does. The distinction that matters operationally is that guardrails carry a pre-committed threshold and a pause-or-rollback consequence; secondary metrics do not.

Structured elaboration

Primary vs. guardrail vs. secondary

An equivalent framing some teams use is proximal vs. distal metrics: a proximal metric sits close to the mechanism of the change (click-through rate on a redesigned button) and moves quickly; a distal metric sits further downstream (long-term retention, lifetime value) and moves slowly but is closer to what the business actually cares about. A guardrail is frequently a distal metric precisely because the harm you are worried about (retention erosion, trust damage) is often slower to appear than the primary win.

Worked proposal for a monetization change (new ad placement)

Primary metric: net revenue per user in the experiment arm. Direction of success: increase. This is the metric the change exists to move.

Guardrail 1: 7-day retention. Direction of harm: decrease. Rationale: an intrusive placement can drive short-term revenue while quietly eroding the reason people come back. Pause/rollback trigger: agreed in advance as a stated relative-drop threshold with the confidence interval's upper bound also below zero (i.e., not just a point estimate dip that could be noise), reviewed before rollout, not chosen after seeing the result.

Guardrail 2: core-task completion rate (the product's main non-monetization action, e.g., completing a search, finishing a checkout, reading an article to completion). Direction of harm: decrease. Rationale: an ad placement that visually or functionally interferes with the primary task is trading long-run product health for short-run revenue.

Guardrail 3: user-initiated complaint or ad-block/opt-out rate. Direction of harm: increase. Rationale: a direct, unambiguous signal of user tolerance that is available faster than retention, useful as an early-warning guardrail even before the retention window has fully played out.

Guardrail 4 (optional, if the surface has one): page load or responsiveness regression, since an added placement can degrade performance in a way that suppresses every other metric indirectly; direction of harm: increase in load time or error rate.

This maps onto the same structure whether you are testing an ad placement, a checkout-flow revenue change (where the natural guardrail set expands to include cart-abandonment rate and support-ticket volume), or a premium-feature paywall (where conversion rate is typically the primary, and DAU, ARPU, and system error rate sit alongside it as guardrails against gating too aggressively or destabilizing the product). The framing also transfers outside pure monetization: for a conversational AI product's response pipeline, the primary might be task-completion rate while the guardrails are safety and quality signals such as a harmful-response rate or an unresolved-escalation rate, because the mechanics of "one thing you're optimizing, several things you refuse to let break" do not change with the domain.

Setting the threshold, not just naming the metric

A guardrail without a pre-committed threshold is not actually a guardrail, it is a chart someone glances at. The threshold should be set from business tolerance for harm (how much retention erosion is worth this much revenue) agreed before the experiment starts, not derived by re-deriving statistical power mid-flight; whether the observed guardrail movement is distinguishable from noise at that threshold is a separate, purely statistical question the analysis answers once data is in, not something this design step needs to resolve.

Worked example

A checkout-flow revenue experiment adds a one-click upsell at the payment step. The team pre-commits four guardrails before launch: cart-abandonment rate (harm: increase), 7-day repeat-purchase rate (harm: decrease), support-ticket volume tagged "checkout confusion" (harm: increase), and page load time at the payment step (harm: increase). Two weeks in, revenue per session is up and three of the four guardrails are flat, but cart-abandonment is up beyond the pre-committed trigger. Because the threshold and the pause rule were set before launch, the team pauses the rollout to investigate the upsell's placement rather than debating in the moment whether the abandonment increase is "bad enough" to matter.

Trade-offs and pitfalls

• Naming too many guardrails dilutes the signal and invites false alarms purely from checking many metrics at once; a handful of well-chosen, harm-specific guardrails beats a long generic list.
• Do not let a metric quietly slide from "secondary" to "guardrail" after the fact because it happened to move in a bad direction; that is choosing your rules after seeing the data, which defeats the purpose of pre-committing thresholds.
• A guardrail with no pre-committed threshold is not enforceable in the moment it matters; agree on the trigger, and who has authority to invoke it, before the experiment ships.

Direct answer

SUTVA, the Stable Unit Treatment Value Assumption, is the assumption underlying a standard A/B test that one unit's observed outcome depends only on which arm that unit itself was assigned to, and not on which arms other units received. It has two components: no interference between units (your outcome is not affected by someone else's assignment) and no hidden variation of treatment (everyone labeled "treatment" received the same treatment). When either fails, the simple difference-in-means between arms is no longer an unbiased estimate of the causal effect you think you are measuring.

Structured elaboration

The two components

1. No interference between units. Unit $i$'s potential outcome under any assignment vector depends only on $i$'s own treatment, not on the treatments assigned to units $j \neq i$. This is the component that breaks in networked or shared-resource products.
2. No hidden variation of treatment (consistency). There is exactly one version of "treatment" and one version of "control"; a unit's potential outcome is well defined given only its treatment label. This breaks when the same nominal arm is implemented differently for different users (different rollout timing, different creative, a bug that only affects some treatment users).

Two real violations and why each biases interpretation

Social feed, no-interference violation. A treated user gets a new sharing feature and posts more; their connections, who are in control, now see more of that content in their own feed even though they were never assigned to treatment. The control group's outcome (engagement) is contaminated by treatment spillover, which pulls control's measured engagement up and understates the true treatment effect: you are comparing "treatment" against "control that is partially treated," not against a clean counterfactual.

Shared inventory or capacity constraint. A promotion arm drives more purchases, which draws down a shared inventory pool or a fixed daily capacity (delivery slots, support-queue capacity) that both arms draw from. Control users now see stockouts or longer wait times caused by treatment's demand, not by anything intrinsic to being in control. This inflates the apparent treatment effect (control looks artificially worse) and, separately, means the effect you measured at the tested traffic share will not hold at 100% rollout, because the resource contention itself scales with the treatment allocation percentage.

In both cases the estimate is not merely noisy, it is biased in a specific, name-able direction, and the bias would not shrink with more sample size because it comes from the assignment mechanism interacting with the product, not from sampling error.

Designing around a suspected violation

Once you suspect interference, the standard fix is to change the unit of randomization to one large enough to contain the spillover, i.e., cluster randomization: randomize by friend-group, geographic market, or server shard instead of by individual, so that most of the interference happens within a cluster (which is internally consistent, either all-treated or all-control) rather than across the treatment/control boundary.

Cluster randomization has its own cost, though: fewer independent units means higher variance for the same total traffic, since the effective sample size is closer to the number of clusters than the number of users. It also requires the outcome to be measurable and meaningful at the cluster level, and it does not eliminate the shared-capacity case unless the constrained resource is itself scoped per cluster.

Detecting a violation you did not design around

If the violation is discovered mid-experiment rather than anticipated, e.g., a caching-configuration bug causes some control users to intermittently render treatment-arm content, the diagnostic sequence is: quantify the leak rate first (what fraction of control exposures actually rendered the treatment experience, pulled directly from logs, not estimated), then decide whether the leak is small enough to bound the bias and proceed with a documented caveat, or large enough that the read is unusable and the fix is to patch the bug and rerun rather than to try to model the contamination away after the fact.

Worked example

A marketplace runs a two-armed test on a checkout redesign, expecting independent per-user outcomes. Mid-experiment, a shared caching layer bug is found: a same-session user occasionally gets served a stale cached page from the opposite arm. Pulling exposure logs, engineering finds this affected roughly 4% of control-arm page views (a number read directly from the cache-hit logs, not assumed). That is a version-of-treatment violation, not an interference violation: some "control" users received a materially different experience than the rest of control, so the control arm is not internally consistent. The team's next step is not to reweight or model this away, because the mechanism (a caching bug) has no principled correction; it is to fix the bug and rerun the experiment cleanly, treating the contaminated run as informative only about the presence of the bug.

Trade-offs and pitfalls

• Do not assume interference is symmetric or negligible just because the product does not look "social." Shared backend resources (queues, inventory, ranking models retrained on pooled data) create interference in products with no visible network feature.
• Cluster randomization trades bias for variance; do not adopt it reflexively for every experiment on a networked product when the actual interference is small relative to the direct effect, since you would be paying a real power cost for a small bias fix.
• A violation discovered after the fact is a data-quality incident, not a modeling problem to be adjusted away; resist the temptation to "correct" biased data with a post hoc statistical patch when the honest fix is to rerun cleanly.