A/B Test Design & Statistical Rigor Questions

Direct answer

When assignment happens per-device (a cookie on web, a device id on the app) but the real unit of interest is the person, users who touch the product on multiple devices get assigned independently on each device, so some of them land in both control and treatment at once. That breaks the assumption that each experimental unit receives exactly one arm: it dilutes the measured treatment effect, since a "contaminated" user's behavior is influenced by both arms, and it can double count outcomes if the same person's actions are logged and analyzed once per device-identity rather than once per person. The fix is to randomize and log at the most persistent identity you actually have, then dedupe and correct for the identities you had to resolve after the fact rather than at assignment time.

Structured elaboration

Where the bias enters

• Assignment stage: a user with a web cookie and a mobile device id gets two independent coin flips. If both land the same way, no harm; if they split, that person is exposed to treatment and control simultaneously, a violation of SUTVA (the assumption that one unit's outcome does not depend on another instance of its own assignment).
• Analysis stage: if the analysis unit is "device" rather than "resolved person," a contaminated person's activity appears once in each arm's numerator, and the two rows are not independent observations even though the analysis code treats them as such, which understates the true variance.
• Coverage bias: identity resolution itself makes mistakes (false merges linking two different people, false splits treating one person as two); if those errors are not random with respect to treatment, they introduce their own bias on top of the contamination.

Practical strategies

1. Randomize at the most stable identity you have. Prefer a logged-in account id over a device id or cookie whenever the user is authenticated; fall back to a deterministic device hash only for logged-out traffic, and treat that population as a separate, lower-confidence stratum in reporting.
2. Log everything needed to resolve identity after the fact. Persist device id, cookie id, and account id (hashed, respecting privacy) on every assignment and every outcome event, even when the assignment itself was made at device level, so contamination can be measured and corrected during analysis rather than discovered too late.
3. Define the primary analysis on the resolved (canonical) identity, not the raw assignment record: after backend identity merge, collapse a contaminated user into a single row and apply an explicit, pre-registered rule for what arm they count as, for example "any-device-treatment counts as treated," reported alongside a stricter "single-device-only" rule as a sensitivity check.
4. Quantify and bound the bias rather than ignore it. Report the primary result plus at least two sensitivity analyses: one restricted to users seen on exactly one device (removes contamination but shrinks the sample), and one using the any-device-treatment rule (keeps the full sample but is a diluted estimate of the true per-exposure effect).
5. Use cluster-robust standard errors at the resolved-identity level so that outcomes from the same person are not treated as independent observations even after collapsing to one row per person, since a person can still contribute multiple sessions or events.

Worked example

Assume 20% of users are active on exactly two devices (web and app) and 80% are single-device (an illustrative, stated split). Assignment happens independently per device with probability 0.5 to treatment. For a two-device user, the four equally likely device-pair outcomes are (control, control), (control, treatment), (treatment, control), (treatment, treatment), each with probability 0.25:
$P(\text{both control}) = P(\text{both treatment}) = 0.25, \qquad P(\text{split, i.e. contaminated}) = 0.5$
So among the 20% of users who are two-device, half get split across arms, which is 10% of the total user base. If a contaminated user's outcome is counted in both the treatment and control totals rather than resolved to one arm, then 10% of the treatment-arm numerator and 10% of the control-arm numerator are contributed by the exact same set of people, which both understates the between-arm difference and violates the independence assumption behind the standard error calculation. Restricting the primary analysis to the 90% of users who are single-device or resolve cleanly to one arm removes the contamination at the cost of 10% of the sample, which should be reflected directly in the power calculation for the corrected analysis.

Trade-offs and pitfalls

• The "any-device-treatment" rule is conservative and interpretable but structurally dilutes the estimated effect toward zero for contaminated users, since they experienced a mix of both arms; don't present it as an unbiased estimate of the pure per-exposure effect.
• Restricting to single-device users is cleaner statistically but changes who the estimate applies to: if multi-device users differ systematically (often more engaged, higher-value), the single-device estimate may not generalize to the full user base.
• Identity resolution is itself a model with false-merge and false-split error rates; a resolution pipeline retrained or changed mid-experiment can shift the contamination rate over time and should be monitored, not assumed constant.
• Logging every identifier needed for later resolution has real privacy and storage cost; agree on hashing and retention policy with the privacy function before instrumenting, not after a contamination investigation is already underway.

Direct answer

Assess heterogeneity with a formal treatment-by-segment interaction test rather than by eyeballing which country's point estimate looks different, and be honest that most individual segments are underpowered relative to the overall test. Because testing many segments at once inflates the odds that one looks significant by chance alone, correct for that multiplicity (a false-discovery-rate procedure for routine screening, Bonferroni's family-wise control when a single false positive would be costly) before trusting any one segment's result, and choose among full pooling (one overall effect), no pooling (an independent estimate per segment), and partial pooling (a hierarchical model that shrinks each segment's estimate toward the overall mean in proportion to how much data that segment actually has); partial pooling is the right production default for exactly this kind of many-segment, uneven-traffic setup, because it avoids both the false confidence of no pooling and the false uniformity of full pooling.

Structured elaboration

Testing for heterogeneity

Fit outcome ~ treatment + segment + treatment:segment, using both device type and region as covariates jointly rather than testing one dimension at a time, since a device-level pattern and a region-level pattern can be confounded with each other if only one is modeled. (In this regression-formula shorthand, ~ means "model the outcome using the terms on the right," and treatment:segment is the interaction term, the piece that tests whether the treatment effect itself varies by segment rather than just shifting the segment's baseline.) Test the joint significance of the interaction terms with a Wald or likelihood-ratio test, or a permutation test as a distribution-free alternative (any of the three answers the same question by a different route, so which one you pick matters less than actually pre-specifying and running one), to get one answer to "is there heterogeneity at all" before looking at any individual segment.

Why per-segment power is the first thing to check

A segment with a fraction of the overall traffic has a much larger minimum detectable effect (MDE) than the pooled test, using the same relationship between sample size and detectable effect as any power calculation:

$\text{MDE} \approx (z_{1-\alpha/2}+z_{1-\beta})\sqrt{\frac{2p(1-p)}{n}}$

For an overall test with 300,000 users per arm and a 5% baseline conversion rate, at $\alpha=0.05$ two-sided and 80% power:

$\text{MDE}_{\text{overall}} \approx (1.9600+0.8416)\sqrt{\frac{2\times0.05\times0.95}{300{,}000}} = 2.8016 \times 0.000563 \approx 0.0016 \ (0.16\text{pp}, \text{about } 3.2\% \text{ relative})$

A single small geo with 3% of that traffic, 9,000 users per arm, has:

$\text{MDE}_{\text{small geo}} \approx 2.8016\sqrt{\frac{2\times0.05\times0.95}{9{,}000}} \approx 0.0091 \ (0.91\text{pp}, \text{about } 18.2\% \text{ relative})$

That is roughly 5.8 times larger (matching $\sqrt{300{,}000/9{,}000}$), meaning that small geo's own data can only reliably detect an effect nearly six times bigger than what the overall test is powered for. A "no significant effect" reading for that geo on its own data is very often just this, not evidence the true effect is zero there.

Correcting for testing many segments at once

Underpowered segments are one problem; testing many of them at the same time creates a second, separate problem. Each segment's interaction test is its own hypothesis test, so scanning fifteen countries or five device types for "which one differs" is exactly the multiple-comparisons setting: even if the treatment effect is truly identical everywhere, running enough segment-level tests makes at least one spuriously significant result likely by chance alone. Two standard corrections apply once the segment set is pre-specified (decided before looking at the results, not assembled from whichever segments already look interesting):

• Bonferroni divides the significance threshold by the number of segments tested (roughly $\alpha/k$ for $k$ segments). It controls the family-wise error rate (FWER), the probability of even a single false positive across the whole set, which makes it the conservative option: reach for it when acting on one wrongly-flagged segment would be expensive, for example a permanent per-country rollout split that is costly to unwind.
• Benjamini-Hochberg (BH) controls the false discovery rate (FDR) instead: the expected proportion of false positives among the segments you flag, not the chance of any false positive at all. It is less conservative than Bonferroni and is the standard choice for segment screening, because the usual goal when scanning many countries or device types is to shortlist which ones deserve a closer look, not to make one irreversible call per segment; tolerating a small, known rate of false leads among the flagged segments is the right trade for not burying the real ones.

Hierarchical shrinkage, covered in the pooling comparison below, answers the same over-testing question through a different mechanism rather than competing with these corrections. Bonferroni and BH both operate on the significance threshold a segment's p-value has to clear before you call it real; partial pooling never runs a per-segment significance decision in the first place; it pulls every segment's raw estimate toward the global mean in proportion to how little data that segment has, so a small segment cannot produce an extreme, attention-grabbing estimate purely from its own noise. A production pipeline commonly uses both together: an FDR-corrected interaction test to decide which segments are worth calling out as genuinely different, and hierarchical shrinkage on the point estimates actually shown to stakeholders, so the reported numbers are already regularized rather than raw.

Full pooling, no pooling, and partial pooling

The shrinkage a hierarchical model applies has a standard closed form: a segment's raw estimate $\hat\theta_i$ with sampling variance $\sigma_i^2$ is pulled toward the grand mean $\bar\theta$ by:

$\lambda_i = \frac{\tau^2}{\tau^2 + \sigma_i^2}, \quad \hat\theta_i^{\text{shrunk}} = \lambda_i \hat\theta_i + (1-\lambda_i)\bar\theta$

where $\tau^2$ is the estimated between-segment variance.

To make this concrete, reuse the same two segments from the MDE comparison above: the overall test's 300,000-users-per-arm scale and the small geo's 9,000-users-per-arm scale, both at the 5% baseline conversion rate already used there. Their sampling variances follow directly from the same standard-error term used in the MDE formula, expressed here in percentage points:

$\sigma_{\text{large}} = \sqrt{\frac{2\times0.05\times0.95}{300{,}000}} \approx 0.0563\text{pp}, \quad \sigma_{\text{large}}^2 \approx 0.00317\ \text{pp}^2$

$\sigma_{\text{small}} = \sqrt{\frac{2\times0.05\times0.95}{9{,}000}} \approx 0.3249\text{pp}, \quad \sigma_{\text{small}}^2 \approx 0.1056\ \text{pp}^2$

Take an illustrative between-segment variance of $\tau^2 = 0.09\ \text{pp}^2$ (true segment effects varying by roughly $\pm0.3$pp around the grand mean is a reasonable order of magnitude here: smaller than the small geo's own sampling noise, larger than the overall test's). The shrinkage weights are:

$\lambda_{\text{large}} = \frac{0.09}{0.09+0.00317} \approx 0.966, \qquad \lambda_{\text{small}} = \frac{0.09}{0.09+0.1056} \approx 0.460$

Suppose the pooled (grand-mean) estimate across all segments is $\bar\theta = 0.20$pp, the large segment's own raw estimate happens to be $\hat\theta_{\text{large}} = 0.25$pp, and the small geo's own raw estimate happens to be $\hat\theta_{\text{small}} = 0.85$pp, more than four times the grand mean at face value. Shrinking each toward $\bar\theta$:

$\hat\theta_{\text{large}}^{\text{shrunk}} = 0.966\times0.25 + 0.034\times0.20 \approx 0.248\text{pp}$

$\hat\theta_{\text{small}}^{\text{shrunk}} = 0.460\times0.85 + 0.540\times0.20 \approx 0.499\text{pp}$

The large segment's raw estimate barely moves, 0.25pp to 0.248pp, because its own sampling variance is tiny next to $\tau^2$, keeping $\lambda_{\text{large}}$ close to 1. The small geo's raw estimate moves a great deal, from 0.85pp down to about 0.50pp, more than half of its apparent excess over the grand mean pulled away, because its sampling variance is large next to $\tau^2$ and $\lambda_{\text{small}}$ sits closer to 0. A striking small-segment number is exactly the case shrinkage is built to discount.

A weakly informative prior centered on the pooled estimate (for a Bayesian implementation) plays the same role as $\tau^2$ in a frequentist multilevel model: it is what lets a small geo borrow strength from the rest of the segments instead of standing entirely on its own thin data. A segment with a small $\sigma_i^2$ (lots of its own data) keeps most of its own estimate; a segment with a large $\sigma_i^2$ (little data) gets pulled hard toward the overall mean. This is the mechanism, not full pooling's blunt "ignore the segment" or no pooling's blunt "trust the segment's noisy number completely."

What to report to stakeholders

For each segment, report the shrunk point estimate, its credible or confidence interval, and something indicating how much shrinkage was applied (effective sample size, or the raw versus shrunk estimate side by side), alongside the single global interaction-test result and which multiplicity correction it used. Explain the shrinkage in plain language for a non-technical audience: "we pulled small countries' estimates toward the overall average because they don't have enough of their own data to stand alone yet," rather than presenting a raw per-country number that a stakeholder might otherwise take at face value.

Recommended rollout strategy

If the global interaction test does not reject homogeneity and the shrunk per-segment estimates are consistent in sign and magnitude with the pooled effect, roll out globally on the strength of the pooled result. If a segment's shrunk estimate remains meaningfully different in sign or magnitude even after shrinkage has pulled it toward the mean, and it survives the multiplicity correction above, that segment is a real candidate for differentiated treatment, but confirm it with a segment-targeted follow-up before committing production logic to a permanent split, the same discipline that applies to any post-hoc-adjacent segment finding.

Trade-offs & pitfalls

• Treating "not significant per segment" as "no effect in that segment." The MDE gap above is the mechanism; a small segment's null result is usually a power problem, not a finding.
• No pooling on small segments overstates confidence in noise. Reporting fifteen independent country-level point estimates without acknowledging their individual uncertainty invites a stakeholder to chase the noisiest ones.
• Full pooling by default hides real, actionable heterogeneity. If the interaction test does reject homogeneity, defaulting to one global number throws away a finding that could inform a real rollout decision.
• Explaining shrinkage badly. If a hierarchical model's output is presented as a black box, a segment owner who sees their raw number "corrected" downward without explanation will reasonably distrust the whole analysis.
• Skipping the multiplicity correction because shrinkage is already in use. Shrinkage regularizes the estimates; it does not by itself control how many segments get flagged as different across the whole set. Treating hierarchical modeling as a substitute for an FDR or family-wise correction, instead of a complementary tool, is how a many-segment scan quietly turns into a fishing expedition again.

Direct answer

SUTVA, the Stable Unit Treatment Value Assumption, is the assumption underlying a standard A/B test that one unit's observed outcome depends only on which arm that unit itself was assigned to, and not on which arms other units received. It has two components: no interference between units (your outcome is not affected by someone else's assignment) and no hidden variation of treatment (everyone labeled "treatment" received the same treatment). When either fails, the simple difference-in-means between arms is no longer an unbiased estimate of the causal effect you think you are measuring.

Structured elaboration

The two components

1. No interference between units. Unit $i$'s potential outcome under any assignment vector depends only on $i$'s own treatment, not on the treatments assigned to units $j \neq i$. This is the component that breaks in networked or shared-resource products.
2. No hidden variation of treatment (consistency). There is exactly one version of "treatment" and one version of "control"; a unit's potential outcome is well defined given only its treatment label. This breaks when the same nominal arm is implemented differently for different users (different rollout timing, different creative, a bug that only affects some treatment users).

Two real violations and why each biases interpretation

Social feed, no-interference violation. A treated user gets a new sharing feature and posts more; their connections, who are in control, now see more of that content in their own feed even though they were never assigned to treatment. The control group's outcome (engagement) is contaminated by treatment spillover, which pulls control's measured engagement up and understates the true treatment effect: you are comparing "treatment" against "control that is partially treated," not against a clean counterfactual.

Shared inventory or capacity constraint. A promotion arm drives more purchases, which draws down a shared inventory pool or a fixed daily capacity (delivery slots, support-queue capacity) that both arms draw from. Control users now see stockouts or longer wait times caused by treatment's demand, not by anything intrinsic to being in control. This inflates the apparent treatment effect (control looks artificially worse) and, separately, means the effect you measured at the tested traffic share will not hold at 100% rollout, because the resource contention itself scales with the treatment allocation percentage.

In both cases the estimate is not merely noisy, it is biased in a specific, name-able direction, and the bias would not shrink with more sample size because it comes from the assignment mechanism interacting with the product, not from sampling error.

Designing around a suspected violation

Once you suspect interference, the standard fix is to change the unit of randomization to one large enough to contain the spillover, i.e., cluster randomization: randomize by friend-group, geographic market, or server shard instead of by individual, so that most of the interference happens within a cluster (which is internally consistent, either all-treated or all-control) rather than across the treatment/control boundary.

Cluster randomization has its own cost, though: fewer independent units means higher variance for the same total traffic, since the effective sample size is closer to the number of clusters than the number of users. It also requires the outcome to be measurable and meaningful at the cluster level, and it does not eliminate the shared-capacity case unless the constrained resource is itself scoped per cluster.

Detecting a violation you did not design around

If the violation is discovered mid-experiment rather than anticipated, e.g., a caching-configuration bug causes some control users to intermittently render treatment-arm content, the diagnostic sequence is: quantify the leak rate first (what fraction of control exposures actually rendered the treatment experience, pulled directly from logs, not estimated), then decide whether the leak is small enough to bound the bias and proceed with a documented caveat, or large enough that the read is unusable and the fix is to patch the bug and rerun rather than to try to model the contamination away after the fact.

Worked example

A marketplace runs a two-armed test on a checkout redesign, expecting independent per-user outcomes. Mid-experiment, a shared caching layer bug is found: a same-session user occasionally gets served a stale cached page from the opposite arm. Pulling exposure logs, engineering finds this affected roughly 4% of control-arm page views (a number read directly from the cache-hit logs, not assumed). That is a version-of-treatment violation, not an interference violation: some "control" users received a materially different experience than the rest of control, so the control arm is not internally consistent. The team's next step is not to reweight or model this away, because the mechanism (a caching bug) has no principled correction; it is to fix the bug and rerun the experiment cleanly, treating the contaminated run as informative only about the presence of the bug.

Trade-offs and pitfalls

• Do not assume interference is symmetric or negligible just because the product does not look "social." Shared backend resources (queues, inventory, ranking models retrained on pooled data) create interference in products with no visible network feature.
• Cluster randomization trades bias for variance; do not adopt it reflexively for every experiment on a networked product when the actual interference is small relative to the direct effect, since you would be paying a real power cost for a small bias fix.
• A violation discovered after the fact is a data-quality incident, not a modeling problem to be adjusted away; resist the temptation to "correct" biased data with a post hoc statistical patch when the honest fix is to rerun cleanly.

Direct answer

The randomization unit should be the largest identity that is (a) stable over the experiment window and (b) matches the unit at which you will measure and report the outcome. For a homepage layout change with user-scoped conversion metrics, that is almost always user id when you have reliable logged-in identity; fall back to device id for logged-out mobile traffic, and treat cookie and session id as fallback-only units because they leak identity across the very boundary you are trying to hold fixed. The mechanism that turns "unit" into an actual bucket assignment is deterministic hash-based bucketing, and its main operational failure mode is touching the hash inputs (the salt or key) mid-experiment. Before any of that, though, you have to define who is even eligible to be in the experiment at all.

Structured elaboration

Defining the eligible population before choosing a unit

Unit choice is a second-order question; the first-order question is which units are even eligible to enter the experiment. For a mobile-only feature (say, a redesign shipped exclusively in the mobile app to a US audience), a desktop-only visitor cannot receive the treatment no matter which arm they land in, so randomizing across your full user base and then measuring outcomes at the account level silently dilutes the experiment: ineligible units get logged into both arms with a null "effect" (they cannot experience the change either way), which pulls the estimated treatment effect toward zero and inflates the sample size needed to detect a real one. The eligible population for a mobile-only US feature is the set of units that are (a) on the mobile platform that ships the feature, (b) in the targeted market (US), and (c) past whatever version or capability gate the feature requires; everyone outside that eligible population should be excluded from the experiment entirely, not folded into control by default. This is a distinct failure mode from picking the wrong unit: a design can choose a perfectly good unit (user id) and still be broken if a third of the "users" randomized into it were structurally incapable of ever seeing the treatment, whether the unit ultimately chosen within that eligible population is user, device, or session id.

Trade-offs by candidate unit

The two axes that matter are: does this unit stay attached to one treatment condition for the life of the experiment, and does it match the level at which you will later compute the metric. Session-level randomization on a homepage layout change fails both: a returning user can see version A on Monday and version B on Wednesday, so "the effect of the layout" is not well defined for that person, and if you then report a user-level conversion rate you are averaging over users who experienced a mix of both conditions.

Target-segment and control-group selection for a personalization test

Personalization experiments add a further wrinkle on top of eligibility and unit choice: because the treatment itself varies per person (each user's personalized experience differs from every other user's), you have to be explicit about two more things: which segment of the eligible population the test targets, and what the control group actually receives. A common setup: the target segment is the subset of eligible users with enough interaction history for the personalization model to act on (say, users with a minimum number of prior sessions); users below that threshold cannot be meaningfully personalized and should either be excluded from the test or routed to a defined fallback, rather than silently folded into a "control" group that has nothing to do with the personalization decision being tested. The control group, correspondingly, should receive a clearly defined non-personalized baseline (a fixed default ranking or layout), not "whatever the legacy system happened to show," so the measured effect is attributable to personalization itself rather than to incidental differences between the two code paths. Get target-segment or control-group definition wrong (an ill-specified segment boundary, or a control group that partially overlaps with treatment logic) and the measured lift reflects a spurious selection effect rather than the personalization algorithm's real value, no matter how correctly the underlying randomization unit and hash mechanism were implemented.

How hash-based deterministic bucketing works

In practice you do not store a per-user assignment row for every experiment. Instead you compute

$\text{bucket}(u) = \text{hash}(u \,\Vert\, \text{salt}) \bmod N$

where $u$ is the chosen unit id (user id, device id, etc.), the salt is a string unique to this experiment (often the experiment name or id), and $N$ is the number of buckets (commonly 100 or 1000 for fine-grained traffic allocation). Buckets are then mapped to arms, e.g. buckets 0-49 to control and 50-99 to treatment for a 50/50 split. Because the hash is deterministic, the same unit id always lands in the same bucket for the same salt, which is what makes the assignment reproducible without a lookup table, and salting per-experiment is what makes assignment to experiment A independent of assignment to experiment B (so the same user can be validly in many concurrent, non-interacting experiments).

Operational pitfalls

• Changing the salt or hashing key mid-experiment. This is the single most common self-inflicted wound. It re-shuffles every unit into a new bucket, silently reassigning some fraction of users from control to treatment (or the reverse) partway through. The experiment now mixes users with a clean single-arm history and users who were exposed to both arms, which is exactly the session-level contamination problem from the table above, except it is invisible unless you log assignment history.
• Reusing a salt across experiments. If two unrelated experiments accidentally share a salt (or one is a substring of the identifier used in the other), their bucket assignments become correlated instead of independent, which breaks the assumption that concurrent experiments do not interfere with each other.
• Changing $N$ or the bucket-to-arm mapping. Even without touching the salt, resizing the traffic split mid-flight (e.g., ramping from 5% to 50%) moves units across the arm boundary unless the mapping is designed to be monotonic (new traffic is added to existing arms rather than everyone being rehashed).
• Identity churn. A user id that gets merged, deleted, or re-issued (account merge, logout/login cycles that mint a new anonymous id) effectively becomes a new hash input mid-experiment, which has the same effect as a salt change for that user.

A finer-grained alternative: per-impression randomization

Every unit above is a person-shaped identity. Some teams instead randomize at the impression level, assigning a fresh coin flip to each page view or ranking request rather than to a person. This is occasionally used for high-frequency, low-persistence decisions (e.g., which of several ranking variants to serve on a given request) where you explicitly do not want a stable per-user experience. It is a different trade entirely from the table above: it eliminates any notion of "this user's assigned arm" (so it cannot answer a question about a durable, user-perceived change like a homepage layout), and it introduces strong intra-user correlation in the outcome data, since one person's many impressions are not independent draws, which inflates the effective variance if you naively treat impressions as independent observations in the analysis. Per-impression randomization is the right tool only when the thing being tested is meant to vary within a single user's experience; for a homepage layout, where the goal is to measure how a stable person-level experience changes behavior, it is the wrong granularity.

Detecting and correcting unit-mismatch after the fact

1. Assignment-churn audit. From the exposure logs, compute the fraction of units that were logged under more than one arm during the experiment window. A near-zero rate is expected; anything material indicates contamination.
2. Pre-period balance check. Compare the two arms on metrics measured before the experiment started (metrics that could not possibly be affected by treatment). An imbalance signals a broken randomization, not a broken hash necessarily, but it is the same diagnostic.
3. Sample ratio mismatch check on the realized split, i.e., does the observed 50/50 (or intended ratio) actually hold at the analysis unit. A skew is a strong signal that the bucketing pipeline itself misbehaved.
4. Timeline reconstruction. If churn is found, check the deployment log for the experiment: a salt, key, or bucket-count change on a specific date will produce a visible step change in the churn-rate-by-day series.
5. Correction paths, in order of preference. Analyze by first-observed assignment only (treat each unit's initial exposure as its assignment, i.e., an intention-to-treat style rule, and accept the resulting dilution of the effect estimate); if the break has a clean date, restrict the analysis window to the stable period before or after it; if contamination is pervasive, drop the experiment's results for the affected window and rerun rather than trying to model around a broken assignment mechanism, since any post hoc adjustment for a data-dependent unit-mismatch is itself a source of bias.

Worked example

Suppose an app-only feature was randomized by session id and you are asked to sanity-check it before trusting the readout. You pull exposure logs and count, per user, the distinct arms they were logged under: 92,000 users saw only control, 91,500 saw only treatment, and 6,500 saw both. Churn rate is $6{,}500 / (92{,}000+91{,}500+6{,}500) \approx 3.4\%$. That is a directly computed, reproducible number from the logs, not an assumption, and a value that high on a homepage-layout test (where the same person plausibly returns within the experiment window) is enough on its own to recommend re-running at user-id granularity rather than trying to salvage the session-level readout.

Trade-offs and pitfalls

• Choosing the "purest" unit (household) is not free: fewer independent units means higher variance per unit, so the same absolute effect needs more households than it would need users to reach the same precision. Unit choice is a bias-versus-noise trade, not a pure bias fix.
• A cookie- or device-based fallback is a compromise you should name explicitly to stakeholders, not a silent substitute for user id; report the estimated multi-device contamination rate alongside the headline result.
• An eligible population that is defined too loosely (e.g., randomizing all traffic instead of just the mobile-only, in-market segment) produces the same kind of diluted, biased-toward-zero readout as a bad unit choice, even when the unit itself is correct.
• Do not "fix" detected contamination by re-including the mixed-exposure users with a different weighting scheme chosen after seeing which way it moves the result; decide the exclusion or ITT rule before looking at the treatment effect.

Direct answer

The primary metric is the single metric that answers "did this change achieve its intended goal," and it is what the ship decision is nominally based on. Guardrail metrics are metrics you are not trying to improve, but are watching to make sure the change does not cause unacceptable harm elsewhere; a guardrail regressing can override a primary metric win. A secondary metric is different from both: it is additional signal you are curious about or want to understand mechanism through, but a secondary metric moving in a bad direction does not, by itself, block a ship decision the way a guardrail breach does. The distinction that matters operationally is that guardrails carry a pre-committed threshold and a pause-or-rollback consequence; secondary metrics do not.

Structured elaboration

Primary vs. guardrail vs. secondary

An equivalent framing some teams use is proximal vs. distal metrics: a proximal metric sits close to the mechanism of the change (click-through rate on a redesigned button) and moves quickly; a distal metric sits further downstream (long-term retention, lifetime value) and moves slowly but is closer to what the business actually cares about. A guardrail is frequently a distal metric precisely because the harm you are worried about (retention erosion, trust damage) is often slower to appear than the primary win.

Worked proposal for a monetization change (new ad placement)

Primary metric: net revenue per user in the experiment arm. Direction of success: increase. This is the metric the change exists to move.

Guardrail 1: 7-day retention. Direction of harm: decrease. Rationale: an intrusive placement can drive short-term revenue while quietly eroding the reason people come back. Pause/rollback trigger: agreed in advance as a stated relative-drop threshold with the confidence interval's upper bound also below zero (i.e., not just a point estimate dip that could be noise), reviewed before rollout, not chosen after seeing the result.

Guardrail 2: core-task completion rate (the product's main non-monetization action, e.g., completing a search, finishing a checkout, reading an article to completion). Direction of harm: decrease. Rationale: an ad placement that visually or functionally interferes with the primary task is trading long-run product health for short-run revenue.

Guardrail 3: user-initiated complaint or ad-block/opt-out rate. Direction of harm: increase. Rationale: a direct, unambiguous signal of user tolerance that is available faster than retention, useful as an early-warning guardrail even before the retention window has fully played out.

Guardrail 4 (optional, if the surface has one): page load or responsiveness regression, since an added placement can degrade performance in a way that suppresses every other metric indirectly; direction of harm: increase in load time or error rate.

This maps onto the same structure whether you are testing an ad placement, a checkout-flow revenue change (where the natural guardrail set expands to include cart-abandonment rate and support-ticket volume), or a premium-feature paywall (where conversion rate is typically the primary, and DAU, ARPU, and system error rate sit alongside it as guardrails against gating too aggressively or destabilizing the product). The framing also transfers outside pure monetization: for a conversational AI product's response pipeline, the primary might be task-completion rate while the guardrails are safety and quality signals such as a harmful-response rate or an unresolved-escalation rate, because the mechanics of "one thing you're optimizing, several things you refuse to let break" do not change with the domain.

Setting the threshold, not just naming the metric

A guardrail without a pre-committed threshold is not actually a guardrail, it is a chart someone glances at. The threshold should be set from business tolerance for harm (how much retention erosion is worth this much revenue) agreed before the experiment starts, not derived by re-deriving statistical power mid-flight; whether the observed guardrail movement is distinguishable from noise at that threshold is a separate, purely statistical question the analysis answers once data is in, not something this design step needs to resolve.

Worked example

A checkout-flow revenue experiment adds a one-click upsell at the payment step. The team pre-commits four guardrails before launch: cart-abandonment rate (harm: increase), 7-day repeat-purchase rate (harm: decrease), support-ticket volume tagged "checkout confusion" (harm: increase), and page load time at the payment step (harm: increase). Two weeks in, revenue per session is up and three of the four guardrails are flat, but cart-abandonment is up beyond the pre-committed trigger. Because the threshold and the pause rule were set before launch, the team pauses the rollout to investigate the upsell's placement rather than debating in the moment whether the abandonment increase is "bad enough" to matter.

Trade-offs and pitfalls

• Naming too many guardrails dilutes the signal and invites false alarms purely from checking many metrics at once; a handful of well-chosen, harm-specific guardrails beats a long generic list.
• Do not let a metric quietly slide from "secondary" to "guardrail" after the fact because it happened to move in a bad direction; that is choosing your rules after seeing the data, which defeats the purpose of pre-committing thresholds.
• A guardrail with no pre-committed threshold is not enforceable in the moment it matters; agree on the trigger, and who has authority to invoke it, before the experiment ships.