A/B Test Design & Statistical Rigor Questions

Direct answer

A heterogeneous treatment effect (HTE) is a real difference in a treatment's effect across subgroups, meaning a feature can genuinely help one segment and hurt another even when the overall average effect looks flat, because a flat average is just a weighted blend of both. The discipline that keeps this useful rather than a source of false discoveries is starting from a short list of subgroups chosen and written down before the test runs, based on a product hypothesis for why that segment might respond differently, rather than slicing every available dimension after the results come in and reporting whichever slice looks interesting.

Structured elaboration

Why a flat average can hide a real split

An average treatment effect (ATE) is a weighted average of segment-level effects. If segment A is half the traffic with a genuine +2.0 percentage point effect, and segment B is the other half with a genuine -1.6 percentage point effect, the pooled effect is:

$\text{ATE} = 0.5 \times 2.0 + 0.5 \times (-1.6) = 0.2 \text{ percentage points}$

A pooled +0.2pp result reads as flat or marginal, and a team that only looks at the ATE would conclude the feature does not matter, when in fact it is a real win for half the population and a real loss for the other half.

A disciplined workflow

1. Pre-specify the subgroup list before running the test. Choose it from a concrete product hypothesis, for example "new users lack context this feature assumes, so we expect a different response than returning users," not from "let's see what breaks out once we have the data." Keep the list short, typically a handful of segments, and write it into the analysis plan alongside the primary metric.
2. Power the subgroup analysis explicitly, not just the overall test. Segments are smaller than the full population, so a segment showing "no significant difference" is very often an underpowered null rather than evidence the segments behave the same; check the segment's own sample size against the effect size you would need to distinguish before treating a null subgroup result as informative.
3. Name the multiplicity problem and route around it, rather than re-deriving it here. Testing several pre-specified subgroups still inflates the chance of a false positive across the set, the same mechanism as testing several metrics; apply a standard multiplicity correction (family-wise or false-discovery-rate methods) to the pre-specified set, and treat any subgroup examined outside that pre-specified list as exploratory by default, no matter how it correlates with the metric.
4. Know the estimation toolkit without needing to build it here. For a short pre-specified list, a direct per-segment intent-to-treat comparison is usually enough. For flexible, higher-dimensional segmentation across many covariates at once, conditional average treatment effect (CATE, the treatment effect estimated for one particular slice of users rather than the population-wide average) estimation via meta-learners (model families purpose-built to estimate that per-slice effect from data) or uplift modeling (the applied name for the same goal: predicting who responds most to the treatment, not just whether the average user responds) is the standard toolkit; the discipline questions above (pre-specification, multiplicity, confirmatory follow-up) apply regardless of which estimation method produced the segment-level number.
5. Treat a post-hoc finding as a hypothesis, not a decision. A subgroup effect that survives pre-specification and multiplicity correction can inform a prioritization or personalization decision directly. A subgroup effect discovered by open-ended slicing after the fact, even a striking one, should be treated as hypothesis-generating only and routed to a dedicated confirmatory experiment on that segment before it drives a shipping decision.
6. Report findings with their status labeled. When presenting a subgroup result to stakeholders, state explicitly whether it was pre-specified or exploratory, and whether a confirmatory step is still required, so a segment finding does not get treated as settled fact before it has earned that status.

A concrete product scenario

A checkout redesign shows a flat, non-significant overall conversion effect. The team had pre-specified a new-user-versus-power-user split before the test, hypothesizing that power users already have an efficient checkout habit that a redesign would disrupt while new users would benefit from the clearer flow. The pre-specified interaction test confirms a real, multiplicity-corrected split matching the illustration above: a genuine gain for new users and a genuine loss for power users. The resulting decision is neither "ship to everyone" (which would hurt power users) nor "scrap the redesign" (which would forgo a real win for new users), but a personalization decision: ship the new checkout to new users only, keep power users on the existing flow, and treat that as the actual outcome of the experiment rather than a footnote to a "no effect" headline.

Trade-offs & pitfalls

• Confusing exploratory with confirmed. The single most common failure mode is presenting a striking post-hoc slice with the same confidence as a pre-specified, corrected result; the two need visibly different treatment in any readout.
• Underpowered subgroup nulls read as "no heterogeneity." A segment too small to detect the effect size in question will always look flat, whether or not a real difference exists; check the power before concluding homogeneity.
• Over-narrow personalization from a single test. One HTE finding is evidence for a segment-specific policy, not proof it will hold up over time or across other metrics; a confirmatory follow-up before fully committing production logic to a segment split is cheap insurance against a finding that was itself a fluke.
• Skipping pre-specification because "we'll just correct for multiplicity later." A multiplicity correction controls the false-positive rate across a stated set of comparisons; it does not rescue a search that had no defined stopping point in the first place.

Direct answer

A lift that spikes in week one and fades to flat by week two could be three different things wearing the same shape: a real novelty effect decaying to its true (smaller or zero) persistent level, a seasonal pattern that has nothing to do with the treatment and would show up in both arms if you looked, or a selection-bias artifact where the traffic source itself (a holiday push or a marketing campaign) skewed an unrepresentative mix of users into treatment. The fastest way to tell them apart is to check whether the decay tracks calendar time (seasonality), user exposure-age (novelty), or arm composition (selection bias), because each explanation leaves a distinct fingerprint in the data, not just a distinct story.

Structured elaboration

The checklist, in order

Disentangling novelty from selection bias when a marketing campaign is the traffic source

This case is harder because both explanations can be true at once and produce the same decaying shape. A marketing campaign that ramps down after week one changes who is arriving, not just how they behave: if campaign-driven traffic is disproportionately assigned into treatment (for example because the campaign linked directly into the treatment experience, or assignment happened downstream of a referral parameter correlated with arm), the week-one spike partially reflects a different, more click-happy population rather than a within-user novelty effect.

• Design fix: randomize on a unit and at a point in the funnel that is independent of campaign exposure, assigning before the user ever sees the campaign-specific landing experience, so acquisition-channel composition is balanced by construction rather than by hope.
• Post-hoc check: compare the new-vs-returning mix and traffic-source mix between arms week by week; if the treatment arm's share of campaign-driven new users is higher than control's, run a balance check on that specific subpopulation, not just the top-line allocation.
• Post-hoc correction: re-run the primary analysis restricted to users who arrived through non-campaign channels, and separately on campaign-driven users; if the effect concentrates in the campaign-driven segment and that segment is imbalanced between arms, the pooled week-one number is not a trustworthy estimate of the treatment effect for the general population.

Redesigning the experiment or analysis window

• Extend the primary analysis window well past the campaign's active period so the segment mix has time to normalize, and pre-register that the primary readout is a later window, not week one.
• Use a difference-in-differences framing (treatment-arm change from a pre-period baseline, minus control-arm change over the same calendar span) so genuine calendar movement common to both arms cancels out instead of being misread as a treatment effect.
• If holiday timing cannot be avoided, consider a staggered or replicated launch (the same treatment introduced to a second, non-holiday cohort later) so you get a second read that is not confounded with that specific calendar event.

Worked example

Suppose the pooled week-one lift is +8% on a baseline conversion rate of 10%, and segment analysis shows the treatment arm received 60% campaign-driven new users in week one versus 40% in control (stated, illustrative inputs). If campaign-driven new users convert at a naturally higher rate, say 14% versus 9% for organic users (illustrative inputs), the arm-level blended rate is a weighted average:
$\text{Treatment rate} = 0.6 \times 0.14 + 0.4 \times 0.09 = 0.084 + 0.036 = 0.120$
$\text{Control rate} = 0.4 \times 0.14 + 0.6 \times 0.09 = 0.056 + 0.054 = 0.110$
That gives an apparent lift of $(0.120 - 0.110)/0.110 \approx 9.1\%$, almost entirely explained by the composition difference rather than any within-user treatment effect, before even considering a real novelty component. This is why segment analysis has to run before you trust the topline number, not after.

Trade-offs and pitfalls

• Restricting analysis to non-campaign traffic to remove selection bias also shrinks your sample and can push you back into an underpowered read, exactly what the power-review step is meant to catch.
• Difference-in-differences assumes both arms would have moved in parallel absent the treatment; a campaign that specifically targets one arm's users breaks that assumption too, so DiD is not a free fix if the selection bias operates through campaign-to-arm assignment rather than through time.
• Extending the window to wait out both seasonality and novelty delays the launch decision; be explicit with stakeholders that "flat by week two" was never a valid stopping rule for this scenario, so the delay does not read as backpedaling.
• Retiring a variant because week two looks flat, without a power review, risks killing a real but modest persistent effect that a two-week window was never sized to detect in the first place.

Direct answer

When assignment happens per-device (a cookie on web, a device id on the app) but the real unit of interest is the person, users who touch the product on multiple devices get assigned independently on each device, so some of them land in both control and treatment at once. That breaks the assumption that each experimental unit receives exactly one arm: it dilutes the measured treatment effect, since a "contaminated" user's behavior is influenced by both arms, and it can double count outcomes if the same person's actions are logged and analyzed once per device-identity rather than once per person. The fix is to randomize and log at the most persistent identity you actually have, then dedupe and correct for the identities you had to resolve after the fact rather than at assignment time.

Structured elaboration

Where the bias enters

• Assignment stage: a user with a web cookie and a mobile device id gets two independent coin flips. If both land the same way, no harm; if they split, that person is exposed to treatment and control simultaneously, a violation of SUTVA (the assumption that one unit's outcome does not depend on another instance of its own assignment).
• Analysis stage: if the analysis unit is "device" rather than "resolved person," a contaminated person's activity appears once in each arm's numerator, and the two rows are not independent observations even though the analysis code treats them as such, which understates the true variance.
• Coverage bias: identity resolution itself makes mistakes (false merges linking two different people, false splits treating one person as two); if those errors are not random with respect to treatment, they introduce their own bias on top of the contamination.

Practical strategies

1. Randomize at the most stable identity you have. Prefer a logged-in account id over a device id or cookie whenever the user is authenticated; fall back to a deterministic device hash only for logged-out traffic, and treat that population as a separate, lower-confidence stratum in reporting.
2. Log everything needed to resolve identity after the fact. Persist device id, cookie id, and account id (hashed, respecting privacy) on every assignment and every outcome event, even when the assignment itself was made at device level, so contamination can be measured and corrected during analysis rather than discovered too late.
3. Define the primary analysis on the resolved (canonical) identity, not the raw assignment record: after backend identity merge, collapse a contaminated user into a single row and apply an explicit, pre-registered rule for what arm they count as, for example "any-device-treatment counts as treated," reported alongside a stricter "single-device-only" rule as a sensitivity check.
4. Quantify and bound the bias rather than ignore it. Report the primary result plus at least two sensitivity analyses: one restricted to users seen on exactly one device (removes contamination but shrinks the sample), and one using the any-device-treatment rule (keeps the full sample but is a diluted estimate of the true per-exposure effect).
5. Use cluster-robust standard errors at the resolved-identity level so that outcomes from the same person are not treated as independent observations even after collapsing to one row per person, since a person can still contribute multiple sessions or events.

Worked example

Assume 20% of users are active on exactly two devices (web and app) and 80% are single-device (an illustrative, stated split). Assignment happens independently per device with probability 0.5 to treatment. For a two-device user, the four equally likely device-pair outcomes are (control, control), (control, treatment), (treatment, control), (treatment, treatment), each with probability 0.25:
$P(\text{both control}) = P(\text{both treatment}) = 0.25, \qquad P(\text{split, i.e. contaminated}) = 0.5$
So among the 20% of users who are two-device, half get split across arms, which is 10% of the total user base. If a contaminated user's outcome is counted in both the treatment and control totals rather than resolved to one arm, then 10% of the treatment-arm numerator and 10% of the control-arm numerator are contributed by the exact same set of people, which both understates the between-arm difference and violates the independence assumption behind the standard error calculation. Restricting the primary analysis to the 90% of users who are single-device or resolve cleanly to one arm removes the contamination at the cost of 10% of the sample, which should be reflected directly in the power calculation for the corrected analysis.

Trade-offs and pitfalls

• The "any-device-treatment" rule is conservative and interpretable but structurally dilutes the estimated effect toward zero for contaminated users, since they experienced a mix of both arms; don't present it as an unbiased estimate of the pure per-exposure effect.
• Restricting to single-device users is cleaner statistically but changes who the estimate applies to: if multi-device users differ systematically (often more engaged, higher-value), the single-device estimate may not generalize to the full user base.
• Identity resolution is itself a model with false-merge and false-split error rates; a resolution pipeline retrained or changed mid-experiment can shift the contamination rate over time and should be monitored, not assumed constant.
• Logging every identifier needed for later resolution has real privacy and storage cost; agree on hashing and retention policy with the privacy function before instrumenting, not after a contamination investigation is already underway.

Direct answer

The primary metric is the single metric that answers "did this change achieve its intended goal," and it is what the ship decision is nominally based on. Guardrail metrics are metrics you are not trying to improve, but are watching to make sure the change does not cause unacceptable harm elsewhere; a guardrail regressing can override a primary metric win. A secondary metric is different from both: it is additional signal you are curious about or want to understand mechanism through, but a secondary metric moving in a bad direction does not, by itself, block a ship decision the way a guardrail breach does. The distinction that matters operationally is that guardrails carry a pre-committed threshold and a pause-or-rollback consequence; secondary metrics do not.

Structured elaboration

Primary vs. guardrail vs. secondary

An equivalent framing some teams use is proximal vs. distal metrics: a proximal metric sits close to the mechanism of the change (click-through rate on a redesigned button) and moves quickly; a distal metric sits further downstream (long-term retention, lifetime value) and moves slowly but is closer to what the business actually cares about. A guardrail is frequently a distal metric precisely because the harm you are worried about (retention erosion, trust damage) is often slower to appear than the primary win.

Worked proposal for a monetization change (new ad placement)

Primary metric: net revenue per user in the experiment arm. Direction of success: increase. This is the metric the change exists to move.

Guardrail 1: 7-day retention. Direction of harm: decrease. Rationale: an intrusive placement can drive short-term revenue while quietly eroding the reason people come back. Pause/rollback trigger: agreed in advance as a stated relative-drop threshold with the confidence interval's upper bound also below zero (i.e., not just a point estimate dip that could be noise), reviewed before rollout, not chosen after seeing the result.

Guardrail 2: core-task completion rate (the product's main non-monetization action, e.g., completing a search, finishing a checkout, reading an article to completion). Direction of harm: decrease. Rationale: an ad placement that visually or functionally interferes with the primary task is trading long-run product health for short-run revenue.

Guardrail 3: user-initiated complaint or ad-block/opt-out rate. Direction of harm: increase. Rationale: a direct, unambiguous signal of user tolerance that is available faster than retention, useful as an early-warning guardrail even before the retention window has fully played out.

Guardrail 4 (optional, if the surface has one): page load or responsiveness regression, since an added placement can degrade performance in a way that suppresses every other metric indirectly; direction of harm: increase in load time or error rate.

This maps onto the same structure whether you are testing an ad placement, a checkout-flow revenue change (where the natural guardrail set expands to include cart-abandonment rate and support-ticket volume), or a premium-feature paywall (where conversion rate is typically the primary, and DAU, ARPU, and system error rate sit alongside it as guardrails against gating too aggressively or destabilizing the product). The framing also transfers outside pure monetization: for a conversational AI product's response pipeline, the primary might be task-completion rate while the guardrails are safety and quality signals such as a harmful-response rate or an unresolved-escalation rate, because the mechanics of "one thing you're optimizing, several things you refuse to let break" do not change with the domain.

Setting the threshold, not just naming the metric

A guardrail without a pre-committed threshold is not actually a guardrail, it is a chart someone glances at. The threshold should be set from business tolerance for harm (how much retention erosion is worth this much revenue) agreed before the experiment starts, not derived by re-deriving statistical power mid-flight; whether the observed guardrail movement is distinguishable from noise at that threshold is a separate, purely statistical question the analysis answers once data is in, not something this design step needs to resolve.

Worked example

A checkout-flow revenue experiment adds a one-click upsell at the payment step. The team pre-commits four guardrails before launch: cart-abandonment rate (harm: increase), 7-day repeat-purchase rate (harm: decrease), support-ticket volume tagged "checkout confusion" (harm: increase), and page load time at the payment step (harm: increase). Two weeks in, revenue per session is up and three of the four guardrails are flat, but cart-abandonment is up beyond the pre-committed trigger. Because the threshold and the pause rule were set before launch, the team pauses the rollout to investigate the upsell's placement rather than debating in the moment whether the abandonment increase is "bad enough" to matter.

Trade-offs and pitfalls

• Naming too many guardrails dilutes the signal and invites false alarms purely from checking many metrics at once; a handful of well-chosen, harm-specific guardrails beats a long generic list.
• Do not let a metric quietly slide from "secondary" to "guardrail" after the fact because it happened to move in a bad direction; that is choosing your rules after seeing the data, which defeats the purpose of pre-committing thresholds.
• A guardrail with no pre-committed threshold is not enforceable in the moment it matters; agree on the trigger, and who has authority to invoke it, before the experiment ships.

Direct answer

An A/A test randomly splits traffic into two groups that receive the identical experience and compares their metrics as if they were a real experiment. You run one to validate the assignment and measurement pipeline before trusting a real A/B result on the same platform: since both groups get the same product, any statistically significant difference between them signals a problem in the pipeline (randomization, instrumentation, or analysis) rather than a real effect, because by construction there is no effect to detect.

Structured elaboration

Two valid use cases

• Validating the assignment and instrumentation pipeline. Confirms that the bucketing hash actually produces the intended split ratio, that each unit sees a stable, single experience, and that event logging correctly attributes actions to the assigned arm end to end (client instrumentation through to the analysis table).
• Establishing a baseline-variance estimate. Because there is no true effect, the spread of the A/A metric difference across many runs (or across a well-chosen resampling of the same data) tells you what "just noise" looks like for this metric on this population, which is useful input for planning: it is a sanity check on your variance assumptions, not a substitute for a proper power calculation.

What to check while it runs

• The realized split ratio against the intended one (a sample-ratio check): meaningfully off the intended ratio (say, 50/50 skewing to 49/51 in a way that recurs, not a single noisy day) points at a bucketing bug before you even look at outcome metrics.
• Core funnel and event counts by arm (sessions, page views, primary conversion event) to confirm the two arms are tracked with equal fidelity, not just equal traffic.
• Whether the metric of interest for the upcoming real experiment behaves as expected in the A/A read, since that is the metric whose baseline variance you actually need.
• Run it for at least one full natural cycle of the traffic (typically a full week, to span weekday/weekend mix) rather than a single day, since a one-day A/A window can look clean by luck or flagged by a day-specific anomaly that has nothing to do with the platform.

An A/A test is, at its core, a targeted way to surface three distinct failure classes: instrumentation errors (events not logged or misattributed), non-random assignment (the bucketing hash is not producing a genuinely random, independent split), and sampling biases (the two arms end up systematically different in composition despite a technically-random split, e.g., a bot-filtering rule that behaves differently by arm). Each class points at a different fix, which is why segmenting the flagged difference (below) matters more than the raw significance flag itself.

Two limitations or common misinterpretations

• A clean A/A result is not proof the pipeline is bug-free. With enough traffic, small true differences in a specific test can still slip through if the bug is intermittent (e.g., only affects a rare browser) or if the metric checked in the A/A test is not the one that will matter in the real experiment. Absence of a flagged difference is reassurance, not a guarantee.
• A single significant A/A result does not, by itself, mean the pipeline is broken. At a conventional significance threshold, some fraction of A/A tests will show a "significant" difference purely by chance even with a perfectly correct pipeline; treat one flagged metric as a prompt to investigate, not as an automatic verdict, especially if you are checking many metrics at once and did not correct for that.

Root-causing a significant A/A result

1. Recheck the sample ratio first. A skewed split is the fastest, most common finding and points straight at a bucketing bug rather than a downstream measurement issue.
2. Segment the difference. Break the flagged metric down by platform, geography, and new-vs-returning user; a difference concentrated in one segment (e.g., one app version) points at an instrumentation bug specific to that segment rather than a global randomization failure.
3. Check for a known confound in how the two arms are served, such as one arm being disproportionately served through a code path with different latency or caching behavior, which is functionally a version-of-treatment bug even though no real treatment was intended.
4. Re-run before escalating, if the first read used a short window: a single noisy day is a weaker signal than a difference that persists across multiple independent A/A windows.
5. If it persists and is not explained by a segment or a known bug, treat the underlying real-experiment platform as unvalidated until the discrepancy is resolved; shipping A/B decisions on top of an unexplained A/A anomaly defeats the purpose of running the check at all.

Worked example

A team runs an A/A test ahead of a planned homepage experiment and flags a significant difference in click-through rate. Step 1, sample ratio: 50.1% vs 49.9%, within normal noise, so not a bucketing problem. Step 2, segmentation: the CTR gap is near zero on Android and web but noticeably present on iOS. Step 3: engineering finds one arm's iOS client is on an older app version with a slightly different default tab order, an artifact of how the A/A test's client-side flag was staged rather than anything about the experiment platform itself. The root cause is a version-of-treatment bug traced to a real, checkable fact (the iOS staging config), not a p-value alone; the fix is correcting the staged rollout, not adjusting the metric.

Trade-offs and pitfalls

• Running A/A tests constantly, on every metric, invites exactly the false-alarm problem described above; use them at meaningful checkpoints (new platform, new metric pipeline, post-incident) rather than as a standing tax on every experiment.
• Do not use a single A/A run's variance estimate as your only power-planning input if you have a more direct historical baseline available; treat it as a cross-check.
• A quiet A/A test on a low-traffic metric provides much weaker reassurance than the same result on a high-traffic metric, because a real problem of a given size is harder to detect with less data; do not treat "clean" as equally strong evidence across metrics of very different volume.