A/B Test Design & Statistical Rigor Questions

Direct answer

The randomization unit should be the largest identity that is (a) stable over the experiment window and (b) matches the unit at which you will measure and report the outcome. For a homepage layout change with user-scoped conversion metrics, that is almost always user id when you have reliable logged-in identity; fall back to device id for logged-out mobile traffic, and treat cookie and session id as fallback-only units because they leak identity across the very boundary you are trying to hold fixed. The mechanism that turns "unit" into an actual bucket assignment is deterministic hash-based bucketing, and its main operational failure mode is touching the hash inputs (the salt or key) mid-experiment. Before any of that, though, you have to define who is even eligible to be in the experiment at all.

Structured elaboration

Defining the eligible population before choosing a unit

Unit choice is a second-order question; the first-order question is which units are even eligible to enter the experiment. For a mobile-only feature (say, a redesign shipped exclusively in the mobile app to a US audience), a desktop-only visitor cannot receive the treatment no matter which arm they land in, so randomizing across your full user base and then measuring outcomes at the account level silently dilutes the experiment: ineligible units get logged into both arms with a null "effect" (they cannot experience the change either way), which pulls the estimated treatment effect toward zero and inflates the sample size needed to detect a real one. The eligible population for a mobile-only US feature is the set of units that are (a) on the mobile platform that ships the feature, (b) in the targeted market (US), and (c) past whatever version or capability gate the feature requires; everyone outside that eligible population should be excluded from the experiment entirely, not folded into control by default. This is a distinct failure mode from picking the wrong unit: a design can choose a perfectly good unit (user id) and still be broken if a third of the "users" randomized into it were structurally incapable of ever seeing the treatment, whether the unit ultimately chosen within that eligible population is user, device, or session id.

Trade-offs by candidate unit

The two axes that matter are: does this unit stay attached to one treatment condition for the life of the experiment, and does it match the level at which you will later compute the metric. Session-level randomization on a homepage layout change fails both: a returning user can see version A on Monday and version B on Wednesday, so "the effect of the layout" is not well defined for that person, and if you then report a user-level conversion rate you are averaging over users who experienced a mix of both conditions.

Target-segment and control-group selection for a personalization test

Personalization experiments add a further wrinkle on top of eligibility and unit choice: because the treatment itself varies per person (each user's personalized experience differs from every other user's), you have to be explicit about two more things: which segment of the eligible population the test targets, and what the control group actually receives. A common setup: the target segment is the subset of eligible users with enough interaction history for the personalization model to act on (say, users with a minimum number of prior sessions); users below that threshold cannot be meaningfully personalized and should either be excluded from the test or routed to a defined fallback, rather than silently folded into a "control" group that has nothing to do with the personalization decision being tested. The control group, correspondingly, should receive a clearly defined non-personalized baseline (a fixed default ranking or layout), not "whatever the legacy system happened to show," so the measured effect is attributable to personalization itself rather than to incidental differences between the two code paths. Get target-segment or control-group definition wrong (an ill-specified segment boundary, or a control group that partially overlaps with treatment logic) and the measured lift reflects a spurious selection effect rather than the personalization algorithm's real value, no matter how correctly the underlying randomization unit and hash mechanism were implemented.

How hash-based deterministic bucketing works

In practice you do not store a per-user assignment row for every experiment. Instead you compute

$\text{bucket}(u) = \text{hash}(u \,\Vert\, \text{salt}) \bmod N$

where $u$ is the chosen unit id (user id, device id, etc.), the salt is a string unique to this experiment (often the experiment name or id), and $N$ is the number of buckets (commonly 100 or 1000 for fine-grained traffic allocation). Buckets are then mapped to arms, e.g. buckets 0-49 to control and 50-99 to treatment for a 50/50 split. Because the hash is deterministic, the same unit id always lands in the same bucket for the same salt, which is what makes the assignment reproducible without a lookup table, and salting per-experiment is what makes assignment to experiment A independent of assignment to experiment B (so the same user can be validly in many concurrent, non-interacting experiments).

Operational pitfalls

• Changing the salt or hashing key mid-experiment. This is the single most common self-inflicted wound. It re-shuffles every unit into a new bucket, silently reassigning some fraction of users from control to treatment (or the reverse) partway through. The experiment now mixes users with a clean single-arm history and users who were exposed to both arms, which is exactly the session-level contamination problem from the table above, except it is invisible unless you log assignment history.
• Reusing a salt across experiments. If two unrelated experiments accidentally share a salt (or one is a substring of the identifier used in the other), their bucket assignments become correlated instead of independent, which breaks the assumption that concurrent experiments do not interfere with each other.
• Changing $N$ or the bucket-to-arm mapping. Even without touching the salt, resizing the traffic split mid-flight (e.g., ramping from 5% to 50%) moves units across the arm boundary unless the mapping is designed to be monotonic (new traffic is added to existing arms rather than everyone being rehashed).
• Identity churn. A user id that gets merged, deleted, or re-issued (account merge, logout/login cycles that mint a new anonymous id) effectively becomes a new hash input mid-experiment, which has the same effect as a salt change for that user.

A finer-grained alternative: per-impression randomization

Every unit above is a person-shaped identity. Some teams instead randomize at the impression level, assigning a fresh coin flip to each page view or ranking request rather than to a person. This is occasionally used for high-frequency, low-persistence decisions (e.g., which of several ranking variants to serve on a given request) where you explicitly do not want a stable per-user experience. It is a different trade entirely from the table above: it eliminates any notion of "this user's assigned arm" (so it cannot answer a question about a durable, user-perceived change like a homepage layout), and it introduces strong intra-user correlation in the outcome data, since one person's many impressions are not independent draws, which inflates the effective variance if you naively treat impressions as independent observations in the analysis. Per-impression randomization is the right tool only when the thing being tested is meant to vary within a single user's experience; for a homepage layout, where the goal is to measure how a stable person-level experience changes behavior, it is the wrong granularity.

Detecting and correcting unit-mismatch after the fact

1. Assignment-churn audit. From the exposure logs, compute the fraction of units that were logged under more than one arm during the experiment window. A near-zero rate is expected; anything material indicates contamination.
2. Pre-period balance check. Compare the two arms on metrics measured before the experiment started (metrics that could not possibly be affected by treatment). An imbalance signals a broken randomization, not a broken hash necessarily, but it is the same diagnostic.
3. Sample ratio mismatch check on the realized split, i.e., does the observed 50/50 (or intended ratio) actually hold at the analysis unit. A skew is a strong signal that the bucketing pipeline itself misbehaved.
4. Timeline reconstruction. If churn is found, check the deployment log for the experiment: a salt, key, or bucket-count change on a specific date will produce a visible step change in the churn-rate-by-day series.
5. Correction paths, in order of preference. Analyze by first-observed assignment only (treat each unit's initial exposure as its assignment, i.e., an intention-to-treat style rule, and accept the resulting dilution of the effect estimate); if the break has a clean date, restrict the analysis window to the stable period before or after it; if contamination is pervasive, drop the experiment's results for the affected window and rerun rather than trying to model around a broken assignment mechanism, since any post hoc adjustment for a data-dependent unit-mismatch is itself a source of bias.

Worked example

Suppose an app-only feature was randomized by session id and you are asked to sanity-check it before trusting the readout. You pull exposure logs and count, per user, the distinct arms they were logged under: 92,000 users saw only control, 91,500 saw only treatment, and 6,500 saw both. Churn rate is $6{,}500 / (92{,}000+91{,}500+6{,}500) \approx 3.4\%$. That is a directly computed, reproducible number from the logs, not an assumption, and a value that high on a homepage-layout test (where the same person plausibly returns within the experiment window) is enough on its own to recommend re-running at user-id granularity rather than trying to salvage the session-level readout.

Trade-offs and pitfalls

• Choosing the "purest" unit (household) is not free: fewer independent units means higher variance per unit, so the same absolute effect needs more households than it would need users to reach the same precision. Unit choice is a bias-versus-noise trade, not a pure bias fix.
• A cookie- or device-based fallback is a compromise you should name explicitly to stakeholders, not a silent substitute for user id; report the estimated multi-device contamination rate alongside the headline result.
• An eligible population that is defined too loosely (e.g., randomizing all traffic instead of just the mobile-only, in-market segment) produces the same kind of diluted, biased-toward-zero readout as a bad unit choice, even when the unit itself is correct.
• Do not "fix" detected contamination by re-including the mixed-exposure users with a different weighting scheme chosen after seeing which way it moves the result; decide the exclusion or ITT rule before looking at the treatment effect.

Direct answer

The primary metric is the single metric that answers "did this change achieve its intended goal," and it is what the ship decision is nominally based on. Guardrail metrics are metrics you are not trying to improve, but are watching to make sure the change does not cause unacceptable harm elsewhere; a guardrail regressing can override a primary metric win. A secondary metric is different from both: it is additional signal you are curious about or want to understand mechanism through, but a secondary metric moving in a bad direction does not, by itself, block a ship decision the way a guardrail breach does. The distinction that matters operationally is that guardrails carry a pre-committed threshold and a pause-or-rollback consequence; secondary metrics do not.

Structured elaboration

Primary vs. guardrail vs. secondary

An equivalent framing some teams use is proximal vs. distal metrics: a proximal metric sits close to the mechanism of the change (click-through rate on a redesigned button) and moves quickly; a distal metric sits further downstream (long-term retention, lifetime value) and moves slowly but is closer to what the business actually cares about. A guardrail is frequently a distal metric precisely because the harm you are worried about (retention erosion, trust damage) is often slower to appear than the primary win.

Worked proposal for a monetization change (new ad placement)

Primary metric: net revenue per user in the experiment arm. Direction of success: increase. This is the metric the change exists to move.

Guardrail 1: 7-day retention. Direction of harm: decrease. Rationale: an intrusive placement can drive short-term revenue while quietly eroding the reason people come back. Pause/rollback trigger: agreed in advance as a stated relative-drop threshold with the confidence interval's upper bound also below zero (i.e., not just a point estimate dip that could be noise), reviewed before rollout, not chosen after seeing the result.

Guardrail 2: core-task completion rate (the product's main non-monetization action, e.g., completing a search, finishing a checkout, reading an article to completion). Direction of harm: decrease. Rationale: an ad placement that visually or functionally interferes with the primary task is trading long-run product health for short-run revenue.

Guardrail 3: user-initiated complaint or ad-block/opt-out rate. Direction of harm: increase. Rationale: a direct, unambiguous signal of user tolerance that is available faster than retention, useful as an early-warning guardrail even before the retention window has fully played out.

Guardrail 4 (optional, if the surface has one): page load or responsiveness regression, since an added placement can degrade performance in a way that suppresses every other metric indirectly; direction of harm: increase in load time or error rate.

This maps onto the same structure whether you are testing an ad placement, a checkout-flow revenue change (where the natural guardrail set expands to include cart-abandonment rate and support-ticket volume), or a premium-feature paywall (where conversion rate is typically the primary, and DAU, ARPU, and system error rate sit alongside it as guardrails against gating too aggressively or destabilizing the product). The framing also transfers outside pure monetization: for a conversational AI product's response pipeline, the primary might be task-completion rate while the guardrails are safety and quality signals such as a harmful-response rate or an unresolved-escalation rate, because the mechanics of "one thing you're optimizing, several things you refuse to let break" do not change with the domain.

Setting the threshold, not just naming the metric

A guardrail without a pre-committed threshold is not actually a guardrail, it is a chart someone glances at. The threshold should be set from business tolerance for harm (how much retention erosion is worth this much revenue) agreed before the experiment starts, not derived by re-deriving statistical power mid-flight; whether the observed guardrail movement is distinguishable from noise at that threshold is a separate, purely statistical question the analysis answers once data is in, not something this design step needs to resolve.

Worked example

A checkout-flow revenue experiment adds a one-click upsell at the payment step. The team pre-commits four guardrails before launch: cart-abandonment rate (harm: increase), 7-day repeat-purchase rate (harm: decrease), support-ticket volume tagged "checkout confusion" (harm: increase), and page load time at the payment step (harm: increase). Two weeks in, revenue per session is up and three of the four guardrails are flat, but cart-abandonment is up beyond the pre-committed trigger. Because the threshold and the pause rule were set before launch, the team pauses the rollout to investigate the upsell's placement rather than debating in the moment whether the abandonment increase is "bad enough" to matter.

Trade-offs and pitfalls

• Naming too many guardrails dilutes the signal and invites false alarms purely from checking many metrics at once; a handful of well-chosen, harm-specific guardrails beats a long generic list.
• Do not let a metric quietly slide from "secondary" to "guardrail" after the fact because it happened to move in a bad direction; that is choosing your rules after seeing the data, which defeats the purpose of pre-committing thresholds.
• A guardrail with no pre-committed threshold is not enforceable in the moment it matters; agree on the trigger, and who has authority to invoke it, before the experiment ships.

Direct answer

A novelty effect is a temporary inflation of an early treatment effect: users explore or click on something purely because it is new, and that extra engagement fades once the feature stops being novel, biasing an early readout upward. A primacy effect (sometimes called a change-aversion or resistance-to-change effect) is the opposite pattern: a change disrupts a habitual workflow, so users are temporarily worse off while they relearn it, biasing an early readout downward, then the effect climbs toward its true level as users adapt. Both biases fade over roughly the same kind of horizon, so trusting a week-one number without checking its trajectory can make you launch a fad or kill a genuine win too early.

Structured elaboration

Mechanism and direction

Diagnostics to tell a spike from a persistent effect

• Time-windowed effect plot: daily or weekly treatment effect with confidence intervals, ideally with a smoothed trend line (LOESS or a spline), not a single pooled average. A genuine effect looks like a roughly flat band around a nonzero value; novelty looks like a spike that decays toward that band; primacy looks like a trough that rises toward it.
• Exposure-age cohorts, not calendar time: plot the effect against days since each user's first exposure for a fixed cohort of users first exposed on the same day, rather than calendar date. A calendar-time plot mixes newly exposed users (still novel-biased) with long-exposed users (already stabilized) every single day, which can mask a real decay curve as a flat line.
• New vs. returning user split: novelty is usually concentrated in users encountering the feature for the first time; if the effect is similar in a segment already exposed for weeks, that argues against novelty as the explanation.
• Change-point or decay model on the daily series: fit a time-varying effect model, effect as a function of exposure age, and test whether the transient component is statistically distinguishable from zero, separately from the asymptotic (persistent) component.
• Placebo check: run the same time-windowed analysis on a pre-launch period with no real treatment; if spike-like patterns appear there too, the "decay" you see in the real experiment may just be normal week-to-week noise, not a novelty artifact.

Adjusting duration and analysis

• Pre-register the analysis window before launch rather than reading the metric the moment it looks good; a fixed rule such as "primary read is the average effect over exposure-days 21 to 35" prevents cherry-picking the peak or the trough.
• Extend the experiment until the exposure-age curve visibly plateaus, or the fitted transient component's confidence interval crosses zero, rather than for a fixed calendar duration chosen in advance.
• Report both the early-window and late-window effect side by side rather than a single blended number; a launch decision based only on the blended average silently averages a fading spike with a stabilizing floor.

Worked example

Two hypothetical (illustrative, not real study data) weekly average-treatment-effect readings for the same nominal conversion metric:

Both curves are converging toward roughly the same persistent level, one from above and one from below, which is exactly the signature that separates them from a flat, genuine effect that would show roughly the same number every week within noise.

If the transient component decays exponentially, $\Delta(t) = C + A e^{-\lambda t}$, where $A$ is the size of the initial novelty or primacy spike above the persistent effect $C$ (the extra amount present at $t=0$ that fades away over time), and the illustrative decay rate is $\lambda = 0.2$ per week, its half-life is:
$t_{1/2} = \frac{\ln 2}{\lambda} = \frac{0.693}{0.2} \approx 3.5 \text{ weeks}$
That is the kind of number worth pre-registering as a decision rule: run at least three half-lives (about 10 to 11 weeks here) before reading the persistent effect $C$, rather than picking an arbitrary duration.

Trade-offs and pitfalls

• Waiting out a full decay curve costs calendar time and opportunity cost on other experiments; for low-stakes features, teams sometimes accept the risk of a novelty-inflated launch decision rather than run for months.
• Segmenting by exposure age needs per-user first-exposure timestamps captured in the assignment log; if you only log calendar-date rollups, you cannot separate calendar effects from exposure-age effects after the fact.
• A curve that looks like decay can just as easily reflect unrelated seasonality (marketing pushes, holidays) that correlates with launch timing; a decay-shaped curve is suggestive, not conclusive, on its own.
• Don't assume every early spike is novelty and every early trough is resistance to change: an early spike can be a genuine effect solving a pent-up need immediately, and an early trough can be a real bug that later gets patched. The pattern is evidence, not proof, and should be paired with qualitative checks (support tickets, session recordings) before concluding the mechanism.

Direct answer

Treat a single surprising segment finding, mobile shows a significant lift while the overall test is flat, as a hypothesis to validate, not a result to act on. Work through data-integrity checks, a formal interaction test with a multiplicity correction (since this segment was very likely noticed after the fact rather than pre-specified), and a set of robustness checks; then use an explicit decision process that weighs the statistical uncertainty against the business value and cost of being wrong, rather than a pure significance threshold, to choose between shipping to that segment, running a confirmatory follow-up, or abandoning the finding.

Structured elaboration

Step 1: verify the data before trusting the effect

• Check assignment balance within mobile specifically: treatment and control counts, and balance on key covariates, within the mobile slice alone, not just in aggregate.
• Check for instrumentation differences: missing events, a different SDK version, or a different exposure window on mobile that could produce a spurious effect having nothing to do with the treatment.
• Check for timing issues: did the mobile rollout start at the same time as the rest of the experiment, and is there any cross-over where a user appears in both device buckets across the test window.

Step 2: test the interaction formally

Fit an interaction model rather than comparing the mobile-only conversion rate to the mobile-only control rate informally:

import statsmodels.formula.api as smf

df["treat"] = df["assignment"].map({"control": 0, "treatment": 1})
model = smf.logit("conversion ~ treat + mobile + treat:mobile + signup_channel", data=df).fit()
print(model.summary())

Illustrative output (a hypothetical summary row, not a real run) would show a coefficient, standard error, z-value, and p-value for each term; the row that matters most here is treat:mobile. A row reading something like treat:mobile coef = 0.18, p = 0.02, alongside a treat main-effect coefficient close to zero and non-significant, is the pattern that supports a genuine mobile-specific effect: the interaction term carries the real signal while the main treatment effect alone looks flat, consistent with the original observation that the overall test showed no lift. A significant coefficient on treat:mobile is what actually supports "the effect really differs by device," rather than the mobile-only point estimate on its own, which can look large purely from within-mobile noise.

Step 3: correct for multiplicity honestly

Ask directly whether mobile was a subgroup chosen before the test ran or one noticed afterward because it happened to look interesting. If it was not pre-specified, and in practice it usually was not when this kind of question comes up, apply a multiplicity correction appropriate to however many segments were actually eyeballed (even informally) before mobile stood out, or at minimum treat the raw p-value as an optimistic upper bound on how surprising this finding really is.

Step 4: check power on the mobile slice itself

Compute the sample size and event count within mobile alone and the confidence interval width on its effect estimate. A wide interval or a small mobile sample means the "significant" reading is fragile, and this matters even more when mobile is a genuinely small-traffic segment (a specific device class or platform with limited volume) rather than merely a smaller slice of a large population: in that case a confirmatory follow-up restricted to the same segment may take a long time to reach adequate power, or may never fully reach the same statistical bar as the overall test, which is itself part of the decision, not a reason to ignore the finding.

Step 5: robustness checks

• Look at related metrics (engagement, retention, complaint or refund rate) to see whether they move in a direction consistent with the primary metric's mobile-specific lift, or whether the primary metric is moving alone in a way that is harder to explain.
• Check whether the effect is stable over the test window or concentrated in a short burst of days.
• Check finer sub-slices of mobile (iOS versus Android, OS version) to rule out the effect actually being driven by one narrow slice within "mobile" rather than the device class as a whole.
• Re-run with alternative covariate adjustment and see whether the interaction coefficient is stable.

Worked example: the decision process

Rather than a bare "p < 0.05 so ship it" rule, weigh four inputs explicitly: how strong the statistical evidence is after the checks above, how large and reliable the resulting business value would be if the effect is real, how costly it is if the segment is shipped and the effect turns out not to be real, and how long a confirmatory follow-up on that segment alone would realistically take to reach adequate power given the segment's own traffic volume.

• Strong evidence, low cost of being wrong, fast to confirm: ship a small, reversible rollout to the segment while a confirmatory read continues, since the downside of being wrong is small and quickly detected.
• Moderate evidence, or the segment is small enough that a proper confirmatory test would take a long time to reach power: this is the case worth naming explicitly, since waiting for full statistical certainty may never be practical for a genuinely small segment. Here, the decision becomes an explicit risk-tolerance call: state the estimated cost of shipping on an unconfirmed finding versus the estimated cost of never acting on a real effect because the segment could never generate enough data to confirm it on its own, and make that trade-off visible to the decision-maker rather than deferring it to a p-value the segment may structurally never be able to produce.
• Weak evidence, or a moderate cost of being wrong: run a dedicated, pre-specified confirmatory experiment targeted at the segment before making any production change, treating the original finding purely as the hypothesis that justified the follow-up.
• Evidence disappears after the data-integrity and robustness checks: abandon the finding and document why, so the same slice does not get re-litigated the next time someone happens to look at it.

Trade-offs & pitfalls

• Treating an unadjusted subgroup p-value as decisive. The interaction test plus a multiplicity correction is what separates a real segment effect from one of several plausible slices that happened to look significant.
• Waiting indefinitely for a small segment to reach the same statistical bar as the overall test. For a genuinely low-traffic segment, that bar may not be reachable on a useful timeline; the decision framework needs to say what happens in that case rather than defaulting to inaction.
• Ignoring instrumentation as a candidate explanation. A device-specific logging or SDK difference is a mundane but common cause of an apparent segment effect and should be ruled out before any statistical machinery is trusted.
• Shipping on a single significant slice with no plan to re-check it. Even a reversible segment rollout should carry a defined follow-up read, not be treated as a closed decision the moment it ships.

Direct answer

An A/A test randomly splits traffic into two groups that receive the identical experience and compares their metrics as if they were a real experiment. You run one to validate the assignment and measurement pipeline before trusting a real A/B result on the same platform: since both groups get the same product, any statistically significant difference between them signals a problem in the pipeline (randomization, instrumentation, or analysis) rather than a real effect, because by construction there is no effect to detect.

Structured elaboration

Two valid use cases

• Validating the assignment and instrumentation pipeline. Confirms that the bucketing hash actually produces the intended split ratio, that each unit sees a stable, single experience, and that event logging correctly attributes actions to the assigned arm end to end (client instrumentation through to the analysis table).
• Establishing a baseline-variance estimate. Because there is no true effect, the spread of the A/A metric difference across many runs (or across a well-chosen resampling of the same data) tells you what "just noise" looks like for this metric on this population, which is useful input for planning: it is a sanity check on your variance assumptions, not a substitute for a proper power calculation.

What to check while it runs

• The realized split ratio against the intended one (a sample-ratio check): meaningfully off the intended ratio (say, 50/50 skewing to 49/51 in a way that recurs, not a single noisy day) points at a bucketing bug before you even look at outcome metrics.
• Core funnel and event counts by arm (sessions, page views, primary conversion event) to confirm the two arms are tracked with equal fidelity, not just equal traffic.
• Whether the metric of interest for the upcoming real experiment behaves as expected in the A/A read, since that is the metric whose baseline variance you actually need.
• Run it for at least one full natural cycle of the traffic (typically a full week, to span weekday/weekend mix) rather than a single day, since a one-day A/A window can look clean by luck or flagged by a day-specific anomaly that has nothing to do with the platform.

An A/A test is, at its core, a targeted way to surface three distinct failure classes: instrumentation errors (events not logged or misattributed), non-random assignment (the bucketing hash is not producing a genuinely random, independent split), and sampling biases (the two arms end up systematically different in composition despite a technically-random split, e.g., a bot-filtering rule that behaves differently by arm). Each class points at a different fix, which is why segmenting the flagged difference (below) matters more than the raw significance flag itself.

Two limitations or common misinterpretations

• A clean A/A result is not proof the pipeline is bug-free. With enough traffic, small true differences in a specific test can still slip through if the bug is intermittent (e.g., only affects a rare browser) or if the metric checked in the A/A test is not the one that will matter in the real experiment. Absence of a flagged difference is reassurance, not a guarantee.
• A single significant A/A result does not, by itself, mean the pipeline is broken. At a conventional significance threshold, some fraction of A/A tests will show a "significant" difference purely by chance even with a perfectly correct pipeline; treat one flagged metric as a prompt to investigate, not as an automatic verdict, especially if you are checking many metrics at once and did not correct for that.

Root-causing a significant A/A result

1. Recheck the sample ratio first. A skewed split is the fastest, most common finding and points straight at a bucketing bug rather than a downstream measurement issue.
2. Segment the difference. Break the flagged metric down by platform, geography, and new-vs-returning user; a difference concentrated in one segment (e.g., one app version) points at an instrumentation bug specific to that segment rather than a global randomization failure.
3. Check for a known confound in how the two arms are served, such as one arm being disproportionately served through a code path with different latency or caching behavior, which is functionally a version-of-treatment bug even though no real treatment was intended.
4. Re-run before escalating, if the first read used a short window: a single noisy day is a weaker signal than a difference that persists across multiple independent A/A windows.
5. If it persists and is not explained by a segment or a known bug, treat the underlying real-experiment platform as unvalidated until the discrepancy is resolved; shipping A/B decisions on top of an unexplained A/A anomaly defeats the purpose of running the check at all.

Worked example

A team runs an A/A test ahead of a planned homepage experiment and flags a significant difference in click-through rate. Step 1, sample ratio: 50.1% vs 49.9%, within normal noise, so not a bucketing problem. Step 2, segmentation: the CTR gap is near zero on Android and web but noticeably present on iOS. Step 3: engineering finds one arm's iOS client is on an older app version with a slightly different default tab order, an artifact of how the A/A test's client-side flag was staged rather than anything about the experiment platform itself. The root cause is a version-of-treatment bug traced to a real, checkable fact (the iOS staging config), not a p-value alone; the fix is correcting the staged rollout, not adjusting the metric.

Trade-offs and pitfalls

• Running A/A tests constantly, on every metric, invites exactly the false-alarm problem described above; use them at meaningful checkpoints (new platform, new metric pipeline, post-incident) rather than as a standing tax on every experiment.
• Do not use a single A/A run's variance estimate as your only power-planning input if you have a more direct historical baseline available; treat it as a cross-check.
• A quiet A/A test on a low-traffic metric provides much weaker reassurance than the same result on a high-traffic metric, because a real problem of a given size is harder to detect with less data; do not treat "clean" as equally strong evidence across metrics of very different volume.