A/B Test Design & Statistical Rigor Questions

Direct answer

A heterogeneous treatment effect (HTE) is a real difference in a treatment's effect across subgroups, meaning a feature can genuinely help one segment and hurt another even when the overall average effect looks flat, because a flat average is just a weighted blend of both. The discipline that keeps this useful rather than a source of false discoveries is starting from a short list of subgroups chosen and written down before the test runs, based on a product hypothesis for why that segment might respond differently, rather than slicing every available dimension after the results come in and reporting whichever slice looks interesting.

Structured elaboration

Why a flat average can hide a real split

An average treatment effect (ATE) is a weighted average of segment-level effects. If segment A is half the traffic with a genuine +2.0 percentage point effect, and segment B is the other half with a genuine -1.6 percentage point effect, the pooled effect is:

$\text{ATE} = 0.5 \times 2.0 + 0.5 \times (-1.6) = 0.2 \text{ percentage points}$

A pooled +0.2pp result reads as flat or marginal, and a team that only looks at the ATE would conclude the feature does not matter, when in fact it is a real win for half the population and a real loss for the other half.

A disciplined workflow

1. Pre-specify the subgroup list before running the test. Choose it from a concrete product hypothesis, for example "new users lack context this feature assumes, so we expect a different response than returning users," not from "let's see what breaks out once we have the data." Keep the list short, typically a handful of segments, and write it into the analysis plan alongside the primary metric.
2. Power the subgroup analysis explicitly, not just the overall test. Segments are smaller than the full population, so a segment showing "no significant difference" is very often an underpowered null rather than evidence the segments behave the same; check the segment's own sample size against the effect size you would need to distinguish before treating a null subgroup result as informative.
3. Name the multiplicity problem and route around it, rather than re-deriving it here. Testing several pre-specified subgroups still inflates the chance of a false positive across the set, the same mechanism as testing several metrics; apply a standard multiplicity correction (family-wise or false-discovery-rate methods) to the pre-specified set, and treat any subgroup examined outside that pre-specified list as exploratory by default, no matter how it correlates with the metric.
4. Know the estimation toolkit without needing to build it here. For a short pre-specified list, a direct per-segment intent-to-treat comparison is usually enough. For flexible, higher-dimensional segmentation across many covariates at once, conditional average treatment effect (CATE, the treatment effect estimated for one particular slice of users rather than the population-wide average) estimation via meta-learners (model families purpose-built to estimate that per-slice effect from data) or uplift modeling (the applied name for the same goal: predicting who responds most to the treatment, not just whether the average user responds) is the standard toolkit; the discipline questions above (pre-specification, multiplicity, confirmatory follow-up) apply regardless of which estimation method produced the segment-level number.
5. Treat a post-hoc finding as a hypothesis, not a decision. A subgroup effect that survives pre-specification and multiplicity correction can inform a prioritization or personalization decision directly. A subgroup effect discovered by open-ended slicing after the fact, even a striking one, should be treated as hypothesis-generating only and routed to a dedicated confirmatory experiment on that segment before it drives a shipping decision.
6. Report findings with their status labeled. When presenting a subgroup result to stakeholders, state explicitly whether it was pre-specified or exploratory, and whether a confirmatory step is still required, so a segment finding does not get treated as settled fact before it has earned that status.

A concrete product scenario

A checkout redesign shows a flat, non-significant overall conversion effect. The team had pre-specified a new-user-versus-power-user split before the test, hypothesizing that power users already have an efficient checkout habit that a redesign would disrupt while new users would benefit from the clearer flow. The pre-specified interaction test confirms a real, multiplicity-corrected split matching the illustration above: a genuine gain for new users and a genuine loss for power users. The resulting decision is neither "ship to everyone" (which would hurt power users) nor "scrap the redesign" (which would forgo a real win for new users), but a personalization decision: ship the new checkout to new users only, keep power users on the existing flow, and treat that as the actual outcome of the experiment rather than a footnote to a "no effect" headline.

Trade-offs & pitfalls

• Confusing exploratory with confirmed. The single most common failure mode is presenting a striking post-hoc slice with the same confidence as a pre-specified, corrected result; the two need visibly different treatment in any readout.
• Underpowered subgroup nulls read as "no heterogeneity." A segment too small to detect the effect size in question will always look flat, whether or not a real difference exists; check the power before concluding homogeneity.
• Over-narrow personalization from a single test. One HTE finding is evidence for a segment-specific policy, not proof it will hold up over time or across other metrics; a confirmatory follow-up before fully committing production logic to a segment split is cheap insurance against a finding that was itself a fluke.
• Skipping pre-specification because "we'll just correct for multiplicity later." A multiplicity correction controls the false-positive rate across a stated set of comparisons; it does not rescue a search that had no defined stopping point in the first place.

Direct answer

When assignment happens per-device (a cookie on web, a device id on the app) but the real unit of interest is the person, users who touch the product on multiple devices get assigned independently on each device, so some of them land in both control and treatment at once. That breaks the assumption that each experimental unit receives exactly one arm: it dilutes the measured treatment effect, since a "contaminated" user's behavior is influenced by both arms, and it can double count outcomes if the same person's actions are logged and analyzed once per device-identity rather than once per person. The fix is to randomize and log at the most persistent identity you actually have, then dedupe and correct for the identities you had to resolve after the fact rather than at assignment time.

Structured elaboration

Where the bias enters

• Assignment stage: a user with a web cookie and a mobile device id gets two independent coin flips. If both land the same way, no harm; if they split, that person is exposed to treatment and control simultaneously, a violation of SUTVA (the assumption that one unit's outcome does not depend on another instance of its own assignment).
• Analysis stage: if the analysis unit is "device" rather than "resolved person," a contaminated person's activity appears once in each arm's numerator, and the two rows are not independent observations even though the analysis code treats them as such, which understates the true variance.
• Coverage bias: identity resolution itself makes mistakes (false merges linking two different people, false splits treating one person as two); if those errors are not random with respect to treatment, they introduce their own bias on top of the contamination.

Practical strategies

1. Randomize at the most stable identity you have. Prefer a logged-in account id over a device id or cookie whenever the user is authenticated; fall back to a deterministic device hash only for logged-out traffic, and treat that population as a separate, lower-confidence stratum in reporting.
2. Log everything needed to resolve identity after the fact. Persist device id, cookie id, and account id (hashed, respecting privacy) on every assignment and every outcome event, even when the assignment itself was made at device level, so contamination can be measured and corrected during analysis rather than discovered too late.
3. Define the primary analysis on the resolved (canonical) identity, not the raw assignment record: after backend identity merge, collapse a contaminated user into a single row and apply an explicit, pre-registered rule for what arm they count as, for example "any-device-treatment counts as treated," reported alongside a stricter "single-device-only" rule as a sensitivity check.
4. Quantify and bound the bias rather than ignore it. Report the primary result plus at least two sensitivity analyses: one restricted to users seen on exactly one device (removes contamination but shrinks the sample), and one using the any-device-treatment rule (keeps the full sample but is a diluted estimate of the true per-exposure effect).
5. Use cluster-robust standard errors at the resolved-identity level so that outcomes from the same person are not treated as independent observations even after collapsing to one row per person, since a person can still contribute multiple sessions or events.

Worked example

Assume 20% of users are active on exactly two devices (web and app) and 80% are single-device (an illustrative, stated split). Assignment happens independently per device with probability 0.5 to treatment. For a two-device user, the four equally likely device-pair outcomes are (control, control), (control, treatment), (treatment, control), (treatment, treatment), each with probability 0.25:
$P(\text{both control}) = P(\text{both treatment}) = 0.25, \qquad P(\text{split, i.e. contaminated}) = 0.5$
So among the 20% of users who are two-device, half get split across arms, which is 10% of the total user base. If a contaminated user's outcome is counted in both the treatment and control totals rather than resolved to one arm, then 10% of the treatment-arm numerator and 10% of the control-arm numerator are contributed by the exact same set of people, which both understates the between-arm difference and violates the independence assumption behind the standard error calculation. Restricting the primary analysis to the 90% of users who are single-device or resolve cleanly to one arm removes the contamination at the cost of 10% of the sample, which should be reflected directly in the power calculation for the corrected analysis.

Trade-offs and pitfalls

• The "any-device-treatment" rule is conservative and interpretable but structurally dilutes the estimated effect toward zero for contaminated users, since they experienced a mix of both arms; don't present it as an unbiased estimate of the pure per-exposure effect.
• Restricting to single-device users is cleaner statistically but changes who the estimate applies to: if multi-device users differ systematically (often more engaged, higher-value), the single-device estimate may not generalize to the full user base.
• Identity resolution is itself a model with false-merge and false-split error rates; a resolution pipeline retrained or changed mid-experiment can shift the contamination rate over time and should be monitored, not assumed constant.
• Logging every identifier needed for later resolution has real privacy and storage cost; agree on hashing and retention policy with the privacy function before instrumenting, not after a contamination investigation is already underway.

Direct answer

The randomization unit should be the largest identity that is (a) stable over the experiment window and (b) matches the unit at which you will measure and report the outcome. For a homepage layout change with user-scoped conversion metrics, that is almost always user id when you have reliable logged-in identity; fall back to device id for logged-out mobile traffic, and treat cookie and session id as fallback-only units because they leak identity across the very boundary you are trying to hold fixed. The mechanism that turns "unit" into an actual bucket assignment is deterministic hash-based bucketing, and its main operational failure mode is touching the hash inputs (the salt or key) mid-experiment. Before any of that, though, you have to define who is even eligible to be in the experiment at all.

Structured elaboration

Defining the eligible population before choosing a unit

Unit choice is a second-order question; the first-order question is which units are even eligible to enter the experiment. For a mobile-only feature (say, a redesign shipped exclusively in the mobile app to a US audience), a desktop-only visitor cannot receive the treatment no matter which arm they land in, so randomizing across your full user base and then measuring outcomes at the account level silently dilutes the experiment: ineligible units get logged into both arms with a null "effect" (they cannot experience the change either way), which pulls the estimated treatment effect toward zero and inflates the sample size needed to detect a real one. The eligible population for a mobile-only US feature is the set of units that are (a) on the mobile platform that ships the feature, (b) in the targeted market (US), and (c) past whatever version or capability gate the feature requires; everyone outside that eligible population should be excluded from the experiment entirely, not folded into control by default. This is a distinct failure mode from picking the wrong unit: a design can choose a perfectly good unit (user id) and still be broken if a third of the "users" randomized into it were structurally incapable of ever seeing the treatment, whether the unit ultimately chosen within that eligible population is user, device, or session id.

Trade-offs by candidate unit

The two axes that matter are: does this unit stay attached to one treatment condition for the life of the experiment, and does it match the level at which you will later compute the metric. Session-level randomization on a homepage layout change fails both: a returning user can see version A on Monday and version B on Wednesday, so "the effect of the layout" is not well defined for that person, and if you then report a user-level conversion rate you are averaging over users who experienced a mix of both conditions.

Target-segment and control-group selection for a personalization test

Personalization experiments add a further wrinkle on top of eligibility and unit choice: because the treatment itself varies per person (each user's personalized experience differs from every other user's), you have to be explicit about two more things: which segment of the eligible population the test targets, and what the control group actually receives. A common setup: the target segment is the subset of eligible users with enough interaction history for the personalization model to act on (say, users with a minimum number of prior sessions); users below that threshold cannot be meaningfully personalized and should either be excluded from the test or routed to a defined fallback, rather than silently folded into a "control" group that has nothing to do with the personalization decision being tested. The control group, correspondingly, should receive a clearly defined non-personalized baseline (a fixed default ranking or layout), not "whatever the legacy system happened to show," so the measured effect is attributable to personalization itself rather than to incidental differences between the two code paths. Get target-segment or control-group definition wrong (an ill-specified segment boundary, or a control group that partially overlaps with treatment logic) and the measured lift reflects a spurious selection effect rather than the personalization algorithm's real value, no matter how correctly the underlying randomization unit and hash mechanism were implemented.

How hash-based deterministic bucketing works

In practice you do not store a per-user assignment row for every experiment. Instead you compute

$\text{bucket}(u) = \text{hash}(u \,\Vert\, \text{salt}) \bmod N$

where $u$ is the chosen unit id (user id, device id, etc.), the salt is a string unique to this experiment (often the experiment name or id), and $N$ is the number of buckets (commonly 100 or 1000 for fine-grained traffic allocation). Buckets are then mapped to arms, e.g. buckets 0-49 to control and 50-99 to treatment for a 50/50 split. Because the hash is deterministic, the same unit id always lands in the same bucket for the same salt, which is what makes the assignment reproducible without a lookup table, and salting per-experiment is what makes assignment to experiment A independent of assignment to experiment B (so the same user can be validly in many concurrent, non-interacting experiments).

Operational pitfalls

• Changing the salt or hashing key mid-experiment. This is the single most common self-inflicted wound. It re-shuffles every unit into a new bucket, silently reassigning some fraction of users from control to treatment (or the reverse) partway through. The experiment now mixes users with a clean single-arm history and users who were exposed to both arms, which is exactly the session-level contamination problem from the table above, except it is invisible unless you log assignment history.
• Reusing a salt across experiments. If two unrelated experiments accidentally share a salt (or one is a substring of the identifier used in the other), their bucket assignments become correlated instead of independent, which breaks the assumption that concurrent experiments do not interfere with each other.
• Changing $N$ or the bucket-to-arm mapping. Even without touching the salt, resizing the traffic split mid-flight (e.g., ramping from 5% to 50%) moves units across the arm boundary unless the mapping is designed to be monotonic (new traffic is added to existing arms rather than everyone being rehashed).
• Identity churn. A user id that gets merged, deleted, or re-issued (account merge, logout/login cycles that mint a new anonymous id) effectively becomes a new hash input mid-experiment, which has the same effect as a salt change for that user.

A finer-grained alternative: per-impression randomization

Every unit above is a person-shaped identity. Some teams instead randomize at the impression level, assigning a fresh coin flip to each page view or ranking request rather than to a person. This is occasionally used for high-frequency, low-persistence decisions (e.g., which of several ranking variants to serve on a given request) where you explicitly do not want a stable per-user experience. It is a different trade entirely from the table above: it eliminates any notion of "this user's assigned arm" (so it cannot answer a question about a durable, user-perceived change like a homepage layout), and it introduces strong intra-user correlation in the outcome data, since one person's many impressions are not independent draws, which inflates the effective variance if you naively treat impressions as independent observations in the analysis. Per-impression randomization is the right tool only when the thing being tested is meant to vary within a single user's experience; for a homepage layout, where the goal is to measure how a stable person-level experience changes behavior, it is the wrong granularity.

Detecting and correcting unit-mismatch after the fact

1. Assignment-churn audit. From the exposure logs, compute the fraction of units that were logged under more than one arm during the experiment window. A near-zero rate is expected; anything material indicates contamination.
2. Pre-period balance check. Compare the two arms on metrics measured before the experiment started (metrics that could not possibly be affected by treatment). An imbalance signals a broken randomization, not a broken hash necessarily, but it is the same diagnostic.
3. Sample ratio mismatch check on the realized split, i.e., does the observed 50/50 (or intended ratio) actually hold at the analysis unit. A skew is a strong signal that the bucketing pipeline itself misbehaved.
4. Timeline reconstruction. If churn is found, check the deployment log for the experiment: a salt, key, or bucket-count change on a specific date will produce a visible step change in the churn-rate-by-day series.
5. Correction paths, in order of preference. Analyze by first-observed assignment only (treat each unit's initial exposure as its assignment, i.e., an intention-to-treat style rule, and accept the resulting dilution of the effect estimate); if the break has a clean date, restrict the analysis window to the stable period before or after it; if contamination is pervasive, drop the experiment's results for the affected window and rerun rather than trying to model around a broken assignment mechanism, since any post hoc adjustment for a data-dependent unit-mismatch is itself a source of bias.

Worked example

Suppose an app-only feature was randomized by session id and you are asked to sanity-check it before trusting the readout. You pull exposure logs and count, per user, the distinct arms they were logged under: 92,000 users saw only control, 91,500 saw only treatment, and 6,500 saw both. Churn rate is $6{,}500 / (92{,}000+91{,}500+6{,}500) \approx 3.4\%$. That is a directly computed, reproducible number from the logs, not an assumption, and a value that high on a homepage-layout test (where the same person plausibly returns within the experiment window) is enough on its own to recommend re-running at user-id granularity rather than trying to salvage the session-level readout.

Trade-offs and pitfalls

• Choosing the "purest" unit (household) is not free: fewer independent units means higher variance per unit, so the same absolute effect needs more households than it would need users to reach the same precision. Unit choice is a bias-versus-noise trade, not a pure bias fix.
• A cookie- or device-based fallback is a compromise you should name explicitly to stakeholders, not a silent substitute for user id; report the estimated multi-device contamination rate alongside the headline result.
• An eligible population that is defined too loosely (e.g., randomizing all traffic instead of just the mobile-only, in-market segment) produces the same kind of diluted, biased-toward-zero readout as a bad unit choice, even when the unit itself is correct.
• Do not "fix" detected contamination by re-including the mixed-exposure users with a different weighting scheme chosen after seeing which way it moves the result; decide the exclusion or ITT rule before looking at the treatment effect.

Direct answer

Beyond CUPED (using a pre-experiment covariate to residualize the outcome), the two other standard variance-reduction levers are stratified (blocked) randomization, which forces balance on a known factor at assignment time instead of hoping random chance balances it, and covariate or regression adjustment, which is the general case of "adjust for a predictive covariate" that CUPED is one specific, pre-experiment-only instance of. Both work by removing a source of outcome variance that is not related to treatment, so the same true effect becomes easier to distinguish from noise; both reduce required sample size roughly in proportion to how much outcome variance the factor explains, and neither invents a new number, they trade a known, explainable source of variance for a smaller residual.

Structured elaboration

Stratified (blocked) randomization

Instead of randomizing the whole population as one pool, split the population into strata on a factor known before assignment (country, device type, new vs. returning user), then randomize independently within each stratum so each arm gets a matched share of every stratum. This removes between-stratum variance from the treatment-effect estimator's variance, because the strata are balanced by design rather than by luck: with plain randomization on a highly imbalanced population, an unlucky split (e.g., treatment skewing toward the low-baseline country) inflates the observed variance of the effect estimate even though the true effect is unaffected.

It is applicable whenever you have a discrete, pre-assignment factor that is known to correlate with the outcome and is stable at randomization time. It differs from covariate adjustment in when the correction happens: stratification acts at assignment time (balance is enforced), while regression adjustment acts at analysis time (balance is estimated and subtracted after the fact). The two are complementary, not substitutes: stratify at assignment for the factors you can, and adjust for continuous covariates at analysis.

Covariate / regression adjustment

This is the general technique of fitting a model for the outcome on one or more covariates (not restricted to pre-experiment-only, unlike CUPED) and using the model to remove predictable variance from the outcome before comparing arms, most simply via ANCOVA (analysis of covariance), a linear regression of $Y$ on the treatment indicator and covariates that removes the variance those covariates explain from the comparison, the same variance-reduction logic as CUPED and stratification, just carried out as a regression rather than a pre-experiment covariate or a balanced split. It is applicable whenever you have covariates, pre-experiment or otherwise as long as they cannot themselves have been affected by treatment, that are predictive of the outcome. CUPED is the special case where the covariate is restricted to a pre-experiment value of the outcome metric itself; regression adjustment generalizes this to any number of eligible covariates and lets you combine several weak predictors into one stronger adjustment.

Effect on sample size and power

For both techniques, if the factor being controlled for explains a fraction $R^2$ of the outcome's variance, the variance of the treatment-effect estimator shrinks by roughly that same factor, and required sample size for a fixed target precision shrinks proportionally, since sample size for a fixed effect and power scales with the variance of the metric. A factor that explains little of the outcome variance buys little; a strong, well-chosen factor can meaningfully shorten the required test duration for the same statistical bar.

Worked example: stratifying a multi-country test

A test is planned across three countries with very different baseline conversion rates: Country A at 4%, Country B at 12%, Country C at 22%, in roughly equal traffic shares (each about one third of total users). Without stratification, plain randomization can by chance send more of one country's traffic to one arm, and even without that bad luck, the pooled outcome variance includes the between-country spread of baseline rates as extra noise the estimator has to average out.

Using the law of total variance, the overall variance of the outcome decomposes as:

$\operatorname{Var}(Y) = \underbrace{\mathbb{E}\big[\operatorname{Var}(Y \mid \text{country})\big]}_{\text{within-country variance}} + \underbrace{\operatorname{Var}\big(\mathbb{E}[Y \mid \text{country}]\big)}_{\text{between-country variance}}$

Stratifying by country and analyzing as a weighted average of within-country treatment effects removes the second (between-country) term from the treatment-effect estimator's variance, since each stratum is separately balanced and the between-stratum spread no longer contributes noise to the comparison. Concretely: with baseline rates of 4%, 12%, 22% and equal stratum weights, the between-country component of variance is

$\bar p = \frac{0.04+0.12+0.22}{3} = 0.1267$

$\operatorname{Var}(\bar p) = \frac{1}{3}\Big[(0.04-\bar p)^2 + (0.12-\bar p)^2 + (0.22-\bar p)^2\Big] = \frac{1}{3}(0.00751 + 0.0000445 + 0.00871) = 0.00542$

That 0.00542 is exactly the between-country variance component the stratified analysis removes from the pooled estimator's variance, computed directly from the three stated baseline rates, not asserted; how large a share of total variance that is depends additionally on the within-country binomial variance at each rate, which you would combine with this term using the same decomposition to get the full picture before quoting an overall percentage reduction.

Implementation for the multi-country case

• Assign the stratum at randomization time using the same deterministic hash-bucketing approach as the overall unit assignment, but nest it: hash within each country separately (or include country in the hash key) so each country independently hits its target split ratio.
• At analysis time, estimate the treatment effect within each country and combine as a weighted average (weighted by stratum size or by inverse variance), rather than pooling raw counts across countries, which is what actually realizes the variance reduction shown above.

Trade-offs and pitfalls

• Stratifying on too many dimensions at once shrinks individual strata until some contain too few units to balance meaningfully, and can create empty or near-empty cells, especially when crossing multiple categorical factors (country times device times cohort).
• A stratification factor chosen because it is convenient rather than because it is predictive buys little variance reduction while adding real implementation complexity; check the factor's explanatory power on historical data before committing the assignment pipeline to it.
• Regression adjustment on covariates measured close to, but not strictly after, the treatment start needs the same scrutiny as CUPED's pre-experiment-only requirement: any covariate that could plausibly be influenced by treatment invalidates the adjustment's unbiasedness, not just its efficiency.

Direct answer

Assess heterogeneity with a formal treatment-by-segment interaction test rather than by eyeballing which country's point estimate looks different, and be honest that most individual segments are underpowered relative to the overall test. Because testing many segments at once inflates the odds that one looks significant by chance alone, correct for that multiplicity (a false-discovery-rate procedure for routine screening, Bonferroni's family-wise control when a single false positive would be costly) before trusting any one segment's result, and choose among full pooling (one overall effect), no pooling (an independent estimate per segment), and partial pooling (a hierarchical model that shrinks each segment's estimate toward the overall mean in proportion to how much data that segment actually has); partial pooling is the right production default for exactly this kind of many-segment, uneven-traffic setup, because it avoids both the false confidence of no pooling and the false uniformity of full pooling.

Structured elaboration

Testing for heterogeneity

Fit outcome ~ treatment + segment + treatment:segment, using both device type and region as covariates jointly rather than testing one dimension at a time, since a device-level pattern and a region-level pattern can be confounded with each other if only one is modeled. (In this regression-formula shorthand, ~ means "model the outcome using the terms on the right," and treatment:segment is the interaction term, the piece that tests whether the treatment effect itself varies by segment rather than just shifting the segment's baseline.) Test the joint significance of the interaction terms with a Wald or likelihood-ratio test, or a permutation test as a distribution-free alternative (any of the three answers the same question by a different route, so which one you pick matters less than actually pre-specifying and running one), to get one answer to "is there heterogeneity at all" before looking at any individual segment.

Why per-segment power is the first thing to check

A segment with a fraction of the overall traffic has a much larger minimum detectable effect (MDE) than the pooled test, using the same relationship between sample size and detectable effect as any power calculation:

$\text{MDE} \approx (z_{1-\alpha/2}+z_{1-\beta})\sqrt{\frac{2p(1-p)}{n}}$

For an overall test with 300,000 users per arm and a 5% baseline conversion rate, at $\alpha=0.05$ two-sided and 80% power:

$\text{MDE}_{\text{overall}} \approx (1.9600+0.8416)\sqrt{\frac{2\times0.05\times0.95}{300{,}000}} = 2.8016 \times 0.000563 \approx 0.0016 \ (0.16\text{pp}, \text{about } 3.2\% \text{ relative})$

A single small geo with 3% of that traffic, 9,000 users per arm, has:

$\text{MDE}_{\text{small geo}} \approx 2.8016\sqrt{\frac{2\times0.05\times0.95}{9{,}000}} \approx 0.0091 \ (0.91\text{pp}, \text{about } 18.2\% \text{ relative})$

That is roughly 5.8 times larger (matching $\sqrt{300{,}000/9{,}000}$), meaning that small geo's own data can only reliably detect an effect nearly six times bigger than what the overall test is powered for. A "no significant effect" reading for that geo on its own data is very often just this, not evidence the true effect is zero there.

Correcting for testing many segments at once

Underpowered segments are one problem; testing many of them at the same time creates a second, separate problem. Each segment's interaction test is its own hypothesis test, so scanning fifteen countries or five device types for "which one differs" is exactly the multiple-comparisons setting: even if the treatment effect is truly identical everywhere, running enough segment-level tests makes at least one spuriously significant result likely by chance alone. Two standard corrections apply once the segment set is pre-specified (decided before looking at the results, not assembled from whichever segments already look interesting):

• Bonferroni divides the significance threshold by the number of segments tested (roughly $\alpha/k$ for $k$ segments). It controls the family-wise error rate (FWER), the probability of even a single false positive across the whole set, which makes it the conservative option: reach for it when acting on one wrongly-flagged segment would be expensive, for example a permanent per-country rollout split that is costly to unwind.
• Benjamini-Hochberg (BH) controls the false discovery rate (FDR) instead: the expected proportion of false positives among the segments you flag, not the chance of any false positive at all. It is less conservative than Bonferroni and is the standard choice for segment screening, because the usual goal when scanning many countries or device types is to shortlist which ones deserve a closer look, not to make one irreversible call per segment; tolerating a small, known rate of false leads among the flagged segments is the right trade for not burying the real ones.

Hierarchical shrinkage, covered in the pooling comparison below, answers the same over-testing question through a different mechanism rather than competing with these corrections. Bonferroni and BH both operate on the significance threshold a segment's p-value has to clear before you call it real; partial pooling never runs a per-segment significance decision in the first place; it pulls every segment's raw estimate toward the global mean in proportion to how little data that segment has, so a small segment cannot produce an extreme, attention-grabbing estimate purely from its own noise. A production pipeline commonly uses both together: an FDR-corrected interaction test to decide which segments are worth calling out as genuinely different, and hierarchical shrinkage on the point estimates actually shown to stakeholders, so the reported numbers are already regularized rather than raw.

Full pooling, no pooling, and partial pooling

The shrinkage a hierarchical model applies has a standard closed form: a segment's raw estimate $\hat\theta_i$ with sampling variance $\sigma_i^2$ is pulled toward the grand mean $\bar\theta$ by:

$\lambda_i = \frac{\tau^2}{\tau^2 + \sigma_i^2}, \quad \hat\theta_i^{\text{shrunk}} = \lambda_i \hat\theta_i + (1-\lambda_i)\bar\theta$

where $\tau^2$ is the estimated between-segment variance.

To make this concrete, reuse the same two segments from the MDE comparison above: the overall test's 300,000-users-per-arm scale and the small geo's 9,000-users-per-arm scale, both at the 5% baseline conversion rate already used there. Their sampling variances follow directly from the same standard-error term used in the MDE formula, expressed here in percentage points:

$\sigma_{\text{large}} = \sqrt{\frac{2\times0.05\times0.95}{300{,}000}} \approx 0.0563\text{pp}, \quad \sigma_{\text{large}}^2 \approx 0.00317\ \text{pp}^2$

$\sigma_{\text{small}} = \sqrt{\frac{2\times0.05\times0.95}{9{,}000}} \approx 0.3249\text{pp}, \quad \sigma_{\text{small}}^2 \approx 0.1056\ \text{pp}^2$

Take an illustrative between-segment variance of $\tau^2 = 0.09\ \text{pp}^2$ (true segment effects varying by roughly $\pm0.3$pp around the grand mean is a reasonable order of magnitude here: smaller than the small geo's own sampling noise, larger than the overall test's). The shrinkage weights are:

$\lambda_{\text{large}} = \frac{0.09}{0.09+0.00317} \approx 0.966, \qquad \lambda_{\text{small}} = \frac{0.09}{0.09+0.1056} \approx 0.460$

Suppose the pooled (grand-mean) estimate across all segments is $\bar\theta = 0.20$pp, the large segment's own raw estimate happens to be $\hat\theta_{\text{large}} = 0.25$pp, and the small geo's own raw estimate happens to be $\hat\theta_{\text{small}} = 0.85$pp, more than four times the grand mean at face value. Shrinking each toward $\bar\theta$:

$\hat\theta_{\text{large}}^{\text{shrunk}} = 0.966\times0.25 + 0.034\times0.20 \approx 0.248\text{pp}$

$\hat\theta_{\text{small}}^{\text{shrunk}} = 0.460\times0.85 + 0.540\times0.20 \approx 0.499\text{pp}$

The large segment's raw estimate barely moves, 0.25pp to 0.248pp, because its own sampling variance is tiny next to $\tau^2$, keeping $\lambda_{\text{large}}$ close to 1. The small geo's raw estimate moves a great deal, from 0.85pp down to about 0.50pp, more than half of its apparent excess over the grand mean pulled away, because its sampling variance is large next to $\tau^2$ and $\lambda_{\text{small}}$ sits closer to 0. A striking small-segment number is exactly the case shrinkage is built to discount.

A weakly informative prior centered on the pooled estimate (for a Bayesian implementation) plays the same role as $\tau^2$ in a frequentist multilevel model: it is what lets a small geo borrow strength from the rest of the segments instead of standing entirely on its own thin data. A segment with a small $\sigma_i^2$ (lots of its own data) keeps most of its own estimate; a segment with a large $\sigma_i^2$ (little data) gets pulled hard toward the overall mean. This is the mechanism, not full pooling's blunt "ignore the segment" or no pooling's blunt "trust the segment's noisy number completely."

What to report to stakeholders

For each segment, report the shrunk point estimate, its credible or confidence interval, and something indicating how much shrinkage was applied (effective sample size, or the raw versus shrunk estimate side by side), alongside the single global interaction-test result and which multiplicity correction it used. Explain the shrinkage in plain language for a non-technical audience: "we pulled small countries' estimates toward the overall average because they don't have enough of their own data to stand alone yet," rather than presenting a raw per-country number that a stakeholder might otherwise take at face value.

Recommended rollout strategy

If the global interaction test does not reject homogeneity and the shrunk per-segment estimates are consistent in sign and magnitude with the pooled effect, roll out globally on the strength of the pooled result. If a segment's shrunk estimate remains meaningfully different in sign or magnitude even after shrinkage has pulled it toward the mean, and it survives the multiplicity correction above, that segment is a real candidate for differentiated treatment, but confirm it with a segment-targeted follow-up before committing production logic to a permanent split, the same discipline that applies to any post-hoc-adjacent segment finding.

Trade-offs & pitfalls

• Treating "not significant per segment" as "no effect in that segment." The MDE gap above is the mechanism; a small segment's null result is usually a power problem, not a finding.
• No pooling on small segments overstates confidence in noise. Reporting fifteen independent country-level point estimates without acknowledging their individual uncertainty invites a stakeholder to chase the noisiest ones.
• Full pooling by default hides real, actionable heterogeneity. If the interaction test does reject homogeneity, defaulting to one global number throws away a finding that could inform a real rollout decision.
• Explaining shrinkage badly. If a hierarchical model's output is presented as a black box, a segment owner who sees their raw number "corrected" downward without explanation will reasonably distrust the whole analysis.
• Skipping the multiplicity correction because shrinkage is already in use. Shrinkage regularizes the estimates; it does not by itself control how many segments get flagged as different across the whole set. Treating hierarchical modeling as a substitute for an FDR or family-wise correction, instead of a complementary tool, is how a many-segment scan quietly turns into a fishing expedition again.