A/B Test Design & Statistical Rigor Questions

Direct answer

Assess heterogeneity with a formal treatment-by-segment interaction test rather than by eyeballing which country's point estimate looks different, and be honest that most individual segments are underpowered relative to the overall test. Because testing many segments at once inflates the odds that one looks significant by chance alone, correct for that multiplicity (a false-discovery-rate procedure for routine screening, Bonferroni's family-wise control when a single false positive would be costly) before trusting any one segment's result, and choose among full pooling (one overall effect), no pooling (an independent estimate per segment), and partial pooling (a hierarchical model that shrinks each segment's estimate toward the overall mean in proportion to how much data that segment actually has); partial pooling is the right production default for exactly this kind of many-segment, uneven-traffic setup, because it avoids both the false confidence of no pooling and the false uniformity of full pooling.

Structured elaboration

Testing for heterogeneity

Fit outcome ~ treatment + segment + treatment:segment, using both device type and region as covariates jointly rather than testing one dimension at a time, since a device-level pattern and a region-level pattern can be confounded with each other if only one is modeled. (In this regression-formula shorthand, ~ means "model the outcome using the terms on the right," and treatment:segment is the interaction term, the piece that tests whether the treatment effect itself varies by segment rather than just shifting the segment's baseline.) Test the joint significance of the interaction terms with a Wald or likelihood-ratio test, or a permutation test as a distribution-free alternative (any of the three answers the same question by a different route, so which one you pick matters less than actually pre-specifying and running one), to get one answer to "is there heterogeneity at all" before looking at any individual segment.

Why per-segment power is the first thing to check

A segment with a fraction of the overall traffic has a much larger minimum detectable effect (MDE) than the pooled test, using the same relationship between sample size and detectable effect as any power calculation:

$\text{MDE} \approx (z_{1-\alpha/2}+z_{1-\beta})\sqrt{\frac{2p(1-p)}{n}}$

For an overall test with 300,000 users per arm and a 5% baseline conversion rate, at $\alpha=0.05$ two-sided and 80% power:

$\text{MDE}_{\text{overall}} \approx (1.9600+0.8416)\sqrt{\frac{2\times0.05\times0.95}{300{,}000}} = 2.8016 \times 0.000563 \approx 0.0016 \ (0.16\text{pp}, \text{about } 3.2\% \text{ relative})$

A single small geo with 3% of that traffic, 9,000 users per arm, has:

$\text{MDE}_{\text{small geo}} \approx 2.8016\sqrt{\frac{2\times0.05\times0.95}{9{,}000}} \approx 0.0091 \ (0.91\text{pp}, \text{about } 18.2\% \text{ relative})$

That is roughly 5.8 times larger (matching $\sqrt{300{,}000/9{,}000}$), meaning that small geo's own data can only reliably detect an effect nearly six times bigger than what the overall test is powered for. A "no significant effect" reading for that geo on its own data is very often just this, not evidence the true effect is zero there.

Correcting for testing many segments at once

Underpowered segments are one problem; testing many of them at the same time creates a second, separate problem. Each segment's interaction test is its own hypothesis test, so scanning fifteen countries or five device types for "which one differs" is exactly the multiple-comparisons setting: even if the treatment effect is truly identical everywhere, running enough segment-level tests makes at least one spuriously significant result likely by chance alone. Two standard corrections apply once the segment set is pre-specified (decided before looking at the results, not assembled from whichever segments already look interesting):

• Bonferroni divides the significance threshold by the number of segments tested (roughly $\alpha/k$ for $k$ segments). It controls the family-wise error rate (FWER), the probability of even a single false positive across the whole set, which makes it the conservative option: reach for it when acting on one wrongly-flagged segment would be expensive, for example a permanent per-country rollout split that is costly to unwind.
• Benjamini-Hochberg (BH) controls the false discovery rate (FDR) instead: the expected proportion of false positives among the segments you flag, not the chance of any false positive at all. It is less conservative than Bonferroni and is the standard choice for segment screening, because the usual goal when scanning many countries or device types is to shortlist which ones deserve a closer look, not to make one irreversible call per segment; tolerating a small, known rate of false leads among the flagged segments is the right trade for not burying the real ones.

Hierarchical shrinkage, covered in the pooling comparison below, answers the same over-testing question through a different mechanism rather than competing with these corrections. Bonferroni and BH both operate on the significance threshold a segment's p-value has to clear before you call it real; partial pooling never runs a per-segment significance decision in the first place; it pulls every segment's raw estimate toward the global mean in proportion to how little data that segment has, so a small segment cannot produce an extreme, attention-grabbing estimate purely from its own noise. A production pipeline commonly uses both together: an FDR-corrected interaction test to decide which segments are worth calling out as genuinely different, and hierarchical shrinkage on the point estimates actually shown to stakeholders, so the reported numbers are already regularized rather than raw.

Full pooling, no pooling, and partial pooling

The shrinkage a hierarchical model applies has a standard closed form: a segment's raw estimate $\hat\theta_i$ with sampling variance $\sigma_i^2$ is pulled toward the grand mean $\bar\theta$ by:

$\lambda_i = \frac{\tau^2}{\tau^2 + \sigma_i^2}, \quad \hat\theta_i^{\text{shrunk}} = \lambda_i \hat\theta_i + (1-\lambda_i)\bar\theta$

where $\tau^2$ is the estimated between-segment variance.

To make this concrete, reuse the same two segments from the MDE comparison above: the overall test's 300,000-users-per-arm scale and the small geo's 9,000-users-per-arm scale, both at the 5% baseline conversion rate already used there. Their sampling variances follow directly from the same standard-error term used in the MDE formula, expressed here in percentage points:

$\sigma_{\text{large}} = \sqrt{\frac{2\times0.05\times0.95}{300{,}000}} \approx 0.0563\text{pp}, \quad \sigma_{\text{large}}^2 \approx 0.00317\ \text{pp}^2$

$\sigma_{\text{small}} = \sqrt{\frac{2\times0.05\times0.95}{9{,}000}} \approx 0.3249\text{pp}, \quad \sigma_{\text{small}}^2 \approx 0.1056\ \text{pp}^2$

Take an illustrative between-segment variance of $\tau^2 = 0.09\ \text{pp}^2$ (true segment effects varying by roughly $\pm0.3$pp around the grand mean is a reasonable order of magnitude here: smaller than the small geo's own sampling noise, larger than the overall test's). The shrinkage weights are:

$\lambda_{\text{large}} = \frac{0.09}{0.09+0.00317} \approx 0.966, \qquad \lambda_{\text{small}} = \frac{0.09}{0.09+0.1056} \approx 0.460$

Suppose the pooled (grand-mean) estimate across all segments is $\bar\theta = 0.20$pp, the large segment's own raw estimate happens to be $\hat\theta_{\text{large}} = 0.25$pp, and the small geo's own raw estimate happens to be $\hat\theta_{\text{small}} = 0.85$pp, more than four times the grand mean at face value. Shrinking each toward $\bar\theta$:

$\hat\theta_{\text{large}}^{\text{shrunk}} = 0.966\times0.25 + 0.034\times0.20 \approx 0.248\text{pp}$

$\hat\theta_{\text{small}}^{\text{shrunk}} = 0.460\times0.85 + 0.540\times0.20 \approx 0.499\text{pp}$

The large segment's raw estimate barely moves, 0.25pp to 0.248pp, because its own sampling variance is tiny next to $\tau^2$, keeping $\lambda_{\text{large}}$ close to 1. The small geo's raw estimate moves a great deal, from 0.85pp down to about 0.50pp, more than half of its apparent excess over the grand mean pulled away, because its sampling variance is large next to $\tau^2$ and $\lambda_{\text{small}}$ sits closer to 0. A striking small-segment number is exactly the case shrinkage is built to discount.

A weakly informative prior centered on the pooled estimate (for a Bayesian implementation) plays the same role as $\tau^2$ in a frequentist multilevel model: it is what lets a small geo borrow strength from the rest of the segments instead of standing entirely on its own thin data. A segment with a small $\sigma_i^2$ (lots of its own data) keeps most of its own estimate; a segment with a large $\sigma_i^2$ (little data) gets pulled hard toward the overall mean. This is the mechanism, not full pooling's blunt "ignore the segment" or no pooling's blunt "trust the segment's noisy number completely."

What to report to stakeholders

For each segment, report the shrunk point estimate, its credible or confidence interval, and something indicating how much shrinkage was applied (effective sample size, or the raw versus shrunk estimate side by side), alongside the single global interaction-test result and which multiplicity correction it used. Explain the shrinkage in plain language for a non-technical audience: "we pulled small countries' estimates toward the overall average because they don't have enough of their own data to stand alone yet," rather than presenting a raw per-country number that a stakeholder might otherwise take at face value.

Recommended rollout strategy

If the global interaction test does not reject homogeneity and the shrunk per-segment estimates are consistent in sign and magnitude with the pooled effect, roll out globally on the strength of the pooled result. If a segment's shrunk estimate remains meaningfully different in sign or magnitude even after shrinkage has pulled it toward the mean, and it survives the multiplicity correction above, that segment is a real candidate for differentiated treatment, but confirm it with a segment-targeted follow-up before committing production logic to a permanent split, the same discipline that applies to any post-hoc-adjacent segment finding.

Trade-offs & pitfalls

• Treating "not significant per segment" as "no effect in that segment." The MDE gap above is the mechanism; a small segment's null result is usually a power problem, not a finding.
• No pooling on small segments overstates confidence in noise. Reporting fifteen independent country-level point estimates without acknowledging their individual uncertainty invites a stakeholder to chase the noisiest ones.
• Full pooling by default hides real, actionable heterogeneity. If the interaction test does reject homogeneity, defaulting to one global number throws away a finding that could inform a real rollout decision.
• Explaining shrinkage badly. If a hierarchical model's output is presented as a black box, a segment owner who sees their raw number "corrected" downward without explanation will reasonably distrust the whole analysis.
• Skipping the multiplicity correction because shrinkage is already in use. Shrinkage regularizes the estimates; it does not by itself control how many segments get flagged as different across the whole set. Treating hierarchical modeling as a substitute for an FDR or family-wise correction, instead of a complementary tool, is how a many-segment scan quietly turns into a fishing expedition again.

Direct answer

A novelty effect is a temporary inflation of an early treatment effect: users explore or click on something purely because it is new, and that extra engagement fades once the feature stops being novel, biasing an early readout upward. A primacy effect (sometimes called a change-aversion or resistance-to-change effect) is the opposite pattern: a change disrupts a habitual workflow, so users are temporarily worse off while they relearn it, biasing an early readout downward, then the effect climbs toward its true level as users adapt. Both biases fade over roughly the same kind of horizon, so trusting a week-one number without checking its trajectory can make you launch a fad or kill a genuine win too early.

Structured elaboration

Mechanism and direction

Diagnostics to tell a spike from a persistent effect

• Time-windowed effect plot: daily or weekly treatment effect with confidence intervals, ideally with a smoothed trend line (LOESS or a spline), not a single pooled average. A genuine effect looks like a roughly flat band around a nonzero value; novelty looks like a spike that decays toward that band; primacy looks like a trough that rises toward it.
• Exposure-age cohorts, not calendar time: plot the effect against days since each user's first exposure for a fixed cohort of users first exposed on the same day, rather than calendar date. A calendar-time plot mixes newly exposed users (still novel-biased) with long-exposed users (already stabilized) every single day, which can mask a real decay curve as a flat line.
• New vs. returning user split: novelty is usually concentrated in users encountering the feature for the first time; if the effect is similar in a segment already exposed for weeks, that argues against novelty as the explanation.
• Change-point or decay model on the daily series: fit a time-varying effect model, effect as a function of exposure age, and test whether the transient component is statistically distinguishable from zero, separately from the asymptotic (persistent) component.
• Placebo check: run the same time-windowed analysis on a pre-launch period with no real treatment; if spike-like patterns appear there too, the "decay" you see in the real experiment may just be normal week-to-week noise, not a novelty artifact.

Adjusting duration and analysis

• Pre-register the analysis window before launch rather than reading the metric the moment it looks good; a fixed rule such as "primary read is the average effect over exposure-days 21 to 35" prevents cherry-picking the peak or the trough.
• Extend the experiment until the exposure-age curve visibly plateaus, or the fitted transient component's confidence interval crosses zero, rather than for a fixed calendar duration chosen in advance.
• Report both the early-window and late-window effect side by side rather than a single blended number; a launch decision based only on the blended average silently averages a fading spike with a stabilizing floor.

Worked example

Two hypothetical (illustrative, not real study data) weekly average-treatment-effect readings for the same nominal conversion metric:

Both curves are converging toward roughly the same persistent level, one from above and one from below, which is exactly the signature that separates them from a flat, genuine effect that would show roughly the same number every week within noise.

If the transient component decays exponentially, $\Delta(t) = C + A e^{-\lambda t}$, where $A$ is the size of the initial novelty or primacy spike above the persistent effect $C$ (the extra amount present at $t=0$ that fades away over time), and the illustrative decay rate is $\lambda = 0.2$ per week, its half-life is:
$t_{1/2} = \frac{\ln 2}{\lambda} = \frac{0.693}{0.2} \approx 3.5 \text{ weeks}$
That is the kind of number worth pre-registering as a decision rule: run at least three half-lives (about 10 to 11 weeks here) before reading the persistent effect $C$, rather than picking an arbitrary duration.

Trade-offs and pitfalls

• Waiting out a full decay curve costs calendar time and opportunity cost on other experiments; for low-stakes features, teams sometimes accept the risk of a novelty-inflated launch decision rather than run for months.
• Segmenting by exposure age needs per-user first-exposure timestamps captured in the assignment log; if you only log calendar-date rollups, you cannot separate calendar effects from exposure-age effects after the fact.
• A curve that looks like decay can just as easily reflect unrelated seasonality (marketing pushes, holidays) that correlates with launch timing; a decay-shaped curve is suggestive, not conclusive, on its own.
• Don't assume every early spike is novelty and every early trough is resistance to change: an early spike can be a genuine effect solving a pent-up need immediately, and an early trough can be a real bug that later gets patched. The pattern is evidence, not proof, and should be paired with qualitative checks (support tickets, session recordings) before concluding the mechanism.

Direct answer

A lift that spikes in week one and fades to flat by week two could be three different things wearing the same shape: a real novelty effect decaying to its true (smaller or zero) persistent level, a seasonal pattern that has nothing to do with the treatment and would show up in both arms if you looked, or a selection-bias artifact where the traffic source itself (a holiday push or a marketing campaign) skewed an unrepresentative mix of users into treatment. The fastest way to tell them apart is to check whether the decay tracks calendar time (seasonality), user exposure-age (novelty), or arm composition (selection bias), because each explanation leaves a distinct fingerprint in the data, not just a distinct story.

Structured elaboration

The checklist, in order

Disentangling novelty from selection bias when a marketing campaign is the traffic source

This case is harder because both explanations can be true at once and produce the same decaying shape. A marketing campaign that ramps down after week one changes who is arriving, not just how they behave: if campaign-driven traffic is disproportionately assigned into treatment (for example because the campaign linked directly into the treatment experience, or assignment happened downstream of a referral parameter correlated with arm), the week-one spike partially reflects a different, more click-happy population rather than a within-user novelty effect.

• Design fix: randomize on a unit and at a point in the funnel that is independent of campaign exposure, assigning before the user ever sees the campaign-specific landing experience, so acquisition-channel composition is balanced by construction rather than by hope.
• Post-hoc check: compare the new-vs-returning mix and traffic-source mix between arms week by week; if the treatment arm's share of campaign-driven new users is higher than control's, run a balance check on that specific subpopulation, not just the top-line allocation.
• Post-hoc correction: re-run the primary analysis restricted to users who arrived through non-campaign channels, and separately on campaign-driven users; if the effect concentrates in the campaign-driven segment and that segment is imbalanced between arms, the pooled week-one number is not a trustworthy estimate of the treatment effect for the general population.

Redesigning the experiment or analysis window

• Extend the primary analysis window well past the campaign's active period so the segment mix has time to normalize, and pre-register that the primary readout is a later window, not week one.
• Use a difference-in-differences framing (treatment-arm change from a pre-period baseline, minus control-arm change over the same calendar span) so genuine calendar movement common to both arms cancels out instead of being misread as a treatment effect.
• If holiday timing cannot be avoided, consider a staggered or replicated launch (the same treatment introduced to a second, non-holiday cohort later) so you get a second read that is not confounded with that specific calendar event.

Worked example

Suppose the pooled week-one lift is +8% on a baseline conversion rate of 10%, and segment analysis shows the treatment arm received 60% campaign-driven new users in week one versus 40% in control (stated, illustrative inputs). If campaign-driven new users convert at a naturally higher rate, say 14% versus 9% for organic users (illustrative inputs), the arm-level blended rate is a weighted average:
$\text{Treatment rate} = 0.6 \times 0.14 + 0.4 \times 0.09 = 0.084 + 0.036 = 0.120$
$\text{Control rate} = 0.4 \times 0.14 + 0.6 \times 0.09 = 0.056 + 0.054 = 0.110$
That gives an apparent lift of $(0.120 - 0.110)/0.110 \approx 9.1\%$, almost entirely explained by the composition difference rather than any within-user treatment effect, before even considering a real novelty component. This is why segment analysis has to run before you trust the topline number, not after.

Trade-offs and pitfalls

• Restricting analysis to non-campaign traffic to remove selection bias also shrinks your sample and can push you back into an underpowered read, exactly what the power-review step is meant to catch.
• Difference-in-differences assumes both arms would have moved in parallel absent the treatment; a campaign that specifically targets one arm's users breaks that assumption too, so DiD is not a free fix if the selection bias operates through campaign-to-arm assignment rather than through time.
• Extending the window to wait out both seasonality and novelty delays the launch decision; be explicit with stakeholders that "flat by week two" was never a valid stopping rule for this scenario, so the delay does not read as backpedaling.
• Retiring a variant because week two looks flat, without a power review, risks killing a real but modest persistent effect that a two-week window was never sized to detect in the first place.

Direct answer

Design this as a pre-registered, longitudinal comparison rather than a single before/after read: fix a small number of windows relative to each user's first exposure, not launch date, in advance, fit a simple decay model to the day-by-day treatment effect, and commit to a decision rule, stated before you see the data, for what pattern of the fitted decay and asymptote counts as real and durable versus novelty that will fade. The goal is to make the durable-vs-fading call a mechanical read of a pre-specified model output, not a judgment call made after watching the curve.

Structured elaboration

Time windows to pre-specify

• Baseline (pre-treatment, roughly two weeks before exposure): confirms no pre-existing difference between the groups on the metric of interest.
• Immediate (days 0 to 7 since first exposure): captures the bulk of any novelty spike.
• Short (days 8 to 30): where a genuine novelty component should be visibly decaying.
• Long (days 91 and beyond, or as far out as the experiment can afford to run): the window whose effect is treated as the primary estimate of the persistent effect, used for the launch decision.
  These are anchored to exposure age, days since each user's own first exposure, not calendar date, so users who join on different days are all compared on the same clock. A calendar-date plot mixes freshly exposed and long-exposed users in the same daily bucket and can mask a real decay curve as a false flat line.

Modeling the decay

Fit the daily or weekly treatment effect to a two-parameter decay-to-asymptote form:
$\Delta(t) = C + A e^{-\lambda t}$
where $t$ is exposure age, $C$ is the persistent (asymptotic) effect, $A$ is the size of the transient novelty component, and $\lambda$ is the decay rate. A purely persistent effect looks like $A \approx 0$, flat from day one; a pure novelty artifact looks like $C \approx 0$, decaying to nothing; most real cases land somewhere in between, with both $A$ and $C$ meaningfully nonzero, meaning some of the early lift really does fade but a smaller durable effect remains.

The decision rule, pre-specified

Commit, before the experiment starts, to a rule such as: the effect is durable if the long-window estimate's confidence interval excludes zero and the fitted persistent component $C$'s confidence interval excludes zero, evaluated no earlier than three estimated half-lives, $3 \times \ln 2/\lambda$, after first exposure. This does three things a post-hoc read cannot: it fixes how long to wait based on the shape of the decay itself rather than an arbitrary calendar deadline, it requires the long-window effect to independently clear significance rather than trusting the fitted curve alone, and it removes the temptation to declare victory the moment the curve looks favorable.

Worked example

Suppose a fitted decay model on the immediate and short windows gives stated, illustrative parameter estimates $A = 6\%$, $C = 2\%$, $\lambda = 0.15$ per week. The half-life of the transient component is:
$t_{1/2} = \frac{\ln 2}{\lambda} = \frac{0.693}{0.15} \approx 4.6 \text{ weeks}$
The pre-specified decision rule requires waiting roughly $3 \times 4.6 \approx 13.9$ weeks, call it 14 weeks, before the long-window read is treated as decisive. At that point, the transient component's contribution has decayed to:
$A \cdot e^{-\lambda \cdot 14} = 6\% \times e^{-0.15 \times 14} = 6\% \times e^{-2.1} \approx 6\% \times 0.122 \approx 0.73\%$
which is small enough that the observed effect at week 14 should be close to the true persistent effect $C$, letting the long-window confidence interval be read as a fair test of durability rather than a mix of fading novelty and true signal.

Trade-offs and pitfalls

• Waiting three half-lives before making the call costs real calendar time and delays every downstream decision riding on this experiment; for a low-stakes cosmetic change, teams often accept a shorter, less rigorous wait rather than the full 14 weeks in the worked example.
• The decay model assumes a single clean exponential; a novelty effect that itself varies by segment, a spike for new users layered with a slower-decaying resistance effect for long-tenured users, will not fit a single two-parameter curve well, and forcing the fit anyway can produce a confidently wrong half-life.
• Anchoring on exposure age rather than calendar date requires per-user first-exposure timestamps captured at assignment time; retrofitting this onto an experiment already running on calendar-date logging means exposure-age curves cannot be reconstructed after the fact.
• A pre-specified decision rule protects against motivated reasoning but is only as good as the pre-specified windows; if the true decay is much slower than assumed when the windows were chosen, day 91 may still be well inside the transient period, so a short pilot or a conservative overestimate of the likely half-life should inform window choice up front, not just the final analysis.

Direct answer

Treat a single surprising segment finding, mobile shows a significant lift while the overall test is flat, as a hypothesis to validate, not a result to act on. Work through data-integrity checks, a formal interaction test with a multiplicity correction (since this segment was very likely noticed after the fact rather than pre-specified), and a set of robustness checks; then use an explicit decision process that weighs the statistical uncertainty against the business value and cost of being wrong, rather than a pure significance threshold, to choose between shipping to that segment, running a confirmatory follow-up, or abandoning the finding.

Structured elaboration

Step 1: verify the data before trusting the effect

• Check assignment balance within mobile specifically: treatment and control counts, and balance on key covariates, within the mobile slice alone, not just in aggregate.
• Check for instrumentation differences: missing events, a different SDK version, or a different exposure window on mobile that could produce a spurious effect having nothing to do with the treatment.
• Check for timing issues: did the mobile rollout start at the same time as the rest of the experiment, and is there any cross-over where a user appears in both device buckets across the test window.

Step 2: test the interaction formally

Fit an interaction model rather than comparing the mobile-only conversion rate to the mobile-only control rate informally:

import statsmodels.formula.api as smf

df["treat"] = df["assignment"].map({"control": 0, "treatment": 1})
model = smf.logit("conversion ~ treat + mobile + treat:mobile + signup_channel", data=df).fit()
print(model.summary())

Illustrative output (a hypothetical summary row, not a real run) would show a coefficient, standard error, z-value, and p-value for each term; the row that matters most here is treat:mobile. A row reading something like treat:mobile coef = 0.18, p = 0.02, alongside a treat main-effect coefficient close to zero and non-significant, is the pattern that supports a genuine mobile-specific effect: the interaction term carries the real signal while the main treatment effect alone looks flat, consistent with the original observation that the overall test showed no lift. A significant coefficient on treat:mobile is what actually supports "the effect really differs by device," rather than the mobile-only point estimate on its own, which can look large purely from within-mobile noise.

Step 3: correct for multiplicity honestly

Ask directly whether mobile was a subgroup chosen before the test ran or one noticed afterward because it happened to look interesting. If it was not pre-specified, and in practice it usually was not when this kind of question comes up, apply a multiplicity correction appropriate to however many segments were actually eyeballed (even informally) before mobile stood out, or at minimum treat the raw p-value as an optimistic upper bound on how surprising this finding really is.

Step 4: check power on the mobile slice itself

Compute the sample size and event count within mobile alone and the confidence interval width on its effect estimate. A wide interval or a small mobile sample means the "significant" reading is fragile, and this matters even more when mobile is a genuinely small-traffic segment (a specific device class or platform with limited volume) rather than merely a smaller slice of a large population: in that case a confirmatory follow-up restricted to the same segment may take a long time to reach adequate power, or may never fully reach the same statistical bar as the overall test, which is itself part of the decision, not a reason to ignore the finding.

Step 5: robustness checks

• Look at related metrics (engagement, retention, complaint or refund rate) to see whether they move in a direction consistent with the primary metric's mobile-specific lift, or whether the primary metric is moving alone in a way that is harder to explain.
• Check whether the effect is stable over the test window or concentrated in a short burst of days.
• Check finer sub-slices of mobile (iOS versus Android, OS version) to rule out the effect actually being driven by one narrow slice within "mobile" rather than the device class as a whole.
• Re-run with alternative covariate adjustment and see whether the interaction coefficient is stable.

Worked example: the decision process

Rather than a bare "p < 0.05 so ship it" rule, weigh four inputs explicitly: how strong the statistical evidence is after the checks above, how large and reliable the resulting business value would be if the effect is real, how costly it is if the segment is shipped and the effect turns out not to be real, and how long a confirmatory follow-up on that segment alone would realistically take to reach adequate power given the segment's own traffic volume.

• Strong evidence, low cost of being wrong, fast to confirm: ship a small, reversible rollout to the segment while a confirmatory read continues, since the downside of being wrong is small and quickly detected.
• Moderate evidence, or the segment is small enough that a proper confirmatory test would take a long time to reach power: this is the case worth naming explicitly, since waiting for full statistical certainty may never be practical for a genuinely small segment. Here, the decision becomes an explicit risk-tolerance call: state the estimated cost of shipping on an unconfirmed finding versus the estimated cost of never acting on a real effect because the segment could never generate enough data to confirm it on its own, and make that trade-off visible to the decision-maker rather than deferring it to a p-value the segment may structurally never be able to produce.
• Weak evidence, or a moderate cost of being wrong: run a dedicated, pre-specified confirmatory experiment targeted at the segment before making any production change, treating the original finding purely as the hypothesis that justified the follow-up.
• Evidence disappears after the data-integrity and robustness checks: abandon the finding and document why, so the same slice does not get re-litigated the next time someone happens to look at it.

Trade-offs & pitfalls

• Treating an unadjusted subgroup p-value as decisive. The interaction test plus a multiplicity correction is what separates a real segment effect from one of several plausible slices that happened to look significant.
• Waiting indefinitely for a small segment to reach the same statistical bar as the overall test. For a genuinely low-traffic segment, that bar may not be reachable on a useful timeline; the decision framework needs to say what happens in that case rather than defaulting to inaction.
• Ignoring instrumentation as a candidate explanation. A device-specific logging or SDK difference is a mundane but common cause of an apparent segment effect and should be ruled out before any statistical machinery is trusted.
• Shipping on a single significant slice with no plan to re-check it. Even a reversible segment rollout should carry a defined follow-up read, not be treated as a closed decision the moment it ships.