A/B Test Design & Statistical Rigor Questions

Direct answer

For a high-variance, long-tailed metric, the highest-leverage combination is usually: a variance-stabilizing transform or a bounded cap to tame the tail, so a handful of extreme users don't dominate the variance, plus a pre-experiment covariate adjustment (CUPED) or stratified randomization to remove variance driven by predictable between-user differences rather than the treatment. These attack two different sources of variance (extreme values, and predictable heterogeneity) and combine rather than being redundant with each other, but every step needs to be validated against historical data before trusting the reduction, because a transform that looks good on paper can distort the very effect you are trying to measure.

Structured elaboration

Data transformations for the tail

• Winsorization: cap the metric at a percentile, for example the 99th, computed from historical pre-experiment data, so extreme values are pulled in rather than dropped. This reduces variance from measurement noise (a bot, a bulk-purchase outlier) but also caps genuinely large real effects, so it should be validated against how much of the historical tail is noise versus real signal.
• Log or Box-Cox transform: compresses a right-skewed distribution so within-group variance shrinks. The trade-off is interpretability: a treatment effect on $\log(1+x)$ is a statement about a multiplicative change, not the raw metric, and has to be back-transformed carefully, since the mean of a log-transformed variable does not simply exponentiate back to the mean of the original.

Covariate-based reduction: CUPED or stratification

• CUPED (Deng et al., WSDM 2013) subtracts a scaled pre-experiment version of the same metric, or a closely related one, using $\theta^* = \mathrm{Cov}(X,Y)/\mathrm{Var}(X)$, which reduces the outcome's variance by a factor of $(1-\rho^2)$ where $\rho$ is the pre/post correlation. It works best when there is enough pre-experiment history to compute a stable $X$ for most users.
• Stratified randomization and analysis: bucket users by a strong predictor, such as spend tier, tenure, or geography, before randomizing, and analyze the treatment effect within strata before combining. This removes the between-stratum component of variance instead of the within-user pre/post correlation; it needs less history than CUPED, working even for users with no pre-experiment data as long as the stratifying variable is known at assignment time, but generally buys less reduction unless the strata are strongly predictive.

Instrumentation changes to support the strategy

• Log a stable pre-experiment window, for example the 28 days before randomization, for the CUPED covariate for every user, computed identically regardless of arm.
• Log the stratification variable at assignment time, not derived after the fact from data that could itself be affected by treatment.
• Cap or flag extreme single-session contributions at the event level, server-side, so winsorization thresholds computed from historical data stay meaningful going forward rather than drifting as product behavior changes.

Validating the reduction on historical data

Backtest the whole pipeline before trusting it in a live experiment: take a completed historical experiment, or two truly random historical splits with no real effect, apply the proposed transform, CUPED adjustment, and stratification, and compare the variance of the adjusted metric to the variance of the raw metric on the exact same underlying data. Report the actual measured ratio, not an assumed one, and separately confirm the point estimate of the (should-be-zero, for a null historical split) effect did not move, the check that the adjustment reduced variance without introducing bias.

Worked example

Take a stated, illustrative backtest on a historical null split (two random halves of historical traffic, no real treatment): raw metric variance $\sigma_Y^2 = 250$ (illustrative units), and the chosen pre-experiment covariate (the prior 28-day value of the same metric) has $\mathrm{Cov}(X,Y) = 100$ and $\mathrm{Var}(X) = 200$ on that same historical data. Then:
$\theta^* = \frac{100}{200} = 0.5, \qquad \rho = \frac{100}{\sqrt{200}\sqrt{250}} = \frac{100}{\sqrt{50{,}000}} \approx \frac{100}{223.6} \approx 0.447$
$\mathrm{Var}(Y') = \sigma_Y^2(1-\rho^2) = 250 \times (1 - 0.2) = 250 \times 0.8 = 200$
CUPED alone gives roughly a 20% variance reduction on this historical backtest. If winsorizing the top 1% of the historical raw metric separately reduces variance from 250 to a stated measured 220, a 12% reduction computed the same way, $(250-220)/250 = 0.12$, before CUPED is applied, and the two are roughly independent sources of variance, applying both in sequence (winsorize, then CUPED on the winsorized metric) is expected to land somewhere between the larger single effect and their combination. The backtest should report the actual combined number measured on the winsorized-then-adjusted metric rather than assuming the two percentages multiply cleanly, since winsorization can itself change $\rho$.

Trade-offs and pitfalls

• Winsorization thresholds fit to historical data and then held fixed can become stale as the product and user base evolve; a threshold that was the 99th percentile last quarter can clip a growing share of real, non-noise values today if left unmonitored.
• Stacking transform, CUPED, and stratification adds real pipeline complexity, more logged fields, more failure points, more places for a bug to silently break the primary metric; each addition should be justified by a measured variance reduction on the backtest, not added by default.
• CUPED requires the covariate to be genuinely pre-experiment; if the "pre-period" window is computed with a pipeline that lags or leaks into the experiment window, this reintroduces the treatment-affected-covariate bias silently, since a backtest on historical null data would not catch a bug that only manifests once treatment exists.
• A large measured variance reduction on a historical backtest is not a guarantee of the same reduction on the live experiment if the live population or the metric's behavior has shifted since the historical window; treat the backtest number as a planning estimate, and re-check variance reduction on the live experiment's own pre-period data before finalizing a launch decision based on it.

Direct answer

Before recommending rollout, I would not treat this as a single significance comparison; I would run a short sequence of checks: confirm the guardrail regression is real and not an artifact, check whether the guardrail movement is even large enough to be distinguishable from noise given the traffic it got (a guardrail is often powered for a much smaller effect than the primary, so "not significant" there can just mean underpowered, not "fine"), rule out a novelty or primacy effect as the explanation for the primary win, and then apply a pre-agreed decision rule rather than a judgment call made after seeing the numbers. If no pre-agreed rule exists, the honest fallback is a staged, guarded rollout with a long-run holdout, not an outright ship.

Structured elaboration

Step 1: Robustness checks on both metrics

• Segment the guardrail regression. Is it concentrated in one platform, cohort, or geography, or spread evenly? A regression concentrated in a narrow segment points at something mechanical (a bug or UX defect specific to that segment) rather than a real, generalizable trade-off.
• Check assignment health. Re-run the same sample-ratio and pre-period balance checks you would run on any experiment; a guardrail move that traces back to a randomization or instrumentation issue is not a real trade-off at all.
• Check the primary win's time course for a novelty effect. A novelty effect is a temporary lift driven by the change being new and attention-grabbing rather than a durable improvement; it typically shows as a large early lift that decays over the experiment window. Plot the primary metric's daily effect size: if it is shrinking over time while the guardrail regression is stable or growing, the primary "win" may partly evaporate on its own before you even weigh the trade-off. The mirror case, a primacy effect, is when existing users are initially resistant to a change (a lift that starts low and grows as people adapt); it matters here mainly as a reason not to over-read a weak early primary result as a fair test either.

Step 2: Is the guardrail result adequately powered

A guardrail that shows "not statistically significant regression" is not the same claim as "no regression." State this as a single check, not a full derivation: given the traffic the experiment actually got, was the guardrail's measurement precise enough to rule out a regression of a magnitude you would actually care about, or is the interval simply too wide to conclude anything either way. If the guardrail is underpowered at the traffic level the primary metric was sized for, that is itself the answer: you do not have enough information yet to trust a rollout, independent of which way the point estimate leans.

Step 3: Weighing a short-term win against a longer-term cost

This is a business trade-off, not a pure statistics question, and the responsible move is to make the trade explicit rather than intuit it. State the primary metric's estimated near-term value and the guardrail's estimated longer-term cost in the same unit (commonly revenue, or another shared north-star), even if one side of that conversion is an approximation, and be explicit about which parts are measured versus assumed. Two structural reasons this is often harder than it looks:

• The primary metric (e.g., short-term engagement or conversion) is usually measured over days, while the guardrail (e.g., retention) compounds over a much longer horizon; a small daily retention hit, if it persists, can outweigh a larger one-time primary gain once compounded over the retention metric's own natural time window.
• The primary effect and the guardrail effect may not be the same size in the population they touch; a lift concentrated in low-value or already-churny users paired with a regression concentrated in high-value users is a worse trade than the same headline numbers spread evenly, which is why the segment check in Step 1 also feeds directly into this weighing step.

Step 4: The decision rule

The rule should exist before you are looking at a live result, exactly like a guardrail threshold. In order of preference:

1. If a pre-committed guardrail threshold and pause rule exist and were breached, honor it. Do not relitigate the threshold after seeing the number; that defeats the purpose of pre-committing it.
2. If no explicit threshold exists, do not ship outright. Treat this as evidence the guardrail set was incomplete going in, fix that for next time, and in the meantime prefer the conservative path below over an ad hoc judgment call.
3. Stage the rollout with a long-run holdout. Ramp exposure gradually (e.g., a small percentage first) while keeping a genuine holdout population unexposed for an extended window well past the point of the initial ship decision, specifically to catch a guardrail effect that is slow to fully appear (churn, trust erosion) even if it looked borderline at the original read.
4. Re-test the specific element suspected of causing the trade-off, isolated from the rest of the change, if the segment and mechanism checks point at one particular piece of the change rather than the whole feature.

This pattern generalizes

The same discipline applies with the trade direction reversed, for example a retention gain paired with an ARPU regression, and to slower-arriving guardrails, for example a generative-AI product where short-term engagement rises but downstream purchases decline over a longer window; both need the same segment, power, and pre-committed-rule checks described above, not a different framework. It also applies to the inverse failure mode: several secondary metrics flag as significant while the primary metric itself is null. That case is a multiple-comparisons risk, not a real signal by default, since checking many metrics at once raises the odds that some look significant purely by chance; treat only the pre-declared guardrails as carrying an automatic mandate to act, and require an unplanned secondary flag to clear a higher, dedicated bar before it changes the decision. Some organizations formalize the whole sequence into an explicit two-stage gate, a short-term engagement stage followed by a separate long-run retention or monetization stage, each with its own pre-declared error-rate control; that is a heavier, more procedural version of the same pre-commitment discipline, and the statistical mechanics of controlling error rates across the two stages belong to hypothesis-testing theory rather than to this design question. For a small, non-significant secondary movement that still looks concerning, the right response is neither to ignore it nor to react to noise in the moment: pre-specify a dedicated, adequately powered follow-up check on that one metric rather than relitigating the current experiment's result under pressure.

Worked example

A feed-ranking change shows a primary click-through lift that is largest in the first three days and roughly half that size by day ten (a decaying pattern read directly off the daily-effect series), alongside a 7-day retention guardrail that moved negative but with a confidence interval that comfortably includes zero. Two things are true at once here: the guardrail result does not clear the bar for "proven regression," and the primary result shows the shape of a novelty effect rather than a stable lift. Given both, the defensible move is neither an unconditional ship (the primary win may partly be novelty, and the guardrail is not cleanly exonerated, just underpowered) nor an unconditional rollback (nothing is proven broken); it is a staged rollout with an extended holdout sized to actually resolve the guardrail question, with a decision point set for after the primary metric's trend has had time to settle.

Trade-offs and pitfalls

• The single biggest mistake in this scenario is treating "guardrail not statistically significant" as "guardrail cleared," when it may simply be underpowered; always check power before treating a null guardrail result as reassurance.
• Deciding the trade-off after seeing which way the numbers lean, rather than applying a rule set before the experiment, is how teams talk themselves into shipping a change they would not have pre-approved.
• A holdout that is too short to catch a slow-moving guardrail effect gives false confidence; size the holdout window to the guardrail's own natural time horizon (e.g., a retention guardrail needs a window long enough for retention itself to be observed), not to the primary metric's faster clock.

Direct answer

Treat a single surprising segment finding, mobile shows a significant lift while the overall test is flat, as a hypothesis to validate, not a result to act on. Work through data-integrity checks, a formal interaction test with a multiplicity correction (since this segment was very likely noticed after the fact rather than pre-specified), and a set of robustness checks; then use an explicit decision process that weighs the statistical uncertainty against the business value and cost of being wrong, rather than a pure significance threshold, to choose between shipping to that segment, running a confirmatory follow-up, or abandoning the finding.

Structured elaboration

Step 1: verify the data before trusting the effect

• Check assignment balance within mobile specifically: treatment and control counts, and balance on key covariates, within the mobile slice alone, not just in aggregate.
• Check for instrumentation differences: missing events, a different SDK version, or a different exposure window on mobile that could produce a spurious effect having nothing to do with the treatment.
• Check for timing issues: did the mobile rollout start at the same time as the rest of the experiment, and is there any cross-over where a user appears in both device buckets across the test window.

Step 2: test the interaction formally

Fit an interaction model rather than comparing the mobile-only conversion rate to the mobile-only control rate informally:

import statsmodels.formula.api as smf

df["treat"] = df["assignment"].map({"control": 0, "treatment": 1})
model = smf.logit("conversion ~ treat + mobile + treat:mobile + signup_channel", data=df).fit()
print(model.summary())

Illustrative output (a hypothetical summary row, not a real run) would show a coefficient, standard error, z-value, and p-value for each term; the row that matters most here is treat:mobile. A row reading something like treat:mobile coef = 0.18, p = 0.02, alongside a treat main-effect coefficient close to zero and non-significant, is the pattern that supports a genuine mobile-specific effect: the interaction term carries the real signal while the main treatment effect alone looks flat, consistent with the original observation that the overall test showed no lift. A significant coefficient on treat:mobile is what actually supports "the effect really differs by device," rather than the mobile-only point estimate on its own, which can look large purely from within-mobile noise.

Step 3: correct for multiplicity honestly

Ask directly whether mobile was a subgroup chosen before the test ran or one noticed afterward because it happened to look interesting. If it was not pre-specified, and in practice it usually was not when this kind of question comes up, apply a multiplicity correction appropriate to however many segments were actually eyeballed (even informally) before mobile stood out, or at minimum treat the raw p-value as an optimistic upper bound on how surprising this finding really is.

Step 4: check power on the mobile slice itself

Compute the sample size and event count within mobile alone and the confidence interval width on its effect estimate. A wide interval or a small mobile sample means the "significant" reading is fragile, and this matters even more when mobile is a genuinely small-traffic segment (a specific device class or platform with limited volume) rather than merely a smaller slice of a large population: in that case a confirmatory follow-up restricted to the same segment may take a long time to reach adequate power, or may never fully reach the same statistical bar as the overall test, which is itself part of the decision, not a reason to ignore the finding.

Step 5: robustness checks

• Look at related metrics (engagement, retention, complaint or refund rate) to see whether they move in a direction consistent with the primary metric's mobile-specific lift, or whether the primary metric is moving alone in a way that is harder to explain.
• Check whether the effect is stable over the test window or concentrated in a short burst of days.
• Check finer sub-slices of mobile (iOS versus Android, OS version) to rule out the effect actually being driven by one narrow slice within "mobile" rather than the device class as a whole.
• Re-run with alternative covariate adjustment and see whether the interaction coefficient is stable.

Worked example: the decision process

Rather than a bare "p < 0.05 so ship it" rule, weigh four inputs explicitly: how strong the statistical evidence is after the checks above, how large and reliable the resulting business value would be if the effect is real, how costly it is if the segment is shipped and the effect turns out not to be real, and how long a confirmatory follow-up on that segment alone would realistically take to reach adequate power given the segment's own traffic volume.

• Strong evidence, low cost of being wrong, fast to confirm: ship a small, reversible rollout to the segment while a confirmatory read continues, since the downside of being wrong is small and quickly detected.
• Moderate evidence, or the segment is small enough that a proper confirmatory test would take a long time to reach power: this is the case worth naming explicitly, since waiting for full statistical certainty may never be practical for a genuinely small segment. Here, the decision becomes an explicit risk-tolerance call: state the estimated cost of shipping on an unconfirmed finding versus the estimated cost of never acting on a real effect because the segment could never generate enough data to confirm it on its own, and make that trade-off visible to the decision-maker rather than deferring it to a p-value the segment may structurally never be able to produce.
• Weak evidence, or a moderate cost of being wrong: run a dedicated, pre-specified confirmatory experiment targeted at the segment before making any production change, treating the original finding purely as the hypothesis that justified the follow-up.
• Evidence disappears after the data-integrity and robustness checks: abandon the finding and document why, so the same slice does not get re-litigated the next time someone happens to look at it.

Trade-offs & pitfalls

• Treating an unadjusted subgroup p-value as decisive. The interaction test plus a multiplicity correction is what separates a real segment effect from one of several plausible slices that happened to look significant.
• Waiting indefinitely for a small segment to reach the same statistical bar as the overall test. For a genuinely low-traffic segment, that bar may not be reachable on a useful timeline; the decision framework needs to say what happens in that case rather than defaulting to inaction.
• Ignoring instrumentation as a candidate explanation. A device-specific logging or SDK difference is a mundane but common cause of an apparent segment effect and should be ruled out before any statistical machinery is trusted.
• Shipping on a single significant slice with no plan to re-check it. Even a reversible segment rollout should carry a defined follow-up read, not be treated as a closed decision the moment it ships.

Direct answer

Beyond CUPED (using a pre-experiment covariate to residualize the outcome), the two other standard variance-reduction levers are stratified (blocked) randomization, which forces balance on a known factor at assignment time instead of hoping random chance balances it, and covariate or regression adjustment, which is the general case of "adjust for a predictive covariate" that CUPED is one specific, pre-experiment-only instance of. Both work by removing a source of outcome variance that is not related to treatment, so the same true effect becomes easier to distinguish from noise; both reduce required sample size roughly in proportion to how much outcome variance the factor explains, and neither invents a new number, they trade a known, explainable source of variance for a smaller residual.

Structured elaboration

Stratified (blocked) randomization

Instead of randomizing the whole population as one pool, split the population into strata on a factor known before assignment (country, device type, new vs. returning user), then randomize independently within each stratum so each arm gets a matched share of every stratum. This removes between-stratum variance from the treatment-effect estimator's variance, because the strata are balanced by design rather than by luck: with plain randomization on a highly imbalanced population, an unlucky split (e.g., treatment skewing toward the low-baseline country) inflates the observed variance of the effect estimate even though the true effect is unaffected.

It is applicable whenever you have a discrete, pre-assignment factor that is known to correlate with the outcome and is stable at randomization time. It differs from covariate adjustment in when the correction happens: stratification acts at assignment time (balance is enforced), while regression adjustment acts at analysis time (balance is estimated and subtracted after the fact). The two are complementary, not substitutes: stratify at assignment for the factors you can, and adjust for continuous covariates at analysis.

Covariate / regression adjustment

This is the general technique of fitting a model for the outcome on one or more covariates (not restricted to pre-experiment-only, unlike CUPED) and using the model to remove predictable variance from the outcome before comparing arms, most simply via ANCOVA (analysis of covariance), a linear regression of $Y$ on the treatment indicator and covariates that removes the variance those covariates explain from the comparison, the same variance-reduction logic as CUPED and stratification, just carried out as a regression rather than a pre-experiment covariate or a balanced split. It is applicable whenever you have covariates, pre-experiment or otherwise as long as they cannot themselves have been affected by treatment, that are predictive of the outcome. CUPED is the special case where the covariate is restricted to a pre-experiment value of the outcome metric itself; regression adjustment generalizes this to any number of eligible covariates and lets you combine several weak predictors into one stronger adjustment.

Effect on sample size and power

For both techniques, if the factor being controlled for explains a fraction $R^2$ of the outcome's variance, the variance of the treatment-effect estimator shrinks by roughly that same factor, and required sample size for a fixed target precision shrinks proportionally, since sample size for a fixed effect and power scales with the variance of the metric. A factor that explains little of the outcome variance buys little; a strong, well-chosen factor can meaningfully shorten the required test duration for the same statistical bar.

Worked example: stratifying a multi-country test

A test is planned across three countries with very different baseline conversion rates: Country A at 4%, Country B at 12%, Country C at 22%, in roughly equal traffic shares (each about one third of total users). Without stratification, plain randomization can by chance send more of one country's traffic to one arm, and even without that bad luck, the pooled outcome variance includes the between-country spread of baseline rates as extra noise the estimator has to average out.

Using the law of total variance, the overall variance of the outcome decomposes as:

$\operatorname{Var}(Y) = \underbrace{\mathbb{E}\big[\operatorname{Var}(Y \mid \text{country})\big]}_{\text{within-country variance}} + \underbrace{\operatorname{Var}\big(\mathbb{E}[Y \mid \text{country}]\big)}_{\text{between-country variance}}$

Stratifying by country and analyzing as a weighted average of within-country treatment effects removes the second (between-country) term from the treatment-effect estimator's variance, since each stratum is separately balanced and the between-stratum spread no longer contributes noise to the comparison. Concretely: with baseline rates of 4%, 12%, 22% and equal stratum weights, the between-country component of variance is

$\bar p = \frac{0.04+0.12+0.22}{3} = 0.1267$

$\operatorname{Var}(\bar p) = \frac{1}{3}\Big[(0.04-\bar p)^2 + (0.12-\bar p)^2 + (0.22-\bar p)^2\Big] = \frac{1}{3}(0.00751 + 0.0000445 + 0.00871) = 0.00542$

That 0.00542 is exactly the between-country variance component the stratified analysis removes from the pooled estimator's variance, computed directly from the three stated baseline rates, not asserted; how large a share of total variance that is depends additionally on the within-country binomial variance at each rate, which you would combine with this term using the same decomposition to get the full picture before quoting an overall percentage reduction.

Implementation for the multi-country case

• Assign the stratum at randomization time using the same deterministic hash-bucketing approach as the overall unit assignment, but nest it: hash within each country separately (or include country in the hash key) so each country independently hits its target split ratio.
• At analysis time, estimate the treatment effect within each country and combine as a weighted average (weighted by stratum size or by inverse variance), rather than pooling raw counts across countries, which is what actually realizes the variance reduction shown above.

Trade-offs and pitfalls

• Stratifying on too many dimensions at once shrinks individual strata until some contain too few units to balance meaningfully, and can create empty or near-empty cells, especially when crossing multiple categorical factors (country times device times cohort).
• A stratification factor chosen because it is convenient rather than because it is predictive buys little variance reduction while adding real implementation complexity; check the factor's explanatory power on historical data before committing the assignment pipeline to it.
• Regression adjustment on covariates measured close to, but not strictly after, the treatment start needs the same scrutiny as CUPED's pre-experiment-only requirement: any covariate that could plausibly be influenced by treatment invalidates the adjustment's unbiasedness, not just its efficiency.

Direct answer

Convert a sample-size target into a run duration in two steps: compute the required sample size per variant with the standard two-proportion test formula, then divide that by how much daily traffic actually lands in each variant (daily uniques times allocation), rounding up. The formula gives you a headcount; the traffic split is what turns a headcount into a calendar.

Structured elaboration

Step 1: the sample-size formula

For a two-sided test comparing baseline conversion $p_1$ against a target conversion $p_2$, at significance $\alpha$ and power $1-\beta$:

$n = \frac{\left(z_{1-\alpha/2}\sqrt{2\bar p(1-\bar p)} + z_{1-\beta}\sqrt{p_1(1-p_1)+p_2(1-p_2)}\right)^2}{(p_2-p_1)^2}, \quad \bar p = \frac{p_1+p_2}{2}$

This is per-variant sample size; assumes a two-sided test with no sequential peeking (one look at the end) and independent, one-conversion-per-user data. Sequential monitoring and multiple-comparisons corrections change this number and are a separate design decision, not part of the raw duration estimate.

Step 2: convert daily traffic into per-variant daily volume

$\text{daily per variant} = \text{daily unique visitors} \times \text{allocation share}$

Step 3: convert sample size into days

$\text{days} = \frac{n}{\text{daily per variant}}$

Round the sample size up (never down, since undershooting trades away the power you asked for) and round the resulting day count up to a whole day. If the metric has meaningful weekday/weekend variation, round further up to a whole number of weeks so every arm sees the same mix of weekdays and weekends; that decision is a separate seasonality question with its own reasoning (see the seasonality-planning answer), not something the raw formula above accounts for.

Assumptions and rounding choices worth stating out loud

• Two-sided test with a single, pre-planned final look; no interim peeking.
• Users are independent and contribute one conversion event each; no repeated exposure double counting.
• All intermediate values kept to about five significant figures before rounding the final answer, and the final day count always rounds up, not to the nearest day.
• A buffer of roughly 10-20% is a common practical addition on top of the raw day count to absorb data loss (bot filtering, QA holds, instrumentation gaps); state it as a buffer, not as part of the statistical requirement.
• If daily traffic is volatile rather than a single stable number, use a conservative (lower) daily estimate for the duration calculation rather than the average, since under-running the test is far more costly than over-running it by a day or two.

Worked example

Inputs: 60,000 daily unique eligible visitors, 50/50 allocation, baseline conversion $p_1 = 3.0\%$, desired minimum detectable effect of 8% relative, $\alpha=0.05$ two-sided ($z_{1-\alpha/2}=1.9600$), power 80% ($z_{1-\beta}=0.8416$).

Absolute MDE: $\Delta = 0.03 \times 0.08 = 0.0024$, so $p_2 = 0.0324$, $\bar p = 0.0312$.

$z_{1-\alpha/2}\sqrt{2\bar p(1-\bar p)} = 1.9600 \times \sqrt{2 \times 0.0312 \times 0.9688} = 1.9600 \times 0.2459 = 0.4819$

$z_{1-\beta}\sqrt{p_1(1-p_1)+p_2(1-p_2)} = 0.8416 \times \sqrt{0.03\times0.97 + 0.0324\times0.9676} = 0.8416 \times 0.2459 = 0.2069$

$n = \frac{(0.4819+0.2069)^2}{(0.0024)^2} = \frac{0.4744}{0.00000576} \approx 82{,}376 \text{ users per variant}$

Daily per variant: $60{,}000 \times 0.5 = 30{,}000$.

$\text{days} = \frac{82{,}376}{30{,}000} \approx 2.75 \rightarrow \text{round up to } 3 \text{ days minimum}$

Because this crosses a weekday/weekend boundary either way, the practical duration recommendation would be at least 7 days (one full week) rather than the bare 3-day statistical minimum, so weekday and weekend behavior are represented in the same proportion in both arms.

Trade-offs & pitfalls

• MDE sensitivity. The required sample size scales with the inverse square of the MDE, so halving the effect you want to detect roughly quadruples the required sample and the resulting duration; a stakeholder asking for a smaller MDE "just to be safe" is asking for a much longer test, not a marginally longer one.
• Binary-outcome formula does not transfer to continuous metrics. Revenue-per-user or time-on-task outcomes use the outcome's variance, not $p(1-p)$, in the same general formula shape; plugging a conversion-rate formula into a continuous metric silently understates or overstates the required sample.
• Shortcutting the calendar rounding. A raw day count under 7 does not mean the test is safe to run for that literal number of days; the weekly-cycle rounding matters as much as the raw arithmetic and is a common place teams cut a real corner under launch pressure.
• Traffic volatility. A single "daily uniques" number hides day-to-day swings; a duration plan built on a lucky high-traffic day will run short in practice.