A/B Test Design & Statistical Rigor Questions

Direct answer

A launch experiment tells you what happens over its own short window; it cannot tell you what happens after novelty fades, after users have had months to adjust their real behavior, or after a pricing algorithm has compounded across several billing cycles. A long-run holdout is a small slice of the eligible population that is deliberately kept on the old experience indefinitely, purely so you still have a counterfactual after everyone else has moved on. Once you roll out to 100%, that counterfactual disappears unless you built one in on purpose, so the holdout is not a nicety, it is the only way to keep answering "compared to what" after full rollout.

Structured elaboration

What the launch experiment structurally cannot tell you

• Novelty and primacy effects. A novelty effect is a short-term bump from users noticing and exploring something new, which fades as the new experience becomes routine. A primacy effect is the opposite pattern: a change that depresses behavior briefly while users re-learn a workflow, then recovers or improves as they adapt. A one- or two-week launch test mostly measures whichever of these dominates early, not the steady-state effect.
• Compounding and delayed effects. A pricing algorithm change or an onboarding flow can shift 90-day retention, churn, or lifetime value in ways that simply have not happened yet by the time the launch test ends. There is nothing to measure early because the outcome has not occurred.
• Post-launch drift. Once a feature is fully shipped, everything else in the product keeps changing around it (other launches, seasonality, market conditions). Without a live control, you cannot separate the feature's ongoing effect from all of that background drift.

Sizing the holdout

Size the holdout the same way you would size any two-arm comparison: pick the smallest long-run effect you would regret missing on your slowest-maturing primary metric, then run the sample-size calculation against your actual eligible population.

$n = \frac{\left(z_{1-\alpha/2}\sqrt{2\bar p(1-\bar p)} + z_{1-\beta}\sqrt{p_1(1-p_1)+p_2(1-p_2)}\right)^2}{(p_2-p_1)^2}, \quad \bar p = \frac{p_1+p_2}{2}$

This gives you the minimum number of users per arm; convert that into a required holdout percentage against your eligible population size (worked below). Round the resulting percentage up, both for attrition out of the holdout itself and because a holdout that is exactly borderline-powered on day one will be underpowered a year later as the population shifts.

How long to maintain it

Tie the minimum duration to the natural maturation window of the outcome you actually care about (a 90-day retention outcome needs at least one full 90-day window past stabilization, not one arbitrary calendar month). Beyond that minimum, treat "keep the holdout" as a decision revisited on a fixed cadence (for example, quarterly) rather than a permanent default:

• If two consecutive readout windows show a stable, well-understood effect, that is the trigger to either retire the holdout or shrink it to a smaller size that still clears the power bar above.
• If the effect is still moving or the product around the feature keeps changing, that is the trigger to keep the holdout at full size.

Surrogate metrics while waiting for the readout

Waiting 90 days for the primary outcome does not mean flying blind for 90 days. Track short-horizon metrics that historically correlate with the long-run outcome, such as week-1 activation or day-7 return rate as leading indicators for 90-day retention, and monitor them on a lightweight, non-primary basis. Two things matter about surrogate metrics: they are for early warning only ("this is trending in a worrying direction, look closer"), and they do not substitute for the long-run readout, because a surrogate can move without the outcome it is supposed to predict actually moving in the same direction once the novelty period ends.

Keeping the holdout uncontaminated

SUTVA, the stable unit treatment value assumption, is the assumption that one unit's outcome does not depend on another unit's treatment assignment. A long-run holdout only tells the truth if it holds: if held-out and treated users interact (shared households, marketplace two-sidedness, referral loops), the "control" group is partly experiencing the treatment through spillover and the comparison is biased. Keep the holdout cohort assignment stable and out of unrelated concurrent experiments on the same surface, and periodically re-check that its demographics still resemble the overall population (holdout users who disproportionately churn out over time silently change what the holdout represents).

Communicating the cost to stakeholders

Frame the holdout as a bounded cost with a defined trigger to shrink it, not an indefinite tax on the business:

• State the cost concretely: holdout size times the per-user value of the already-measured launch uplift times the time period, so a stakeholder sees an opportunity-cost number in the same units as their other decisions, not an abstract appeal to rigor.
• Pair that cost with what it buys: the ability to catch a long-run reversal (a change that looked good for two weeks but erodes retention over two quarters) before it has already happened to 100% of users.
• Offer a shrinking schedule tied to the review cadence above, so "keep a holdout forever" is never the actual proposal on the table.

Worked example

Suppose the primary long-run outcome is 90-day retention, currently at a 40% baseline, and the team wants to be able to detect a 1 percentage point absolute erosion (39% vs 40%) at $\alpha=0.05$ two-sided, 80% power ($z_{1-\alpha/2}=1.9600$, $z_{1-\beta}=0.8416$).

$\bar p = \frac{0.40+0.39}{2} = 0.395$

$z_{1-\alpha/2}\sqrt{2\bar p(1-\bar p)} = 1.9600 \times \sqrt{2 \times 0.395 \times 0.605} = 1.9600 \times 0.6910 = 1.3544$

$z_{1-\beta}\sqrt{p_1(1-p_1)+p_2(1-p_2)} = 0.8416 \times \sqrt{0.40\times0.60 + 0.39\times0.61} = 0.8416 \times 0.6931 = 0.5833$

$n = \frac{(1.3544+0.5833)^2}{(0.01)^2} = \frac{3.7511}{0.0001} \approx 37{,}513 \text{ users per arm}$

Against a population of 2,000,000 monthly eligible users, a 1% holdout (20,000 users) falls short of this bar, a 2% holdout (40,000 users) clears it with a small margin, and a 3% holdout (60,000 users) clears it comfortably and leaves room for attrition. That is the actual decision: 2% is the honest minimum for this MDE, 3% is the safer operating choice, and anything above that is buying detection of a smaller effect than the team said it cared about, at a cost that keeps growing.

Trade-offs & pitfalls

• Conflating a holdout with a canary. A canary (see ramp and staged rollout) exists to catch acute, short-term harm during rollout. A holdout exists to measure a slow-moving counterfactual after rollout is complete. Sizing and duration logic for one does not transfer to the other; a 2% canary held for three days answers a completely different question than a 2% holdout held for two quarters.
• Letting the holdout go stale. A holdout that was correctly sized against last year's population and last year's MDE can silently become underpowered as traffic composition shifts; the sizing calculation is not a one-time exercise.
• Treating "permanent" as the default answer. The strongest version of this answer is a holdout with an explicit re-evaluation trigger, not an open-ended commitment that stakeholders correctly resent paying for indefinitely.

Direct answer

SUTVA, the Stable Unit Treatment Value Assumption, is the assumption underlying a standard A/B test that one unit's observed outcome depends only on which arm that unit itself was assigned to, and not on which arms other units received. It has two components: no interference between units (your outcome is not affected by someone else's assignment) and no hidden variation of treatment (everyone labeled "treatment" received the same treatment). When either fails, the simple difference-in-means between arms is no longer an unbiased estimate of the causal effect you think you are measuring.

Structured elaboration

The two components

1. No interference between units. Unit $i$'s potential outcome under any assignment vector depends only on $i$'s own treatment, not on the treatments assigned to units $j \neq i$. This is the component that breaks in networked or shared-resource products.
2. No hidden variation of treatment (consistency). There is exactly one version of "treatment" and one version of "control"; a unit's potential outcome is well defined given only its treatment label. This breaks when the same nominal arm is implemented differently for different users (different rollout timing, different creative, a bug that only affects some treatment users).

Two real violations and why each biases interpretation

Social feed, no-interference violation. A treated user gets a new sharing feature and posts more; their connections, who are in control, now see more of that content in their own feed even though they were never assigned to treatment. The control group's outcome (engagement) is contaminated by treatment spillover, which pulls control's measured engagement up and understates the true treatment effect: you are comparing "treatment" against "control that is partially treated," not against a clean counterfactual.

Shared inventory or capacity constraint. A promotion arm drives more purchases, which draws down a shared inventory pool or a fixed daily capacity (delivery slots, support-queue capacity) that both arms draw from. Control users now see stockouts or longer wait times caused by treatment's demand, not by anything intrinsic to being in control. This inflates the apparent treatment effect (control looks artificially worse) and, separately, means the effect you measured at the tested traffic share will not hold at 100% rollout, because the resource contention itself scales with the treatment allocation percentage.

In both cases the estimate is not merely noisy, it is biased in a specific, name-able direction, and the bias would not shrink with more sample size because it comes from the assignment mechanism interacting with the product, not from sampling error.

Designing around a suspected violation

Once you suspect interference, the standard fix is to change the unit of randomization to one large enough to contain the spillover, i.e., cluster randomization: randomize by friend-group, geographic market, or server shard instead of by individual, so that most of the interference happens within a cluster (which is internally consistent, either all-treated or all-control) rather than across the treatment/control boundary.

Cluster randomization has its own cost, though: fewer independent units means higher variance for the same total traffic, since the effective sample size is closer to the number of clusters than the number of users. It also requires the outcome to be measurable and meaningful at the cluster level, and it does not eliminate the shared-capacity case unless the constrained resource is itself scoped per cluster.

Detecting a violation you did not design around

If the violation is discovered mid-experiment rather than anticipated, e.g., a caching-configuration bug causes some control users to intermittently render treatment-arm content, the diagnostic sequence is: quantify the leak rate first (what fraction of control exposures actually rendered the treatment experience, pulled directly from logs, not estimated), then decide whether the leak is small enough to bound the bias and proceed with a documented caveat, or large enough that the read is unusable and the fix is to patch the bug and rerun rather than to try to model the contamination away after the fact.

Worked example

A marketplace runs a two-armed test on a checkout redesign, expecting independent per-user outcomes. Mid-experiment, a shared caching layer bug is found: a same-session user occasionally gets served a stale cached page from the opposite arm. Pulling exposure logs, engineering finds this affected roughly 4% of control-arm page views (a number read directly from the cache-hit logs, not assumed). That is a version-of-treatment violation, not an interference violation: some "control" users received a materially different experience than the rest of control, so the control arm is not internally consistent. The team's next step is not to reweight or model this away, because the mechanism (a caching bug) has no principled correction; it is to fix the bug and rerun the experiment cleanly, treating the contaminated run as informative only about the presence of the bug.

Trade-offs and pitfalls

• Do not assume interference is symmetric or negligible just because the product does not look "social." Shared backend resources (queues, inventory, ranking models retrained on pooled data) create interference in products with no visible network feature.
• Cluster randomization trades bias for variance; do not adopt it reflexively for every experiment on a networked product when the actual interference is small relative to the direct effect, since you would be paying a real power cost for a small bias fix.
• A violation discovered after the fact is a data-quality incident, not a modeling problem to be adjusted away; resist the temptation to "correct" biased data with a post hoc statistical patch when the honest fix is to rerun cleanly.

Direct answer

Before recommending rollout, I would not treat this as a single significance comparison; I would run a short sequence of checks: confirm the guardrail regression is real and not an artifact, check whether the guardrail movement is even large enough to be distinguishable from noise given the traffic it got (a guardrail is often powered for a much smaller effect than the primary, so "not significant" there can just mean underpowered, not "fine"), rule out a novelty or primacy effect as the explanation for the primary win, and then apply a pre-agreed decision rule rather than a judgment call made after seeing the numbers. If no pre-agreed rule exists, the honest fallback is a staged, guarded rollout with a long-run holdout, not an outright ship.

Structured elaboration

Step 1: Robustness checks on both metrics

• Segment the guardrail regression. Is it concentrated in one platform, cohort, or geography, or spread evenly? A regression concentrated in a narrow segment points at something mechanical (a bug or UX defect specific to that segment) rather than a real, generalizable trade-off.
• Check assignment health. Re-run the same sample-ratio and pre-period balance checks you would run on any experiment; a guardrail move that traces back to a randomization or instrumentation issue is not a real trade-off at all.
• Check the primary win's time course for a novelty effect. A novelty effect is a temporary lift driven by the change being new and attention-grabbing rather than a durable improvement; it typically shows as a large early lift that decays over the experiment window. Plot the primary metric's daily effect size: if it is shrinking over time while the guardrail regression is stable or growing, the primary "win" may partly evaporate on its own before you even weigh the trade-off. The mirror case, a primacy effect, is when existing users are initially resistant to a change (a lift that starts low and grows as people adapt); it matters here mainly as a reason not to over-read a weak early primary result as a fair test either.

Step 2: Is the guardrail result adequately powered

A guardrail that shows "not statistically significant regression" is not the same claim as "no regression." State this as a single check, not a full derivation: given the traffic the experiment actually got, was the guardrail's measurement precise enough to rule out a regression of a magnitude you would actually care about, or is the interval simply too wide to conclude anything either way. If the guardrail is underpowered at the traffic level the primary metric was sized for, that is itself the answer: you do not have enough information yet to trust a rollout, independent of which way the point estimate leans.

Step 3: Weighing a short-term win against a longer-term cost

This is a business trade-off, not a pure statistics question, and the responsible move is to make the trade explicit rather than intuit it. State the primary metric's estimated near-term value and the guardrail's estimated longer-term cost in the same unit (commonly revenue, or another shared north-star), even if one side of that conversion is an approximation, and be explicit about which parts are measured versus assumed. Two structural reasons this is often harder than it looks:

• The primary metric (e.g., short-term engagement or conversion) is usually measured over days, while the guardrail (e.g., retention) compounds over a much longer horizon; a small daily retention hit, if it persists, can outweigh a larger one-time primary gain once compounded over the retention metric's own natural time window.
• The primary effect and the guardrail effect may not be the same size in the population they touch; a lift concentrated in low-value or already-churny users paired with a regression concentrated in high-value users is a worse trade than the same headline numbers spread evenly, which is why the segment check in Step 1 also feeds directly into this weighing step.

Step 4: The decision rule

The rule should exist before you are looking at a live result, exactly like a guardrail threshold. In order of preference:

1. If a pre-committed guardrail threshold and pause rule exist and were breached, honor it. Do not relitigate the threshold after seeing the number; that defeats the purpose of pre-committing it.
2. If no explicit threshold exists, do not ship outright. Treat this as evidence the guardrail set was incomplete going in, fix that for next time, and in the meantime prefer the conservative path below over an ad hoc judgment call.
3. Stage the rollout with a long-run holdout. Ramp exposure gradually (e.g., a small percentage first) while keeping a genuine holdout population unexposed for an extended window well past the point of the initial ship decision, specifically to catch a guardrail effect that is slow to fully appear (churn, trust erosion) even if it looked borderline at the original read.
4. Re-test the specific element suspected of causing the trade-off, isolated from the rest of the change, if the segment and mechanism checks point at one particular piece of the change rather than the whole feature.

This pattern generalizes

The same discipline applies with the trade direction reversed, for example a retention gain paired with an ARPU regression, and to slower-arriving guardrails, for example a generative-AI product where short-term engagement rises but downstream purchases decline over a longer window; both need the same segment, power, and pre-committed-rule checks described above, not a different framework. It also applies to the inverse failure mode: several secondary metrics flag as significant while the primary metric itself is null. That case is a multiple-comparisons risk, not a real signal by default, since checking many metrics at once raises the odds that some look significant purely by chance; treat only the pre-declared guardrails as carrying an automatic mandate to act, and require an unplanned secondary flag to clear a higher, dedicated bar before it changes the decision. Some organizations formalize the whole sequence into an explicit two-stage gate, a short-term engagement stage followed by a separate long-run retention or monetization stage, each with its own pre-declared error-rate control; that is a heavier, more procedural version of the same pre-commitment discipline, and the statistical mechanics of controlling error rates across the two stages belong to hypothesis-testing theory rather than to this design question. For a small, non-significant secondary movement that still looks concerning, the right response is neither to ignore it nor to react to noise in the moment: pre-specify a dedicated, adequately powered follow-up check on that one metric rather than relitigating the current experiment's result under pressure.

Worked example

A feed-ranking change shows a primary click-through lift that is largest in the first three days and roughly half that size by day ten (a decaying pattern read directly off the daily-effect series), alongside a 7-day retention guardrail that moved negative but with a confidence interval that comfortably includes zero. Two things are true at once here: the guardrail result does not clear the bar for "proven regression," and the primary result shows the shape of a novelty effect rather than a stable lift. Given both, the defensible move is neither an unconditional ship (the primary win may partly be novelty, and the guardrail is not cleanly exonerated, just underpowered) nor an unconditional rollback (nothing is proven broken); it is a staged rollout with an extended holdout sized to actually resolve the guardrail question, with a decision point set for after the primary metric's trend has had time to settle.

Trade-offs and pitfalls

• The single biggest mistake in this scenario is treating "guardrail not statistically significant" as "guardrail cleared," when it may simply be underpowered; always check power before treating a null guardrail result as reassurance.
• Deciding the trade-off after seeing which way the numbers lean, rather than applying a rule set before the experiment, is how teams talk themselves into shipping a change they would not have pre-approved.
• A holdout that is too short to catch a slow-moving guardrail effect gives false confidence; size the holdout window to the guardrail's own natural time horizon (e.g., a retention guardrail needs a window long enough for retention itself to be observed), not to the primary metric's faster clock.

Direct answer

For a high-variance, long-tailed metric, the highest-leverage combination is usually: a variance-stabilizing transform or a bounded cap to tame the tail, so a handful of extreme users don't dominate the variance, plus a pre-experiment covariate adjustment (CUPED) or stratified randomization to remove variance driven by predictable between-user differences rather than the treatment. These attack two different sources of variance (extreme values, and predictable heterogeneity) and combine rather than being redundant with each other, but every step needs to be validated against historical data before trusting the reduction, because a transform that looks good on paper can distort the very effect you are trying to measure.

Structured elaboration

Data transformations for the tail

• Winsorization: cap the metric at a percentile, for example the 99th, computed from historical pre-experiment data, so extreme values are pulled in rather than dropped. This reduces variance from measurement noise (a bot, a bulk-purchase outlier) but also caps genuinely large real effects, so it should be validated against how much of the historical tail is noise versus real signal.
• Log or Box-Cox transform: compresses a right-skewed distribution so within-group variance shrinks. The trade-off is interpretability: a treatment effect on $\log(1+x)$ is a statement about a multiplicative change, not the raw metric, and has to be back-transformed carefully, since the mean of a log-transformed variable does not simply exponentiate back to the mean of the original.

Covariate-based reduction: CUPED or stratification

• CUPED (Deng et al., WSDM 2013) subtracts a scaled pre-experiment version of the same metric, or a closely related one, using $\theta^* = \mathrm{Cov}(X,Y)/\mathrm{Var}(X)$, which reduces the outcome's variance by a factor of $(1-\rho^2)$ where $\rho$ is the pre/post correlation. It works best when there is enough pre-experiment history to compute a stable $X$ for most users.
• Stratified randomization and analysis: bucket users by a strong predictor, such as spend tier, tenure, or geography, before randomizing, and analyze the treatment effect within strata before combining. This removes the between-stratum component of variance instead of the within-user pre/post correlation; it needs less history than CUPED, working even for users with no pre-experiment data as long as the stratifying variable is known at assignment time, but generally buys less reduction unless the strata are strongly predictive.

Instrumentation changes to support the strategy

• Log a stable pre-experiment window, for example the 28 days before randomization, for the CUPED covariate for every user, computed identically regardless of arm.
• Log the stratification variable at assignment time, not derived after the fact from data that could itself be affected by treatment.
• Cap or flag extreme single-session contributions at the event level, server-side, so winsorization thresholds computed from historical data stay meaningful going forward rather than drifting as product behavior changes.

Validating the reduction on historical data

Backtest the whole pipeline before trusting it in a live experiment: take a completed historical experiment, or two truly random historical splits with no real effect, apply the proposed transform, CUPED adjustment, and stratification, and compare the variance of the adjusted metric to the variance of the raw metric on the exact same underlying data. Report the actual measured ratio, not an assumed one, and separately confirm the point estimate of the (should-be-zero, for a null historical split) effect did not move, the check that the adjustment reduced variance without introducing bias.

Worked example

Take a stated, illustrative backtest on a historical null split (two random halves of historical traffic, no real treatment): raw metric variance $\sigma_Y^2 = 250$ (illustrative units), and the chosen pre-experiment covariate (the prior 28-day value of the same metric) has $\mathrm{Cov}(X,Y) = 100$ and $\mathrm{Var}(X) = 200$ on that same historical data. Then:
$\theta^* = \frac{100}{200} = 0.5, \qquad \rho = \frac{100}{\sqrt{200}\sqrt{250}} = \frac{100}{\sqrt{50{,}000}} \approx \frac{100}{223.6} \approx 0.447$
$\mathrm{Var}(Y') = \sigma_Y^2(1-\rho^2) = 250 \times (1 - 0.2) = 250 \times 0.8 = 200$
CUPED alone gives roughly a 20% variance reduction on this historical backtest. If winsorizing the top 1% of the historical raw metric separately reduces variance from 250 to a stated measured 220, a 12% reduction computed the same way, $(250-220)/250 = 0.12$, before CUPED is applied, and the two are roughly independent sources of variance, applying both in sequence (winsorize, then CUPED on the winsorized metric) is expected to land somewhere between the larger single effect and their combination. The backtest should report the actual combined number measured on the winsorized-then-adjusted metric rather than assuming the two percentages multiply cleanly, since winsorization can itself change $\rho$.

Trade-offs and pitfalls

• Winsorization thresholds fit to historical data and then held fixed can become stale as the product and user base evolve; a threshold that was the 99th percentile last quarter can clip a growing share of real, non-noise values today if left unmonitored.
• Stacking transform, CUPED, and stratification adds real pipeline complexity, more logged fields, more failure points, more places for a bug to silently break the primary metric; each addition should be justified by a measured variance reduction on the backtest, not added by default.
• CUPED requires the covariate to be genuinely pre-experiment; if the "pre-period" window is computed with a pipeline that lags or leaks into the experiment window, this reintroduces the treatment-affected-covariate bias silently, since a backtest on historical null data would not catch a bug that only manifests once treatment exists.
• A large measured variance reduction on a historical backtest is not a guarantee of the same reduction on the live experiment if the live population or the metric's behavior has shifted since the historical window; treat the backtest number as a planning estimate, and re-check variance reduction on the live experiment's own pre-period data before finalizing a launch decision based on it.

Direct answer

A standard user-level A/B test assumes each user's outcome depends only on their own assignment, an assumption called SUTVA (the stable unit treatment value assumption: no interference between units). On a social or messaging product that assumption is false by design, since treating one user changes what their connections see and can do, so a connection's outcome now depends on someone else's assignment too. Two designs address this: graph-cluster randomization, which partitions the social graph into clusters of densely connected users and randomizes whole clusters together so most interference happens within a cluster rather than leaking across the treatment/control boundary, and ego-network (egocentric) randomization, which measures outcomes as a function of a user's own neighborhood so you can directly compare people who have different fractions of treated friends. A third, simpler option when interference is more diffuse than a friend graph, staggered or geography-based rollout, trades away the direct/indirect decomposition for a much simpler design.

Structured elaboration

Why the standard design breaks

In plain user-level random assignment, a treated user's treated friends can amplify or dampen the effect for their untreated friends. That means a "control" user's outcome is not actually independent of the experiment: it depends on how many of their friends landed in treatment. The measured effect on control users is no longer a clean baseline, so the naive treatment-minus-control difference is biased, usually understating the total impact because part of the effect has leaked into the control group.

Design options

Randomization unit and exposure mapping, worked for graph-cluster randomization

• Unit: partition the social graph into non-overlapping clusters that maximize within-cluster edges and minimize between-cluster edges, then randomize entire clusters, not individual users, to treatment or control.
• Exposure mapping: for each user $i$, define $Z_i \in \{0,1\}$ as their own cluster's arm, and define a continuous exposure variable $G_i$ as the fraction of user $i$'s friends who are treated. Because clustering concentrates friends within the same cluster, most users will have $G_i$ close to 0 or close to 1, with only users near a cluster boundary landing at an intermediate exposure level, which isolates a smaller boundary group to study the spillover on while the bulk of users give a cleaner direct-effect read.

Estimating direct and indirect effects

Once users are bucketed by their own assignment $Z_i$ and a discretized exposure level $G_i$ (for example low, medium, high fraction of treated friends), two effects fall out of the comparisons:

• Direct effect: the difference in outcome between treated and untreated users who have the same exposure level, holding friend exposure fixed.
• Indirect (spillover) effect: the difference in outcome between untreated users at a higher exposure level versus untreated users at exposure level zero, what having more treated friends adds even without being treated yourself.
  A cluster boundary is never perfectly clean, so estimators typically reweight by each user's actual exposure level rather than their intended cluster assignment, and standard errors are computed at the cluster level, not the user level, since users within a cluster are not independent observations.

Worked example

Suppose graph clustering produces 40 clusters of roughly equal size, 20 assigned to treatment and 20 to control (illustrative, stated setup). Within a typical cluster, suppose 85% of a user's friends fall inside their own cluster and 15% fall outside it (a stated clustering-quality figure). For a user in a treatment cluster, assuming friends outside the cluster are treated at the overall population rate of 0.5, the expected fraction of treated friends is approximately:
$G_i \approx 0.85 \times 1 + 0.15 \times 0.5 = 0.85 + 0.075 = 0.925$
For a user in a control cluster:
$G_i \approx 0.85 \times 0 + 0.15 \times 0.5 = 0.075$
Clustering pushes most users toward exposure levels near 1 or near 0 rather than near the population average of 0.5, which is what makes the direct effect (treated vs. untreated at matched exposure) and the indirect effect (untreated-high-exposure vs. untreated-zero-exposure, here roughly the group of control-cluster users near a 0.075 treated-friend fraction) separately estimable, instead of both being smeared into the middle.

Trade-offs and pitfalls

• Graph clustering is itself an approximation; a poor clustering with low within-cluster edge density leaves exposure levels clustered near 0.5 for most users, exactly the regime where direct and indirect effects are hardest to tell apart, so validate cluster quality before trusting the design.
• Fewer, larger clusters approach plain random-user assignment (bad for the reason above); more, smaller clusters isolate interference better but reduce the effective number of independent units for inference, hurting power, a genuine trade-off in the design itself.
• Ego-network randomization directly measures spillover but generally needs a much larger sample than a plain user-level test to get precise exposure-level estimates, since it effectively estimates several treatment effects instead of one.
• None of these designs eliminate interference entirely, they contain and quantify it; if the product mechanism is genuinely viral, with treated users actively recruiting friends across cluster boundaries, even a well-clustered design can still leak, so pair the statistical design with a product-level sanity check on how far effects typically travel through the graph.