A/B Test Design & Statistical Rigor Questions

Direct answer

Before recommending rollout, I would not treat this as a single significance comparison; I would run a short sequence of checks: confirm the guardrail regression is real and not an artifact, check whether the guardrail movement is even large enough to be distinguishable from noise given the traffic it got (a guardrail is often powered for a much smaller effect than the primary, so "not significant" there can just mean underpowered, not "fine"), rule out a novelty or primacy effect as the explanation for the primary win, and then apply a pre-agreed decision rule rather than a judgment call made after seeing the numbers. If no pre-agreed rule exists, the honest fallback is a staged, guarded rollout with a long-run holdout, not an outright ship.

Structured elaboration

Step 1: Robustness checks on both metrics

• Segment the guardrail regression. Is it concentrated in one platform, cohort, or geography, or spread evenly? A regression concentrated in a narrow segment points at something mechanical (a bug or UX defect specific to that segment) rather than a real, generalizable trade-off.
• Check assignment health. Re-run the same sample-ratio and pre-period balance checks you would run on any experiment; a guardrail move that traces back to a randomization or instrumentation issue is not a real trade-off at all.
• Check the primary win's time course for a novelty effect. A novelty effect is a temporary lift driven by the change being new and attention-grabbing rather than a durable improvement; it typically shows as a large early lift that decays over the experiment window. Plot the primary metric's daily effect size: if it is shrinking over time while the guardrail regression is stable or growing, the primary "win" may partly evaporate on its own before you even weigh the trade-off. The mirror case, a primacy effect, is when existing users are initially resistant to a change (a lift that starts low and grows as people adapt); it matters here mainly as a reason not to over-read a weak early primary result as a fair test either.

Step 2: Is the guardrail result adequately powered

A guardrail that shows "not statistically significant regression" is not the same claim as "no regression." State this as a single check, not a full derivation: given the traffic the experiment actually got, was the guardrail's measurement precise enough to rule out a regression of a magnitude you would actually care about, or is the interval simply too wide to conclude anything either way. If the guardrail is underpowered at the traffic level the primary metric was sized for, that is itself the answer: you do not have enough information yet to trust a rollout, independent of which way the point estimate leans.

Step 3: Weighing a short-term win against a longer-term cost

This is a business trade-off, not a pure statistics question, and the responsible move is to make the trade explicit rather than intuit it. State the primary metric's estimated near-term value and the guardrail's estimated longer-term cost in the same unit (commonly revenue, or another shared north-star), even if one side of that conversion is an approximation, and be explicit about which parts are measured versus assumed. Two structural reasons this is often harder than it looks:

• The primary metric (e.g., short-term engagement or conversion) is usually measured over days, while the guardrail (e.g., retention) compounds over a much longer horizon; a small daily retention hit, if it persists, can outweigh a larger one-time primary gain once compounded over the retention metric's own natural time window.
• The primary effect and the guardrail effect may not be the same size in the population they touch; a lift concentrated in low-value or already-churny users paired with a regression concentrated in high-value users is a worse trade than the same headline numbers spread evenly, which is why the segment check in Step 1 also feeds directly into this weighing step.

Step 4: The decision rule

The rule should exist before you are looking at a live result, exactly like a guardrail threshold. In order of preference:

1. If a pre-committed guardrail threshold and pause rule exist and were breached, honor it. Do not relitigate the threshold after seeing the number; that defeats the purpose of pre-committing it.
2. If no explicit threshold exists, do not ship outright. Treat this as evidence the guardrail set was incomplete going in, fix that for next time, and in the meantime prefer the conservative path below over an ad hoc judgment call.
3. Stage the rollout with a long-run holdout. Ramp exposure gradually (e.g., a small percentage first) while keeping a genuine holdout population unexposed for an extended window well past the point of the initial ship decision, specifically to catch a guardrail effect that is slow to fully appear (churn, trust erosion) even if it looked borderline at the original read.
4. Re-test the specific element suspected of causing the trade-off, isolated from the rest of the change, if the segment and mechanism checks point at one particular piece of the change rather than the whole feature.

This pattern generalizes

The same discipline applies with the trade direction reversed, for example a retention gain paired with an ARPU regression, and to slower-arriving guardrails, for example a generative-AI product where short-term engagement rises but downstream purchases decline over a longer window; both need the same segment, power, and pre-committed-rule checks described above, not a different framework. It also applies to the inverse failure mode: several secondary metrics flag as significant while the primary metric itself is null. That case is a multiple-comparisons risk, not a real signal by default, since checking many metrics at once raises the odds that some look significant purely by chance; treat only the pre-declared guardrails as carrying an automatic mandate to act, and require an unplanned secondary flag to clear a higher, dedicated bar before it changes the decision. Some organizations formalize the whole sequence into an explicit two-stage gate, a short-term engagement stage followed by a separate long-run retention or monetization stage, each with its own pre-declared error-rate control; that is a heavier, more procedural version of the same pre-commitment discipline, and the statistical mechanics of controlling error rates across the two stages belong to hypothesis-testing theory rather than to this design question. For a small, non-significant secondary movement that still looks concerning, the right response is neither to ignore it nor to react to noise in the moment: pre-specify a dedicated, adequately powered follow-up check on that one metric rather than relitigating the current experiment's result under pressure.

Worked example

A feed-ranking change shows a primary click-through lift that is largest in the first three days and roughly half that size by day ten (a decaying pattern read directly off the daily-effect series), alongside a 7-day retention guardrail that moved negative but with a confidence interval that comfortably includes zero. Two things are true at once here: the guardrail result does not clear the bar for "proven regression," and the primary result shows the shape of a novelty effect rather than a stable lift. Given both, the defensible move is neither an unconditional ship (the primary win may partly be novelty, and the guardrail is not cleanly exonerated, just underpowered) nor an unconditional rollback (nothing is proven broken); it is a staged rollout with an extended holdout sized to actually resolve the guardrail question, with a decision point set for after the primary metric's trend has had time to settle.

Trade-offs and pitfalls

• The single biggest mistake in this scenario is treating "guardrail not statistically significant" as "guardrail cleared," when it may simply be underpowered; always check power before treating a null guardrail result as reassurance.
• Deciding the trade-off after seeing which way the numbers lean, rather than applying a rule set before the experiment, is how teams talk themselves into shipping a change they would not have pre-approved.
• A holdout that is too short to catch a slow-moving guardrail effect gives false confidence; size the holdout window to the guardrail's own natural time horizon (e.g., a retention guardrail needs a window long enough for retention itself to be observed), not to the primary metric's faster clock.

Direct answer

A novelty effect is a temporary inflation of an early treatment effect: users explore or click on something purely because it is new, and that extra engagement fades once the feature stops being novel, biasing an early readout upward. A primacy effect (sometimes called a change-aversion or resistance-to-change effect) is the opposite pattern: a change disrupts a habitual workflow, so users are temporarily worse off while they relearn it, biasing an early readout downward, then the effect climbs toward its true level as users adapt. Both biases fade over roughly the same kind of horizon, so trusting a week-one number without checking its trajectory can make you launch a fad or kill a genuine win too early.

Structured elaboration

Mechanism and direction

Diagnostics to tell a spike from a persistent effect

• Time-windowed effect plot: daily or weekly treatment effect with confidence intervals, ideally with a smoothed trend line (LOESS or a spline), not a single pooled average. A genuine effect looks like a roughly flat band around a nonzero value; novelty looks like a spike that decays toward that band; primacy looks like a trough that rises toward it.
• Exposure-age cohorts, not calendar time: plot the effect against days since each user's first exposure for a fixed cohort of users first exposed on the same day, rather than calendar date. A calendar-time plot mixes newly exposed users (still novel-biased) with long-exposed users (already stabilized) every single day, which can mask a real decay curve as a flat line.
• New vs. returning user split: novelty is usually concentrated in users encountering the feature for the first time; if the effect is similar in a segment already exposed for weeks, that argues against novelty as the explanation.
• Change-point or decay model on the daily series: fit a time-varying effect model, effect as a function of exposure age, and test whether the transient component is statistically distinguishable from zero, separately from the asymptotic (persistent) component.
• Placebo check: run the same time-windowed analysis on a pre-launch period with no real treatment; if spike-like patterns appear there too, the "decay" you see in the real experiment may just be normal week-to-week noise, not a novelty artifact.

Adjusting duration and analysis

• Pre-register the analysis window before launch rather than reading the metric the moment it looks good; a fixed rule such as "primary read is the average effect over exposure-days 21 to 35" prevents cherry-picking the peak or the trough.
• Extend the experiment until the exposure-age curve visibly plateaus, or the fitted transient component's confidence interval crosses zero, rather than for a fixed calendar duration chosen in advance.
• Report both the early-window and late-window effect side by side rather than a single blended number; a launch decision based only on the blended average silently averages a fading spike with a stabilizing floor.

Worked example

Two hypothetical (illustrative, not real study data) weekly average-treatment-effect readings for the same nominal conversion metric:

Both curves are converging toward roughly the same persistent level, one from above and one from below, which is exactly the signature that separates them from a flat, genuine effect that would show roughly the same number every week within noise.

If the transient component decays exponentially, $\Delta(t) = C + A e^{-\lambda t}$, where $A$ is the size of the initial novelty or primacy spike above the persistent effect $C$ (the extra amount present at $t=0$ that fades away over time), and the illustrative decay rate is $\lambda = 0.2$ per week, its half-life is:
$t_{1/2} = \frac{\ln 2}{\lambda} = \frac{0.693}{0.2} \approx 3.5 \text{ weeks}$
That is the kind of number worth pre-registering as a decision rule: run at least three half-lives (about 10 to 11 weeks here) before reading the persistent effect $C$, rather than picking an arbitrary duration.

Trade-offs and pitfalls

• Waiting out a full decay curve costs calendar time and opportunity cost on other experiments; for low-stakes features, teams sometimes accept the risk of a novelty-inflated launch decision rather than run for months.
• Segmenting by exposure age needs per-user first-exposure timestamps captured in the assignment log; if you only log calendar-date rollups, you cannot separate calendar effects from exposure-age effects after the fact.
• A curve that looks like decay can just as easily reflect unrelated seasonality (marketing pushes, holidays) that correlates with launch timing; a decay-shaped curve is suggestive, not conclusive, on its own.
• Don't assume every early spike is novelty and every early trough is resistance to change: an early spike can be a genuine effect solving a pent-up need immediately, and an early trough can be a real bug that later gets patched. The pattern is evidence, not proof, and should be paired with qualitative checks (support tickets, session recordings) before concluding the mechanism.

Direct answer

A lift that spikes in week one and fades to flat by week two could be three different things wearing the same shape: a real novelty effect decaying to its true (smaller or zero) persistent level, a seasonal pattern that has nothing to do with the treatment and would show up in both arms if you looked, or a selection-bias artifact where the traffic source itself (a holiday push or a marketing campaign) skewed an unrepresentative mix of users into treatment. The fastest way to tell them apart is to check whether the decay tracks calendar time (seasonality), user exposure-age (novelty), or arm composition (selection bias), because each explanation leaves a distinct fingerprint in the data, not just a distinct story.

Structured elaboration

The checklist, in order

Disentangling novelty from selection bias when a marketing campaign is the traffic source

This case is harder because both explanations can be true at once and produce the same decaying shape. A marketing campaign that ramps down after week one changes who is arriving, not just how they behave: if campaign-driven traffic is disproportionately assigned into treatment (for example because the campaign linked directly into the treatment experience, or assignment happened downstream of a referral parameter correlated with arm), the week-one spike partially reflects a different, more click-happy population rather than a within-user novelty effect.

• Design fix: randomize on a unit and at a point in the funnel that is independent of campaign exposure, assigning before the user ever sees the campaign-specific landing experience, so acquisition-channel composition is balanced by construction rather than by hope.
• Post-hoc check: compare the new-vs-returning mix and traffic-source mix between arms week by week; if the treatment arm's share of campaign-driven new users is higher than control's, run a balance check on that specific subpopulation, not just the top-line allocation.
• Post-hoc correction: re-run the primary analysis restricted to users who arrived through non-campaign channels, and separately on campaign-driven users; if the effect concentrates in the campaign-driven segment and that segment is imbalanced between arms, the pooled week-one number is not a trustworthy estimate of the treatment effect for the general population.

Redesigning the experiment or analysis window

• Extend the primary analysis window well past the campaign's active period so the segment mix has time to normalize, and pre-register that the primary readout is a later window, not week one.
• Use a difference-in-differences framing (treatment-arm change from a pre-period baseline, minus control-arm change over the same calendar span) so genuine calendar movement common to both arms cancels out instead of being misread as a treatment effect.
• If holiday timing cannot be avoided, consider a staggered or replicated launch (the same treatment introduced to a second, non-holiday cohort later) so you get a second read that is not confounded with that specific calendar event.

Worked example

Suppose the pooled week-one lift is +8% on a baseline conversion rate of 10%, and segment analysis shows the treatment arm received 60% campaign-driven new users in week one versus 40% in control (stated, illustrative inputs). If campaign-driven new users convert at a naturally higher rate, say 14% versus 9% for organic users (illustrative inputs), the arm-level blended rate is a weighted average:
$\text{Treatment rate} = 0.6 \times 0.14 + 0.4 \times 0.09 = 0.084 + 0.036 = 0.120$
$\text{Control rate} = 0.4 \times 0.14 + 0.6 \times 0.09 = 0.056 + 0.054 = 0.110$
That gives an apparent lift of $(0.120 - 0.110)/0.110 \approx 9.1\%$, almost entirely explained by the composition difference rather than any within-user treatment effect, before even considering a real novelty component. This is why segment analysis has to run before you trust the topline number, not after.

Trade-offs and pitfalls

• Restricting analysis to non-campaign traffic to remove selection bias also shrinks your sample and can push you back into an underpowered read, exactly what the power-review step is meant to catch.
• Difference-in-differences assumes both arms would have moved in parallel absent the treatment; a campaign that specifically targets one arm's users breaks that assumption too, so DiD is not a free fix if the selection bias operates through campaign-to-arm assignment rather than through time.
• Extending the window to wait out both seasonality and novelty delays the launch decision; be explicit with stakeholders that "flat by week two" was never a valid stopping rule for this scenario, so the delay does not read as backpedaling.
• Retiring a variant because week two looks flat, without a power review, risks killing a real but modest persistent effect that a two-week window was never sized to detect in the first place.

Direct answer

The randomization unit should be the largest identity that is (a) stable over the experiment window and (b) matches the unit at which you will measure and report the outcome. For a homepage layout change with user-scoped conversion metrics, that is almost always user id when you have reliable logged-in identity; fall back to device id for logged-out mobile traffic, and treat cookie and session id as fallback-only units because they leak identity across the very boundary you are trying to hold fixed. The mechanism that turns "unit" into an actual bucket assignment is deterministic hash-based bucketing, and its main operational failure mode is touching the hash inputs (the salt or key) mid-experiment. Before any of that, though, you have to define who is even eligible to be in the experiment at all.

Structured elaboration

Defining the eligible population before choosing a unit

Unit choice is a second-order question; the first-order question is which units are even eligible to enter the experiment. For a mobile-only feature (say, a redesign shipped exclusively in the mobile app to a US audience), a desktop-only visitor cannot receive the treatment no matter which arm they land in, so randomizing across your full user base and then measuring outcomes at the account level silently dilutes the experiment: ineligible units get logged into both arms with a null "effect" (they cannot experience the change either way), which pulls the estimated treatment effect toward zero and inflates the sample size needed to detect a real one. The eligible population for a mobile-only US feature is the set of units that are (a) on the mobile platform that ships the feature, (b) in the targeted market (US), and (c) past whatever version or capability gate the feature requires; everyone outside that eligible population should be excluded from the experiment entirely, not folded into control by default. This is a distinct failure mode from picking the wrong unit: a design can choose a perfectly good unit (user id) and still be broken if a third of the "users" randomized into it were structurally incapable of ever seeing the treatment, whether the unit ultimately chosen within that eligible population is user, device, or session id.

Trade-offs by candidate unit

The two axes that matter are: does this unit stay attached to one treatment condition for the life of the experiment, and does it match the level at which you will later compute the metric. Session-level randomization on a homepage layout change fails both: a returning user can see version A on Monday and version B on Wednesday, so "the effect of the layout" is not well defined for that person, and if you then report a user-level conversion rate you are averaging over users who experienced a mix of both conditions.

Target-segment and control-group selection for a personalization test

Personalization experiments add a further wrinkle on top of eligibility and unit choice: because the treatment itself varies per person (each user's personalized experience differs from every other user's), you have to be explicit about two more things: which segment of the eligible population the test targets, and what the control group actually receives. A common setup: the target segment is the subset of eligible users with enough interaction history for the personalization model to act on (say, users with a minimum number of prior sessions); users below that threshold cannot be meaningfully personalized and should either be excluded from the test or routed to a defined fallback, rather than silently folded into a "control" group that has nothing to do with the personalization decision being tested. The control group, correspondingly, should receive a clearly defined non-personalized baseline (a fixed default ranking or layout), not "whatever the legacy system happened to show," so the measured effect is attributable to personalization itself rather than to incidental differences between the two code paths. Get target-segment or control-group definition wrong (an ill-specified segment boundary, or a control group that partially overlaps with treatment logic) and the measured lift reflects a spurious selection effect rather than the personalization algorithm's real value, no matter how correctly the underlying randomization unit and hash mechanism were implemented.

How hash-based deterministic bucketing works

In practice you do not store a per-user assignment row for every experiment. Instead you compute

$\text{bucket}(u) = \text{hash}(u \,\Vert\, \text{salt}) \bmod N$

where $u$ is the chosen unit id (user id, device id, etc.), the salt is a string unique to this experiment (often the experiment name or id), and $N$ is the number of buckets (commonly 100 or 1000 for fine-grained traffic allocation). Buckets are then mapped to arms, e.g. buckets 0-49 to control and 50-99 to treatment for a 50/50 split. Because the hash is deterministic, the same unit id always lands in the same bucket for the same salt, which is what makes the assignment reproducible without a lookup table, and salting per-experiment is what makes assignment to experiment A independent of assignment to experiment B (so the same user can be validly in many concurrent, non-interacting experiments).

Operational pitfalls

• Changing the salt or hashing key mid-experiment. This is the single most common self-inflicted wound. It re-shuffles every unit into a new bucket, silently reassigning some fraction of users from control to treatment (or the reverse) partway through. The experiment now mixes users with a clean single-arm history and users who were exposed to both arms, which is exactly the session-level contamination problem from the table above, except it is invisible unless you log assignment history.
• Reusing a salt across experiments. If two unrelated experiments accidentally share a salt (or one is a substring of the identifier used in the other), their bucket assignments become correlated instead of independent, which breaks the assumption that concurrent experiments do not interfere with each other.
• Changing $N$ or the bucket-to-arm mapping. Even without touching the salt, resizing the traffic split mid-flight (e.g., ramping from 5% to 50%) moves units across the arm boundary unless the mapping is designed to be monotonic (new traffic is added to existing arms rather than everyone being rehashed).
• Identity churn. A user id that gets merged, deleted, or re-issued (account merge, logout/login cycles that mint a new anonymous id) effectively becomes a new hash input mid-experiment, which has the same effect as a salt change for that user.

A finer-grained alternative: per-impression randomization

Every unit above is a person-shaped identity. Some teams instead randomize at the impression level, assigning a fresh coin flip to each page view or ranking request rather than to a person. This is occasionally used for high-frequency, low-persistence decisions (e.g., which of several ranking variants to serve on a given request) where you explicitly do not want a stable per-user experience. It is a different trade entirely from the table above: it eliminates any notion of "this user's assigned arm" (so it cannot answer a question about a durable, user-perceived change like a homepage layout), and it introduces strong intra-user correlation in the outcome data, since one person's many impressions are not independent draws, which inflates the effective variance if you naively treat impressions as independent observations in the analysis. Per-impression randomization is the right tool only when the thing being tested is meant to vary within a single user's experience; for a homepage layout, where the goal is to measure how a stable person-level experience changes behavior, it is the wrong granularity.

Detecting and correcting unit-mismatch after the fact

1. Assignment-churn audit. From the exposure logs, compute the fraction of units that were logged under more than one arm during the experiment window. A near-zero rate is expected; anything material indicates contamination.
2. Pre-period balance check. Compare the two arms on metrics measured before the experiment started (metrics that could not possibly be affected by treatment). An imbalance signals a broken randomization, not a broken hash necessarily, but it is the same diagnostic.
3. Sample ratio mismatch check on the realized split, i.e., does the observed 50/50 (or intended ratio) actually hold at the analysis unit. A skew is a strong signal that the bucketing pipeline itself misbehaved.
4. Timeline reconstruction. If churn is found, check the deployment log for the experiment: a salt, key, or bucket-count change on a specific date will produce a visible step change in the churn-rate-by-day series.
5. Correction paths, in order of preference. Analyze by first-observed assignment only (treat each unit's initial exposure as its assignment, i.e., an intention-to-treat style rule, and accept the resulting dilution of the effect estimate); if the break has a clean date, restrict the analysis window to the stable period before or after it; if contamination is pervasive, drop the experiment's results for the affected window and rerun rather than trying to model around a broken assignment mechanism, since any post hoc adjustment for a data-dependent unit-mismatch is itself a source of bias.

Worked example

Suppose an app-only feature was randomized by session id and you are asked to sanity-check it before trusting the readout. You pull exposure logs and count, per user, the distinct arms they were logged under: 92,000 users saw only control, 91,500 saw only treatment, and 6,500 saw both. Churn rate is $6{,}500 / (92{,}000+91{,}500+6{,}500) \approx 3.4\%$. That is a directly computed, reproducible number from the logs, not an assumption, and a value that high on a homepage-layout test (where the same person plausibly returns within the experiment window) is enough on its own to recommend re-running at user-id granularity rather than trying to salvage the session-level readout.

Trade-offs and pitfalls

• Choosing the "purest" unit (household) is not free: fewer independent units means higher variance per unit, so the same absolute effect needs more households than it would need users to reach the same precision. Unit choice is a bias-versus-noise trade, not a pure bias fix.
• A cookie- or device-based fallback is a compromise you should name explicitly to stakeholders, not a silent substitute for user id; report the estimated multi-device contamination rate alongside the headline result.
• An eligible population that is defined too loosely (e.g., randomizing all traffic instead of just the mobile-only, in-market segment) produces the same kind of diluted, biased-toward-zero readout as a bad unit choice, even when the unit itself is correct.
• Do not "fix" detected contamination by re-including the mixed-exposure users with a different weighting scheme chosen after seeing which way it moves the result; decide the exclusion or ITT rule before looking at the treatment effect.

Direct answer

A standard user-level A/B test assumes each user's outcome depends only on their own assignment, an assumption called SUTVA (the stable unit treatment value assumption: no interference between units). On a social or messaging product that assumption is false by design, since treating one user changes what their connections see and can do, so a connection's outcome now depends on someone else's assignment too. Two designs address this: graph-cluster randomization, which partitions the social graph into clusters of densely connected users and randomizes whole clusters together so most interference happens within a cluster rather than leaking across the treatment/control boundary, and ego-network (egocentric) randomization, which measures outcomes as a function of a user's own neighborhood so you can directly compare people who have different fractions of treated friends. A third, simpler option when interference is more diffuse than a friend graph, staggered or geography-based rollout, trades away the direct/indirect decomposition for a much simpler design.

Structured elaboration

Why the standard design breaks

In plain user-level random assignment, a treated user's treated friends can amplify or dampen the effect for their untreated friends. That means a "control" user's outcome is not actually independent of the experiment: it depends on how many of their friends landed in treatment. The measured effect on control users is no longer a clean baseline, so the naive treatment-minus-control difference is biased, usually understating the total impact because part of the effect has leaked into the control group.

Design options

Randomization unit and exposure mapping, worked for graph-cluster randomization

• Unit: partition the social graph into non-overlapping clusters that maximize within-cluster edges and minimize between-cluster edges, then randomize entire clusters, not individual users, to treatment or control.
• Exposure mapping: for each user $i$, define $Z_i \in \{0,1\}$ as their own cluster's arm, and define a continuous exposure variable $G_i$ as the fraction of user $i$'s friends who are treated. Because clustering concentrates friends within the same cluster, most users will have $G_i$ close to 0 or close to 1, with only users near a cluster boundary landing at an intermediate exposure level, which isolates a smaller boundary group to study the spillover on while the bulk of users give a cleaner direct-effect read.

Estimating direct and indirect effects

Once users are bucketed by their own assignment $Z_i$ and a discretized exposure level $G_i$ (for example low, medium, high fraction of treated friends), two effects fall out of the comparisons:

• Direct effect: the difference in outcome between treated and untreated users who have the same exposure level, holding friend exposure fixed.
• Indirect (spillover) effect: the difference in outcome between untreated users at a higher exposure level versus untreated users at exposure level zero, what having more treated friends adds even without being treated yourself.
  A cluster boundary is never perfectly clean, so estimators typically reweight by each user's actual exposure level rather than their intended cluster assignment, and standard errors are computed at the cluster level, not the user level, since users within a cluster are not independent observations.

Worked example

Suppose graph clustering produces 40 clusters of roughly equal size, 20 assigned to treatment and 20 to control (illustrative, stated setup). Within a typical cluster, suppose 85% of a user's friends fall inside their own cluster and 15% fall outside it (a stated clustering-quality figure). For a user in a treatment cluster, assuming friends outside the cluster are treated at the overall population rate of 0.5, the expected fraction of treated friends is approximately:
$G_i \approx 0.85 \times 1 + 0.15 \times 0.5 = 0.85 + 0.075 = 0.925$
For a user in a control cluster:
$G_i \approx 0.85 \times 0 + 0.15 \times 0.5 = 0.075$
Clustering pushes most users toward exposure levels near 1 or near 0 rather than near the population average of 0.5, which is what makes the direct effect (treated vs. untreated at matched exposure) and the indirect effect (untreated-high-exposure vs. untreated-zero-exposure, here roughly the group of control-cluster users near a 0.075 treated-friend fraction) separately estimable, instead of both being smeared into the middle.

Trade-offs and pitfalls

• Graph clustering is itself an approximation; a poor clustering with low within-cluster edge density leaves exposure levels clustered near 0.5 for most users, exactly the regime where direct and indirect effects are hardest to tell apart, so validate cluster quality before trusting the design.
• Fewer, larger clusters approach plain random-user assignment (bad for the reason above); more, smaller clusters isolate interference better but reduce the effective number of independent units for inference, hurting power, a genuine trade-off in the design itself.
• Ego-network randomization directly measures spillover but generally needs a much larger sample than a plain user-level test to get precise exposure-level estimates, since it effectively estimates several treatment effects instead of one.
• None of these designs eliminate interference entirely, they contain and quantify it; if the product mechanism is genuinely viral, with treated users actively recruiting friends across cluster boundaries, even a well-clustered design can still leak, so pair the statistical design with a product-level sanity check on how far effects typically travel through the graph.