Advanced Experimentation Designs Questions

Direct answer: With only 120 eligible users worldwide, abandon a standard between-subject parallel A/B test (verified underpowered below) and instead reach for a WITHIN-SUBJECT (paired) design, a switchback/time-block design if the feature has a natural on/off toggle per user or per small cluster, a holdout ramp with close qualitative tracking, or a case-study/qualitative approach if even a paired design cannot reach adequate power. Choose a cluster-randomized or switchback design over individual-level randomization specifically when the 120 users cluster naturally into groups (teams, organizations, workspaces) where within-cluster interference is likely, e.g. developers on the same team influence each other's usage patterns, in which case individual-level randomization would violate SUTVA within the cluster the same way social-network interference does at consumer scale (this topic's sibling a-b-test-design-and-statistical-rigor owns that consumer-scale case; the tiny-population, small-cluster version is this topic's ground).

Why a standard parallel test is underpowered (executed): with N=120 total (60/arm), baseline conversion 30%, and even a generous 33% relative lift target (0.30 to 0.40, larger than most real product effects), achieved power is only about 21%. The required N for 80% power at that same generous MDE would be 712 total (356/arm), nearly 6x the entire available population. Any smaller, more realistic effect size would need even more, confirming a between-subject design simply cannot work at this scale.

Why a within-subject (paired) design helps: pairing removes BETWEEN-user variance from the comparison, since each user serves as their own control (measured before and after, or under both conditions). With within-user correlation $\rho=0.5$ (a moderate, realistic value for a within-user metric measured twice), the paired-design effective standard-deviation multiplier is $\sqrt{2(1-\rho)}=1.0$, versus $\sqrt{2}\approx1.41$ for an independent between-subject comparison, roughly HALVING the required sample size for the same power and MDE (since required N scales with the square of that multiplier, $1.0^2/1.41^2\approx0.5$). For 120 users, this can be the difference between an infeasible and a feasible test.

Practical plan:

• Metric selection: choose a metric with LOW within-user noise and short time-to-signal (e.g. task-completion rate or time-to-first-success on the developer-facing feature) rather than a slow, high-variance business metric like long-term retention, which would need far more than 120 users regardless of design.
• Instrumentation: capture per-user, per-period usage tightly enough to compute a clean pre/post or on/off comparison per user, and log enough contextual detail (which org/team each user belongs to) to check for and adjust for within-cluster correlation if the 120 users cluster into teams.
• Design choice given clustering: if the 120 users sit in, say, 15 teams of 8, prefer CLUSTER-randomization or a team-level switchback over individual randomization, since individually randomizing within a team risks contamination (control users on a team where teammates are treated may indirectly experience the feature via shared docs/workflows); if the users are genuinely independent (different companies, no interaction), individual-level within-subject randomization is fine and simpler.
• Rollout recommendation: run a holdout-ramp: enable the feature for a majority of the 120 (or all, using a within-subject pre/post design with a smaller true holdout group retained purely to distinguish the treatment effect from a general time trend), supplement the quantitative signal with structured qualitative feedback (brief surveys or interviews with a sample of the 120), since at this population size qualitative and case-study evidence carries proportionally MORE weight in the final ship/no-ship decision than it would in a consumer-scale test, and treat the quantitative estimate as directional rather than a precise point estimate given the residual power limitations even after pairing.

Direct answer: Choose a factorial (or multivariate/MVT) design over a sequence of separate A/B tests when you have two or more candidate changes you plausibly want to ship TOGETHER, you suspect (or must rule out) an interaction between them, and you have enough concurrent traffic to split into cells without starving any one cell's power. Choose sequential single-factor A/B tests when changes are independent, traffic is scarce, or you need to ship and learn on the first change before the second is even designed.

Trade-offs

• Speed of learning: factorial wins when the alternative is truly SEQUENTIAL (test A, wait, then test B). For the same total sample budget, a k-factor design estimates all k main effects roughly as precisely as k separate two-arm tests run one after another with the SAME total budget split across them, but the factorial gets all k answers CONCURRENTLY instead of one-after-another, and gets the interaction essentially for free.
• Statistical power for main effects versus interactions: each main effect in a factorial is estimated by comparing half the cells against the other half (pooling over the other factor), so main-effect power is comparable to a standalone two-arm test at the same total N. The INTERACTION contrast is NOT pooled the same way: it needs the same per-cell sample size as a single unpooled cell-vs-cell comparison, so detecting an interaction of the same absolute size as a main effect costs roughly 2x the total sample of detecting that main effect alone (verified numerically in S1).
• Interpretability: a single-factor sequential test has a trivial reading ("does this metric move"). A factorial with a significant interaction requires explaining conditional effects ("A only works when B is on"), which is harder to communicate and can confuse stakeholders unless you plan for it (see S6).
• Operational complexity: factorial designs need every combination of treatments to be simultaneously deployable and measurable, more engineering coordination, more QA surface (interaction bugs, cross-feature conflicts), and a bigger simultaneous "blast radius" if something goes wrong, versus shipping and validating one change at a time.

Worked confirmation (executed simulation, not just cited): simulating 2 factors with true additive effects of size 0.12 (in outcome-SD units) each, total budget N=4000, alpha=0.05, over 4000 replications:

• FACTORIAL (4 cells of 1000 each): power(A) = 0.965, power(B) = 0.961
• SEQUENTIAL (2 separate 2-arm tests, same total budget split in half, 1000/arm each): power(A) = 0.771, power(B) = 0.759

For the identical total sample budget, the factorial design delivered materially higher power on BOTH main effects (0.96+ vs 0.77) because every unit in a factorial cell contributes to estimating EVERY factor's main effect (a unit in the (A=1,B=1) cell informs both the A comparison and the B comparison), while the sequential approach wastes half its budget on a control arm that is redundant across the two separate tests.

Business scenario where factorial is clearly preferable: a subscription product wants to test (1) a new pricing page copy and (2) a redesigned onboarding flow. Product suspects the pricing framing only lands if users have already been walked through the new onboarding (an interaction hypothesis: pricing-copy effect is conditional on onboarding version). Running these sequentially would answer "does new pricing help" and "does new onboarding help" separately but could never detect that new pricing ONLY helps when paired with new onboarding, exactly the finding the business needs to decide whether to ship them together, separately, or not at all. A 2x2 factorial answers all three questions (both main effects plus the interaction) in one concurrent test.

When NOT to use factorial: if you have 4+ factors, a FULL factorial's cell count (2^4 = 16 cells) can quickly outstrip available traffic; that is exactly the constrained-traffic case that motivates fractional factorial designs (S2, S3) rather than abandoning multi-factor testing altogether.

Direct answer: Structure the explanation as five short bullets/slides: (1) the question tested, in plain business language; (2) the headline result, including whether an interaction was found and what that means in plain terms; (3) 2-3 concrete business-impact scenarios tied to the interaction; (4) the uncertainty around the estimate, expressed as a range, not a p-value; (5) your recommended next action.

Worked structure, continuing the S1 pricing x layout example:

1. The question tested: "We tested a lower price point and a redesigned layout, separately and together, to see if either helps checkout conversion, and whether they work better as a pair."
2. Headline result: "The new layout alone did not move conversion. The lower price alone barely moved it. But the lower price PAIRED WITH the new layout increased conversion by [X]%, roughly [3x] more than either change alone. In plain terms: the discount only really lands when the page also feels redesigned and trustworthy; on the old layout it barely registers."
3. Business-impact scenarios: "If we ship both together: we expect roughly [$Y] in incremental monthly revenue, at the cost of a lower average order value from the discount. If we ship the layout alone: essentially no measurable revenue change, but a UX-quality win we can still justify separately. If we ship the discount alone (old layout): we lose margin with almost no conversion gain, this is the option to avoid."
4. Uncertainty: "We are confident the two changes need each other, this isn't a coin flip; the range on the combined lift is [X1]% to [X2]% with high confidence, meaning even in the more conservative case it still clears our bar to ship."
5. Recommended next action: "Ship the combined pricing-plus-layout change to 100% of traffic, and separately roll back or archive the discount-alone and layout-alone paths, since neither pays for itself independently."

Why this structure works for a non-technical audience: it leads with the DECISION-RELEVANT finding (the two changes must be shipped together, not separately) rather than the statistical mechanism (an interaction term was significant), translates "interaction" into "only works when paired with" language a business audience immediately understands, and puts uncertainty in dollar/percentage-range terms rather than a p-value or confidence-interval width that would require re-explaining what a p-value means.

Direct answer: With a 2% baseline conversion rate, 100k eligible users/day, 80% power, and alpha 0.05, and assuming a 15% relative minimum detectable effect (the question does not state one explicitly, so I am stating this assumption up front, a common convention for this kind of sizing exercise; new baseline = 2.30%), the design needs roughly 18,346 users PER CELL to power the main effects, or about 36,693 users per cell to ALSO power the interaction at the same MDE. Traffic should be split evenly, 25% into each of the 4 cells (control/control, pricing-B/layout-old, pricing-A/layout-new, pricing-B/layout-new), randomized independently on pricing and layout so every user has an equal chance of any of the 4 combinations.

Sample-size computation (executed, two-proportion z-test):

For a MAIN EFFECT comparison (pooling the 2 cells that share a pricing level against the other 2), $p_1=0.02$, $p_2=0.023$ (15% relative lift):

$$n = \frac{\left(z_{\alpha/2}\sqrt{2\bar{p}(1-\bar{p})} + z_{\beta}\sqrt{p_1(1-p_1)+p_2(1-p_2)}\right)^2}{(p_2-p_1)^2}$$

gives n = 36,693 per POOLED arm (i.e. per 2-cell group). Since each pooled arm is 2 of the 4 cells, that is 18,346 users per CELL for main-effect power, 73,386 total across all 4 cells, which at 100k/day takes about 0.73 days.

For the INTERACTION contrast (an unpooled, single-cell-vs-single-cell comparison, since the interaction estimate uses the +1/-1/-1/+1 contrast across all four individual cells rather than pooled halves), the SAME formula applied per cell (no pooling) requires 36,693 users per cell, 146,771 total, about 1.47 days at 100k/day, exactly 2.00x the main-effect total. This is the general result cited in S0: interaction detection at the same MDE costs about 2x a main effect's sample size.

Traffic allocation: randomize pricing (A vs B) and layout (old vs new) INDEPENDENTLY via two separate hash-based coin flips per user (verified orthogonal in S9), producing four roughly-equal 25% cells automatically; do not manually stratify into 4 named buckets, since independent-factor randomization already balances the design and keeps it robust to later adding a third factor.

Analysis and interpretation: fit

$$y = \beta_0 + \beta_1 \cdot \text{Pricing} + \beta_2 \cdot \text{Layout} + \beta_3 \cdot (\text{Pricing} \times \text{Layout}) + \varepsilon$$

on the binary conversion outcome (linear probability model or logistic regression with an interaction term). $\beta_1$ is the pricing main effect averaged over both layouts, $\beta_2$ the layout main effect averaged over both pricing versions, and $\beta_3$ the interaction: how much the pricing effect CHANGES when layout changes (equivalently, how much the layout effect changes when pricing changes). Run the interaction test FIRST: if $\beta_3$ is significant, report cell-specific conversion rates (all 4 cells) rather than a single "average" pricing effect, since the average would be misleading if the effect flips sign across layouts. If the interaction is not significant, report the two main effects as if from independent tests and recommend shipping whichever combination of winning main effects performs best. Present results to stakeholders as: (1) the interaction test verdict first, in plain language, (2) a 2x2 table of observed conversion rates per cell with confidence intervals, (3) the recommended cell to ship, and (4) the guardrail-metric check on that winning cell (per this topic's sibling a-b-test-design-and-statistical-rigor, which owns guardrail-metric mechanics).