Advanced Experimentation Designs Questions

Direct answer: With only 120 eligible users worldwide, abandon a standard between-subject parallel A/B test (verified underpowered below) and instead reach for a WITHIN-SUBJECT (paired) design, a switchback/time-block design if the feature has a natural on/off toggle per user or per small cluster, a holdout ramp with close qualitative tracking, or a case-study/qualitative approach if even a paired design cannot reach adequate power. Choose a cluster-randomized or switchback design over individual-level randomization specifically when the 120 users cluster naturally into groups (teams, organizations, workspaces) where within-cluster interference is likely, e.g. developers on the same team influence each other's usage patterns, in which case individual-level randomization would violate SUTVA within the cluster the same way social-network interference does at consumer scale (this topic's sibling a-b-test-design-and-statistical-rigor owns that consumer-scale case; the tiny-population, small-cluster version is this topic's ground).

Why a standard parallel test is underpowered (executed): with N=120 total (60/arm), baseline conversion 30%, and even a generous 33% relative lift target (0.30 to 0.40, larger than most real product effects), achieved power is only about 21%. The required N for 80% power at that same generous MDE would be 712 total (356/arm), nearly 6x the entire available population. Any smaller, more realistic effect size would need even more, confirming a between-subject design simply cannot work at this scale.

Why a within-subject (paired) design helps: pairing removes BETWEEN-user variance from the comparison, since each user serves as their own control (measured before and after, or under both conditions). With within-user correlation $\rho=0.5$ (a moderate, realistic value for a within-user metric measured twice), the paired-design effective standard-deviation multiplier is $\sqrt{2(1-\rho)}=1.0$, versus $\sqrt{2}\approx1.41$ for an independent between-subject comparison, roughly HALVING the required sample size for the same power and MDE (since required N scales with the square of that multiplier, $1.0^2/1.41^2\approx0.5$). For 120 users, this can be the difference between an infeasible and a feasible test.

Practical plan:

• Metric selection: choose a metric with LOW within-user noise and short time-to-signal (e.g. task-completion rate or time-to-first-success on the developer-facing feature) rather than a slow, high-variance business metric like long-term retention, which would need far more than 120 users regardless of design.
• Instrumentation: capture per-user, per-period usage tightly enough to compute a clean pre/post or on/off comparison per user, and log enough contextual detail (which org/team each user belongs to) to check for and adjust for within-cluster correlation if the 120 users cluster into teams.
• Design choice given clustering: if the 120 users sit in, say, 15 teams of 8, prefer CLUSTER-randomization or a team-level switchback over individual randomization, since individually randomizing within a team risks contamination (control users on a team where teammates are treated may indirectly experience the feature via shared docs/workflows); if the users are genuinely independent (different companies, no interaction), individual-level within-subject randomization is fine and simpler.
• Rollout recommendation: run a holdout-ramp: enable the feature for a majority of the 120 (or all, using a within-subject pre/post design with a smaller true holdout group retained purely to distinguish the treatment effect from a general time trend), supplement the quantitative signal with structured qualitative feedback (brief surveys or interviews with a sample of the 120), since at this population size qualitative and case-study evidence carries proportionally MORE weight in the final ship/no-ship decision than it would in a consumer-scale test, and treat the quantitative estimate as directional rather than a precise point estimate given the residual power limitations even after pairing.

Direct answer: Fit a regression of the shared outcome on both experiments' assignment indicators PLUS their interaction term ($y = \beta_0 + \beta_1 A + \beta_2 B + \beta_3 (A \times B) + \varepsilon$); a statistically significant $\beta_3$ means assignment to A modified B's effect (or vice versa, the interaction is symmetric), the signature of cross-experiment contamination. Confirm the regression-based finding with a permutation test on $\beta_3$ (assumption-free, robust to the regression's parametric assumptions) before treating it as real. When multiple upstream experiments feed the same downstream metric, use a similar interaction-regression or a structural/layered-holdout approach to attribute the downstream change to individual upstream sources.

Interaction-term regression, verified working (executed): I simulated 20,000 users independently assigned to experiments A and B, with a planted TRUE contamination effect of $\beta_3 = 0.06$ (i.e. A's effect on the shared metric genuinely depends on B's assignment) plus small independent main effects and noise. The fitted model recovered $\hat{\beta}_3 = 0.0607$ (standard error 0.0042, t=14.31, p ≈ 0), correctly detecting the planted contamination at high confidence.

Permutation test (executed, model-assumption-free confirmation): I re-ran the same interaction estimate 500 times, each time RANDOMLY SHUFFLING experiment B's assignment (which destroys any true A x B relationship while preserving each variable's marginal distribution) and re-fitting the interaction coefficient. The permutation p-value, the fraction of shuffled-B interaction estimates at least as extreme as the observed $\hat{\beta}_3$, was $p_{\text{perm}}=0.004$, well below 0.05, confirming the interaction-regression result is not an artifact of the regression's normality/homoscedasticity assumptions.

Why run both: the analytic regression t-test is fast and gives a standard error and confidence interval for the SIZE of the contamination, useful for deciding whether it is business-meaningful, not just statistically detectable; the permutation test is slower but assumption-free, a useful independent confirmation particularly if the outcome metric is skewed or the sample is modest enough that the regression's asymptotic normality assumption is questionable.

Attributing a shared downstream metric to individual upstream experiments (multiple teams feeding the same funnel): (1) if the upstream experiments are ORTHOGONALLY assigned (verified independent per S9), extend the same interaction-regression approach to include ALL upstream assignment indicators plus their pairwise interactions with each other on the shared downstream outcome, the main-effect coefficients then give each experiment's OWN attributable contribution net of the others, and any significant interaction terms flag genuine cross-experiment effects rather than clean independent attribution; (2) if the upstream experiments are NOT independent, for example nested or sequential in the funnel (experiment 1 changes who even reaches the point where experiment 2 applies), a pure regression attribution is unreliable, and a LAYERED-HOLDOUT design, where a small slice of traffic is held out of ALL upstream experiments simultaneously and used as the pure counterfactual baseline for the downstream metric, gives a cleaner (if smaller-sample) attribution than trying to decompose correlated upstream effects via regression alone; (3) whichever approach is used, be explicit with stakeholders about the SCALABILITY limits, attribution via pairwise-interaction regression grows combinatorially in the number of upstream experiments and quickly becomes underpowered past a handful, and the layered-holdout approach has a real operational cost (traffic held back from every upstream experiment simultaneously), so in practice this kind of full attribution analysis is reserved for genuinely contested or high-stakes downstream-metric disputes between teams, not run routinely on every pair of concurrent experiments.

Direct answer: With 5 binary features and only 8 arms available (2^3), use a quarter-fraction 2^(5-2) design with generators D=AB and E=AC. This yields the standard minimum-aberration Resolution III design for this factor count. A is relatively safe to interpret (its only alias is the four-letter word ABCDE, negligible in practice); D and E carry real risk since each is 1:1 aliased with a plausible two-factor interaction (AB and AC respectively).

Constructing the design (executed): start from the full 2^3 factorial on A, B, C (8 runs), then set D = A·B and E = A·C:

Full alias structure (derived from the defining relations $I=ABD=ACE=BCDE$, the third relation is the product of the first two, word lengths 3, 3, 4, so this is Resolution III):

• A: aliased with BD (via ABD), CE (via ACE), and the weak ABCDE (via BCDE)
• B: aliased with AD, ABCE, CDE
• C: aliased with ABCD, AE, BDE
• D: aliased with AB, ACDE, BCE
• E: aliased with ABDE, AC, BCD

Interpretation guidance: A, B, and C are each cleanly separated from any OTHER main effect and only alias with 2-factor interactions (BD/CE for A, AD for B, AE for C respectively) or weaker higher-order words; if the business has no strong prior that those specific pairwise interactions are large, A/B/C main effects can be reported with reasonable confidence. D is aliased 1:1 with AB and E with AC, both entirely plausible 2-factor interactions a stakeholder might expect (e.g. if D and E were chosen as the two "riskiest" features, expect their main-effect estimates might really be picking up an AB or AC interaction): flag D and E's point estimates as "confounded, needs a confirmatory follow-up" rather than reporting them at face value.

Generalizing to the 3-headline/2-hero-image, high-interpretability landing-page case: this is a 3x2 mixed-level design (6 cells if run as a full factorial, which is small enough to run FULL rather than fractional; fractional designs earn their keep specifically when the FULL factorial's cell count would outstrip traffic, e.g. 4+ binary factors). Since interpretability is explicitly required here and 6 cells is affordable, recommend the FULL factorial over any fraction: fractional factorial trades interpretability for run-count savings, and this scenario does not need that trade.

Comparing full factorial, fractional factorial, and sequential targeted experiments (per aa4819c0's ask, when total traffic is limited across multiple segments): full factorial gives the cleanest, alias-free estimates but its cell count grows exponentially with factor count, quickly becoming infeasible; fractional factorial trades some interpretability (confounded higher-order effects) for a linear run-count, and is the right default once a full factorial would need more cells than traffic supports; sequential targeted experiments (test the highest-uncertainty factor first, then the next) avoid aliasing entirely but pay the SPEED cost quantified in S0 (roughly half the power per unit time, or double the calendar time, versus a concurrent design at the same total budget) and cannot detect interactions between factors tested in different sequential waves. Recommendation: default to full factorial while cells stay affordable (roughly up to 3-4 binary factors depending on traffic), switch to a minimum-aberration fractional design once traffic-per-cell would otherwise drop below your power floor, and reserve pure sequential testing for genuinely independent, low-interaction-risk factors or truly traffic-starved situations.

Direct answer: Formulate this as: minimize the variance of the interaction-effect estimator, $\text{Var}(\hat{\theta}_{AB}) = \sum_{i=1}^{4} \sigma_i^2/n_i$ (the four-cell +1/-1/-1/+1 contrast estimator's variance, assuming independent cells with per-cell variance $\sigma_i^2$), subject to a total-cost budget constraint $\sum_i c_i n_i = B$, where $n_i$ (the sample size in cell i) are the decision variables, $c_i$ the known per-user cost or complexity weight of cell i, and B the fixed budget.

Deriving the allocation rule: using a Lagrangian, $L = \sum_i \sigma_i^2/n_i + \lambda\left(\sum_i c_i n_i - B\right)$, setting $\partial L/\partial n_i = -\sigma_i^2/n_i^2 + \lambda c_i = 0$ gives $n_i \propto \sigma_i/\sqrt{c_i}$: allocate MORE sample to cells that are noisier (higher $\sigma_i$) and LESS to cells that are more expensive (higher $c_i$), the classic Neyman-allocation rule generalized to cost-weighted variance minimization.

Worked numeric example (executed): suppose all four cells have the same outcome variance ($\sigma_i=0.14$ for all i, e.g. a conversion-rate-like metric), but cell 4 (both treatments simultaneously on) is 5x costlier per user, for instance because it requires manual QA/concierge support, so $c = [1, 1, 1, 5]$, budget B = 400,000 cost-units.

• EQUAL allocation (n=50,000 in every cell, spending the budget evenly): $\text{Var}(\hat{\theta}_{AB}) = 1.568 \times 10^{-6}$.
• NEYMAN-OPTIMAL allocation ($n_i \propto \sigma_i/\sqrt{c_i}$): n* = [76,393, 76,393, 76,393, 34,164] (budget check: $\sum c_i n_i^* = 400{,}000$, exact), giving $\text{Var}(\hat{\theta}_{AB}) = 1.343 \times 10^{-6}$.

That is a 14.3% variance reduction versus equal allocation for the SAME total cost budget, purely from shifting sample away from the expensive cell 4 toward the three equally-noisy, cheaper cells.

Practical simplifications: (1) if per-cell variances are genuinely similar (the common case for conversion-rate-like metrics with similar baseline rates), the allocation collapses to pure cost-weighting, $n_i \propto 1/\sqrt{c_i}$, which is easy to compute without needing pilot-data variance estimates; (2) if BOTH cost and variance vary substantially and in the SAME direction (e.g. the noisier cell is also the pricier one), the two effects partially cancel and the gain from optimizing over equal allocation shrinks, in a version of this example with cell 4 at both 3x cost AND roughly 2x the SD, the improvement dropped to about 1%, so it is worth checking numerically rather than assuming Neyman allocation always helps a lot; (3) in practice, run this as a one-time pre-launch calculation using pilot-period variance and known engineering-cost estimates, not as a live-adjusting allocation, since re-allocating traffic mid-experiment reintroduces the kind of assignment-pipeline risk the sibling topic's A/A-test validation content is designed to catch.

Direct answer: Choose a factorial (or multivariate/MVT) design over a sequence of separate A/B tests when you have two or more candidate changes you plausibly want to ship TOGETHER, you suspect (or must rule out) an interaction between them, and you have enough concurrent traffic to split into cells without starving any one cell's power. Choose sequential single-factor A/B tests when changes are independent, traffic is scarce, or you need to ship and learn on the first change before the second is even designed.

Trade-offs

• Speed of learning: factorial wins when the alternative is truly SEQUENTIAL (test A, wait, then test B). For the same total sample budget, a k-factor design estimates all k main effects roughly as precisely as k separate two-arm tests run one after another with the SAME total budget split across them, but the factorial gets all k answers CONCURRENTLY instead of one-after-another, and gets the interaction essentially for free.
• Statistical power for main effects versus interactions: each main effect in a factorial is estimated by comparing half the cells against the other half (pooling over the other factor), so main-effect power is comparable to a standalone two-arm test at the same total N. The INTERACTION contrast is NOT pooled the same way: it needs the same per-cell sample size as a single unpooled cell-vs-cell comparison, so detecting an interaction of the same absolute size as a main effect costs roughly 2x the total sample of detecting that main effect alone (verified numerically in S1).
• Interpretability: a single-factor sequential test has a trivial reading ("does this metric move"). A factorial with a significant interaction requires explaining conditional effects ("A only works when B is on"), which is harder to communicate and can confuse stakeholders unless you plan for it (see S6).
• Operational complexity: factorial designs need every combination of treatments to be simultaneously deployable and measurable, more engineering coordination, more QA surface (interaction bugs, cross-feature conflicts), and a bigger simultaneous "blast radius" if something goes wrong, versus shipping and validating one change at a time.

Worked confirmation (executed simulation, not just cited): simulating 2 factors with true additive effects of size 0.12 (in outcome-SD units) each, total budget N=4000, alpha=0.05, over 4000 replications:

• FACTORIAL (4 cells of 1000 each): power(A) = 0.965, power(B) = 0.961
• SEQUENTIAL (2 separate 2-arm tests, same total budget split in half, 1000/arm each): power(A) = 0.771, power(B) = 0.759

For the identical total sample budget, the factorial design delivered materially higher power on BOTH main effects (0.96+ vs 0.77) because every unit in a factorial cell contributes to estimating EVERY factor's main effect (a unit in the (A=1,B=1) cell informs both the A comparison and the B comparison), while the sequential approach wastes half its budget on a control arm that is redundant across the two separate tests.

Business scenario where factorial is clearly preferable: a subscription product wants to test (1) a new pricing page copy and (2) a redesigned onboarding flow. Product suspects the pricing framing only lands if users have already been walked through the new onboarding (an interaction hypothesis: pricing-copy effect is conditional on onboarding version). Running these sequentially would answer "does new pricing help" and "does new onboarding help" separately but could never detect that new pricing ONLY helps when paired with new onboarding, exactly the finding the business needs to decide whether to ship them together, separately, or not at all. A 2x2 factorial answers all three questions (both main effects plus the interaction) in one concurrent test.

When NOT to use factorial: if you have 4+ factors, a FULL factorial's cell count (2^4 = 16 cells) can quickly outstrip available traffic; that is exactly the constrained-traffic case that motivates fractional factorial designs (S2, S3) rather than abandoning multi-factor testing altogether.