Advanced Experimentation Designs Questions

Direct answer: Across 81 cells, do NOT read raw p < 0.05 at face value; control the false discovery rate (FDR) with Benjamini-Hochberg (or a variant if cells are correlated), separate robust main effects from interactions using replication and hierarchical shrinkage rather than trusting any single cell's raw estimate, and prioritize follow-ups by effect-size-times-confidence divided by rollout cost. Separately, for the SAME total daily traffic budget, a single 10-armed concurrent multivariate test reaches a decision in about 1.8x LESS calendar time than 9 sequential pairwise A/B tests, and does so using LESS total user-sample, not more.

FDR control (Benjamini-Hochberg, executed on a simulated 81-cell scenario): I planted 6 truly real effects (small p-values) among 75 truly null cells and ran BH at target FDR q=0.10. Sorting all 81 p-values ascending and finding the LARGEST rank k where $p_{(k)} \le q \cdot k/m$ (m=81) gave a cutoff of $p \le 0.0056$ at k=5, rejecting exactly 5 cells: all 5 were true discoveries, 0 false discoveries (realized FDR = 0.00, at or under the 10% target). For contrast, an UNCORRECTED alpha=0.05 across all 81 cells would flag about 9 cells, and on the 75 truly-null cells alone would be expected to false-positive about $75 \times 0.05 = 3.75$ cells purely by chance, exactly the multiple-comparisons trap this scale of test creates. Always run BH (or a dependence-aware variant if metrics are correlated) before reporting "significant cells" from a large multi-factor grid.

Separating robust main effects from interactions: (1) require BH-adjusted significance, not raw p, before treating a cell as real; (2) check whether an apparently large interaction cell is explainable by two large main effects alone (i.e. does the observed cell mean roughly equal the additive prediction from the two main effects, or does it deviate meaningfully, the deviation is the true interaction signal); (3) where traffic allows, hold out a slice of the 81-cell data or run a small confirmatory replication on the top few BH-surviving cells before committing to a business narrative, since even a well-controlled 10% FDR still means roughly 1 in 10 flagged cells could be noise; (4) prefer hierarchical/partial-pooling shrinkage over raw per-cell means when cells have very different sample sizes, so a lucky small-N cell does not get reported at face value.

Prioritized follow-ups: rank BH-surviving cells by (estimated effect size) x (posterior confidence, e.g. 1 minus BH-adjusted p) divided by (engineering/rollout cost of shipping that cell's combination), and recommend confirmatory single-factor or targeted 2-way follow-up tests on the top 2-3 ranked cells rather than attempting to roll out all 5 simultaneously (which would itself become an uncontrolled concurrent-experiment interference problem, see S9).

Throughput comparison (executed, baseline 5%, 100k visitors/day, 80% power, alpha 0.05): at a 10% relative MDE (baseline 5% to alternative 5.5%), the two-proportion sample size is 31,233 users per arm.

• Approach A, single 10-armed concurrent test (1 shared baseline + 9 variants, all accruing simultaneously): needs 312,334 users total (31,233 per arm x 10 arms), which at 100k/day takes about 3.12 days.
• Approach B, 9 SEQUENTIAL pairwise tests (each re-drawing its own fresh baseline arm, run one after another): needs 62,467 users per test (2 x 31,233) x 9 tests = 562,202 total, which takes about 5.62 days end-to-end.

Approach A finishes about 1.8x faster AND uses about 1.8x LESS total sample (312,334 vs 562,202), because the concurrent design shares ONE baseline arm across all 9 comparisons instead of re-drawing a fresh baseline 9 times. This is the same underlying mechanism as S0's power result: shared/concurrent accrual is more efficient than sequential re-baselining, on both the speed axis and the total-sample axis.

Direct answer: Aliasing happens when a fractional factorial design cannot separate the influence of two (or more) different effects because they change in exactly the same pattern across the runs that were actually included in the reduced design; the two effects are then statistically indistinguishable from each other using that data alone. Design resolution describes how BAD the confounding is: Resolution III means main effects are aliased with two-factor interactions (2fi's); Resolution IV means main effects are clear of 2fi's but 2fi's are aliased with each other; Resolution V means main effects and 2fi's are all clear of each other (only 2fi's alias with three-factor interactions, which are usually negligible). Higher resolution costs more runs for the same number of factors.

Deriving the alias structure for the 2^(3-1) half-fraction, generator C = AB:

The half-fraction design matrix (executed):

Since C = A·B (the defining generator), multiplying both sides by C gives $C \cdot C = A \cdot B \cdot C$, and since $C^2 = I$ (the identity column, all +1), the DEFINING RELATION is $I = ABC$.

To find what any effect is aliased with, multiply it by the defining relation and simplify using $X^2 = I$:

• $A \text{ aliased with } A \cdot ABC = A^2BC = BC$
• $B \text{ aliased with } B \cdot ABC = AB^2C = AC$
• $C \text{ aliased with } C \cdot ABC = ABC^2 = AB$

So main effect A is aliased with interaction BC precisely BECAUSE the defining relation is $I=ABC$: the "A" column in the half-fraction and the "BC" column (computed as B times C) are numerically identical (both equal +1,-1,-1,+1 down the four runs), so no linear model fit to this data alone can tell whether an observed effect on that column comes from A, from BC, or some mixture of both. This is a Resolution III design (shortest word in the defining relation, "ABC," has length 3).

De-aliasing via fold-over (executed simulation, single-factor construction): run a second block, a "fold-over on A": flip the SIGN of column A only, keep the B and C run-values exactly as they were in block 1 (do not re-derive C from the flipped A):

Checking the ABC column on this second block gives -1 on every run, so its defining relation is $I=-ABC$ (flipping one letter's sign in an odd-length word flips the word's overall sign). That sign flip is the whole trick: in block 1, A is aliased with $+BC$ (its contrast estimates $\text{effect}(A)+\text{effect}(BC)$); in the fold-over block, A is aliased with $-BC$ (its contrast estimates $\text{effect}(A)-\text{effect}(BC)$). Simulating a true response with A's true effect = 5 and BC's true effect = 3 (noise SD 0.5, averaged over 5000 replications): block 1's A-contrast converges to 8.00 (matching $5+3$), and the fold-over block's A-contrast converges to 2.00 (matching $5-3$). AVERAGING the two blocks' A-contrast estimates, $(8.00+2.00)/2=5.00$, isolates A alone (true A = 5, exact match); the HALF-DIFFERENCE, $(8.00-2.00)/2=3.00$, isolates BC alone (true BC = 3, exact match). The general mechanism: fold over on the ONE factor you need de-aliased (flip only its column, keep the others fixed), which flips that factor's alias sign but not the alias's own true effect, then the sum and difference of the two blocks' contrast estimates separate the two confounded effects cleanly.

Direct answer: Fit a regression of the shared outcome on both experiments' assignment indicators PLUS their interaction term ($y = \beta_0 + \beta_1 A + \beta_2 B + \beta_3 (A \times B) + \varepsilon$); a statistically significant $\beta_3$ means assignment to A modified B's effect (or vice versa, the interaction is symmetric), the signature of cross-experiment contamination. Confirm the regression-based finding with a permutation test on $\beta_3$ (assumption-free, robust to the regression's parametric assumptions) before treating it as real. When multiple upstream experiments feed the same downstream metric, use a similar interaction-regression or a structural/layered-holdout approach to attribute the downstream change to individual upstream sources.

Interaction-term regression, verified working (executed): I simulated 20,000 users independently assigned to experiments A and B, with a planted TRUE contamination effect of $\beta_3 = 0.06$ (i.e. A's effect on the shared metric genuinely depends on B's assignment) plus small independent main effects and noise. The fitted model recovered $\hat{\beta}_3 = 0.0607$ (standard error 0.0042, t=14.31, p ≈ 0), correctly detecting the planted contamination at high confidence.

Permutation test (executed, model-assumption-free confirmation): I re-ran the same interaction estimate 500 times, each time RANDOMLY SHUFFLING experiment B's assignment (which destroys any true A x B relationship while preserving each variable's marginal distribution) and re-fitting the interaction coefficient. The permutation p-value, the fraction of shuffled-B interaction estimates at least as extreme as the observed $\hat{\beta}_3$, was $p_{\text{perm}}=0.004$, well below 0.05, confirming the interaction-regression result is not an artifact of the regression's normality/homoscedasticity assumptions.

Why run both: the analytic regression t-test is fast and gives a standard error and confidence interval for the SIZE of the contamination, useful for deciding whether it is business-meaningful, not just statistically detectable; the permutation test is slower but assumption-free, a useful independent confirmation particularly if the outcome metric is skewed or the sample is modest enough that the regression's asymptotic normality assumption is questionable.

Attributing a shared downstream metric to individual upstream experiments (multiple teams feeding the same funnel): (1) if the upstream experiments are ORTHOGONALLY assigned (verified independent per S9), extend the same interaction-regression approach to include ALL upstream assignment indicators plus their pairwise interactions with each other on the shared downstream outcome, the main-effect coefficients then give each experiment's OWN attributable contribution net of the others, and any significant interaction terms flag genuine cross-experiment effects rather than clean independent attribution; (2) if the upstream experiments are NOT independent, for example nested or sequential in the funnel (experiment 1 changes who even reaches the point where experiment 2 applies), a pure regression attribution is unreliable, and a LAYERED-HOLDOUT design, where a small slice of traffic is held out of ALL upstream experiments simultaneously and used as the pure counterfactual baseline for the downstream metric, gives a cleaner (if smaller-sample) attribution than trying to decompose correlated upstream effects via regression alone; (3) whichever approach is used, be explicit with stakeholders about the SCALABILITY limits, attribution via pairwise-interaction regression grows combinatorially in the number of upstream experiments and quickly becomes underpowered past a handful, and the layered-holdout approach has a real operational cost (traffic held back from every upstream experiment simultaneously), so in practice this kind of full attribution analysis is reserved for genuinely contested or high-stakes downstream-metric disputes between teams, not run routinely on every pair of concurrent experiments.

Direct answer: With a 2% baseline conversion rate, 100k eligible users/day, 80% power, and alpha 0.05, and assuming a 15% relative minimum detectable effect (the question does not state one explicitly, so I am stating this assumption up front, a common convention for this kind of sizing exercise; new baseline = 2.30%), the design needs roughly 18,346 users PER CELL to power the main effects, or about 36,693 users per cell to ALSO power the interaction at the same MDE. Traffic should be split evenly, 25% into each of the 4 cells (control/control, pricing-B/layout-old, pricing-A/layout-new, pricing-B/layout-new), randomized independently on pricing and layout so every user has an equal chance of any of the 4 combinations.

Sample-size computation (executed, two-proportion z-test):

For a MAIN EFFECT comparison (pooling the 2 cells that share a pricing level against the other 2), $p_1=0.02$, $p_2=0.023$ (15% relative lift):

$$n = \frac{\left(z_{\alpha/2}\sqrt{2\bar{p}(1-\bar{p})} + z_{\beta}\sqrt{p_1(1-p_1)+p_2(1-p_2)}\right)^2}{(p_2-p_1)^2}$$

gives n = 36,693 per POOLED arm (i.e. per 2-cell group). Since each pooled arm is 2 of the 4 cells, that is 18,346 users per CELL for main-effect power, 73,386 total across all 4 cells, which at 100k/day takes about 0.73 days.

For the INTERACTION contrast (an unpooled, single-cell-vs-single-cell comparison, since the interaction estimate uses the +1/-1/-1/+1 contrast across all four individual cells rather than pooled halves), the SAME formula applied per cell (no pooling) requires 36,693 users per cell, 146,771 total, about 1.47 days at 100k/day, exactly 2.00x the main-effect total. This is the general result cited in S0: interaction detection at the same MDE costs about 2x a main effect's sample size.

Traffic allocation: randomize pricing (A vs B) and layout (old vs new) INDEPENDENTLY via two separate hash-based coin flips per user (verified orthogonal in S9), producing four roughly-equal 25% cells automatically; do not manually stratify into 4 named buckets, since independent-factor randomization already balances the design and keeps it robust to later adding a third factor.

Analysis and interpretation: fit

$$y = \beta_0 + \beta_1 \cdot \text{Pricing} + \beta_2 \cdot \text{Layout} + \beta_3 \cdot (\text{Pricing} \times \text{Layout}) + \varepsilon$$

on the binary conversion outcome (linear probability model or logistic regression with an interaction term). $\beta_1$ is the pricing main effect averaged over both layouts, $\beta_2$ the layout main effect averaged over both pricing versions, and $\beta_3$ the interaction: how much the pricing effect CHANGES when layout changes (equivalently, how much the layout effect changes when pricing changes). Run the interaction test FIRST: if $\beta_3$ is significant, report cell-specific conversion rates (all 4 cells) rather than a single "average" pricing effect, since the average would be misleading if the effect flips sign across layouts. If the interaction is not significant, report the two main effects as if from independent tests and recommend shipping whichever combination of winning main effects performs best. Present results to stakeholders as: (1) the interaction test verdict first, in plain language, (2) a 2x2 table of observed conversion rates per cell with confidence intervals, (3) the recommended cell to ship, and (4) the guardrail-metric check on that winning cell (per this topic's sibling a-b-test-design-and-statistical-rigor, which owns guardrail-metric mechanics).

Direct answer: Choose a factorial (or multivariate/MVT) design over a sequence of separate A/B tests when you have two or more candidate changes you plausibly want to ship TOGETHER, you suspect (or must rule out) an interaction between them, and you have enough concurrent traffic to split into cells without starving any one cell's power. Choose sequential single-factor A/B tests when changes are independent, traffic is scarce, or you need to ship and learn on the first change before the second is even designed.

Trade-offs

• Speed of learning: factorial wins when the alternative is truly SEQUENTIAL (test A, wait, then test B). For the same total sample budget, a k-factor design estimates all k main effects roughly as precisely as k separate two-arm tests run one after another with the SAME total budget split across them, but the factorial gets all k answers CONCURRENTLY instead of one-after-another, and gets the interaction essentially for free.
• Statistical power for main effects versus interactions: each main effect in a factorial is estimated by comparing half the cells against the other half (pooling over the other factor), so main-effect power is comparable to a standalone two-arm test at the same total N. The INTERACTION contrast is NOT pooled the same way: it needs the same per-cell sample size as a single unpooled cell-vs-cell comparison, so detecting an interaction of the same absolute size as a main effect costs roughly 2x the total sample of detecting that main effect alone (verified numerically in S1).
• Interpretability: a single-factor sequential test has a trivial reading ("does this metric move"). A factorial with a significant interaction requires explaining conditional effects ("A only works when B is on"), which is harder to communicate and can confuse stakeholders unless you plan for it (see S6).
• Operational complexity: factorial designs need every combination of treatments to be simultaneously deployable and measurable, more engineering coordination, more QA surface (interaction bugs, cross-feature conflicts), and a bigger simultaneous "blast radius" if something goes wrong, versus shipping and validating one change at a time.

Worked confirmation (executed simulation, not just cited): simulating 2 factors with true additive effects of size 0.12 (in outcome-SD units) each, total budget N=4000, alpha=0.05, over 4000 replications:

• FACTORIAL (4 cells of 1000 each): power(A) = 0.965, power(B) = 0.961
• SEQUENTIAL (2 separate 2-arm tests, same total budget split in half, 1000/arm each): power(A) = 0.771, power(B) = 0.759

For the identical total sample budget, the factorial design delivered materially higher power on BOTH main effects (0.96+ vs 0.77) because every unit in a factorial cell contributes to estimating EVERY factor's main effect (a unit in the (A=1,B=1) cell informs both the A comparison and the B comparison), while the sequential approach wastes half its budget on a control arm that is redundant across the two separate tests.

Business scenario where factorial is clearly preferable: a subscription product wants to test (1) a new pricing page copy and (2) a redesigned onboarding flow. Product suspects the pricing framing only lands if users have already been walked through the new onboarding (an interaction hypothesis: pricing-copy effect is conditional on onboarding version). Running these sequentially would answer "does new pricing help" and "does new onboarding help" separately but could never detect that new pricing ONLY helps when paired with new onboarding, exactly the finding the business needs to decide whether to ship them together, separately, or not at all. A 2x2 factorial answers all three questions (both main effects plus the interaction) in one concurrent test.

When NOT to use factorial: if you have 4+ factors, a FULL factorial's cell count (2^4 = 16 cells) can quickly outstrip available traffic; that is exactly the constrained-traffic case that motivates fractional factorial designs (S2, S3) rather than abandoning multi-factor testing altogether.