Advanced Experimentation Designs Questions

Direct answer: Structure the explanation as five short bullets/slides: (1) the question tested, in plain business language; (2) the headline result, including whether an interaction was found and what that means in plain terms; (3) 2-3 concrete business-impact scenarios tied to the interaction; (4) the uncertainty around the estimate, expressed as a range, not a p-value; (5) your recommended next action.

Worked structure, continuing the S1 pricing x layout example:

1. The question tested: "We tested a lower price point and a redesigned layout, separately and together, to see if either helps checkout conversion, and whether they work better as a pair."
2. Headline result: "The new layout alone did not move conversion. The lower price alone barely moved it. But the lower price PAIRED WITH the new layout increased conversion by [X]%, roughly [3x] more than either change alone. In plain terms: the discount only really lands when the page also feels redesigned and trustworthy; on the old layout it barely registers."
3. Business-impact scenarios: "If we ship both together: we expect roughly [$Y] in incremental monthly revenue, at the cost of a lower average order value from the discount. If we ship the layout alone: essentially no measurable revenue change, but a UX-quality win we can still justify separately. If we ship the discount alone (old layout): we lose margin with almost no conversion gain, this is the option to avoid."
4. Uncertainty: "We are confident the two changes need each other, this isn't a coin flip; the range on the combined lift is [X1]% to [X2]% with high confidence, meaning even in the more conservative case it still clears our bar to ship."
5. Recommended next action: "Ship the combined pricing-plus-layout change to 100% of traffic, and separately roll back or archive the discount-alone and layout-alone paths, since neither pays for itself independently."

Why this structure works for a non-technical audience: it leads with the DECISION-RELEVANT finding (the two changes must be shipped together, not separately) rather than the statistical mechanism (an interaction term was significant), translates "interaction" into "only works when paired with" language a business audience immediately understands, and puts uncertainty in dollar/percentage-range terms rather than a p-value or confidence-interval width that would require re-explaining what a p-value means.

Direct answer: Across 81 cells, do NOT read raw p < 0.05 at face value; control the false discovery rate (FDR) with Benjamini-Hochberg (or a variant if cells are correlated), separate robust main effects from interactions using replication and hierarchical shrinkage rather than trusting any single cell's raw estimate, and prioritize follow-ups by effect-size-times-confidence divided by rollout cost. Separately, for the SAME total daily traffic budget, a single 10-armed concurrent multivariate test reaches a decision in about 1.8x LESS calendar time than 9 sequential pairwise A/B tests, and does so using LESS total user-sample, not more.

FDR control (Benjamini-Hochberg, executed on a simulated 81-cell scenario): I planted 6 truly real effects (small p-values) among 75 truly null cells and ran BH at target FDR q=0.10. Sorting all 81 p-values ascending and finding the LARGEST rank k where $p_{(k)} \le q \cdot k/m$ (m=81) gave a cutoff of $p \le 0.0056$ at k=5, rejecting exactly 5 cells: all 5 were true discoveries, 0 false discoveries (realized FDR = 0.00, at or under the 10% target). For contrast, an UNCORRECTED alpha=0.05 across all 81 cells would flag about 9 cells, and on the 75 truly-null cells alone would be expected to false-positive about $75 \times 0.05 = 3.75$ cells purely by chance, exactly the multiple-comparisons trap this scale of test creates. Always run BH (or a dependence-aware variant if metrics are correlated) before reporting "significant cells" from a large multi-factor grid.

Separating robust main effects from interactions: (1) require BH-adjusted significance, not raw p, before treating a cell as real; (2) check whether an apparently large interaction cell is explainable by two large main effects alone (i.e. does the observed cell mean roughly equal the additive prediction from the two main effects, or does it deviate meaningfully, the deviation is the true interaction signal); (3) where traffic allows, hold out a slice of the 81-cell data or run a small confirmatory replication on the top few BH-surviving cells before committing to a business narrative, since even a well-controlled 10% FDR still means roughly 1 in 10 flagged cells could be noise; (4) prefer hierarchical/partial-pooling shrinkage over raw per-cell means when cells have very different sample sizes, so a lucky small-N cell does not get reported at face value.

Prioritized follow-ups: rank BH-surviving cells by (estimated effect size) x (posterior confidence, e.g. 1 minus BH-adjusted p) divided by (engineering/rollout cost of shipping that cell's combination), and recommend confirmatory single-factor or targeted 2-way follow-up tests on the top 2-3 ranked cells rather than attempting to roll out all 5 simultaneously (which would itself become an uncontrolled concurrent-experiment interference problem, see S9).

Throughput comparison (executed, baseline 5%, 100k visitors/day, 80% power, alpha 0.05): at a 10% relative MDE (baseline 5% to alternative 5.5%), the two-proportion sample size is 31,233 users per arm.

• Approach A, single 10-armed concurrent test (1 shared baseline + 9 variants, all accruing simultaneously): needs 312,334 users total (31,233 per arm x 10 arms), which at 100k/day takes about 3.12 days.
• Approach B, 9 SEQUENTIAL pairwise tests (each re-drawing its own fresh baseline arm, run one after another): needs 62,467 users per test (2 x 31,233) x 9 tests = 562,202 total, which takes about 5.62 days end-to-end.

Approach A finishes about 1.8x faster AND uses about 1.8x LESS total sample (312,334 vs 562,202), because the concurrent design shares ONE baseline arm across all 9 comparisons instead of re-drawing a fresh baseline 9 times. This is the same underlying mechanism as S0's power result: shared/concurrent accrual is more efficient than sequential re-baselining, on both the speed axis and the total-sample axis.

Direct answer: Choose a factorial (or multivariate/MVT) design over a sequence of separate A/B tests when you have two or more candidate changes you plausibly want to ship TOGETHER, you suspect (or must rule out) an interaction between them, and you have enough concurrent traffic to split into cells without starving any one cell's power. Choose sequential single-factor A/B tests when changes are independent, traffic is scarce, or you need to ship and learn on the first change before the second is even designed.

Trade-offs

• Speed of learning: factorial wins when the alternative is truly SEQUENTIAL (test A, wait, then test B). For the same total sample budget, a k-factor design estimates all k main effects roughly as precisely as k separate two-arm tests run one after another with the SAME total budget split across them, but the factorial gets all k answers CONCURRENTLY instead of one-after-another, and gets the interaction essentially for free.
• Statistical power for main effects versus interactions: each main effect in a factorial is estimated by comparing half the cells against the other half (pooling over the other factor), so main-effect power is comparable to a standalone two-arm test at the same total N. The INTERACTION contrast is NOT pooled the same way: it needs the same per-cell sample size as a single unpooled cell-vs-cell comparison, so detecting an interaction of the same absolute size as a main effect costs roughly 2x the total sample of detecting that main effect alone (verified numerically in S1).
• Interpretability: a single-factor sequential test has a trivial reading ("does this metric move"). A factorial with a significant interaction requires explaining conditional effects ("A only works when B is on"), which is harder to communicate and can confuse stakeholders unless you plan for it (see S6).
• Operational complexity: factorial designs need every combination of treatments to be simultaneously deployable and measurable, more engineering coordination, more QA surface (interaction bugs, cross-feature conflicts), and a bigger simultaneous "blast radius" if something goes wrong, versus shipping and validating one change at a time.

Worked confirmation (executed simulation, not just cited): simulating 2 factors with true additive effects of size 0.12 (in outcome-SD units) each, total budget N=4000, alpha=0.05, over 4000 replications:

• FACTORIAL (4 cells of 1000 each): power(A) = 0.965, power(B) = 0.961
• SEQUENTIAL (2 separate 2-arm tests, same total budget split in half, 1000/arm each): power(A) = 0.771, power(B) = 0.759

For the identical total sample budget, the factorial design delivered materially higher power on BOTH main effects (0.96+ vs 0.77) because every unit in a factorial cell contributes to estimating EVERY factor's main effect (a unit in the (A=1,B=1) cell informs both the A comparison and the B comparison), while the sequential approach wastes half its budget on a control arm that is redundant across the two separate tests.

Business scenario where factorial is clearly preferable: a subscription product wants to test (1) a new pricing page copy and (2) a redesigned onboarding flow. Product suspects the pricing framing only lands if users have already been walked through the new onboarding (an interaction hypothesis: pricing-copy effect is conditional on onboarding version). Running these sequentially would answer "does new pricing help" and "does new onboarding help" separately but could never detect that new pricing ONLY helps when paired with new onboarding, exactly the finding the business needs to decide whether to ship them together, separately, or not at all. A 2x2 factorial answers all three questions (both main effects plus the interaction) in one concurrent test.

When NOT to use factorial: if you have 4+ factors, a FULL factorial's cell count (2^4 = 16 cells) can quickly outstrip available traffic; that is exactly the constrained-traffic case that motivates fractional factorial designs (S2, S3) rather than abandoning multi-factor testing altogether.

Direct answer: Fit a regression of the shared outcome on both experiments' assignment indicators PLUS their interaction term ($y = \beta_0 + \beta_1 A + \beta_2 B + \beta_3 (A \times B) + \varepsilon$); a statistically significant $\beta_3$ means assignment to A modified B's effect (or vice versa, the interaction is symmetric), the signature of cross-experiment contamination. Confirm the regression-based finding with a permutation test on $\beta_3$ (assumption-free, robust to the regression's parametric assumptions) before treating it as real. When multiple upstream experiments feed the same downstream metric, use a similar interaction-regression or a structural/layered-holdout approach to attribute the downstream change to individual upstream sources.

Interaction-term regression, verified working (executed): I simulated 20,000 users independently assigned to experiments A and B, with a planted TRUE contamination effect of $\beta_3 = 0.06$ (i.e. A's effect on the shared metric genuinely depends on B's assignment) plus small independent main effects and noise. The fitted model recovered $\hat{\beta}_3 = 0.0607$ (standard error 0.0042, t=14.31, p ≈ 0), correctly detecting the planted contamination at high confidence.

Permutation test (executed, model-assumption-free confirmation): I re-ran the same interaction estimate 500 times, each time RANDOMLY SHUFFLING experiment B's assignment (which destroys any true A x B relationship while preserving each variable's marginal distribution) and re-fitting the interaction coefficient. The permutation p-value, the fraction of shuffled-B interaction estimates at least as extreme as the observed $\hat{\beta}_3$, was $p_{\text{perm}}=0.004$, well below 0.05, confirming the interaction-regression result is not an artifact of the regression's normality/homoscedasticity assumptions.

Why run both: the analytic regression t-test is fast and gives a standard error and confidence interval for the SIZE of the contamination, useful for deciding whether it is business-meaningful, not just statistically detectable; the permutation test is slower but assumption-free, a useful independent confirmation particularly if the outcome metric is skewed or the sample is modest enough that the regression's asymptotic normality assumption is questionable.

Attributing a shared downstream metric to individual upstream experiments (multiple teams feeding the same funnel): (1) if the upstream experiments are ORTHOGONALLY assigned (verified independent per S9), extend the same interaction-regression approach to include ALL upstream assignment indicators plus their pairwise interactions with each other on the shared downstream outcome, the main-effect coefficients then give each experiment's OWN attributable contribution net of the others, and any significant interaction terms flag genuine cross-experiment effects rather than clean independent attribution; (2) if the upstream experiments are NOT independent, for example nested or sequential in the funnel (experiment 1 changes who even reaches the point where experiment 2 applies), a pure regression attribution is unreliable, and a LAYERED-HOLDOUT design, where a small slice of traffic is held out of ALL upstream experiments simultaneously and used as the pure counterfactual baseline for the downstream metric, gives a cleaner (if smaller-sample) attribution than trying to decompose correlated upstream effects via regression alone; (3) whichever approach is used, be explicit with stakeholders about the SCALABILITY limits, attribution via pairwise-interaction regression grows combinatorially in the number of upstream experiments and quickly becomes underpowered past a handful, and the layered-holdout approach has a real operational cost (traffic held back from every upstream experiment simultaneously), so in practice this kind of full attribution analysis is reserved for genuinely contested or high-stakes downstream-metric disputes between teams, not run routinely on every pair of concurrent experiments.

Direct answer: With a 2% baseline conversion rate, 100k eligible users/day, 80% power, and alpha 0.05, and assuming a 15% relative minimum detectable effect (the question does not state one explicitly, so I am stating this assumption up front, a common convention for this kind of sizing exercise; new baseline = 2.30%), the design needs roughly 18,346 users PER CELL to power the main effects, or about 36,693 users per cell to ALSO power the interaction at the same MDE. Traffic should be split evenly, 25% into each of the 4 cells (control/control, pricing-B/layout-old, pricing-A/layout-new, pricing-B/layout-new), randomized independently on pricing and layout so every user has an equal chance of any of the 4 combinations.

Sample-size computation (executed, two-proportion z-test):

For a MAIN EFFECT comparison (pooling the 2 cells that share a pricing level against the other 2), $p_1=0.02$, $p_2=0.023$ (15% relative lift):

$$n = \frac{\left(z_{\alpha/2}\sqrt{2\bar{p}(1-\bar{p})} + z_{\beta}\sqrt{p_1(1-p_1)+p_2(1-p_2)}\right)^2}{(p_2-p_1)^2}$$

gives n = 36,693 per POOLED arm (i.e. per 2-cell group). Since each pooled arm is 2 of the 4 cells, that is 18,346 users per CELL for main-effect power, 73,386 total across all 4 cells, which at 100k/day takes about 0.73 days.

For the INTERACTION contrast (an unpooled, single-cell-vs-single-cell comparison, since the interaction estimate uses the +1/-1/-1/+1 contrast across all four individual cells rather than pooled halves), the SAME formula applied per cell (no pooling) requires 36,693 users per cell, 146,771 total, about 1.47 days at 100k/day, exactly 2.00x the main-effect total. This is the general result cited in S0: interaction detection at the same MDE costs about 2x a main effect's sample size.

Traffic allocation: randomize pricing (A vs B) and layout (old vs new) INDEPENDENTLY via two separate hash-based coin flips per user (verified orthogonal in S9), producing four roughly-equal 25% cells automatically; do not manually stratify into 4 named buckets, since independent-factor randomization already balances the design and keeps it robust to later adding a third factor.

Analysis and interpretation: fit

$$y = \beta_0 + \beta_1 \cdot \text{Pricing} + \beta_2 \cdot \text{Layout} + \beta_3 \cdot (\text{Pricing} \times \text{Layout}) + \varepsilon$$

on the binary conversion outcome (linear probability model or logistic regression with an interaction term). $\beta_1$ is the pricing main effect averaged over both layouts, $\beta_2$ the layout main effect averaged over both pricing versions, and $\beta_3$ the interaction: how much the pricing effect CHANGES when layout changes (equivalently, how much the layout effect changes when pricing changes). Run the interaction test FIRST: if $\beta_3$ is significant, report cell-specific conversion rates (all 4 cells) rather than a single "average" pricing effect, since the average would be misleading if the effect flips sign across layouts. If the interaction is not significant, report the two main effects as if from independent tests and recommend shipping whichever combination of winning main effects performs best. Present results to stakeholders as: (1) the interaction test verdict first, in plain language, (2) a 2x2 table of observed conversion rates per cell with confidence intervals, (3) the recommended cell to ship, and (4) the guardrail-metric check on that winning cell (per this topic's sibling a-b-test-design-and-statistical-rigor, which owns guardrail-metric mechanics).