Advanced Experimentation Designs Questions

Direct answer: Choose a factorial (or multivariate/MVT) design over a sequence of separate A/B tests when you have two or more candidate changes you plausibly want to ship TOGETHER, you suspect (or must rule out) an interaction between them, and you have enough concurrent traffic to split into cells without starving any one cell's power. Choose sequential single-factor A/B tests when changes are independent, traffic is scarce, or you need to ship and learn on the first change before the second is even designed.

Trade-offs

• Speed of learning: factorial wins when the alternative is truly SEQUENTIAL (test A, wait, then test B). For the same total sample budget, a k-factor design estimates all k main effects roughly as precisely as k separate two-arm tests run one after another with the SAME total budget split across them, but the factorial gets all k answers CONCURRENTLY instead of one-after-another, and gets the interaction essentially for free.
• Statistical power for main effects versus interactions: each main effect in a factorial is estimated by comparing half the cells against the other half (pooling over the other factor), so main-effect power is comparable to a standalone two-arm test at the same total N. The INTERACTION contrast is NOT pooled the same way: it needs the same per-cell sample size as a single unpooled cell-vs-cell comparison, so detecting an interaction of the same absolute size as a main effect costs roughly 2x the total sample of detecting that main effect alone (verified numerically in S1).
• Interpretability: a single-factor sequential test has a trivial reading ("does this metric move"). A factorial with a significant interaction requires explaining conditional effects ("A only works when B is on"), which is harder to communicate and can confuse stakeholders unless you plan for it (see S6).
• Operational complexity: factorial designs need every combination of treatments to be simultaneously deployable and measurable, more engineering coordination, more QA surface (interaction bugs, cross-feature conflicts), and a bigger simultaneous "blast radius" if something goes wrong, versus shipping and validating one change at a time.

Worked confirmation (executed simulation, not just cited): simulating 2 factors with true additive effects of size 0.12 (in outcome-SD units) each, total budget N=4000, alpha=0.05, over 4000 replications:

• FACTORIAL (4 cells of 1000 each): power(A) = 0.965, power(B) = 0.961
• SEQUENTIAL (2 separate 2-arm tests, same total budget split in half, 1000/arm each): power(A) = 0.771, power(B) = 0.759

For the identical total sample budget, the factorial design delivered materially higher power on BOTH main effects (0.96+ vs 0.77) because every unit in a factorial cell contributes to estimating EVERY factor's main effect (a unit in the (A=1,B=1) cell informs both the A comparison and the B comparison), while the sequential approach wastes half its budget on a control arm that is redundant across the two separate tests.

Business scenario where factorial is clearly preferable: a subscription product wants to test (1) a new pricing page copy and (2) a redesigned onboarding flow. Product suspects the pricing framing only lands if users have already been walked through the new onboarding (an interaction hypothesis: pricing-copy effect is conditional on onboarding version). Running these sequentially would answer "does new pricing help" and "does new onboarding help" separately but could never detect that new pricing ONLY helps when paired with new onboarding, exactly the finding the business needs to decide whether to ship them together, separately, or not at all. A 2x2 factorial answers all three questions (both main effects plus the interaction) in one concurrent test.

When NOT to use factorial: if you have 4+ factors, a FULL factorial's cell count (2^4 = 16 cells) can quickly outstrip available traffic; that is exactly the constrained-traffic case that motivates fractional factorial designs (S2, S3) rather than abandoning multi-factor testing altogether.

Direct answer: Aliasing happens when a fractional factorial design cannot separate the influence of two (or more) different effects because they change in exactly the same pattern across the runs that were actually included in the reduced design; the two effects are then statistically indistinguishable from each other using that data alone. Design resolution describes how BAD the confounding is: Resolution III means main effects are aliased with two-factor interactions (2fi's); Resolution IV means main effects are clear of 2fi's but 2fi's are aliased with each other; Resolution V means main effects and 2fi's are all clear of each other (only 2fi's alias with three-factor interactions, which are usually negligible). Higher resolution costs more runs for the same number of factors.

Deriving the alias structure for the 2^(3-1) half-fraction, generator C = AB:

The half-fraction design matrix (executed):

Since C = A·B (the defining generator), multiplying both sides by C gives $C \cdot C = A \cdot B \cdot C$, and since $C^2 = I$ (the identity column, all +1), the DEFINING RELATION is $I = ABC$.

To find what any effect is aliased with, multiply it by the defining relation and simplify using $X^2 = I$:

• $A \text{ aliased with } A \cdot ABC = A^2BC = BC$
• $B \text{ aliased with } B \cdot ABC = AB^2C = AC$
• $C \text{ aliased with } C \cdot ABC = ABC^2 = AB$

So main effect A is aliased with interaction BC precisely BECAUSE the defining relation is $I=ABC$: the "A" column in the half-fraction and the "BC" column (computed as B times C) are numerically identical (both equal +1,-1,-1,+1 down the four runs), so no linear model fit to this data alone can tell whether an observed effect on that column comes from A, from BC, or some mixture of both. This is a Resolution III design (shortest word in the defining relation, "ABC," has length 3).

De-aliasing via fold-over (executed simulation, single-factor construction): run a second block, a "fold-over on A": flip the SIGN of column A only, keep the B and C run-values exactly as they were in block 1 (do not re-derive C from the flipped A):

Checking the ABC column on this second block gives -1 on every run, so its defining relation is $I=-ABC$ (flipping one letter's sign in an odd-length word flips the word's overall sign). That sign flip is the whole trick: in block 1, A is aliased with $+BC$ (its contrast estimates $\text{effect}(A)+\text{effect}(BC)$); in the fold-over block, A is aliased with $-BC$ (its contrast estimates $\text{effect}(A)-\text{effect}(BC)$). Simulating a true response with A's true effect = 5 and BC's true effect = 3 (noise SD 0.5, averaged over 5000 replications): block 1's A-contrast converges to 8.00 (matching $5+3$), and the fold-over block's A-contrast converges to 2.00 (matching $5-3$). AVERAGING the two blocks' A-contrast estimates, $(8.00+2.00)/2=5.00$, isolates A alone (true A = 5, exact match); the HALF-DIFFERENCE, $(8.00-2.00)/2=3.00$, isolates BC alone (true BC = 3, exact match). The general mechanism: fold over on the ONE factor you need de-aliased (flip only its column, keep the others fixed), which flips that factor's alias sign but not the alias's own true effect, then the sum and difference of the two blocks' contrast estimates separate the two confounded effects cleanly.

Direct answer: Fit a regression of the shared outcome on both experiments' assignment indicators PLUS their interaction term ($y = \beta_0 + \beta_1 A + \beta_2 B + \beta_3 (A \times B) + \varepsilon$); a statistically significant $\beta_3$ means assignment to A modified B's effect (or vice versa, the interaction is symmetric), the signature of cross-experiment contamination. Confirm the regression-based finding with a permutation test on $\beta_3$ (assumption-free, robust to the regression's parametric assumptions) before treating it as real. When multiple upstream experiments feed the same downstream metric, use a similar interaction-regression or a structural/layered-holdout approach to attribute the downstream change to individual upstream sources.

Interaction-term regression, verified working (executed): I simulated 20,000 users independently assigned to experiments A and B, with a planted TRUE contamination effect of $\beta_3 = 0.06$ (i.e. A's effect on the shared metric genuinely depends on B's assignment) plus small independent main effects and noise. The fitted model recovered $\hat{\beta}_3 = 0.0607$ (standard error 0.0042, t=14.31, p ≈ 0), correctly detecting the planted contamination at high confidence.

Permutation test (executed, model-assumption-free confirmation): I re-ran the same interaction estimate 500 times, each time RANDOMLY SHUFFLING experiment B's assignment (which destroys any true A x B relationship while preserving each variable's marginal distribution) and re-fitting the interaction coefficient. The permutation p-value, the fraction of shuffled-B interaction estimates at least as extreme as the observed $\hat{\beta}_3$, was $p_{\text{perm}}=0.004$, well below 0.05, confirming the interaction-regression result is not an artifact of the regression's normality/homoscedasticity assumptions.

Why run both: the analytic regression t-test is fast and gives a standard error and confidence interval for the SIZE of the contamination, useful for deciding whether it is business-meaningful, not just statistically detectable; the permutation test is slower but assumption-free, a useful independent confirmation particularly if the outcome metric is skewed or the sample is modest enough that the regression's asymptotic normality assumption is questionable.

Attributing a shared downstream metric to individual upstream experiments (multiple teams feeding the same funnel): (1) if the upstream experiments are ORTHOGONALLY assigned (verified independent per S9), extend the same interaction-regression approach to include ALL upstream assignment indicators plus their pairwise interactions with each other on the shared downstream outcome, the main-effect coefficients then give each experiment's OWN attributable contribution net of the others, and any significant interaction terms flag genuine cross-experiment effects rather than clean independent attribution; (2) if the upstream experiments are NOT independent, for example nested or sequential in the funnel (experiment 1 changes who even reaches the point where experiment 2 applies), a pure regression attribution is unreliable, and a LAYERED-HOLDOUT design, where a small slice of traffic is held out of ALL upstream experiments simultaneously and used as the pure counterfactual baseline for the downstream metric, gives a cleaner (if smaller-sample) attribution than trying to decompose correlated upstream effects via regression alone; (3) whichever approach is used, be explicit with stakeholders about the SCALABILITY limits, attribution via pairwise-interaction regression grows combinatorially in the number of upstream experiments and quickly becomes underpowered past a handful, and the layered-holdout approach has a real operational cost (traffic held back from every upstream experiment simultaneously), so in practice this kind of full attribution analysis is reserved for genuinely contested or high-stakes downstream-metric disputes between teams, not run routinely on every pair of concurrent experiments.

Direct answer: Across 81 cells, do NOT read raw p < 0.05 at face value; control the false discovery rate (FDR) with Benjamini-Hochberg (or a variant if cells are correlated), separate robust main effects from interactions using replication and hierarchical shrinkage rather than trusting any single cell's raw estimate, and prioritize follow-ups by effect-size-times-confidence divided by rollout cost. Separately, for the SAME total daily traffic budget, a single 10-armed concurrent multivariate test reaches a decision in about 1.8x LESS calendar time than 9 sequential pairwise A/B tests, and does so using LESS total user-sample, not more.

FDR control (Benjamini-Hochberg, executed on a simulated 81-cell scenario): I planted 6 truly real effects (small p-values) among 75 truly null cells and ran BH at target FDR q=0.10. Sorting all 81 p-values ascending and finding the LARGEST rank k where $p_{(k)} \le q \cdot k/m$ (m=81) gave a cutoff of $p \le 0.0056$ at k=5, rejecting exactly 5 cells: all 5 were true discoveries, 0 false discoveries (realized FDR = 0.00, at or under the 10% target). For contrast, an UNCORRECTED alpha=0.05 across all 81 cells would flag about 9 cells, and on the 75 truly-null cells alone would be expected to false-positive about $75 \times 0.05 = 3.75$ cells purely by chance, exactly the multiple-comparisons trap this scale of test creates. Always run BH (or a dependence-aware variant if metrics are correlated) before reporting "significant cells" from a large multi-factor grid.

Separating robust main effects from interactions: (1) require BH-adjusted significance, not raw p, before treating a cell as real; (2) check whether an apparently large interaction cell is explainable by two large main effects alone (i.e. does the observed cell mean roughly equal the additive prediction from the two main effects, or does it deviate meaningfully, the deviation is the true interaction signal); (3) where traffic allows, hold out a slice of the 81-cell data or run a small confirmatory replication on the top few BH-surviving cells before committing to a business narrative, since even a well-controlled 10% FDR still means roughly 1 in 10 flagged cells could be noise; (4) prefer hierarchical/partial-pooling shrinkage over raw per-cell means when cells have very different sample sizes, so a lucky small-N cell does not get reported at face value.

Prioritized follow-ups: rank BH-surviving cells by (estimated effect size) x (posterior confidence, e.g. 1 minus BH-adjusted p) divided by (engineering/rollout cost of shipping that cell's combination), and recommend confirmatory single-factor or targeted 2-way follow-up tests on the top 2-3 ranked cells rather than attempting to roll out all 5 simultaneously (which would itself become an uncontrolled concurrent-experiment interference problem, see S9).

Throughput comparison (executed, baseline 5%, 100k visitors/day, 80% power, alpha 0.05): at a 10% relative MDE (baseline 5% to alternative 5.5%), the two-proportion sample size is 31,233 users per arm.

• Approach A, single 10-armed concurrent test (1 shared baseline + 9 variants, all accruing simultaneously): needs 312,334 users total (31,233 per arm x 10 arms), which at 100k/day takes about 3.12 days.
• Approach B, 9 SEQUENTIAL pairwise tests (each re-drawing its own fresh baseline arm, run one after another): needs 62,467 users per test (2 x 31,233) x 9 tests = 562,202 total, which takes about 5.62 days end-to-end.

Approach A finishes about 1.8x faster AND uses about 1.8x LESS total sample (312,334 vs 562,202), because the concurrent design shares ONE baseline arm across all 9 comparisons instead of re-drawing a fresh baseline 9 times. This is the same underlying mechanism as S0's power result: shared/concurrent accrual is more efficient than sequential re-baselining, on both the speed axis and the total-sample axis.

Direct answer

Reach for a multi-armed bandit like Thompson sampling when you have several candidate arms, want to minimize traffic lost to weaker variants while you learn, and don't need a clean, pre-registered effect size for a one-time launch decision. Switch back to a fixed-horizon or sequential "always-valid" A/B test when you need a defensible causal estimate for stakeholders, want balanced data across arms for later subgroup analysis, or the reward isn't stationary enough for adaptive allocation to converge safely.

Structured elaboration

Bandits optimize for regret, not inference. Thompson sampling keeps a posterior distribution over each arm's conversion rate and samples an arm in proportion to its probability of being best, so traffic shifts toward the winner as evidence accumulates instead of staying fixed at, say, a 50/50 split for the whole test. The classic Lai and Robbins (1985) result shows a well-tuned bandit's expected cumulative regret grows only

$O(\log N)$

with traffic $N$, versus $O(N)$ for a fixed split that keeps sending a constant share of traffic to the losing arm for the entire experiment.

Fixed-horizon and sequential tests optimize for valid inference. A fixed-horizon test commits its sample size up front and analyzes once at the end. A sequential "always-valid" test (for example a mixture sequential probability ratio test) is built to let you monitor the result continuously while still controlling the true Type I error rate, the false-positive rate under the null hypothesis, which naive repeated peeking on a plain z-test does not.

The switch-back triggers: you need one clean, defensible number for a launch review; you need roughly equal data on every arm to slice by segment afterward; or the environment is non-stationary (novelty effects, a shifting user base) in a way that breaks the bandit's assumption that each arm's true reward rate holds still while it learns.

Worked example

Suppose true conversion rates are $p_A = 10\%$ and $p_B = 12\%$, and you route 100,000 users through a fixed 50/50 split. In expectation, 50,000 users see the weaker arm A, so the cost of learning relative to serving B from day one is

$50{,}000 \times (0.12 - 0.10) = 1{,}000 \text{ lost conversions}$

A bandit does not eliminate this cost, but it shifts traffic toward B well before all 100,000 users are spent, so its realized regret lands well under that 1,000-conversion bound.

Now the peeking side: if you instead ran that fixed z-test and checked significance 10 times as data trickled in, stopping the moment $p<0.05$, treating each look as an independent 5% chance of a false positive gives an illustrative upper bound of

$1-(1-0.05)^{10} \approx 1 - 0.95^{10} \approx 40\%$

versus the nominal 5%. Consecutive looks are not actually independent, they share the accumulating data and are positively correlated, so the real inflation is lower than this bound. The classic reference values for equally spaced peeks at a nominal 5% (Armitage, McPherson, and Rowe, 1969) are about 14% real false-positive rate after 5 looks and about 37% after 100 looks. Either way, unadjusted peeking silently multiplies your real false-positive rate several times past the number on the significance test. An always-valid sequential test exists specifically to let you monitor that often without this inflation.

Trade-offs and pitfalls

• Bandits starve inferior arms of traffic, so you get little data to analyze subgroup or heterogeneous effects on those arms later.
• A bandit's final estimate comes from adaptively-collected data, not a clean p-value; a launch decision often still needs a separate valid-inference step on top of it.
• Bandits assume roughly stationary reward. Novelty effects or a shifting user mix mid-test can lock allocation onto a stale winner before the environment settles.
• Sequential and always-valid tests need their monitoring boundary (an alpha-spending function or similar) designed before the test starts; bolting continuous peeking onto a plain fixed-horizon z-test is the classic mistake that produces the inflated false-positive rate shown above.

What the interviewer probes next

Expect a follow-up on how you'd notice a bandit has locked onto a stale winner, how you'd still get a defensible effect-size read after adaptive allocation, and how Thompson sampling's exploration compares to epsilon-greedy or upper-confidence-bound approaches under a tight regret budget.