Advanced Experimentation Designs Questions

Direct answer: Choose a factorial (or multivariate/MVT) design over a sequence of separate A/B tests when you have two or more candidate changes you plausibly want to ship TOGETHER, you suspect (or must rule out) an interaction between them, and you have enough concurrent traffic to split into cells without starving any one cell's power. Choose sequential single-factor A/B tests when changes are independent, traffic is scarce, or you need to ship and learn on the first change before the second is even designed.

Trade-offs

• Speed of learning: factorial wins when the alternative is truly SEQUENTIAL (test A, wait, then test B). For the same total sample budget, a k-factor design estimates all k main effects roughly as precisely as k separate two-arm tests run one after another with the SAME total budget split across them, but the factorial gets all k answers CONCURRENTLY instead of one-after-another, and gets the interaction essentially for free.
• Statistical power for main effects versus interactions: each main effect in a factorial is estimated by comparing half the cells against the other half (pooling over the other factor), so main-effect power is comparable to a standalone two-arm test at the same total N. The INTERACTION contrast is NOT pooled the same way: it needs the same per-cell sample size as a single unpooled cell-vs-cell comparison, so detecting an interaction of the same absolute size as a main effect costs roughly 2x the total sample of detecting that main effect alone (verified numerically in S1).
• Interpretability: a single-factor sequential test has a trivial reading ("does this metric move"). A factorial with a significant interaction requires explaining conditional effects ("A only works when B is on"), which is harder to communicate and can confuse stakeholders unless you plan for it (see S6).
• Operational complexity: factorial designs need every combination of treatments to be simultaneously deployable and measurable, more engineering coordination, more QA surface (interaction bugs, cross-feature conflicts), and a bigger simultaneous "blast radius" if something goes wrong, versus shipping and validating one change at a time.

Worked confirmation (executed simulation, not just cited): simulating 2 factors with true additive effects of size 0.12 (in outcome-SD units) each, total budget N=4000, alpha=0.05, over 4000 replications:

• FACTORIAL (4 cells of 1000 each): power(A) = 0.965, power(B) = 0.961
• SEQUENTIAL (2 separate 2-arm tests, same total budget split in half, 1000/arm each): power(A) = 0.771, power(B) = 0.759

For the identical total sample budget, the factorial design delivered materially higher power on BOTH main effects (0.96+ vs 0.77) because every unit in a factorial cell contributes to estimating EVERY factor's main effect (a unit in the (A=1,B=1) cell informs both the A comparison and the B comparison), while the sequential approach wastes half its budget on a control arm that is redundant across the two separate tests.

Business scenario where factorial is clearly preferable: a subscription product wants to test (1) a new pricing page copy and (2) a redesigned onboarding flow. Product suspects the pricing framing only lands if users have already been walked through the new onboarding (an interaction hypothesis: pricing-copy effect is conditional on onboarding version). Running these sequentially would answer "does new pricing help" and "does new onboarding help" separately but could never detect that new pricing ONLY helps when paired with new onboarding, exactly the finding the business needs to decide whether to ship them together, separately, or not at all. A 2x2 factorial answers all three questions (both main effects plus the interaction) in one concurrent test.

When NOT to use factorial: if you have 4+ factors, a FULL factorial's cell count (2^4 = 16 cells) can quickly outstrip available traffic; that is exactly the constrained-traffic case that motivates fractional factorial designs (S2, S3) rather than abandoning multi-factor testing altogether.

Direct answer: With only 120 eligible users worldwide, abandon a standard between-subject parallel A/B test (verified underpowered below) and instead reach for a WITHIN-SUBJECT (paired) design, a switchback/time-block design if the feature has a natural on/off toggle per user or per small cluster, a holdout ramp with close qualitative tracking, or a case-study/qualitative approach if even a paired design cannot reach adequate power. Choose a cluster-randomized or switchback design over individual-level randomization specifically when the 120 users cluster naturally into groups (teams, organizations, workspaces) where within-cluster interference is likely, e.g. developers on the same team influence each other's usage patterns, in which case individual-level randomization would violate SUTVA within the cluster the same way social-network interference does at consumer scale (this topic's sibling a-b-test-design-and-statistical-rigor owns that consumer-scale case; the tiny-population, small-cluster version is this topic's ground).

Why a standard parallel test is underpowered (executed): with N=120 total (60/arm), baseline conversion 30%, and even a generous 33% relative lift target (0.30 to 0.40, larger than most real product effects), achieved power is only about 21%. The required N for 80% power at that same generous MDE would be 712 total (356/arm), nearly 6x the entire available population. Any smaller, more realistic effect size would need even more, confirming a between-subject design simply cannot work at this scale.

Why a within-subject (paired) design helps: pairing removes BETWEEN-user variance from the comparison, since each user serves as their own control (measured before and after, or under both conditions). With within-user correlation $\rho=0.5$ (a moderate, realistic value for a within-user metric measured twice), the paired-design effective standard-deviation multiplier is $\sqrt{2(1-\rho)}=1.0$, versus $\sqrt{2}\approx1.41$ for an independent between-subject comparison, roughly HALVING the required sample size for the same power and MDE (since required N scales with the square of that multiplier, $1.0^2/1.41^2\approx0.5$). For 120 users, this can be the difference between an infeasible and a feasible test.

Practical plan:

• Metric selection: choose a metric with LOW within-user noise and short time-to-signal (e.g. task-completion rate or time-to-first-success on the developer-facing feature) rather than a slow, high-variance business metric like long-term retention, which would need far more than 120 users regardless of design.
• Instrumentation: capture per-user, per-period usage tightly enough to compute a clean pre/post or on/off comparison per user, and log enough contextual detail (which org/team each user belongs to) to check for and adjust for within-cluster correlation if the 120 users cluster into teams.
• Design choice given clustering: if the 120 users sit in, say, 15 teams of 8, prefer CLUSTER-randomization or a team-level switchback over individual randomization, since individually randomizing within a team risks contamination (control users on a team where teammates are treated may indirectly experience the feature via shared docs/workflows); if the users are genuinely independent (different companies, no interaction), individual-level within-subject randomization is fine and simpler.
• Rollout recommendation: run a holdout-ramp: enable the feature for a majority of the 120 (or all, using a within-subject pre/post design with a smaller true holdout group retained purely to distinguish the treatment effect from a general time trend), supplement the quantitative signal with structured qualitative feedback (brief surveys or interviews with a sample of the 120), since at this population size qualitative and case-study evidence carries proportionally MORE weight in the final ship/no-ship decision than it would in a consumer-scale test, and treat the quantitative estimate as directional rather than a precise point estimate given the residual power limitations even after pairing.

Direct answer

Reach for a multi-armed bandit like Thompson sampling when you have several candidate arms, want to minimize traffic lost to weaker variants while you learn, and don't need a clean, pre-registered effect size for a one-time launch decision. Switch back to a fixed-horizon or sequential "always-valid" A/B test when you need a defensible causal estimate for stakeholders, want balanced data across arms for later subgroup analysis, or the reward isn't stationary enough for adaptive allocation to converge safely.

Structured elaboration

Bandits optimize for regret, not inference. Thompson sampling keeps a posterior distribution over each arm's conversion rate and samples an arm in proportion to its probability of being best, so traffic shifts toward the winner as evidence accumulates instead of staying fixed at, say, a 50/50 split for the whole test. The classic Lai and Robbins (1985) result shows a well-tuned bandit's expected cumulative regret grows only

$O(\log N)$

with traffic $N$, versus $O(N)$ for a fixed split that keeps sending a constant share of traffic to the losing arm for the entire experiment.

Fixed-horizon and sequential tests optimize for valid inference. A fixed-horizon test commits its sample size up front and analyzes once at the end. A sequential "always-valid" test (for example a mixture sequential probability ratio test) is built to let you monitor the result continuously while still controlling the true Type I error rate, the false-positive rate under the null hypothesis, which naive repeated peeking on a plain z-test does not.

The switch-back triggers: you need one clean, defensible number for a launch review; you need roughly equal data on every arm to slice by segment afterward; or the environment is non-stationary (novelty effects, a shifting user base) in a way that breaks the bandit's assumption that each arm's true reward rate holds still while it learns.

Worked example

Suppose true conversion rates are $p_A = 10\%$ and $p_B = 12\%$, and you route 100,000 users through a fixed 50/50 split. In expectation, 50,000 users see the weaker arm A, so the cost of learning relative to serving B from day one is

$50{,}000 \times (0.12 - 0.10) = 1{,}000 \text{ lost conversions}$

A bandit does not eliminate this cost, but it shifts traffic toward B well before all 100,000 users are spent, so its realized regret lands well under that 1,000-conversion bound.

Now the peeking side: if you instead ran that fixed z-test and checked significance 10 times as data trickled in, stopping the moment $p<0.05$, treating each look as an independent 5% chance of a false positive gives an illustrative upper bound of

$1-(1-0.05)^{10} \approx 1 - 0.95^{10} \approx 40\%$

versus the nominal 5%. Consecutive looks are not actually independent, they share the accumulating data and are positively correlated, so the real inflation is lower than this bound. The classic reference values for equally spaced peeks at a nominal 5% (Armitage, McPherson, and Rowe, 1969) are about 14% real false-positive rate after 5 looks and about 37% after 100 looks. Either way, unadjusted peeking silently multiplies your real false-positive rate several times past the number on the significance test. An always-valid sequential test exists specifically to let you monitor that often without this inflation.

Trade-offs and pitfalls

• Bandits starve inferior arms of traffic, so you get little data to analyze subgroup or heterogeneous effects on those arms later.
• A bandit's final estimate comes from adaptively-collected data, not a clean p-value; a launch decision often still needs a separate valid-inference step on top of it.
• Bandits assume roughly stationary reward. Novelty effects or a shifting user mix mid-test can lock allocation onto a stale winner before the environment settles.
• Sequential and always-valid tests need their monitoring boundary (an alpha-spending function or similar) designed before the test starts; bolting continuous peeking onto a plain fixed-horizon z-test is the classic mistake that produces the inflated false-positive rate shown above.

What the interviewer probes next

Expect a follow-up on how you'd notice a bandit has locked onto a stale winner, how you'd still get a defensible effect-size read after adaptive allocation, and how Thompson sampling's exploration compares to epsilon-greedy or upper-confidence-bound approaches under a tight regret budget.

Direct answer: Structure the explanation as five short bullets/slides: (1) the question tested, in plain business language; (2) the headline result, including whether an interaction was found and what that means in plain terms; (3) 2-3 concrete business-impact scenarios tied to the interaction; (4) the uncertainty around the estimate, expressed as a range, not a p-value; (5) your recommended next action.

Worked structure, continuing the S1 pricing x layout example:

1. The question tested: "We tested a lower price point and a redesigned layout, separately and together, to see if either helps checkout conversion, and whether they work better as a pair."
2. Headline result: "The new layout alone did not move conversion. The lower price alone barely moved it. But the lower price PAIRED WITH the new layout increased conversion by [X]%, roughly [3x] more than either change alone. In plain terms: the discount only really lands when the page also feels redesigned and trustworthy; on the old layout it barely registers."
3. Business-impact scenarios: "If we ship both together: we expect roughly [$Y] in incremental monthly revenue, at the cost of a lower average order value from the discount. If we ship the layout alone: essentially no measurable revenue change, but a UX-quality win we can still justify separately. If we ship the discount alone (old layout): we lose margin with almost no conversion gain, this is the option to avoid."
4. Uncertainty: "We are confident the two changes need each other, this isn't a coin flip; the range on the combined lift is [X1]% to [X2]% with high confidence, meaning even in the more conservative case it still clears our bar to ship."
5. Recommended next action: "Ship the combined pricing-plus-layout change to 100% of traffic, and separately roll back or archive the discount-alone and layout-alone paths, since neither pays for itself independently."

Why this structure works for a non-technical audience: it leads with the DECISION-RELEVANT finding (the two changes must be shipped together, not separately) rather than the statistical mechanism (an interaction term was significant), translates "interaction" into "only works when paired with" language a business audience immediately understands, and puts uncertainty in dollar/percentage-range terms rather than a p-value or confidence-interval width that would require re-explaining what a p-value means.

Direct answer: Aliasing happens when a fractional factorial design cannot separate the influence of two (or more) different effects because they change in exactly the same pattern across the runs that were actually included in the reduced design; the two effects are then statistically indistinguishable from each other using that data alone. Design resolution describes how BAD the confounding is: Resolution III means main effects are aliased with two-factor interactions (2fi's); Resolution IV means main effects are clear of 2fi's but 2fi's are aliased with each other; Resolution V means main effects and 2fi's are all clear of each other (only 2fi's alias with three-factor interactions, which are usually negligible). Higher resolution costs more runs for the same number of factors.

Deriving the alias structure for the 2^(3-1) half-fraction, generator C = AB:

The half-fraction design matrix (executed):

Since C = A·B (the defining generator), multiplying both sides by C gives $C \cdot C = A \cdot B \cdot C$, and since $C^2 = I$ (the identity column, all +1), the DEFINING RELATION is $I = ABC$.

To find what any effect is aliased with, multiply it by the defining relation and simplify using $X^2 = I$:

• $A \text{ aliased with } A \cdot ABC = A^2BC = BC$
• $B \text{ aliased with } B \cdot ABC = AB^2C = AC$
• $C \text{ aliased with } C \cdot ABC = ABC^2 = AB$

So main effect A is aliased with interaction BC precisely BECAUSE the defining relation is $I=ABC$: the "A" column in the half-fraction and the "BC" column (computed as B times C) are numerically identical (both equal +1,-1,-1,+1 down the four runs), so no linear model fit to this data alone can tell whether an observed effect on that column comes from A, from BC, or some mixture of both. This is a Resolution III design (shortest word in the defining relation, "ABC," has length 3).

De-aliasing via fold-over (executed simulation, single-factor construction): run a second block, a "fold-over on A": flip the SIGN of column A only, keep the B and C run-values exactly as they were in block 1 (do not re-derive C from the flipped A):

Checking the ABC column on this second block gives -1 on every run, so its defining relation is $I=-ABC$ (flipping one letter's sign in an odd-length word flips the word's overall sign). That sign flip is the whole trick: in block 1, A is aliased with $+BC$ (its contrast estimates $\text{effect}(A)+\text{effect}(BC)$); in the fold-over block, A is aliased with $-BC$ (its contrast estimates $\text{effect}(A)-\text{effect}(BC)$). Simulating a true response with A's true effect = 5 and BC's true effect = 3 (noise SD 0.5, averaged over 5000 replications): block 1's A-contrast converges to 8.00 (matching $5+3$), and the fold-over block's A-contrast converges to 2.00 (matching $5-3$). AVERAGING the two blocks' A-contrast estimates, $(8.00+2.00)/2=5.00$, isolates A alone (true A = 5, exact match); the HALF-DIFFERENCE, $(8.00-2.00)/2=3.00$, isolates BC alone (true BC = 3, exact match). The general mechanism: fold over on the ONE factor you need de-aliased (flip only its column, keep the others fixed), which flips that factor's alias sign but not the alias's own true effect, then the sum and difference of the two blocks' contrast estimates separate the two confounded effects cleanly.