Advanced Experimentation Designs Questions

Direct answer: Across 81 cells, do NOT read raw p < 0.05 at face value; control the false discovery rate (FDR) with Benjamini-Hochberg (or a variant if cells are correlated), separate robust main effects from interactions using replication and hierarchical shrinkage rather than trusting any single cell's raw estimate, and prioritize follow-ups by effect-size-times-confidence divided by rollout cost. Separately, for the SAME total daily traffic budget, a single 10-armed concurrent multivariate test reaches a decision in about 1.8x LESS calendar time than 9 sequential pairwise A/B tests, and does so using LESS total user-sample, not more.

FDR control (Benjamini-Hochberg, executed on a simulated 81-cell scenario): I planted 6 truly real effects (small p-values) among 75 truly null cells and ran BH at target FDR q=0.10. Sorting all 81 p-values ascending and finding the LARGEST rank k where $p_{(k)} \le q \cdot k/m$ (m=81) gave a cutoff of $p \le 0.0056$ at k=5, rejecting exactly 5 cells: all 5 were true discoveries, 0 false discoveries (realized FDR = 0.00, at or under the 10% target). For contrast, an UNCORRECTED alpha=0.05 across all 81 cells would flag about 9 cells, and on the 75 truly-null cells alone would be expected to false-positive about $75 \times 0.05 = 3.75$ cells purely by chance, exactly the multiple-comparisons trap this scale of test creates. Always run BH (or a dependence-aware variant if metrics are correlated) before reporting "significant cells" from a large multi-factor grid.

Separating robust main effects from interactions: (1) require BH-adjusted significance, not raw p, before treating a cell as real; (2) check whether an apparently large interaction cell is explainable by two large main effects alone (i.e. does the observed cell mean roughly equal the additive prediction from the two main effects, or does it deviate meaningfully, the deviation is the true interaction signal); (3) where traffic allows, hold out a slice of the 81-cell data or run a small confirmatory replication on the top few BH-surviving cells before committing to a business narrative, since even a well-controlled 10% FDR still means roughly 1 in 10 flagged cells could be noise; (4) prefer hierarchical/partial-pooling shrinkage over raw per-cell means when cells have very different sample sizes, so a lucky small-N cell does not get reported at face value.

Prioritized follow-ups: rank BH-surviving cells by (estimated effect size) x (posterior confidence, e.g. 1 minus BH-adjusted p) divided by (engineering/rollout cost of shipping that cell's combination), and recommend confirmatory single-factor or targeted 2-way follow-up tests on the top 2-3 ranked cells rather than attempting to roll out all 5 simultaneously (which would itself become an uncontrolled concurrent-experiment interference problem, see S9).

Throughput comparison (executed, baseline 5%, 100k visitors/day, 80% power, alpha 0.05): at a 10% relative MDE (baseline 5% to alternative 5.5%), the two-proportion sample size is 31,233 users per arm.

• Approach A, single 10-armed concurrent test (1 shared baseline + 9 variants, all accruing simultaneously): needs 312,334 users total (31,233 per arm x 10 arms), which at 100k/day takes about 3.12 days.
• Approach B, 9 SEQUENTIAL pairwise tests (each re-drawing its own fresh baseline arm, run one after another): needs 62,467 users per test (2 x 31,233) x 9 tests = 562,202 total, which takes about 5.62 days end-to-end.

Approach A finishes about 1.8x faster AND uses about 1.8x LESS total sample (312,334 vs 562,202), because the concurrent design shares ONE baseline arm across all 9 comparisons instead of re-drawing a fresh baseline 9 times. This is the same underlying mechanism as S0's power result: shared/concurrent accrual is more efficient than sequential re-baselining, on both the speed axis and the total-sample axis.

Direct answer: With only 120 eligible users worldwide, abandon a standard between-subject parallel A/B test (verified underpowered below) and instead reach for a WITHIN-SUBJECT (paired) design, a switchback/time-block design if the feature has a natural on/off toggle per user or per small cluster, a holdout ramp with close qualitative tracking, or a case-study/qualitative approach if even a paired design cannot reach adequate power. Choose a cluster-randomized or switchback design over individual-level randomization specifically when the 120 users cluster naturally into groups (teams, organizations, workspaces) where within-cluster interference is likely, e.g. developers on the same team influence each other's usage patterns, in which case individual-level randomization would violate SUTVA within the cluster the same way social-network interference does at consumer scale (this topic's sibling a-b-test-design-and-statistical-rigor owns that consumer-scale case; the tiny-population, small-cluster version is this topic's ground).

Why a standard parallel test is underpowered (executed): with N=120 total (60/arm), baseline conversion 30%, and even a generous 33% relative lift target (0.30 to 0.40, larger than most real product effects), achieved power is only about 21%. The required N for 80% power at that same generous MDE would be 712 total (356/arm), nearly 6x the entire available population. Any smaller, more realistic effect size would need even more, confirming a between-subject design simply cannot work at this scale.

Why a within-subject (paired) design helps: pairing removes BETWEEN-user variance from the comparison, since each user serves as their own control (measured before and after, or under both conditions). With within-user correlation $\rho=0.5$ (a moderate, realistic value for a within-user metric measured twice), the paired-design effective standard-deviation multiplier is $\sqrt{2(1-\rho)}=1.0$, versus $\sqrt{2}\approx1.41$ for an independent between-subject comparison, roughly HALVING the required sample size for the same power and MDE (since required N scales with the square of that multiplier, $1.0^2/1.41^2\approx0.5$). For 120 users, this can be the difference between an infeasible and a feasible test.

Practical plan:

• Metric selection: choose a metric with LOW within-user noise and short time-to-signal (e.g. task-completion rate or time-to-first-success on the developer-facing feature) rather than a slow, high-variance business metric like long-term retention, which would need far more than 120 users regardless of design.
• Instrumentation: capture per-user, per-period usage tightly enough to compute a clean pre/post or on/off comparison per user, and log enough contextual detail (which org/team each user belongs to) to check for and adjust for within-cluster correlation if the 120 users cluster into teams.
• Design choice given clustering: if the 120 users sit in, say, 15 teams of 8, prefer CLUSTER-randomization or a team-level switchback over individual randomization, since individually randomizing within a team risks contamination (control users on a team where teammates are treated may indirectly experience the feature via shared docs/workflows); if the users are genuinely independent (different companies, no interaction), individual-level within-subject randomization is fine and simpler.
• Rollout recommendation: run a holdout-ramp: enable the feature for a majority of the 120 (or all, using a within-subject pre/post design with a smaller true holdout group retained purely to distinguish the treatment effect from a general time trend), supplement the quantitative signal with structured qualitative feedback (brief surveys or interviews with a sample of the 120), since at this population size qualitative and case-study evidence carries proportionally MORE weight in the final ship/no-ship decision than it would in a consumer-scale test, and treat the quantitative estimate as directional rather than a precise point estimate given the residual power limitations even after pairing.

Direct answer: Formulate this as: minimize the variance of the interaction-effect estimator, $\text{Var}(\hat{\theta}_{AB}) = \sum_{i=1}^{4} \sigma_i^2/n_i$ (the four-cell +1/-1/-1/+1 contrast estimator's variance, assuming independent cells with per-cell variance $\sigma_i^2$), subject to a total-cost budget constraint $\sum_i c_i n_i = B$, where $n_i$ (the sample size in cell i) are the decision variables, $c_i$ the known per-user cost or complexity weight of cell i, and B the fixed budget.

Deriving the allocation rule: using a Lagrangian, $L = \sum_i \sigma_i^2/n_i + \lambda\left(\sum_i c_i n_i - B\right)$, setting $\partial L/\partial n_i = -\sigma_i^2/n_i^2 + \lambda c_i = 0$ gives $n_i \propto \sigma_i/\sqrt{c_i}$: allocate MORE sample to cells that are noisier (higher $\sigma_i$) and LESS to cells that are more expensive (higher $c_i$), the classic Neyman-allocation rule generalized to cost-weighted variance minimization.

Worked numeric example (executed): suppose all four cells have the same outcome variance ($\sigma_i=0.14$ for all i, e.g. a conversion-rate-like metric), but cell 4 (both treatments simultaneously on) is 5x costlier per user, for instance because it requires manual QA/concierge support, so $c = [1, 1, 1, 5]$, budget B = 400,000 cost-units.

• EQUAL allocation (n=50,000 in every cell, spending the budget evenly): $\text{Var}(\hat{\theta}_{AB}) = 1.568 \times 10^{-6}$.
• NEYMAN-OPTIMAL allocation ($n_i \propto \sigma_i/\sqrt{c_i}$): n* = [76,393, 76,393, 76,393, 34,164] (budget check: $\sum c_i n_i^* = 400{,}000$, exact), giving $\text{Var}(\hat{\theta}_{AB}) = 1.343 \times 10^{-6}$.

That is a 14.3% variance reduction versus equal allocation for the SAME total cost budget, purely from shifting sample away from the expensive cell 4 toward the three equally-noisy, cheaper cells.

Practical simplifications: (1) if per-cell variances are genuinely similar (the common case for conversion-rate-like metrics with similar baseline rates), the allocation collapses to pure cost-weighting, $n_i \propto 1/\sqrt{c_i}$, which is easy to compute without needing pilot-data variance estimates; (2) if BOTH cost and variance vary substantially and in the SAME direction (e.g. the noisier cell is also the pricier one), the two effects partially cancel and the gain from optimizing over equal allocation shrinks, in a version of this example with cell 4 at both 3x cost AND roughly 2x the SD, the improvement dropped to about 1%, so it is worth checking numerically rather than assuming Neyman allocation always helps a lot; (3) in practice, run this as a one-time pre-launch calculation using pilot-period variance and known engineering-cost estimates, not as a live-adjusting allocation, since re-allocating traffic mid-experiment reintroduces the kind of assignment-pipeline risk the sibling topic's A/A-test validation content is designed to catch.

Direct answer: Aliasing happens when a fractional factorial design cannot separate the influence of two (or more) different effects because they change in exactly the same pattern across the runs that were actually included in the reduced design; the two effects are then statistically indistinguishable from each other using that data alone. Design resolution describes how BAD the confounding is: Resolution III means main effects are aliased with two-factor interactions (2fi's); Resolution IV means main effects are clear of 2fi's but 2fi's are aliased with each other; Resolution V means main effects and 2fi's are all clear of each other (only 2fi's alias with three-factor interactions, which are usually negligible). Higher resolution costs more runs for the same number of factors.

Deriving the alias structure for the 2^(3-1) half-fraction, generator C = AB:

The half-fraction design matrix (executed):

Since C = A·B (the defining generator), multiplying both sides by C gives $C \cdot C = A \cdot B \cdot C$, and since $C^2 = I$ (the identity column, all +1), the DEFINING RELATION is $I = ABC$.

To find what any effect is aliased with, multiply it by the defining relation and simplify using $X^2 = I$:

• $A \text{ aliased with } A \cdot ABC = A^2BC = BC$
• $B \text{ aliased with } B \cdot ABC = AB^2C = AC$
• $C \text{ aliased with } C \cdot ABC = ABC^2 = AB$

So main effect A is aliased with interaction BC precisely BECAUSE the defining relation is $I=ABC$: the "A" column in the half-fraction and the "BC" column (computed as B times C) are numerically identical (both equal +1,-1,-1,+1 down the four runs), so no linear model fit to this data alone can tell whether an observed effect on that column comes from A, from BC, or some mixture of both. This is a Resolution III design (shortest word in the defining relation, "ABC," has length 3).

De-aliasing via fold-over (executed simulation, single-factor construction): run a second block, a "fold-over on A": flip the SIGN of column A only, keep the B and C run-values exactly as they were in block 1 (do not re-derive C from the flipped A):

Checking the ABC column on this second block gives -1 on every run, so its defining relation is $I=-ABC$ (flipping one letter's sign in an odd-length word flips the word's overall sign). That sign flip is the whole trick: in block 1, A is aliased with $+BC$ (its contrast estimates $\text{effect}(A)+\text{effect}(BC)$); in the fold-over block, A is aliased with $-BC$ (its contrast estimates $\text{effect}(A)-\text{effect}(BC)$). Simulating a true response with A's true effect = 5 and BC's true effect = 3 (noise SD 0.5, averaged over 5000 replications): block 1's A-contrast converges to 8.00 (matching $5+3$), and the fold-over block's A-contrast converges to 2.00 (matching $5-3$). AVERAGING the two blocks' A-contrast estimates, $(8.00+2.00)/2=5.00$, isolates A alone (true A = 5, exact match); the HALF-DIFFERENCE, $(8.00-2.00)/2=3.00$, isolates BC alone (true BC = 3, exact match). The general mechanism: fold over on the ONE factor you need de-aliased (flip only its column, keep the others fixed), which flips that factor's alias sign but not the alias's own true effect, then the sum and difference of the two blocks' contrast estimates separate the two confounded effects cleanly.

Direct answer

Reach for a multi-armed bandit like Thompson sampling when you have several candidate arms, want to minimize traffic lost to weaker variants while you learn, and don't need a clean, pre-registered effect size for a one-time launch decision. Switch back to a fixed-horizon or sequential "always-valid" A/B test when you need a defensible causal estimate for stakeholders, want balanced data across arms for later subgroup analysis, or the reward isn't stationary enough for adaptive allocation to converge safely.

Structured elaboration

Bandits optimize for regret, not inference. Thompson sampling keeps a posterior distribution over each arm's conversion rate and samples an arm in proportion to its probability of being best, so traffic shifts toward the winner as evidence accumulates instead of staying fixed at, say, a 50/50 split for the whole test. The classic Lai and Robbins (1985) result shows a well-tuned bandit's expected cumulative regret grows only

$O(\log N)$

with traffic $N$, versus $O(N)$ for a fixed split that keeps sending a constant share of traffic to the losing arm for the entire experiment.

Fixed-horizon and sequential tests optimize for valid inference. A fixed-horizon test commits its sample size up front and analyzes once at the end. A sequential "always-valid" test (for example a mixture sequential probability ratio test) is built to let you monitor the result continuously while still controlling the true Type I error rate, the false-positive rate under the null hypothesis, which naive repeated peeking on a plain z-test does not.

The switch-back triggers: you need one clean, defensible number for a launch review; you need roughly equal data on every arm to slice by segment afterward; or the environment is non-stationary (novelty effects, a shifting user base) in a way that breaks the bandit's assumption that each arm's true reward rate holds still while it learns.

Worked example

Suppose true conversion rates are $p_A = 10\%$ and $p_B = 12\%$, and you route 100,000 users through a fixed 50/50 split. In expectation, 50,000 users see the weaker arm A, so the cost of learning relative to serving B from day one is

$50{,}000 \times (0.12 - 0.10) = 1{,}000 \text{ lost conversions}$

A bandit does not eliminate this cost, but it shifts traffic toward B well before all 100,000 users are spent, so its realized regret lands well under that 1,000-conversion bound.

Now the peeking side: if you instead ran that fixed z-test and checked significance 10 times as data trickled in, stopping the moment $p<0.05$, treating each look as an independent 5% chance of a false positive gives an illustrative upper bound of

$1-(1-0.05)^{10} \approx 1 - 0.95^{10} \approx 40\%$

versus the nominal 5%. Consecutive looks are not actually independent, they share the accumulating data and are positively correlated, so the real inflation is lower than this bound. The classic reference values for equally spaced peeks at a nominal 5% (Armitage, McPherson, and Rowe, 1969) are about 14% real false-positive rate after 5 looks and about 37% after 100 looks. Either way, unadjusted peeking silently multiplies your real false-positive rate several times past the number on the significance test. An always-valid sequential test exists specifically to let you monitor that often without this inflation.

Trade-offs and pitfalls

• Bandits starve inferior arms of traffic, so you get little data to analyze subgroup or heterogeneous effects on those arms later.
• A bandit's final estimate comes from adaptively-collected data, not a clean p-value; a launch decision often still needs a separate valid-inference step on top of it.
• Bandits assume roughly stationary reward. Novelty effects or a shifting user mix mid-test can lock allocation onto a stale winner before the environment settles.
• Sequential and always-valid tests need their monitoring boundary (an alpha-spending function or similar) designed before the test starts; bolting continuous peeking onto a plain fixed-horizon z-test is the classic mistake that produces the inflated false-positive rate shown above.

What the interviewer probes next

Expect a follow-up on how you'd notice a bandit has locked onto a stale winner, how you'd still get a defensible effect-size read after adaptive allocation, and how Thompson sampling's exploration compares to epsilon-greedy or upper-confidence-bound approaches under a tight regret budget.