Advanced Experimentation Designs Questions

Direct answer: With a 2% baseline conversion rate, 100k eligible users/day, 80% power, and alpha 0.05, and assuming a 15% relative minimum detectable effect (the question does not state one explicitly, so I am stating this assumption up front, a common convention for this kind of sizing exercise; new baseline = 2.30%), the design needs roughly 18,346 users PER CELL to power the main effects, or about 36,693 users per cell to ALSO power the interaction at the same MDE. Traffic should be split evenly, 25% into each of the 4 cells (control/control, pricing-B/layout-old, pricing-A/layout-new, pricing-B/layout-new), randomized independently on pricing and layout so every user has an equal chance of any of the 4 combinations.

Sample-size computation (executed, two-proportion z-test):

For a MAIN EFFECT comparison (pooling the 2 cells that share a pricing level against the other 2), $p_1=0.02$, $p_2=0.023$ (15% relative lift):

$$n = \frac{\left(z_{\alpha/2}\sqrt{2\bar{p}(1-\bar{p})} + z_{\beta}\sqrt{p_1(1-p_1)+p_2(1-p_2)}\right)^2}{(p_2-p_1)^2}$$

gives n = 36,693 per POOLED arm (i.e. per 2-cell group). Since each pooled arm is 2 of the 4 cells, that is 18,346 users per CELL for main-effect power, 73,386 total across all 4 cells, which at 100k/day takes about 0.73 days.

For the INTERACTION contrast (an unpooled, single-cell-vs-single-cell comparison, since the interaction estimate uses the +1/-1/-1/+1 contrast across all four individual cells rather than pooled halves), the SAME formula applied per cell (no pooling) requires 36,693 users per cell, 146,771 total, about 1.47 days at 100k/day, exactly 2.00x the main-effect total. This is the general result cited in S0: interaction detection at the same MDE costs about 2x a main effect's sample size.

Traffic allocation: randomize pricing (A vs B) and layout (old vs new) INDEPENDENTLY via two separate hash-based coin flips per user (verified orthogonal in S9), producing four roughly-equal 25% cells automatically; do not manually stratify into 4 named buckets, since independent-factor randomization already balances the design and keeps it robust to later adding a third factor.

Analysis and interpretation: fit

$$y = \beta_0 + \beta_1 \cdot \text{Pricing} + \beta_2 \cdot \text{Layout} + \beta_3 \cdot (\text{Pricing} \times \text{Layout}) + \varepsilon$$

on the binary conversion outcome (linear probability model or logistic regression with an interaction term). $\beta_1$ is the pricing main effect averaged over both layouts, $\beta_2$ the layout main effect averaged over both pricing versions, and $\beta_3$ the interaction: how much the pricing effect CHANGES when layout changes (equivalently, how much the layout effect changes when pricing changes). Run the interaction test FIRST: if $\beta_3$ is significant, report cell-specific conversion rates (all 4 cells) rather than a single "average" pricing effect, since the average would be misleading if the effect flips sign across layouts. If the interaction is not significant, report the two main effects as if from independent tests and recommend shipping whichever combination of winning main effects performs best. Present results to stakeholders as: (1) the interaction test verdict first, in plain language, (2) a 2x2 table of observed conversion rates per cell with confidence intervals, (3) the recommended cell to ship, and (4) the guardrail-metric check on that winning cell (per this topic's sibling a-b-test-design-and-statistical-rigor, which owns guardrail-metric mechanics).

Direct answer: With 5 binary features and only 8 arms available (2^3), use a quarter-fraction 2^(5-2) design with generators D=AB and E=AC. This yields the standard minimum-aberration Resolution III design for this factor count. A is relatively safe to interpret (its only alias is the four-letter word ABCDE, negligible in practice); D and E carry real risk since each is 1:1 aliased with a plausible two-factor interaction (AB and AC respectively).

Constructing the design (executed): start from the full 2^3 factorial on A, B, C (8 runs), then set D = A·B and E = A·C:

Full alias structure (derived from the defining relations $I=ABD=ACE=BCDE$, the third relation is the product of the first two, word lengths 3, 3, 4, so this is Resolution III):

• A: aliased with BD (via ABD), CE (via ACE), and the weak ABCDE (via BCDE)
• B: aliased with AD, ABCE, CDE
• C: aliased with ABCD, AE, BDE
• D: aliased with AB, ACDE, BCE
• E: aliased with ABDE, AC, BCD

Interpretation guidance: A, B, and C are each cleanly separated from any OTHER main effect and only alias with 2-factor interactions (BD/CE for A, AD for B, AE for C respectively) or weaker higher-order words; if the business has no strong prior that those specific pairwise interactions are large, A/B/C main effects can be reported with reasonable confidence. D is aliased 1:1 with AB and E with AC, both entirely plausible 2-factor interactions a stakeholder might expect (e.g. if D and E were chosen as the two "riskiest" features, expect their main-effect estimates might really be picking up an AB or AC interaction): flag D and E's point estimates as "confounded, needs a confirmatory follow-up" rather than reporting them at face value.

Generalizing to the 3-headline/2-hero-image, high-interpretability landing-page case: this is a 3x2 mixed-level design (6 cells if run as a full factorial, which is small enough to run FULL rather than fractional; fractional designs earn their keep specifically when the FULL factorial's cell count would outstrip traffic, e.g. 4+ binary factors). Since interpretability is explicitly required here and 6 cells is affordable, recommend the FULL factorial over any fraction: fractional factorial trades interpretability for run-count savings, and this scenario does not need that trade.

Comparing full factorial, fractional factorial, and sequential targeted experiments (per aa4819c0's ask, when total traffic is limited across multiple segments): full factorial gives the cleanest, alias-free estimates but its cell count grows exponentially with factor count, quickly becoming infeasible; fractional factorial trades some interpretability (confounded higher-order effects) for a linear run-count, and is the right default once a full factorial would need more cells than traffic supports; sequential targeted experiments (test the highest-uncertainty factor first, then the next) avoid aliasing entirely but pay the SPEED cost quantified in S0 (roughly half the power per unit time, or double the calendar time, versus a concurrent design at the same total budget) and cannot detect interactions between factors tested in different sequential waves. Recommendation: default to full factorial while cells stay affordable (roughly up to 3-4 binary factors depending on traffic), switch to a minimum-aberration fractional design once traffic-per-cell would otherwise drop below your power floor, and reserve pure sequential testing for genuinely independent, low-interaction-risk factors or truly traffic-starved situations.

Direct answer: Choose a factorial (or multivariate/MVT) design over a sequence of separate A/B tests when you have two or more candidate changes you plausibly want to ship TOGETHER, you suspect (or must rule out) an interaction between them, and you have enough concurrent traffic to split into cells without starving any one cell's power. Choose sequential single-factor A/B tests when changes are independent, traffic is scarce, or you need to ship and learn on the first change before the second is even designed.

Trade-offs

• Speed of learning: factorial wins when the alternative is truly SEQUENTIAL (test A, wait, then test B). For the same total sample budget, a k-factor design estimates all k main effects roughly as precisely as k separate two-arm tests run one after another with the SAME total budget split across them, but the factorial gets all k answers CONCURRENTLY instead of one-after-another, and gets the interaction essentially for free.
• Statistical power for main effects versus interactions: each main effect in a factorial is estimated by comparing half the cells against the other half (pooling over the other factor), so main-effect power is comparable to a standalone two-arm test at the same total N. The INTERACTION contrast is NOT pooled the same way: it needs the same per-cell sample size as a single unpooled cell-vs-cell comparison, so detecting an interaction of the same absolute size as a main effect costs roughly 2x the total sample of detecting that main effect alone (verified numerically in S1).
• Interpretability: a single-factor sequential test has a trivial reading ("does this metric move"). A factorial with a significant interaction requires explaining conditional effects ("A only works when B is on"), which is harder to communicate and can confuse stakeholders unless you plan for it (see S6).
• Operational complexity: factorial designs need every combination of treatments to be simultaneously deployable and measurable, more engineering coordination, more QA surface (interaction bugs, cross-feature conflicts), and a bigger simultaneous "blast radius" if something goes wrong, versus shipping and validating one change at a time.

Worked confirmation (executed simulation, not just cited): simulating 2 factors with true additive effects of size 0.12 (in outcome-SD units) each, total budget N=4000, alpha=0.05, over 4000 replications:

• FACTORIAL (4 cells of 1000 each): power(A) = 0.965, power(B) = 0.961
• SEQUENTIAL (2 separate 2-arm tests, same total budget split in half, 1000/arm each): power(A) = 0.771, power(B) = 0.759

For the identical total sample budget, the factorial design delivered materially higher power on BOTH main effects (0.96+ vs 0.77) because every unit in a factorial cell contributes to estimating EVERY factor's main effect (a unit in the (A=1,B=1) cell informs both the A comparison and the B comparison), while the sequential approach wastes half its budget on a control arm that is redundant across the two separate tests.

Business scenario where factorial is clearly preferable: a subscription product wants to test (1) a new pricing page copy and (2) a redesigned onboarding flow. Product suspects the pricing framing only lands if users have already been walked through the new onboarding (an interaction hypothesis: pricing-copy effect is conditional on onboarding version). Running these sequentially would answer "does new pricing help" and "does new onboarding help" separately but could never detect that new pricing ONLY helps when paired with new onboarding, exactly the finding the business needs to decide whether to ship them together, separately, or not at all. A 2x2 factorial answers all three questions (both main effects plus the interaction) in one concurrent test.

When NOT to use factorial: if you have 4+ factors, a FULL factorial's cell count (2^4 = 16 cells) can quickly outstrip available traffic; that is exactly the constrained-traffic case that motivates fractional factorial designs (S2, S3) rather than abandoning multi-factor testing altogether.

Direct answer: A switchback experiment assigns TIME BLOCKS (for example, whole weeks or days for an entire market or region), not individual users, to control or treatment, alternating on a schedule; it is the standard fix when individual-level randomization is invalid because treated and control units share a common resource pool (drivers, inventory, ad auction) that lets treatment "leak" into control, the SUTVA-violation problem this topic's sibling a-b-test-design-and-statistical-rigor closes for USER-level interference but does not cover for TIME-block designs.

Randomizing the schedule: do not simply alternate control-treatment-control-treatment on a fixed clock; randomize the block ORDER (e.g. randomly assign each week to control or treatment subject to a balance constraint, roughly half of each) so that any systematic weekly pattern (day-of-week mix, holiday timing) is not perfectly confounded with the treatment schedule, and randomize the STARTING phase across different markets/regions if running the switchback in parallel across several geographies, so a market-specific event does not land on the same treatment/control alignment everywhere.

Washout period and carryover: a washout period is a buffer at the START of each new block (for example, the first hour of a new week) during which data is DISCARDED from analysis, to let effects from the PRIOR block's treatment (driver repositioning, cached recommendations, habituation) dissipate before the new block's measurement window begins. Choose the washout length from domain knowledge of how fast the system's state resets (e.g. driver-supply rebalancing might take 30-60 minutes in a ride-hailing market, materially shorter than the week-long block itself). To DETECT carryover statistically: fit a model with a lagged treatment indicator (whether the PREVIOUS block was treatment) alongside the current block's treatment indicator; a significant lagged-treatment coefficient is evidence of carryover that the washout period is not fully absorbing, and the washout should be lengthened.

Controlling time trends and seasonality (executed simulation): naive difference-in-means between treatment and control blocks is BIASED whenever the treatment schedule is correlated with time, which is common in real switchback schedules (e.g. an operationally simpler "2 weeks control, 2 weeks treatment" alternating pattern, rather than fully independent per-week randomization). I simulated 40 weekly blocks on a 2-week-alternating schedule, a true treatment effect of 0.05, a linear time trend of 0.004/week, and noise SD 0.05, and repeated 3,000 times:

• naive diff-in-means: mean bias = +0.0077 (about 15% relative bias on the true 0.05 effect), RMSE = 0.0175
• regression adjusting for the linear time trend (fitting $y = \beta_0 + \beta_1 \cdot \text{week} + \beta_2 \cdot \text{treat} + \varepsilon$ and reading $\beta_2$): mean bias = -0.0003 (essentially unbiased), RMSE = 0.0158

This confirms the analysis MUST include a time-trend term (or week fixed effects, or a seasonal/day-of-week control), not a raw difference in block means, whenever the treatment schedule is not perfectly independent of time, which it rarely is in operationally realistic switchback schedules.

Sample size / number of blocks: switchback power depends on the NUMBER OF INDEPENDENT TIME BLOCKS, not the number of users within a block (users within a block are correlated through the shared time-block treatment, so they do not each contribute independent information the way users in a standard parallel A/B test do); as a rule of thumb, plan for at least 15 to 20 independent blocks per arm to get a usable estimate of the block-to-block variance needed for valid inference (analogous to needing enough clusters in a cluster-randomized trial), and use a block-level (not user-level) t-test or randomization-inference test on the block averages, whose within-block sample size mainly reduces MEASUREMENT NOISE per block rather than increasing the effective number of independent observations.

Two cases where a crossover/switchback design is inappropriate: (1) when the treatment has a PERSISTENT, long-lasting effect that does not wash out within a reasonable block length, for example a pricing change that shifts long-term customer trust or a feature that permanently changes user habits, the "carryover" here is not a short-term artifact to buffer away but the actual long-run effect you are trying to measure, and a switchback would systematically underestimate it; (2) when the unit being switched (a whole market or region) is small enough, or the treatment disruptive enough, that ALTERNATING it repeatedly causes operational instability or a bad user experience (users noticing prices or features flip back and forth week to week), which is both a product-quality risk and a threat to validity if users start changing their OWN behavior in anticipation of the alternation.

Direct answer: Structure the explanation as five short bullets/slides: (1) the question tested, in plain business language; (2) the headline result, including whether an interaction was found and what that means in plain terms; (3) 2-3 concrete business-impact scenarios tied to the interaction; (4) the uncertainty around the estimate, expressed as a range, not a p-value; (5) your recommended next action.

Worked structure, continuing the S1 pricing x layout example:

1. The question tested: "We tested a lower price point and a redesigned layout, separately and together, to see if either helps checkout conversion, and whether they work better as a pair."
2. Headline result: "The new layout alone did not move conversion. The lower price alone barely moved it. But the lower price PAIRED WITH the new layout increased conversion by [X]%, roughly [3x] more than either change alone. In plain terms: the discount only really lands when the page also feels redesigned and trustworthy; on the old layout it barely registers."
3. Business-impact scenarios: "If we ship both together: we expect roughly [$Y] in incremental monthly revenue, at the cost of a lower average order value from the discount. If we ship the layout alone: essentially no measurable revenue change, but a UX-quality win we can still justify separately. If we ship the discount alone (old layout): we lose margin with almost no conversion gain, this is the option to avoid."
4. Uncertainty: "We are confident the two changes need each other, this isn't a coin flip; the range on the combined lift is [X1]% to [X2]% with high confidence, meaning even in the more conservative case it still clears our bar to ship."
5. Recommended next action: "Ship the combined pricing-plus-layout change to 100% of traffic, and separately roll back or archive the discount-alone and layout-alone paths, since neither pays for itself independently."

Why this structure works for a non-technical audience: it leads with the DECISION-RELEVANT finding (the two changes must be shipped together, not separately) rather than the statistical mechanism (an interaction term was significant), translates "interaction" into "only works when paired with" language a business audience immediately understands, and puts uncertainty in dollar/percentage-range terms rather than a p-value or confidence-interval width that would require re-explaining what a p-value means.