Advanced Experimentation Designs Questions

Direct answer: Across 81 cells, do NOT read raw p < 0.05 at face value; control the false discovery rate (FDR) with Benjamini-Hochberg (or a variant if cells are correlated), separate robust main effects from interactions using replication and hierarchical shrinkage rather than trusting any single cell's raw estimate, and prioritize follow-ups by effect-size-times-confidence divided by rollout cost. Separately, for the SAME total daily traffic budget, a single 10-armed concurrent multivariate test reaches a decision in about 1.8x LESS calendar time than 9 sequential pairwise A/B tests, and does so using LESS total user-sample, not more.

FDR control (Benjamini-Hochberg, executed on a simulated 81-cell scenario): I planted 6 truly real effects (small p-values) among 75 truly null cells and ran BH at target FDR q=0.10. Sorting all 81 p-values ascending and finding the LARGEST rank k where $p_{(k)} \le q \cdot k/m$ (m=81) gave a cutoff of $p \le 0.0056$ at k=5, rejecting exactly 5 cells: all 5 were true discoveries, 0 false discoveries (realized FDR = 0.00, at or under the 10% target). For contrast, an UNCORRECTED alpha=0.05 across all 81 cells would flag about 9 cells, and on the 75 truly-null cells alone would be expected to false-positive about $75 \times 0.05 = 3.75$ cells purely by chance, exactly the multiple-comparisons trap this scale of test creates. Always run BH (or a dependence-aware variant if metrics are correlated) before reporting "significant cells" from a large multi-factor grid.

Separating robust main effects from interactions: (1) require BH-adjusted significance, not raw p, before treating a cell as real; (2) check whether an apparently large interaction cell is explainable by two large main effects alone (i.e. does the observed cell mean roughly equal the additive prediction from the two main effects, or does it deviate meaningfully, the deviation is the true interaction signal); (3) where traffic allows, hold out a slice of the 81-cell data or run a small confirmatory replication on the top few BH-surviving cells before committing to a business narrative, since even a well-controlled 10% FDR still means roughly 1 in 10 flagged cells could be noise; (4) prefer hierarchical/partial-pooling shrinkage over raw per-cell means when cells have very different sample sizes, so a lucky small-N cell does not get reported at face value.

Prioritized follow-ups: rank BH-surviving cells by (estimated effect size) x (posterior confidence, e.g. 1 minus BH-adjusted p) divided by (engineering/rollout cost of shipping that cell's combination), and recommend confirmatory single-factor or targeted 2-way follow-up tests on the top 2-3 ranked cells rather than attempting to roll out all 5 simultaneously (which would itself become an uncontrolled concurrent-experiment interference problem, see S9).

Throughput comparison (executed, baseline 5%, 100k visitors/day, 80% power, alpha 0.05): at a 10% relative MDE (baseline 5% to alternative 5.5%), the two-proportion sample size is 31,233 users per arm.

• Approach A, single 10-armed concurrent test (1 shared baseline + 9 variants, all accruing simultaneously): needs 312,334 users total (31,233 per arm x 10 arms), which at 100k/day takes about 3.12 days.
• Approach B, 9 SEQUENTIAL pairwise tests (each re-drawing its own fresh baseline arm, run one after another): needs 62,467 users per test (2 x 31,233) x 9 tests = 562,202 total, which takes about 5.62 days end-to-end.

Approach A finishes about 1.8x faster AND uses about 1.8x LESS total sample (312,334 vs 562,202), because the concurrent design shares ONE baseline arm across all 9 comparisons instead of re-drawing a fresh baseline 9 times. This is the same underlying mechanism as S0's power result: shared/concurrent accrual is more efficient than sequential re-baselining, on both the speed axis and the total-sample axis.

Direct answer: With only 120 eligible users worldwide, abandon a standard between-subject parallel A/B test (verified underpowered below) and instead reach for a WITHIN-SUBJECT (paired) design, a switchback/time-block design if the feature has a natural on/off toggle per user or per small cluster, a holdout ramp with close qualitative tracking, or a case-study/qualitative approach if even a paired design cannot reach adequate power. Choose a cluster-randomized or switchback design over individual-level randomization specifically when the 120 users cluster naturally into groups (teams, organizations, workspaces) where within-cluster interference is likely, e.g. developers on the same team influence each other's usage patterns, in which case individual-level randomization would violate SUTVA within the cluster the same way social-network interference does at consumer scale (this topic's sibling a-b-test-design-and-statistical-rigor owns that consumer-scale case; the tiny-population, small-cluster version is this topic's ground).

Why a standard parallel test is underpowered (executed): with N=120 total (60/arm), baseline conversion 30%, and even a generous 33% relative lift target (0.30 to 0.40, larger than most real product effects), achieved power is only about 21%. The required N for 80% power at that same generous MDE would be 712 total (356/arm), nearly 6x the entire available population. Any smaller, more realistic effect size would need even more, confirming a between-subject design simply cannot work at this scale.

Why a within-subject (paired) design helps: pairing removes BETWEEN-user variance from the comparison, since each user serves as their own control (measured before and after, or under both conditions). With within-user correlation $\rho=0.5$ (a moderate, realistic value for a within-user metric measured twice), the paired-design effective standard-deviation multiplier is $\sqrt{2(1-\rho)}=1.0$, versus $\sqrt{2}\approx1.41$ for an independent between-subject comparison, roughly HALVING the required sample size for the same power and MDE (since required N scales with the square of that multiplier, $1.0^2/1.41^2\approx0.5$). For 120 users, this can be the difference between an infeasible and a feasible test.

Practical plan:

• Metric selection: choose a metric with LOW within-user noise and short time-to-signal (e.g. task-completion rate or time-to-first-success on the developer-facing feature) rather than a slow, high-variance business metric like long-term retention, which would need far more than 120 users regardless of design.
• Instrumentation: capture per-user, per-period usage tightly enough to compute a clean pre/post or on/off comparison per user, and log enough contextual detail (which org/team each user belongs to) to check for and adjust for within-cluster correlation if the 120 users cluster into teams.
• Design choice given clustering: if the 120 users sit in, say, 15 teams of 8, prefer CLUSTER-randomization or a team-level switchback over individual randomization, since individually randomizing within a team risks contamination (control users on a team where teammates are treated may indirectly experience the feature via shared docs/workflows); if the users are genuinely independent (different companies, no interaction), individual-level within-subject randomization is fine and simpler.
• Rollout recommendation: run a holdout-ramp: enable the feature for a majority of the 120 (or all, using a within-subject pre/post design with a smaller true holdout group retained purely to distinguish the treatment effect from a general time trend), supplement the quantitative signal with structured qualitative feedback (brief surveys or interviews with a sample of the 120), since at this population size qualitative and case-study evidence carries proportionally MORE weight in the final ship/no-ship decision than it would in a consumer-scale test, and treat the quantitative estimate as directional rather than a precise point estimate given the residual power limitations even after pairing.

Direct answer: Aliasing happens when a fractional factorial design cannot separate the influence of two (or more) different effects because they change in exactly the same pattern across the runs that were actually included in the reduced design; the two effects are then statistically indistinguishable from each other using that data alone. Design resolution describes how BAD the confounding is: Resolution III means main effects are aliased with two-factor interactions (2fi's); Resolution IV means main effects are clear of 2fi's but 2fi's are aliased with each other; Resolution V means main effects and 2fi's are all clear of each other (only 2fi's alias with three-factor interactions, which are usually negligible). Higher resolution costs more runs for the same number of factors.

Deriving the alias structure for the 2^(3-1) half-fraction, generator C = AB:

The half-fraction design matrix (executed):

Since C = A·B (the defining generator), multiplying both sides by C gives $C \cdot C = A \cdot B \cdot C$, and since $C^2 = I$ (the identity column, all +1), the DEFINING RELATION is $I = ABC$.

To find what any effect is aliased with, multiply it by the defining relation and simplify using $X^2 = I$:

• $A \text{ aliased with } A \cdot ABC = A^2BC = BC$
• $B \text{ aliased with } B \cdot ABC = AB^2C = AC$
• $C \text{ aliased with } C \cdot ABC = ABC^2 = AB$

So main effect A is aliased with interaction BC precisely BECAUSE the defining relation is $I=ABC$: the "A" column in the half-fraction and the "BC" column (computed as B times C) are numerically identical (both equal +1,-1,-1,+1 down the four runs), so no linear model fit to this data alone can tell whether an observed effect on that column comes from A, from BC, or some mixture of both. This is a Resolution III design (shortest word in the defining relation, "ABC," has length 3).

De-aliasing via fold-over (executed simulation, single-factor construction): run a second block, a "fold-over on A": flip the SIGN of column A only, keep the B and C run-values exactly as they were in block 1 (do not re-derive C from the flipped A):

Checking the ABC column on this second block gives -1 on every run, so its defining relation is $I=-ABC$ (flipping one letter's sign in an odd-length word flips the word's overall sign). That sign flip is the whole trick: in block 1, A is aliased with $+BC$ (its contrast estimates $\text{effect}(A)+\text{effect}(BC)$); in the fold-over block, A is aliased with $-BC$ (its contrast estimates $\text{effect}(A)-\text{effect}(BC)$). Simulating a true response with A's true effect = 5 and BC's true effect = 3 (noise SD 0.5, averaged over 5000 replications): block 1's A-contrast converges to 8.00 (matching $5+3$), and the fold-over block's A-contrast converges to 2.00 (matching $5-3$). AVERAGING the two blocks' A-contrast estimates, $(8.00+2.00)/2=5.00$, isolates A alone (true A = 5, exact match); the HALF-DIFFERENCE, $(8.00-2.00)/2=3.00$, isolates BC alone (true BC = 3, exact match). The general mechanism: fold over on the ONE factor you need de-aliased (flip only its column, keep the others fixed), which flips that factor's alias sign but not the alias's own true effect, then the sum and difference of the two blocks' contrast estimates separate the two confounded effects cleanly.

Direct answer: With a 2% baseline conversion rate, 100k eligible users/day, 80% power, and alpha 0.05, and assuming a 15% relative minimum detectable effect (the question does not state one explicitly, so I am stating this assumption up front, a common convention for this kind of sizing exercise; new baseline = 2.30%), the design needs roughly 18,346 users PER CELL to power the main effects, or about 36,693 users per cell to ALSO power the interaction at the same MDE. Traffic should be split evenly, 25% into each of the 4 cells (control/control, pricing-B/layout-old, pricing-A/layout-new, pricing-B/layout-new), randomized independently on pricing and layout so every user has an equal chance of any of the 4 combinations.

Sample-size computation (executed, two-proportion z-test):

For a MAIN EFFECT comparison (pooling the 2 cells that share a pricing level against the other 2), $p_1=0.02$, $p_2=0.023$ (15% relative lift):

$$n = \frac{\left(z_{\alpha/2}\sqrt{2\bar{p}(1-\bar{p})} + z_{\beta}\sqrt{p_1(1-p_1)+p_2(1-p_2)}\right)^2}{(p_2-p_1)^2}$$

gives n = 36,693 per POOLED arm (i.e. per 2-cell group). Since each pooled arm is 2 of the 4 cells, that is 18,346 users per CELL for main-effect power, 73,386 total across all 4 cells, which at 100k/day takes about 0.73 days.

For the INTERACTION contrast (an unpooled, single-cell-vs-single-cell comparison, since the interaction estimate uses the +1/-1/-1/+1 contrast across all four individual cells rather than pooled halves), the SAME formula applied per cell (no pooling) requires 36,693 users per cell, 146,771 total, about 1.47 days at 100k/day, exactly 2.00x the main-effect total. This is the general result cited in S0: interaction detection at the same MDE costs about 2x a main effect's sample size.

Traffic allocation: randomize pricing (A vs B) and layout (old vs new) INDEPENDENTLY via two separate hash-based coin flips per user (verified orthogonal in S9), producing four roughly-equal 25% cells automatically; do not manually stratify into 4 named buckets, since independent-factor randomization already balances the design and keeps it robust to later adding a third factor.

Analysis and interpretation: fit

$$y = \beta_0 + \beta_1 \cdot \text{Pricing} + \beta_2 \cdot \text{Layout} + \beta_3 \cdot (\text{Pricing} \times \text{Layout}) + \varepsilon$$

on the binary conversion outcome (linear probability model or logistic regression with an interaction term). $\beta_1$ is the pricing main effect averaged over both layouts, $\beta_2$ the layout main effect averaged over both pricing versions, and $\beta_3$ the interaction: how much the pricing effect CHANGES when layout changes (equivalently, how much the layout effect changes when pricing changes). Run the interaction test FIRST: if $\beta_3$ is significant, report cell-specific conversion rates (all 4 cells) rather than a single "average" pricing effect, since the average would be misleading if the effect flips sign across layouts. If the interaction is not significant, report the two main effects as if from independent tests and recommend shipping whichever combination of winning main effects performs best. Present results to stakeholders as: (1) the interaction test verdict first, in plain language, (2) a 2x2 table of observed conversion rates per cell with confidence intervals, (3) the recommended cell to ship, and (4) the guardrail-metric check on that winning cell (per this topic's sibling a-b-test-design-and-statistical-rigor, which owns guardrail-metric mechanics).

Direct answer: With 5 binary features and only 8 arms available (2^3), use a quarter-fraction 2^(5-2) design with generators D=AB and E=AC. This yields the standard minimum-aberration Resolution III design for this factor count. A is relatively safe to interpret (its only alias is the four-letter word ABCDE, negligible in practice); D and E carry real risk since each is 1:1 aliased with a plausible two-factor interaction (AB and AC respectively).

Constructing the design (executed): start from the full 2^3 factorial on A, B, C (8 runs), then set D = A·B and E = A·C:

Full alias structure (derived from the defining relations $I=ABD=ACE=BCDE$, the third relation is the product of the first two, word lengths 3, 3, 4, so this is Resolution III):

• A: aliased with BD (via ABD), CE (via ACE), and the weak ABCDE (via BCDE)
• B: aliased with AD, ABCE, CDE
• C: aliased with ABCD, AE, BDE
• D: aliased with AB, ACDE, BCE
• E: aliased with ABDE, AC, BCD

Interpretation guidance: A, B, and C are each cleanly separated from any OTHER main effect and only alias with 2-factor interactions (BD/CE for A, AD for B, AE for C respectively) or weaker higher-order words; if the business has no strong prior that those specific pairwise interactions are large, A/B/C main effects can be reported with reasonable confidence. D is aliased 1:1 with AB and E with AC, both entirely plausible 2-factor interactions a stakeholder might expect (e.g. if D and E were chosen as the two "riskiest" features, expect their main-effect estimates might really be picking up an AB or AC interaction): flag D and E's point estimates as "confounded, needs a confirmatory follow-up" rather than reporting them at face value.

Generalizing to the 3-headline/2-hero-image, high-interpretability landing-page case: this is a 3x2 mixed-level design (6 cells if run as a full factorial, which is small enough to run FULL rather than fractional; fractional designs earn their keep specifically when the FULL factorial's cell count would outstrip traffic, e.g. 4+ binary factors). Since interpretability is explicitly required here and 6 cells is affordable, recommend the FULL factorial over any fraction: fractional factorial trades interpretability for run-count savings, and this scenario does not need that trade.

Comparing full factorial, fractional factorial, and sequential targeted experiments (per aa4819c0's ask, when total traffic is limited across multiple segments): full factorial gives the cleanest, alias-free estimates but its cell count grows exponentially with factor count, quickly becoming infeasible; fractional factorial trades some interpretability (confounded higher-order effects) for a linear run-count, and is the right default once a full factorial would need more cells than traffic supports; sequential targeted experiments (test the highest-uncertainty factor first, then the next) avoid aliasing entirely but pay the SPEED cost quantified in S0 (roughly half the power per unit time, or double the calendar time, versus a concurrent design at the same total budget) and cannot detect interactions between factors tested in different sequential waves. Recommendation: default to full factorial while cells stay affordable (roughly up to 3-4 binary factors depending on traffic), switch to a minimum-aberration fractional design once traffic-per-cell would otherwise drop below your power floor, and reserve pure sequential testing for genuinely independent, low-interaction-risk factors or truly traffic-starved situations.