Experimentation and Validation Questions
Deciding what to test and how to learn it cheaply before committing product investment: identifying the riskiest assumption, writing a falsifiable hypothesis, choosing between a prototype, fake door, concierge test, pilot, beta or live experiment, sequencing validation steps and budgeting effort across assumptions, setting go/no-go criteria and rollback triggers in advance, and combining qualitative and quantitative evidence, including when they disagree. Covers pilot design (participants, comparison group, guardrails, threats to validity), offline checks of a model idea before small online trials, limited trials of vendors or architectures, and validating demand under thin data or privacy limits. Assesses hypothesis-driven product development and evidence-based decisions, including validation of technical and operational bets. Excludes the statistical design of a single online test, interpreting finished experiment results, experimentation platform engineering, user research methods, ordering tests across a program, and non-randomized causal inference.
You need to decide between a monolith and a modular-service architecture but want to limit customer risk. How would you validate the choice with a limited trial before committing, and what evidence would settle it?
Sample Answer
The idea in plain words
A monolith is one program that is built, deployed and run as a single unit. A modular-service architecture splits it into separate programs that call each other over the network. Each shape is a bet on specific benefits (a bet here means a choice that only pays off if a stated benefit actually appears): independent deploys, independent scaling, fault isolation, team autonomy. Each also has costs: network calls add delay and failure points, and there is more to operate. Instead of arguing the choice in general, name the benefit you are betting on, split out one piece, serve a small share of real traffic from it with an instant way back, and compare agreed measures before and after. The evidence that settles it is delivery speed, reliability, latency and cost for that piece, measured against the monolith's own baseline.
Step 1: write the bet as a testable statement
For example: "Extracting the notifications module into its own service lets its team deploy at least twice as often, without a meaningful rise in the change failure rate, adding more than 20 ms at p99 (the time within which 99 percent of requests finish), or raising run cost by more than 30 percent. Here a meaningful rise means more than 2 points." All of these numbers are illustrative thresholds you would set with the team. If no one can name the benefit, that is itself the answer: stay with the monolith, perhaps organised into well-separated modules inside it (a modular monolith), which gives much of the structure at a fraction of the operating cost.
Step 2: choose the slice for the trial
Pick a part that has a narrow, well-defined interface (few callers), real traffic (so the numbers mean something), an owning team, and a modest blast radius if it fails (blast radius is how much of the product, and how many customers, are hurt when this one part breaks). Notifications, search indexing or report generation often fit; the payment path usually does not. The slice must own its data. If the new service still reads and writes the monolith's database tables, the boundary is fake and the trial hides the coupling it was meant to expose.
Step 3: limit customer risk while testing
- Contract tests (automated checks that the monolith and the service still agree on what each may send and expect back) that pin the interface between them.
- Shadow (dark) launch: the monolith still answers customers, while the same requests are also sent to the service and the answers and latency compared. Run until the mismatches are explained. The copy must have no customer-visible side effects: for a module that sends messages, charges cards or writes records customers can see, point the service at a stub that records what it would have done (or at internal test recipients), otherwise customers receive every message twice.
- Feature flag with instant rollback: a switch that sends traffic back to the code inside the monolith within minutes. Keep that code in place for the whole trial.
- Ramp by cohort (a cohort is a group of users moved over together): internal users first, then a small share of low-risk customers (for example 5 percent), then widen. Avoid customers with strict service agreements (written promises about uptime or response time) until the later stages.
- Time limit and stop rule fixed before the trial: a customer-visible incident traced to the boundary pauses the ramp, and the on-call engineer (whoever is responsible for responding to alerts out of hours) can trigger the rollback.
Step 4: the evidence that settles it
Compare the trial period (assume 8 weeks) with the same module's recent past:
| Measure | What it tells you | Illustrative pass mark |
|---|---|---|
| Deploys per week by the owning team | Independence | At least twice the baseline |
| Change failure rate (share of deploys that cause a rollback or incident) | Whether speed came with breakage | Not worse than baseline plus 2 points |
| p99 latency added by the new hop (one extra network call from one program to another) | Cost of the network | Within 20 ms |
| Availability of the affected feature (the share of time it works for users) | Cost of the extra dependency | Meets the feature's target (worked example follows) |
| Run cost (infrastructure plus on-call time) | Operating overhead | Within 30 percent of baseline |
| Incidents traced to the boundary | Hidden coupling | None that are customer-visible |
These marks are an illustration of the form. The bet written in Step 1 decides which measures lead: here deploys per week shows whether the benefit arrived, and availability and p99 latency show the price customers would feel; change failure rate, run cost and boundary incidents check for hidden costs. A real trial also covers security review, data consistency, the team's own view of the extra cognitive load (the mental effort of keeping several services and their interactions in your head) and who may declare the trial over.
Worked example: why availability and latency are measured, not assumed. Suppose the monolith meets 99.9 percent availability and the new service also 99.9 percent, with independent failures (one failing does not make the other more likely to fail), and every request needs both. Availability multiplies: 0.999×0.999=0.998001, about 99.80 percent. In a 30-day month (43,200 minutes) that allows (1−0.998001)×43,200≈86.4 minutes of downtime instead of 0.001×43,200=43.2 minutes. Timeouts, fallbacks (serve a degraded result if the service is slow) and retries are how a team claws that back, and the trial tests whether they work. For latency, assume 8 ms at p99 per call and two calls per request, in sequence: a rough estimate of 2×8=16 ms added, inside the 20 ms mark. (Percentiles do not add exactly. For well-behaved latencies the 99th percentile of the total is a little below the sum of the two, because the slowest 1 percent of calls rarely land on the same request. For heavy-tailed latencies, where an occasional call is very slow, the total can even exceed the sum, so the trial measures the added p99 end to end instead of relying on the sum.)
Caution about small samples. Assume the team deploys 1 a week before the trial, which is 8 deployments in 8 weeks, and 2 a week during it, which is 16. Suppose the baseline had 1 failed deploy in 8 (12.5 percent) and the trial has 1 in 16 (6.25 percent). The trial looks half as failure-prone, yet the difference is a single event, and one more failure in the trial would move its rate by 6.25 points, more than three times the 2-point margin in the pass mark (6.25/2≈3.1). So at this volume the rate cannot meet or miss that mark in any reliable way. Do not decide on the rate alone; read each failure and ask whether the boundary caused it.
What I would decide
My default is to stay a modular monolith and make the boundary real inside it first, which is cheap and reversible. Then run the trial for one slice where a concrete benefit is expected. Commit to the full split only if the pass marks are met and the team can describe the next two services it would extract and why. The answer flips toward services if one component needs very different scaling from the rest, if a compliance boundary requires isolation, or if teams are blocked by each other's release schedules. It flips toward the monolith if the added hop breaks the latency mark or the team spends its time on cross-service debugging.
Common mistakes
- Splitting the code but not the data.
- Choosing the most critical module for the trial, so a failure is expensive.
- Comparing against an imagined baseline instead of the module's recent history.
- Agreeing the pass marks after seeing results.
- Counting a handful of deploys or incidents as a rate.
Your team is debating whether to A/B test a variant that might lift conversion but adds UI complexity, or simply ship one polished experience. How do you decide whether this change deserves a test at all, and who do you involve?
Sample Answer
The idea in plain words
An A/B test (also called a controlled experiment) randomly shows one version to some visitors and another to the rest, then compares an outcome such as conversion (the share of visitors who complete the goal action). It is worth running when three things hold: you are genuinely unsure which version is better, being wrong costs something real, and your traffic can answer the question in a time you can wait. If any of those fails, a test is the wrong tool and you should decide another way. So the question to answer is not "can we test it?" but "what would we do differently with the answer, and is it worth the time and traffic to get it?"
The decision in five questions, asked in this order
- Is the outcome uncertain? If the team agrees the change is obviously good (a bug fix, a legal requirement, an accessibility repair), ship it. If the team is split, that disagreement is itself evidence that a test may pay.
- What does being wrong cost, and can it be undone? A reversible change with a small downside can simply ship with monitoring. A change that adds lasting UI complexity is costly to remove later, so a measured result is worth more here.
- How big an effect would justify the complexity? Convert the complexity into a break-even lift (worked example below).
- Can the traffic detect that effect in an acceptable time? Compute it before deciding (below).
- Is the outcome measurable in the test window? Effects on retention, trust or brand may take months to appear; a two-week conversion test would miss them.
Worked example
Assumed inputs, with plausible ranges: 40,000 eligible visitors a week (range 10,000 to 200,000); baseline conversion 5% (range 2% to 10%); 50 dollars of value per conversion; the variant's ongoing complexity cost (extra code to maintain, slower future changes) 150,000 dollars a year (range 50,000 to 300,000).
Break-even lift. The complexity cost per week is 150,000 / 52 = 2,885 dollars. That needs 2,885 / 50 = 57.7 extra conversions a week, which is 57.7 / 40,000 = 0.144 percentage points (pp), or 0.144 / 5.0 = 2.9% relative to the 5% baseline. Note the two conventions: an absolute lift is the difference in rates (5.0% to 5.2% is 0.2 pp); a relative lift divides by the baseline (0.2 / 5.0 = 4%).
Sample needed. For 80% power (the chance of detecting a real effect of the stated size) at the usual 5% two-sided significance level (when there is truly no effect, the test wrongly reports one, in either direction, only 5% of the time), a standard approximation for the visitors needed per arm is n≈16p(1−p)/d2, where p is the baseline rate and d the absolute difference to detect. The 16 comes from the fuller formula n=2(zα/2+zβ)2p(1−p)/d2, with zα/2=1.96 for the 5% two-sided level and zβ=0.84 for 80% power: 2 x (1.96 + 0.84)^2 = 15.7, which rounds to 16. "Per arm" counts one group (control or variant), so the total is twice that, and the weeks column is the total divided by 40,000 visitors a week.
| Effect to detect | Absolute difference d | Visitors per arm | Total visitors | Weeks at 40,000 a week |
|---|---|---|---|---|
| 4% relative | 0.002 | 16 x 0.05 x 0.95 / 0.002^2 = 190,000 | 380,000 | 9.5 |
| 10% relative | 0.005 | 16 x 0.05 x 0.95 / 0.005^2 = 30,400 | 60,800 | 1.52 |
Reading it. The break-even lift is 2.9% relative, but a test that could reliably detect a lift that small would run more than 9 weeks. If the design team credibly expects a 10% relative lift, the test takes about two weeks, which is a cheap price for avoiding a permanent complexity cost on a guess. If the most anyone expects is a few percent, a test cannot resolve it in a reasonable time, and the choice is between shipping on judgment or not building the variant.
Recommendation for the stated situation
Test it, if the proposed variant could plausibly produce a lift of about 10% relative or more: run it for two full weeks (full weeks, so weekday and weekend behaviour are both covered), with conversion as the primary metric (the single number the ship decision rests on) and a few guardrails (numbers that must not get worse even if the primary metric improves) set in advance (for example refund rate, page speed and support contacts). If the realistic lift is only a few percent, do not run a long, weak test. Ship the polished single experience, roll it out in stages with a small holdout group (a slice of users who keep the old experience for comparison), and decide on the variant only if a stronger version of the idea appears.
Who to involve, and why
| Person | What they contribute |
|---|---|
| Product manager (the decision owner) | States the decision the result will drive and the break-even lift |
| Designer or user researcher | Says what lift the design can plausibly deliver and whether a prototype test would settle usability first |
| Engineer | Estimates the lasting complexity cost, instruments the metric (adds the tracking so the outcome is recorded reliably), and checks the variant can be switched off cleanly |
| Data analyst or scientist | Checks baseline, traffic, test duration and guardrails before launch |
| Support, legal or finance (when the change touches pricing, claims or billing) | Sets limits on what can be shown to whom |
Agree the ship rule with these people before launch, so no one reopens it after the result.
Common mistakes
- Testing everything by reflex, including changes that are obvious or tiny.
- Running a test too small to detect the effect that matters, then calling a flat result "no effect".
- Ignoring the complexity cost, which a conversion lift alone does not measure.
- Choosing the metric and the ship rule after seeing results.
- Forgetting that a test measures the test window, not long-term behaviour.
Your A/B test result and your customer interview findings point in different directions on a go/no-go decision. How do you combine the two kinds of evidence, when would you run each, and how do you resolve the disagreement?
Sample Answer
The idea in plain words
An A/B test shows two versions to randomly split groups and measures what people did. A customer interview asks a small number of people what they thought and why. When they disagree, the useful question is not "which one wins?" but "what is each one actually measuring?" The test measures behaviour, on average, over a window of days. Interviews explain reasons and expose segments and risks, but cover a handful of people, each reporting on themselves. A disagreement usually means they are measuring different things (a different time horizon, a different group, behaviour versus stated feeling), and the job is to find which gap explains it and run the cheapest check that separates the explanations.
What each source can and cannot settle
| Question | A/B test | Interviews |
|---|---|---|
| Did the metric move, and by how much? | Yes, with an interval | No, too few people and no control group |
| Why did it move? | Only by inference | Yes, in the person's own words |
| Who is helped or hurt? | If segments were planned and have enough data | Yes, for the few people met, but not their share of users |
| Will it last? | Only for the length of the test | Hints at it (annoyance, confusion) but does not measure it |
| Is the metric the right one? | No | Yes, this is where "we measured the wrong thing" surfaces |
When to run each
Interviews come first, to find the problem and the right metric before the test. The test comes second, to measure size on that metric. Interviews come third, aimed at the surprises, with participants picked from the test's own data (for example people who bought and people who did not, in each arm), because a random chat after the fact explains little. A go/no-go usually needs all three; the failure mode is running only the one that is cheapest.
How to resolve the disagreement, in order
- Check each source is valid. For the test: was the traffic split as planned (a mismatch between planned and observed split, called sample ratio mismatch, suggests broken assignment), did logging work, was the test long enough to cover weekly cycles? For the interviews: who was recruited, were questions leading, and how many said the thing compared with how many would by chance?
- Write down what each source measures. For example "the test says first purchases rose; the interviews say the page felt pushy". These can both be true.
- Turn the interview claim into a prediction the data can check. If the page felt pushy and users resent it, then they should buy again less often. That is checkable in logs already collected.
- Slice by the people you interviewed. Check the same metric for those segments before arguing about averages.
- If the data cannot separate the explanations, run one cheap targeted follow-up (a longer holdout, a version without the contested element, a short survey to a sample from each arm), and say in advance what each outcome would mean.
- Decide with a stated rule, and record which evidence would reverse it.
Worked example: a checkout page with a countdown timer
All numbers are assumptions chosen for the example, not benchmarks.
The test. 40,000 users per arm, randomized by user. First-purchase rate: control 4,000 / 40,000 = 10.0%; treatment 4,240 / 40,000 = 10.6%. The difference is 10.6%−10.0%=0.6 points absolute, or 0.6/10.0=6.0% relative. The standard error (SE) is the typical size of the random wobble in a measured difference from one sample to the next. For a difference in two proportions, the unpooled version (each arm uses its own rate, rather than one blended rate) is
SE=np1(1−p1)+np2(1−p2)=400000.1×0.9+400000.106×0.894≈0.00215The 95% interval is 0.006±1.96×0.00215, which is 0.18 to 1.02 points. The 1.96 is the bell-curve cut-off that leaves 2.5% in each tail, so "estimate plus or minus 1.96 SE" is a 95% interval. In decision terms: the lift is probably real, and plausibly anywhere from small to about one point.
The interviews. 12 treatment-arm users were interviewed, and 8 of the 12 called the countdown timer "pushy". Before reading this as a finding, compare it with chance. If opinion were split evenly, the probability of 8 or more of 12 on one named side is the sum of the binomial terms 495 + 220 + 66 + 12 + 1 = 794 out of 4,096 possible outcomes, which is 0.194. Where the counts come from: if each of the 12 people were a fair coin flip between "pushy" and "not pushy", there are 212=4096 equally likely patterns of answers. The number of patterns with exactly k people saying "pushy" is "12 choose k" =12!/(k!(12−k)!), which is 495 for k=8, 220 for 9, 66 for 10, 12 for 11 and 1 for 12. Adding the five counts gives the 794 patterns with 8 or more. A result that happens about one time in five by luck is a reason to look, not a measurement. It is a hypothesis.
Turn the hypothesis into a check. If pushiness hurts later behaviour, repeat purchasing should suffer. Using the same assigned users (all 40,000 per arm, not only the buyers), assume 30-day repeat purchasers are 1,200 in each arm: 1,200 / 40,000 = 3.00% in both. The 95% interval for the difference is 0±1.96×2×0.03×0.97/40000=0±0.0024, so -0.24 to +0.24 points.
Note the base. Among buyers only, repeat rates are 1,200 / 4,000 = 30.0% (control) and 1,200 / 4,240 = 28.3% (treatment), which looks like treatment buyers repeat less. That comparison is not a fair test, because the treatment arm turned extra people into buyers and those people may be less committed. Comparing everyone who was assigned keeps the randomization intact.
Reading the two together. The timer added first purchases (+0.6 points, which is 240 extra first-time buyers per arm) and the number of repeat purchasers did not move (0.0 points, interval -0.24 to +0.24). Two stories fit that. If the 240 extra buyers were as loyal as the existing buyers, who repeat at 30%, repeat purchasers would rise by about 72, which is 72 / 40,000 = 0.18 points, and that sits inside the interval. If the timer wins less loyal buyers, or irritates some people who would have come back anyway, the rise would be smaller or the count would stay flat or fall. The data cannot separate the two, so the interviews' concern stays a live hypothesis that the data neither confirms nor rules out, and the flat repeat result is the reason to run a better-powered follow-up, not a verdict.
What I would decide
Hold the full rollout and iterate. Ship a version without the countdown (or with a softer one) to a test against the current control, with the 30-day repeat rate per assigned user as a pre-registered guardrail (a protective metric whose tolerance is written down and agreed before the data arrives, so it cannot be chosen after the result), and a rule written beforehand: ship if first purchases rise with the interval above zero and the repeat rate's lower end is no worse than -0.2 points. At 40,000 users per arm the repeat-rate interval is about plus or minus 0.24 points on each side, wider than the -0.2 margin, so the margin cannot be read at that size. The walk-through: the interval's half-width is 1.96×2p(1−p)/n and for it to shrink to 0.002 (0.2 points) with p=0.03, assuming both arms share a 3% rate, n=2×1.962×0.03×0.97/0.0022=2×3.8416×0.0291/0.000004≈55,895, about 55,900 users per arm. That size only makes the interval as narrow as the margin. If the true difference is zero, the measured lower end lands above -0.2 only about half the time at that size (the estimate is as likely to fall below zero as above it), so a harmless version would fail the rule half the time. To pass a harmless version with 80% probability the size must also include power: n=2(1.96+0.8416)2×0.0291/0.0022≈114,200 users per arm, about 2.9 times the first test's 40,000. Plan the sample from that second figure. The cost is time: the first-purchase read arrives within the enrolment period, but the 30-day repeat read needs 30 days of follow-up after the last user enters. What would flip it: if the business model is one-off purchases, the repeat metric matters less and the existing +0.6 points could justify shipping with the interviews noted as brand risk.
The reverse conflict: the test is flat, interviews are enthusiastic
Common explanations in the order to check them. First, dilution (the lift is spread thin because most treated users never meet the change): if only 25% of treatment users ever reach the new feature, a 0.6 point overall lift corresponds to about 0.6/0.25=2.4 points among those who reach it, assuming no effect on the rest. Check exposure rates and compare reachers with a matching group. Second, stated versus revealed preference (what people say they like versus what their actions show they choose): people say they like things they do not use. Third, the metric: the feature may help in ways the primary metric does not capture. A flat result with a wide interval is inconclusive, not proof of no effect.
Common mistakes
- Averaging the two ("half-way ship") or deferring to the louder source.
- Treating 8 of 12 as a majority of customers.
- Comparing only buyers across arms after the treatment changed who buys.
- Running the follow-up without saying in advance what result would change the decision.
You want to pilot a new enterprise feature with 10 customers out of a base of 200. How would you choose them, what terms would let you roll back, and how would you judge whether the pilot result will hold for the other 190?
Sample Answer
The idea in plain words
A pilot with 10 of 200 customers is a small, controlled trial. It answers two different questions that are easy to blur: "does this work and what breaks?" and "will the result hold for everyone else?". Ten accounts can answer the first well and the second only roughly. So the plan has three jobs: choose the ten so that they resemble the rest rather than flatter the feature, write terms that let either side stop cleanly, and treat the pilot as the first wave of a staged rollout rather than as proof.
1. How to choose the 10
Set eligibility first. Require that the account uses the relevant part of the product, is on a current version, can switch the feature on technically, and has a contact willing to give weekly feedback. Exclude accounts in an active escalation or a renewal negotiation in the next 60 days (assumption), because a failure there costs real revenue and muddies the signal. Write down these exclusions: every one is a way the pilot differs from the other 190.
Stratify, then draw at random inside each group. A stratum is a subgroup that you expect to behave differently, here by account size. Picking the friendliest customers by hand gives you champions, who are enthusiastic and forgiving, and inflates the result (selection bias). Instead split the eligible accounts into strata and draw from each at random.
Assume the 200 accounts are 100 small, 70 mid-size and 30 large. A proportional split of 10 would be 10 x 100/200 = 5 small, 10 x 70/200 = 3.5 mid-size and 10 x 30/200 = 1.5 large. For an enterprise feature the large accounts are where the risk and revenue sit, so I would deliberately oversample them: 4 small, 3 mid-size, 3 large. Oversampling is fine as long as you weight the results back to the real mix (shown below). The reason is that the raw pilot rate describes a pilot in which 30% of accounts are large (3 of 10), while only 15% of the real base is large (30 of 200). If large accounts behave differently, the raw rate is tilted toward them; weighting restores the real proportions.
Invite more than you need. Not everyone accepts, so invite about 16 (assumption) to land 10. Record who declines and why. Decliners are data about the 190 who would not volunteer.
2. Terms that allow a rollback
A written pilot agreement, in plain language, with these terms:
- Time-boxed. For example 8 weeks (assumption), extendable only by mutual consent. No obligation on either side to continue.
- Either side can stop. Short notice (for example 5 business days) with no penalty to the customer.
- Pilot status is explicit. The feature can change or be withdrawn, support is best-effort rather than the standard service commitment, and the customer's existing workflows must keep working.
- Price. Free or discounted for the pilot, with no price promised for the general release unless stated.
- Data. What the feature stores, and how the customer's data is exported or deleted on exit.
Technical side, so that the legal right to roll back is real:
- A per-account feature flag (a switch that turns the feature on or off for one account without a new release).
- Changes that only add things (new tables or settings) and never overwrite existing data, so switching off restores the previous behaviour.
- A rehearsal before launch: switch the feature off on a test account and confirm its old workflow works.
- Named triggers for rolling back an account: a data integrity or security problem, a serious incident, the customer asking to stop, or pilot support load beyond a set limit.
3. Will the result hold for the other 190?
Treat this as an evidence question and check five things.
-
Compare the ten with the rest on measurable traits (size, industry, tenure, current usage, integrations). Big gaps tell you which segments the pilot says little about.
-
Check for pilot-only advantages. Pilot accounts usually get direct access to the team and hand-holding. Measure it, then scale it. If pilot accounts took 6 support hours each per month (assumption), the other 190 would need 190 x 6 = 1,140 hours a month. At 160 working hours per person-month (assumption), 1,140 / 160 = 7.1, about 7 people. If standard onboarding is meant to cost about 1 hour per account per month (assumption), the need would be 190 / 160 = 1.2 people. The pilot result only transfers if the feature can be adopted without the white-glove help.
-
Weight the result back to the real mix. Suppose, as an illustration, 3 of 4 small accounts, 2 of 3 mid-size and 1 of 3 large reach the target usage by week 6. The raw rate is 6 / 10 = 60%. Weighting each stratum by its true share (0.50, 0.35, 0.15):
textWeighted adoption = 0.50 x (3/4) + 0.35 x (2/3) + 0.15 x (1/3) = 0.375 + 0.2333 + 0.05 = 0.658 (about 66%)Each pilot account carries a weight equal to its stratum's true share divided by its share of the pilot: a large account stands for 0.15 / 0.30 = 0.5 of an account's worth of the real mix (it is over-represented), while a small account stands for 0.50 / 0.40 = 1.25. The weighted figure is higher than the raw one because the pilot oversampled the large accounts, which adopted least. Weighting corrects the mix but not the noise: each stratum has only 3 or 4 accounts.
-
Respect the sample size. An exact 95% confidence interval for 6 successes out of 10 runs from about 26% to 88% (the Clopper-Pearson method). "Exact" means it is built from the binomial counts directly rather than a bell-curve approximation. The lower end is the true adoption rate at which seeing 6 or more adopters out of 10 would happen only 2.5% of the time; the upper end is the rate at which seeing 6 or fewer would happen only 2.5% of the time. Rates below 26% or above 88% would rarely produce a result like ours. Ten accounts cannot separate a feature that 30% of customers would adopt from one that 85% would. A confidence interval is a range of true rates consistent with the data under that method; it is not the probability that the true rate lies inside it. The program in section 4 below solves those two tail conditions, so you can reproduce the interval.
-
Prefer behaviour over opinion. Use usage logs (did they use it in weeks 4 to 6?) and commitments (a signed intent to buy at the target price), not just friendly feedback.
4. Decide in advance what the pilot has to show
Because 10 accounts are few, use the pilot as a screen and a learning tool, with bars written beforehand (all assumptions): at least 6 of 10 reach target usage by week 6, no severe incident, and at least 3 sign an intent to purchase at the target price.
A small sample is useful for detecting failure. If the true adoption rate were 50%, every one of the 1,024 (that is, 2 to the power 10) possible success-or-failure patterns across 10 accounts would be equally likely. Exactly 0 successes can happen in 1 way, exactly 1 success in 10 ways (any one of the ten accounts), and exactly 2 successes in 45 ways (the number of pairs among ten accounts, 10 x 9 / 2). So seeing 2 or fewer of 10 would have probability (1 + 10 + 45) / 1024 = 56 / 1024 = 0.055, about 5.5%, so such a result would be fair grounds to stop or redesign. A pass is different: it justifies the next wave, not a launch. The program below checks the 5.5%, the 26% to 88% interval and the weighting:
from math import comb
from scipy import stats
from scipy.optimize import brentq
# (a) the stop-rule probability: 2 or fewer successes out of 10 when the true rate is 50%
ways = [comb(10, k) for k in (0, 1, 2)]
print("ways to get 0, 1, 2 successes:", ways, "| all sequences:", 2**10,
"| probability:", round(sum(ways) / 2**10, 4))
# (b) exact 95% interval for 6 successes out of 10
k, n = 6, 10
lo = brentq(lambda p: stats.binom.sf(k - 1, n, p) - 0.025, 1e-9, 1 - 1e-9) # P(6 or more) = 2.5%
hi = brentq(lambda p: stats.binom.cdf(k, n, p) - 0.025, 1e-9, 1 - 1e-9) # P(6 or fewer) = 2.5%
print("interval by solving the two tail conditions:", round(lo, 3), "to", round(hi, 3))
print("scipy exact interval:", [round(x, 3) for x in stats.binomtest(k, n).proportion_ci(0.95, method="exact")])
# (c) weighting back to the real mix
true_share = {"small": 100 / 200, "mid": 70 / 200, "large": 30 / 200}
pilot_n = {"small": 4, "mid": 3, "large": 3}
adopted = {"small": 3, "mid": 2, "large": 1}
for s in true_share:
print(s, "weight per pilot account:", round(true_share[s] / (pilot_n[s] / 10), 2))
print("raw rate:", sum(adopted.values()) / 10,
"| weighted rate:", round(sum(true_share[s] * adopted[s] / pilot_n[s] for s in true_share), 3))
Output:
ways to get 0, 1, 2 successes: [1, 10, 45] | all sequences: 1024 | probability: 0.0547
interval by solving the two tail conditions: 0.262 to 0.878
scipy exact interval: [0.262, 0.878]
small weight per pilot account: 1.25
mid weight per pilot account: 1.17
large weight per pilot account: 0.5
raw rate: 0.6 | weighted rate: 0.658
The next wave is the real test of the 190. Draw about 30 more accounts at random from the remaining 190 (assumption), use standard onboarding with no special support, and compare their results with the pilot's. If they match, widen further (for example two waves of about 80 each). If they fall well short, the pilot's extra attention was part of the result.
Common mistakes
- Choosing the ten by who is easiest to work with.
- Ignoring who declined.
- Reading a 6-of-10 result as "60% of customers will adopt".
- Letting the pilot team's extra effort go unmeasured.
- Writing a contract that lets customers leave but gives no technical way to switch the feature off cleanly.
You hold a risky design assumption and have little engineering time. Walk me through the lightest experiment you would run to gain confidence: what you would build, who you would show it to, and the rule you would use to continue or stop.
Sample Answer
The idea in plain words
When engineering time is scarce, do not build the product to test the design. Build the cheapest thing that lets a real person behave as if the design were true, and watch what they do. The belief you are testing is your riskiest assumption: the belief that, if wrong, sinks the design and that you have the least evidence for. The lightest experiments fake the expensive part. A Wizard of Oz test (a person quietly does by hand what the finished software would do) or a concierge test (you openly deliver the service by hand to a few customers) both need almost no engineering. The continue-or-stop rule is written before anyone is shown anything, as a count of observed behaviours, so the result cannot be talked into whatever the team hoped.
Step 1: turn the design risk into one testable statement
Take the one assumption you would least like to be wrong about and state it so that a person's behaviour can contradict it. "Team leads will like an auto-ranked task list" cannot fail. "At least 7 of 10 team leads will plan their week from the order we give them, without re-sorting it" can.
Check three things before you build anything:
- Is it behaviour, not opinion? What people do (re-sorted the list or did not) beats what they say they would do.
- Does it isolate the risky part? Test only the assumption that could kill the design. Login, polish and edge cases are not part of the test.
- Could it come out the wrong way? If every possible result leads to "continue", it is a demo, not an experiment.
Step 2: choose what to build (the lightest thing that still produces real behaviour)
| Option | What you build | Engineering time | Good for |
|---|---|---|---|
| Clickable mock-up | Linked screens in a design tool | None | "Do people understand it and find the next step?" |
| Wizard of Oz | Real-looking screen, a person produces the output behind it | Almost none | "Do people act on output of this kind and quality?" |
| Concierge | You deliver the outcome by hand, openly | None | "Is the outcome valuable enough to come back for?" |
| Fake door | A button or page for a feature that does not exist yet; count clicks | A few hours | "Does anyone reach for this at all?" |
For the example below the risk is whether people trust the machine-made output, so a Wizard of Oz test fits: a spreadsheet the product manager fills in by hand, shared as if a tool had produced it.
Step 3: choose who sees it
Pick people who have the problem now and are not your friends: 10 current team leads who run weekly planning, recruited from real customers or internal users outside your own team. Avoid people who owe you a favour, because they will be polite. Ten is a small number on purpose; it is enough to see a large effect and too few to prove a small one, and the rule below is built around that.
Step 4: write the rule before the first session
Example inputs (assumptions for illustration, not data): 10 team leads; the design only pays off if most leads act on the ranked order. State three outcomes in advance:
- Continue if 7 or more of 10 plan from the given order without re-sorting.
- Stop or redesign the risky part if 3 or fewer do.
- Iterate once if 4 to 6 do: change the one thing leads said they distrusted, and run a second round with 10 different leads.
What this rule can and cannot tell you
A 10-person count is noisy, so check how the rule behaves under two assumed "true" acceptance rates, using the binomial distribution, which gives the chance of exactly k successes in 10 independent tries when each try has the same success probability. The 30% and 70% rows are two assumed cases: one a clearly weak design, one a clearly good one, chosen as mirror images around 50%, which is why the two rows are the same numbers swapped (the chance of 7 or more at 70% equals the chance of 3 or fewer at 30%, because counting successes at 70% is counting failures at 30%). Here is one term by hand. The chance of exactly 7 of 10 at a 30% true rate is the number of ways to pick which 7 accept, (710)=120, times the chance of any one such pattern, 0.37×0.73. That is 120×0.0002187×0.343≈0.0090. Add the terms for 8, 9 and 10 and "7 or more" at 30% is about 0.0106. The program below computes all the table cells.
from math import comb
from scipy.stats import binom
n = 10
total = 0.0
for k in (7, 8, 9, 10):
term = comb(n, k) * 0.3**k * 0.7**(n - k)
total += term
print(f"exactly {k} of 10 at 30%: C(10,{k}) = {comb(n, k)}, probability {term:.6f}")
print(f"7 or more at 30%: {total:.4f}")
for p in (0.3, 0.7):
stop = binom.cdf(3, n, p)
cont = binom.sf(6, n, p)
print(f"true rate {p}: continue {cont:.4f}, stop {stop:.4f}, iterate {1 - cont - stop:.4f}")
Output:
exactly 7 of 10 at 30%: C(10,7) = 120, probability 0.009002
exactly 8 of 10 at 30%: C(10,8) = 45, probability 0.001447
exactly 9 of 10 at 30%: C(10,9) = 10, probability 0.000138
exactly 10 of 10 at 30%: C(10,10) = 1, probability 0.000006
7 or more at 30%: 0.0106
true rate 0.3: continue 0.0106, stop 0.6496, iterate 0.3398
true rate 0.7: continue 0.6496, stop 0.0106, iterate 0.3398
The table, from those sums:
| True share of leads who would accept the order | Chance the test says "continue" (7 or more) | Chance it says "stop" (3 or fewer) | Chance it says "iterate" (4 to 6) |
|---|---|---|---|
| 30% (a design not worth building) | 0.0106 | 0.6496 | 0.3398 |
| 70% (a design worth building) | 0.6496 | 0.0106 | 0.3398 |
Read it this way. If the design were really a 30% design, the rule would wrongly say "continue" about 1 time in 100, so a pass is real evidence. But if the design were a genuine 70% design, the rule would say "continue" only about 65 times in 100 and would send the rest to a second round. A small test with a strict bar is good at catching a bad idea and slow at confirming a good one, which is the right way round when engineering time is the scarce thing.
Also record why the people who re-sorted did so. One lead who says "I did not trust the third item" is a design fix; five leads re-sorting for five different reasons is a stop.
The judgment
Run the Wizard of Oz test in one week, with the product manager filling the spreadsheet by hand. Spend no engineering hours on this round: the counts come from watching sessions. If the count reaches 7 of 10, ask engineering for the thinnest real build of the ranking, which is now the justified cost. If it lands at 4 to 6, spend the second round on the single most-cited objection. If it is 3 or fewer, stop and record what you learned so the design is not rediscovered later.
What would change this plan: if the risk is technical ("can the ranking be computed at this quality at all?") then a human cannot fake it and a short engineering spike with a pass bar on a sample of real data is the lightest test instead. If you can reach only 4 people, the count rule is too coarse and you should use the sessions to find why people act as they do rather than to count.
Common mistakes
- Testing opinions ("would you use this?") instead of observing a choice.
- Building the whole flow when only one step is risky.
- Choosing the pass bar after seeing the results.
- Recruiting colleagues or fans, then reading their politeness as demand.
- Treating "iterate" as a free pass: write down in advance what must change and how many rounds you allow (here, one).
Unlock Full Question Bank
Get access to all 31 Experimentation and Validation interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.