Google Product Manager (Junior Level 1-2 Years) Interview Preparation Guide

Proposal: Fund 6-week external bootcamp for 5 product & design hires

Objective

• Rapidly upskill five new product & design hires to reduce ramp time, improve discovery discipline, and increase cross-functional delivery quality within 3 months.

Expected outcomes

• Standardized discovery/process language across PMs and designers
• Faster delivery of customer-validated features
• Higher usability and fewer rework cycles

Specific KPIs (with baselines & targets)

• Ramp time to full productivity (baseline: 12 weeks) → target: 8 weeks (–33%)
• Feature cycle time (idea→release) for small enhancements (baseline: 6 weeks) → target: 4.5 weeks (–25%)
• Usability task success rate on prototype tests (baseline: 72%) → target: ≥85%
• Number of post-release defects attributed to requirements/UX (baseline: 10/month) → target: ≤6/month
• Customer NPS or feature satisfaction for shipped features (baseline: +5) → target: +10
• Percentage of features launched with validated hypotheses (A/B or user test) (baseline: 40%) → target: 75%

Cost breakdown (6 weeks, 5 people)

• Bootcamp fees: $6,000 per person → $30,000
• Materials & tools: $500 per person → $2,500
• Backfill / temporary coverage (contractors or redistributed time): $2,000 per person → $10,000
• Travel / lodging (if applicable): $1,000 per person → $5,000
  Total one-time investment: $47,500

Expected ROI (conservative)

• If faster delivery reduces time-to-revenue for 2 medium features this quarter, incremental revenue or cost savings estimated ≥ $120,000 — 2.5x payback in first 6 months.

Risk mitigation

• Select bootcamp with proven curriculum and alumni references; run a 1-week pilot module for 1 hire if uncertain
• Stagger training where possible to maintain coverage; designate backups for key tasks
• Pair training with real project: trainees must apply a capstone project directly tied to roadmap
• Post-bootcamp mentorship: assign senior PM/designer buddies and biweekly brown-bag sessions
• Measurement plan and go/no-go checkpoints at week 6 and week 12; if KPIs not on track, implement targeted coaching.

Monitoring plan (who/what/when)

• Baseline capture (Product Ops) week 0: ramp metrics, cycle time, usability scores
• Weekly: training progress reports; HR tracks completion and engagement
• 6 weeks (end of bootcamp): capstone demo and immediate usability tests; update KPIs
• 12 weeks and 24 weeks: Product Ops to report KPIs vs baseline (ramp time, feature cycle time, defects, validated features, satisfaction)
• Dashboard: create a short dashboard in Looker/Tableau tracking the KPIs; monthly review with manager and stakeholders
• Success criteria for continued investment: meeting ≥3 of 5 KPI targets by week 24 and demonstrable capstone impact on roadmap.

Recommendation
Approve $47,500 for a 6-week cohort with the capstone requirement and the monitoring plan above to achieve faster, higher-quality delivery and a forecasted >2x return within 6–12 months.

Direct answer

A Type I error is rejecting a true null hypothesis (a false positive: concluding there's an effect when there isn't one); a Type II error is failing to reject a false null (a false negative: missing a real effect). The significance level alpha ($\alpha$) is the Type I error rate you're willing to accept; beta ($\beta$) is the Type II error rate, and power ($1-\beta$) is the chance of detecting a true effect. The right alpha is not a universal default. It should reflect how expensive a false positive is relative to a false negative for the specific decision being made.

Structured elaboration

The error matrix

Business examples

• Type I error (false positive) is costly: shipping an expensive personalization system because a test showed a "significant" lift that was actually noise. You pay ongoing engineering and infrastructure cost, plus the opportunity cost of the team not working on something that actually helps.
• Type II error (false negative) is costly: killing a checkout redesign because the test result wasn't significant, when the redesign truly would have lifted conversion. You lose the revenue and any competitive advantage from never shipping it.

Setting alpha and beta from cost, not convention

For a fixed sample size, $\alpha$ and $\beta$ trade off against each other: tightening $\alpha$ (fewer false positives) pushes power down (more false negatives) unless you also grow $n$. The textbook defaults ($\alpha=0.05$, power $=0.80$) are a starting point, not a business decision. Set $\alpha$ lower when a false positive is expensive to reverse (a payments flow, a compliance-adjacent surface, a change that's hard to roll back); accept a higher $\alpha$, or invest in a larger sample, when missing a true win is the more expensive mistake.

Worked example

A team wants to detect a 2 percentage-point lift on a checkout metric, baseline $p_0=0.20$, $p_1=0.22$, with $n=2{,}000$ users already collected per arm (fixed. This cohort won't grow before the decision is needed).

Holding $n$ fixed, power at two different alpha values (two-sided z-test for two proportions):

(computed from the standard two-proportion power formula with scipy.stats.norm)

Insisting on $\alpha=0.01$ for this "high-stakes" surface, without collecting more data, drops power from 34% to 15%: an 85% chance of missing a real 2-point lift entirely. If the team genuinely needs $\alpha=0.01$, the fix is a larger $n$, not accepting a test that's this underpowered.

Trade-offs & pitfalls

• Alpha and beta are symmetric-looking numbers with asymmetric real-world stakes; picking $\alpha=0.05$ by convention without asking what each error costs is the most common mistake.
• Tightening alpha without growing the sample size silently starves the test of power. Teams sometimes do this and then are surprised the "safer" test never finds anything.
• Failing to reject $H_0$ is not proof of no effect. It can just as easily mean the test was underpowered for the true effect size, so a null result should be read alongside the achieved power, not treated as a settled "no."

Direct answer

I would not pick a winner. A large analyst number and a weak survey can both be true (willingness to pay is the most a buyer says they would spend; purchase intent is how likely they say they are to buy): the analyst figure usually measures what the whole category spends, while the survey measures whether your target buyers would buy your product. I would first diagnose each source, then test the gap with evidence of real behaviour, and I would agree the postpone triggers before collecting any new data.

Step 1: Interrogate both signals

• Analyst report: what exactly does the $2B include (which buyers, regions, services plus software)? Is it measured spend or a forecast? When was it published, who commissioned it, and can the method be traced?
• Survey: who answered (recruited from a general panel, meaning a paid pool of anyone willing to take surveys, or people who own this budget)? How many? Did the question describe the product concretely? Stated purchase intent (what people say they would do) tends to differ from what they do, so treat it as a rough signal, not a measurement.

Step 2: Reconcile

Re-cut the survey to the people who actually match your buyer. If 120 people answered but only 30 own the budget, that is 25 percent of the sample, and the weak result may come mostly from the other 90. Then compare definitions by narrowing the headline to the part you could serve. Illustrative numbers: of the $2B, perhaps only 40 percent is software (the other 60 percent, $1.2B, is services you would not sell), leaving $800M; if only 30 percent of that is in the regions and company sizes you target, the serviceable market is about $240M (2,000 x 0.4 x 0.3). A $240M market with a weak re-cut survey is a very different bet from a $2B market, and the two sources now disagree far less.

Step 3: Triangulate with behaviour-based research

Triangulation means checking a conclusion against independent sources that fail in different ways.

1. Interview buyers about what they pay today and what workarounds (manual or makeshift ways they solve the problem now) they run (current spend is stronger evidence than a future promise).
2. Expert calls with people who sold or bought in this category to learn why deals stall.
3. Public filings of incumbents (the established companies already selling in the space) to see whether revenue in your segment matches the analyst story.
4. A small commitment test: ask about 10 qualified budget owners for a paid pilot (a small, time-limited paid trial), deposit or signed letter of intent (a written, non-binding statement that they intend to buy under agreed terms).

Postpone triggers (set in advance, illustrative thresholds)

Postpone building if (a) the re-cut survey of true budget owners is still weak, (b) fewer than 2 of those 10 commit to a paid pilot, and (c) buyers describe their current workaround as good enough. The 2-of-10 line (20 percent) is a judgment call, not a benchmark: it is a placeholder chosen so that a result clearly below it would worry me, because if few warm, qualified prospects commit money, the sales funnel probably cannot reach the customer count the business needs. I would postpone only when all three conditions hold together; if only one or two hold, I would run one more targeted test (for example a different segment or price) before deciding. Adapt it by working backward from how many customers you need and what share of conversations you can realistically convert. Ten people is directional, so the thresholds decide whether to keep investing, not whether the market exists.

Trade-offs and pitfalls

• Dismissing the survey because the analyst number is bigger, or the reverse, is the common error.
• A low result from the wrong respondents proves nothing about the right ones.
• Deciding the trigger after seeing the data invites rationalisation.

Situation: On a launch project, my team owned the API work and the partner team owned the customer-facing workflow. We both wanted the same release date, but their priority was polish while mine was integration stability.

Task: I needed to keep both sides moving even as trade-offs came up.

Action: I set up a shared plan with one clear owner per dependency, then separated must-have work from nice-to-have work. I also defined the term "trade-off" for the group as a choice where we gain one benefit by giving up another, so the conversation stayed concrete. When design wanted an extra step and engineering needed more time for testing, I asked, "What is the smallest version that still protects the user and the launch date?" We agreed to ship the core flow first, keep one optional enhancement for later, and review progress twice a week.

Result: We delivered the shared goal on time with a smaller scope, and both teams felt heard. I learned that progress keeps moving when you make the decision criteria explicit instead of debating opinions.

Direct answer
Quantitative evidence (metrics, logs, traces) tells you what happened and when with precision but can miss context and intent; qualitative evidence (engineer interviews, notes, chat logs) fills in the why and the human decision-making, but is subject to memory bias and self-justification. Weigh them together, and when they conflict, treat the disagreement itself as a finding worth investigating rather than picking whichever is more convenient.

Structured elaboration

• Quantitative evidence is precise and timestamped, which makes it the backbone of any timeline, but it can be silent on intent and context: a metric shows latency spiked at 14:03, but not why an engineer chose to deploy at that specific moment or what they believed was true when they did.
• Qualitative evidence captures reasoning and context that logs can't ("I deployed because the dashboard looked fine and I didn't know about the downstream dependency"), but human memory reconstructs events after the fact, often unconsciously smoothing over uncertainty or minimizing one's own role, so it should never override hard timestamped data when the two genuinely conflict.
• Correlating them into one timeline: anchor the timeline on quantitative events (deploys, alerts, metric changes) first, since those are objective and timestamped, then layer qualitative context alongside each event (what the engineer believed, what they were looking at, why they made a given call) as annotation, not as competing facts.
• When they conflict: if an engineer recalls checking a dashboard that logs show wasn't accessed, that's not necessarily dishonesty, memory under stress is genuinely unreliable, but it IS worth investigating why the gap exists: was there a different dashboard, a misremembered timestamp, or a real gap in what was actually checked before the decision was made. The conflict itself, not just its resolution, is often informative about where the process broke down.

Worked example
An engineer recalls seeing a warning-level alert before deploying and deciding it looked minor enough to proceed. Logs show no alert fired until four minutes after the deploy. Rather than concluding the engineer is simply wrong or dismissing the recollection, the investigation digs further and finds the engineer was actually looking at a stale, cached view of the dashboard that hadn't refreshed in several minutes, itself a real and separately worth-fixing gap (a dashboard that can silently show stale data during exactly the moment it matters most). The quantitative record established what actually happened; the qualitative account, once reconciled rather than dismissed, revealed a genuine, previously-unknown contributing factor that the logs alone would never have surfaced.

Trade-offs and pitfalls
The most common mistake is treating quantitative data as always authoritative and qualitative accounts as merely decorative color, which misses genuine contributing factors that only surface through human context. The opposite mistake, treating a confident personal recollection as more reliable than the logs when they conflict, risks building the postmortem's conclusion on a memory distortion. The discipline is to anchor on timestamped data but take conflicting qualitative accounts seriously enough to investigate the gap, not dismiss either source reflexively.

Direct answer

I would treat the threat as information about an unmet problem, not an order. I would lead the negotiation as one accountable owner with my executive sponsor (a senior leader on our side who owns the relationship) and the customer's sponsor (the senior person on their side who backs the purchase) present, find the problem beneath the requested feature, offer alternatives that solve it without damaging the product, avoid dated feature promises tied to the threat, and write the final decision in a decision record.

Before the meeting: size the stakes

• Share of annual recurring revenue (ARR, the yearly subscription revenue) and the renewal date.
• What the contract says, so nothing is promised that legal has not seen.
• Engineering's estimate of build cost and of the ongoing cost of keeping the variant working.
• Which other customers asked for something similar.
• The customer's real alternative: another vendor, building it themselves, or living without it.
• Whether the person making the threat signs the renewal.

In the meeting: ask about the problem

"Walk me through the last time this blocked you." "What happens to your team if it does not exist by renewal?" "Who else is affected?"

Alternatives, cheapest for the product first

Options 1 to 3 are the usual first moves, 4 is the usual route when the need is real, and 5 and 6 are last resorts.

1. Existing capability used differently, or configuration.
2. Integration through the public API, by the customer or our solutions team.
3. A generalized version: a smaller capability that solves the underlying need for many customers, built on our normal release rhythm (cadence).
4. A design partnership: the customer co-designs and tests that general version, in return for early access.
5. Paid custom work, isolated, with explicit support terms and an end date (a sunset date, when the arrangement stops).
6. Decline, with a clear reason.

What I would say to the customer

"I want you renewing, and I want your approval problem solved. I cannot commit to building that exact approval chain by a date, because it would stay in the product and slow every later change we make for all customers. What I can commit to is this: a manual second-approver checklist from next week, a design review of configurable approval rules with your compliance team by the end of next month, and a review date at the start of next quarter."

If they push for the feature and date in the contract, I take it to legal and my sponsor and offer the commitment to the problem and the review date instead.

Roadmap commitment under a churn threat

If a commitment is the price of staying, I make it a commitment to a problem and a review date, not a feature and a ship date. Engineering owns dates. That protects roadmap discipline, the engineering cadence, and the contract risk together, because a promised date that slips is worse than an honest "design review by June".

Worked example (illustrative)

A customer worth $900k of $18M ARR (5%) renews in five months and demands a hard-coded approval chain for their organization. Engineering estimates one quarter of the year (about 13 weeks) of two engineers to build it, so about 26 engineer-weeks, plus keeping that variant (a customer-specific version of the product that must keep working alongside the main one) working through every later change to approvals. Discovery shows the real problem: their compliance team needs two named approvers on purchases above a limit.

Illustrative cost comparison at $4,000 per engineer-week: the custom chain is 26 weeks = $104,000, plus about 6 weeks a year of upkeep = $24,000 a year, indefinitely. The general rules might cost 36 weeks = $144,000, but upkeep is part of normal product work and the rules also serve other customers who ask. Against that, the revenue at risk is $900k times the chance they really leave: even at 25% that is $225k. So money alone does not rule out the custom build. What tips it is that the variant's upkeep never ends and every later customer will ask for their own.

Proposal: configurable approval rules, built for everyone on the next planned cycle, with the customer as design partner. Interim workaround: a manual second-approver checklist. A decision record is a short written note of what was decided, why, and when to revisit it. Excerpt:

Decision:  Build configurable approval rules (general), not a custom approval chain.
Context:   Customer needs two approvers above a spend limit; renewal in 5 months.
Options:   Custom chain (rejected: permanent variant), config rules (chosen), decline.
Our commitment: Design review with their team by end of next month (owner: PM).
Their commitment: Two people available for design feedback and testing.
Review date: Start of next quarter. Reopen if design review shows the rules cannot express their policy.

Trade-offs and pitfalls

• A renewal at risk is real, but so is the compounding cost of a one-off variant that every later customer will ask for.
• Do not assume the threat is a bluff, and do not accept it as a deadline without checking who is making it.
• Do not let sales agree to the feature in a side channel. One owner, one record.
• If they leave anyway, record that as a known cost of the decision and learn from it.

Approach: treat this as an attribution problem — test whether the competitor release is a statistically and practically significant driver of the Q3 churn spike versus other explanations.

1. Define metrics

• Primary: gross churn rate (customers cancelled / starting customers), net MRR churn, retention curve (D30/D90), survival function.
• Secondary: engagement (DAU/MAU, key feature usage), NPS/CSAT change, support-ticket volume and sentiment, win/loss sales notes.

2. Cohort construction

• Time cohorts: monthly cohorts (signups) and pre/post cohorts relative to competitor release date (e.g., 60 days before vs 60 after).
• Behavioral cohorts: high vs low engagement, by plan tier, by region, by contract type.
• Exposed vs unexposed: define "exposed" as users who used overlapping features in the prior 30 days or visited competitor pages (if tracked).

3. Tests / analyses

• Descriptive: plot churn rate by cohort and by segment; overlay competitor release timeline.
• Difference-in-differences (DiD): compare churn trends pre/post between exposed and unexposed cohorts to isolate release effect.
• Survival analysis / Cox proportional hazards: model time-to-churn with competitor-release as time-dependent covariate and control variables (engagement, plan, tenure).
• Logistic regression / propensity score matching: predict churn with a binary exposure to competitor and control for confounders; PSM to balance covariates and estimate ATT.
• Funnel & feature-usage analysis: see if usage of features that competitor replicates dropped before churn.
• Text analysis on support tickets and cancellation reasons to quantify mentions of competitor.

4. Sanity checks & confounders

• Data quality: verify timestamps, dedupe subscriptions, check cohorts have sufficient N.
• Seasonal/marketing confounders: control for price changes, campaigns, outages, contract renewals, economic events.
• Parallel trends check for DiD, balance checks for PSM.
• Robustness: vary exposure window sizes, use placebo release dates, check geographical markets where competitor wasn't launched.
• Compute practical significance: convert relative increase to absolute lost MRR and CLTV impact.

5. Outcome and next steps

• If models show significant effect: quantify lost MRR, prioritize retention offers for exposed users, product differentiation roadmap, and track churn after mitigations.
• If not: investigate alternative causes (bugs, pricing, onboarding) and repeat analyses.

This sequence gives a defensible, actionable answer linking statistical evidence to product decisions.

Direct answer

In a metrics review, a stakeholder pointed out that our headline engagement score, a composite metric meant to summarize overall product health, was climbing even while support complaints about a specific onboarding step were increasing, and argued the metric was hiding a real problem rather than reflecting one. Instead of defending the aggregate number, I decomposed it, formed a specific hypothesis about which onboarding step was actually driving the complaints, ran a targeted experiment on that step, tracked both the step-level metric and a broader guardrail metric, and reported back a result that had actually cleared a statistical bar, not just moved in a favorable direction.

Structured elaboration

Treat a metric critique as a decomposition problem, not a defense of the number. When someone says a metric is misleading, the useful response is to break the composite down into its parts and check each one against the specific complaint, rather than arguing the aggregate is fine because it's trending up. I pulled apart the engagement score into its underlying components and checked each against the complaint pattern in support tickets, which narrowed the search to one specific step rather than the whole onboarding flow.

Form a hypothesis specific enough to be wrong. A vague hypothesis like "onboarding could be smoother" cannot be tested cleanly. I stated the hypothesis precisely: a specific mandatory step in onboarding was causing enough friction that some meaningful share of users disengaged there, and that this friction was not visible in the composite score because it was being offset by strong engagement from users who made it past that step.

Design the change or experiment to isolate that specific cause. I changed only the flagged step, shortening it, rather than making several onboarding changes at once, so that if the metrics moved, I could actually attribute the movement to that step rather than to a bundle of unrelated changes.

Track the metric tied to the hypothesis and a broader guardrail. I tracked step-level completion rate as the primary metric, since that maps directly to the hypothesis, and a downstream activation metric as a guardrail, to make sure a win on the specific step wasn't just shifting friction to a later point in the funnel rather than actually resolving it.

Report a result that clears a real statistical bar, and be honest about what it does and doesn't prove. I checked whether the observed improvement in step completion was large enough and consistent enough to be unlikely due to chance, rather than reporting a favorable-looking number from a short or small sample as if it were conclusive, and I reported the guardrail metric's outcome alongside the primary result even though it wasn't the headline finding.

Worked example

The critique came from a growth stakeholder reviewing our monthly metrics, who noted that engagement score had been rising for two straight months while support tickets mentioning "can't get past setup" had also been rising over the same period, an inconsistency that suggested the composite score was not capturing something real. Breaking the score into its components and cross-referencing against the support ticket themes pointed to one specific step: a mandatory multi-field profile-setup screen early in onboarding.

My hypothesis was that this step's length and the number of required fields were causing a meaningful share of users to abandon onboarding entirely, while the users who did push through were unusually engaged afterward, which is what let the aggregate score keep climbing even as this specific problem worsened. I built a shortened version of the step, reducing it to only the fields actually needed to use the core product, and ran it as a controlled test against the original version, changing nothing else in the flow. I tracked completion rate at that specific step as the primary metric and a downstream activation metric (whether users reached a meaningful first-use milestone) as a guardrail. The shortened step's completion rate rose from 61% to 74%, a 13-point lift that held up under a standard significance check across the two-week test, meaning it was unlikely to be random variation, and the downstream activation guardrail moved in the same positive direction, from 48% to 52%, rather than staying flat or declining, which ruled out the concern that we were just shifting the friction further down the funnel. I reported both results back to the stakeholder who raised the original critique, along with the corrected view of what the composite engagement score had been masking.

Trade-offs and pitfalls

The clearest pitfall is chasing a single vocal critique without checking whether it represents a broader pattern; I cross-referenced the stakeholder's specific complaint against the support ticket data before committing engineering time to a fix, rather than assuming one comment meant the whole flow was broken. A second pitfall is treating a statistically significant result as automatically meaningful; a tiny, technically significant improvement in a huge sample can be true and still not matter much to the business, so I looked at the size of the effect and its business consequence, not only whether a significance check passed. I also try to remember that composite metrics can always hide local problems by design, which is why pairing quantitative dashboards with qualitative signals like support tickets, rather than trusting the headline number alone, is what actually surfaced this issue in the first place.

Direct answer
Build the rubric around whether the writing actually works for its reader: does it state its point clearly, fit the audience it's for, give the reader something to do with it, and use a tone appropriate to its purpose, then justify each criterion by what a failure on it costs the reader.

Structured elaboration
Proposed criteria, with the reasoning for each:

1. Clear main point: can a reader state the document's core message in one sentence after reading it? Justification: this is the single biggest failure mode in unclear writing, so it anchors the rubric.
2. Appropriate structure: does the important information come early, with supporting detail after, rather than requiring the reader to read to the end to find the point?
3. Audience fit: is the level of jargon and assumed background knowledge appropriate for who's actually going to read this, rather than written for the author's own level of familiarity?
4. Concision: is there padding, hedging, or restatement that could be cut without losing meaning?
5. Actionable next step: if the document implies an action or a decision, is that action stated explicitly, rather than left for the reader to infer?
6. Precision: are claims specific and checkable, or do vague quantifiers stand in for actual numbers where numbers were available?
7. Tone fit: is the tone appropriate to the stakes and relationship, neither over-casual for a high-stakes audience nor needlessly formal for a quick internal note?
8. Honesty about caveats: does the document surface real limitations or risks, rather than smoothing them over to look cleaner?

Worked example
Applying this to a short vendor-status email: main point ("vendor is delayed two weeks") is clear in the first sentence; structure is fine; audience fit is appropriate (no unnecessary jargon for a business reader); concision is good at three sentences; the next step (approve a revised deadline) is explicitly stated; precision holds (a specific date is given, not "soon"); tone is appropriately direct without being alarmist; and the caveat (a small risk of a further one-week slip) is honestly included rather than hidden. That's 8 for 8, which is a genuinely well-written status update by this rubric.

Trade-offs and pitfalls

• A rubric with too many criteria becomes tedious to apply consistently; six to eight, as here, is usually enough to catch the failure modes that matter most without turning review into a lengthy checklist exercise.
• Some criteria trade off against each other (concision versus caveats, for instance); the rubric should make clear that cutting a genuine caveat to satisfy concision is a failure, not a win, on this rubric.
• A rubric like this works best as a discussion tool during review, not as a rigid pass/fail gate; a document can reasonably fail one criterion (say, tone) for a good reason specific to its context.

Direct answer

For a baseline conversion rate of 10% and a target 10% relative lift (an absolute increase to 11%), at 80% power and a 5% two-sided significance level, the required sample size is 14,751 users per variant (29,502 total), computed with the standard two-proportion z-test sample-size formula and verified in code rather than looked up. The two biggest caveats to state alongside that number: the formula assumes a fixed sample size decided in advance and analyzed once (not repeatedly peeked at), and it assumes exactly one comparison, running the same kind of test many times across funnel steps and segments requires correcting the significance threshold, which materially increases the required sample size.

Structured elaboration

The formula. For detecting a difference between baseline conversion rate $p_1$ and target rate $p_2$, with two-sided significance level $\alpha$ and power $1-\beta$, the required sample size per variant is

where $\bar p = (p_1 + p_2)/2$ is the pooled average rate used for the critical-value term, $z_{\alpha/2}$ is the standard normal critical value for the chosen two-sided significance level, and $z_\beta$ is the standard normal value corresponding to the chosen power.

Every calculation step, for $p_1 = 0.10$, $p_2 = 0.11$, $\alpha = 0.05$, power $= 0.80$:

• $z_{\alpha/2} = z_{0.025} = 1.9600$ (the value beyond which 2.5% of the standard normal distribution lies on each tail, for a 5% two-sided test)
• $z_{\beta} = z_{0.20} = 0.8416$ (the value corresponding to 80% power)
• $\bar p = (0.10 + 0.11)/2 = 0.105$
• Critical-value term: $z_{\alpha/2}\sqrt{2 \bar p(1-\bar p)} = 1.9600 \times \sqrt{2 \times 0.105 \times 0.895} = 1.9600 \times \sqrt{0.1880} = 1.9600 \times 0.4335 = 0.8497$
• Power term: $z_\beta \sqrt{p_1(1-p_1) + p_2(1-p_2)} = 0.8416 \times \sqrt{0.10 \times 0.90 + 0.11 \times 0.89} = 0.8416 \times \sqrt{0.09 + 0.0979} = 0.8416 \times \sqrt{0.1879} = 0.8416 \times 0.4335 = 0.3648$
• Sum, squared, over $(p_2-p_1)^2 = 0.01^2 = 0.0001$: $n = (0.8497 + 0.3648)^2 / 0.0001 = 1.2145^2 / 0.0001 = 1.4751 / 0.0001 = 14{,}751$ (rounding the intermediate sum up slightly at each displayed step; the exact, unrounded computation below gives 14,750.79, rounded up to 14,751)

Final answer: n = 14,751 per variant, 29,502 total across both variants.

Approximations and caveats in the formula itself. This is the normal (Wald-style) approximation to the binomial, accurate for the sample sizes this kind of calculation typically produces, but technically an approximation, not an exact result; some practitioners add a small continuity correction for extra conservatism, which this answer omits since the effect is negligible at n in the thousands. The formula also assumes a FIXED sample size decided before the test starts and analyzed exactly once at the end; a test monitored continuously and stopped as soon as significance is first observed (without a formal sequential-testing correction) will falsely "succeed" far more often than the stated 5% significance level implies, because repeated looks each carry their own chance of a false positive.

Worked example

All arithmetic verified in Python (statistics.NormalDist, no external library required), not by hand or by eyeballing z-scores:

import math
from statistics import NormalDist
nd = NormalDist()

def sample_size_two_proportion(p1, p2, alpha=0.05, power=0.8):
    z_alpha = nd.inv_cdf(1 - alpha / 2)
    z_beta = nd.inv_cdf(power)
    p_bar = (p1 + p2) / 2
    term1 = z_alpha * math.sqrt(2 * p_bar * (1 - p_bar))
    term2 = z_beta * math.sqrt(p1 * (1 - p1) + p2 * (1 - p2))
    n = ((term1 + term2) ** 2) / ((p2 - p1) ** 2)
    return n, math.ceil(n)

print("Primary (10% -> 11%):", sample_size_two_proportion(0.10, 0.11))
print("Companion (5% -> 6%):", sample_size_two_proportion(0.05, 0.06))
m = 10
print(f"Bonferroni-corrected, m={m} concurrent tests:",
      sample_size_two_proportion(0.10, 0.11, alpha=0.05 / m))

Output (actually executed):

Primary (10% -> 11%): (14750.79046904495, 14751)
Companion (5% -> 6%): (8157.731447849276, 8158)
Bonferroni-corrected, m=10 concurrent tests: (25019.652834268625, 25020)

Companion instance, at different baseline and lift numbers. For a baseline of 5% detecting a 20% relative lift (to 6% absolute), holding power and significance fixed at the same 80% and 5% two-sided, the required sample size is 8,158 per variant, noticeably smaller than the primary case despite testing a comparable relative lift, because the absolute gap between the two rates (1 percentage point in both cases) sits on a smaller baseline variance term at 5% than at 10%.

Multiplicity: what changes when many tests run concurrently

Running this same kind of sample-size-and-significance calculation independently across many funnel steps and many segments (checking, say, 10 different step-by-segment slices in the same rollout) inflates the true chance of at least one false positive far above the nominal 5% significance level per test; with 10 independent tests each run at 5%, the chance of at least one false "significant" result by pure chance is $1 - (1-0.05)^{10} \approx 40\%$, not 5%. Two standard corrections address this, with different trade-offs:

• Bonferroni correction. Divide the target significance level by the number of comparisons, $\alpha_{\text{corrected}} = \alpha / m$, and use THAT corrected alpha in the sample-size formula's $z_{\alpha/2}$ term. This is simple and conservative (it controls the probability of ANY false positive across all $m$ tests, the family-wise error rate), but it is a strict, often overly cautious correction, and it directly and substantially inflates the required sample size, computed above: for $m=10$, $\alpha_{\text{corrected}} = 0.005$, giving $z_{\alpha/2} = 2.8070$ instead of 1.9600, and the required sample size rises to 25,020 per variant, a 1.696x inflation over the uncorrected 14,751.
• Benjamini-Hochberg procedure. Instead of fixing a single stricter alpha for every test in advance, this method controls the false discovery rate (FDR, the expected proportion of "significant" results that are actually false positives, rather than the probability of any false positive at all) by ranking all $m$ tests' p-values after the data is collected and comparing each rank's p-value to a rank-dependent threshold. It is meaningfully more powerful than Bonferroni for the same nominal error-control target, since it does not treat every test as if it needed the full worst-case correction, but its correction is inherently a function of the actual observed p-values, not a single fixed number known before the study runs, so it is typically applied at the ANALYSIS stage after data collection rather than used directly, in closed form, to size the study in advance the way Bonferroni's fixed corrected alpha can be.

Trade-offs and pitfalls

• The Bonferroni-inflated sample size (25,020 per variant for 10 concurrent tests) is a real operational cost, not just an abstract statistical nicety; a team that runs many simultaneous funnel-step experiments without planning for this will either under-power every individual test or need to accept a much longer data-collection window, and this trade-off should be surfaced to stakeholders before the rollout, not discovered afterward when none of the ten tests reach significance.
• A common mistake is applying a multiplicity correction to the significance threshold used for ANALYSIS while never adjusting the sample-size calculation that determined how long the test would run, which produces an experiment that was never actually powered to detect the effect at the stricter, corrected threshold it will be judged against.
• Continuously monitoring a running test and stopping the moment it crosses significance, without a sequential-testing correction, invalidates the 5% significance-level guarantee this calculation is built on; if early stopping is operationally necessary, use a sequential testing method (such as a group-sequential design with pre-specified alpha spending) rather than an ad hoc "check daily, stop when significant" practice layered on top of a fixed-sample-size calculation.
• The formula's power term uses $p_1$ and $p_2$'s own individual variances (the more standard, slightly more accurate approach), while the critical-value term uses the pooled $\bar p$; a simplified version some calculators use applies $\bar p$ to BOTH terms, which is a reasonable approximation when $p_1$ and $p_2$ are close together (verified directly: this simplification gives 14,752 for the primary case here, versus 14,751 with the more precise unpooled power term, a 0.01% difference) but diverges more as the two rates get further apart, so know which version a given calculator or teammate is using before comparing numbers.