Statistical Inference and Hypothesis Testing Questions

Three common sampling biases in user research and product analytics are selection bias, survivorship bias, and non-response bias. Each systematically skews who or what ends up in a dataset relative to the population the team actually wants to understand, and each has a distinct, practical mitigation.

Structured elaboration

Selection bias

The recruitment or logging mechanism itself favors certain users over others, so the sample never represented the target population to begin with.

• Product example: recruiting research participants exclusively through an in-app "give feedback" banner over-represents highly engaged power users and misses casual or lapsed users entirely.
• Mitigation: diversify recruitment channels (email, support tickets, targeted outreach to inactive users, paid panels), and when analyzing logged data, weight or stratify results by known usage segments so heavy users do not dominate the conclusion.

Survivorship bias

The dataset only contains entities that "survived" some earlier filter, silently excluding the ones that dropped out, which are often the most informative cases for understanding why people leave.

• Product example: analyzing feature usage only among currently active accounts misses everyone who abandoned the product during onboarding, so the data looks healthier than the funnel actually is.
• Mitigation: instrument and retain events from the full funnel, including users who never became active, and deliberately recruit recent churners or drop-outs for exit interviews rather than only studying who remains.

Non-response bias

Among people invited to respond (a survey, an in-app prompt), the ones who actually respond differ systematically from the ones who do not, in ways related to the very thing being measured.

• Product example: a satisfaction survey sent right after a purchase mostly reaches people who are already positive enough to engage further, biasing an NPS score upward relative to the full user base.
• Mitigation: keep surveys short to raise response rates, follow up with non-responders (or a random subsample of them), and compare respondent demographics or behavior to the overall population to check how representative the response set actually is; apply response weighting when a gap is found.

Worked example

A team wants to know why a new onboarding flow has a 40% completion rate. If they only interview users pulled from a "rate your experience" prompt shown after successful completion, they have compounded two biases at once: survivorship (only completers are in the sampling frame at all) and non-response (only the completers who bothered to respond to the prompt are actually interviewed). The resulting interviews will describe the experience of the roughly 40% who finished and felt strongly enough to respond, and say nothing about the 60% who dropped out, which is the group the team most needed to understand. Fixing this requires reaching users at the point of drop-off (an exit-intent prompt, or a follow-up email to users who started but did not finish), not just the users who reached the end.

Trade-offs & pitfalls

• These three biases often compound in the same study (as above), so checking for one is not enough; ask separately who was even eligible to be seen, who dropped out along the way, and who actually responded.
• Fixing selection bias with broader recruitment can raise the cost and time of a study; weighting analysis by known segments is often a cheaper partial fix when full re-recruitment is not practical.
• Comparing respondents to the overall population only catches non-response bias on variables you can actually measure for non-respondents (demographics, usage tier); it cannot catch bias on unmeasured traits.
• Passive telemetry (event logs) is not automatically unbiased just because it does not rely on someone opting in; the underlying feature or platform itself can have selection effects, such as only being available on newer devices.

Direct answer

Simpson's paradox is when a trend that appears in an aggregated dataset reverses, or disappears, once the data is broken into meaningful subgroups. It happens when a subgroup variable is both correlated with the outcome and unevenly distributed across the groups being compared, so the aggregate is really comparing different mixes of subgroups rather than like-for-like. In A/B testing this can flip an experiment's headline verdict if traffic composition differs between arms.

Structured elaboration

Mechanism. The paradox needs two ingredients: a segment (e.g. new vs returning users) whose baseline conversion rate differs a lot, and a segment mix that differs between the two experiment arms, whether from a randomization bug, non-random routing, or the arms genuinely attracting different user mixes (for example, if the treatment changes something only new users see first). When the high-baseline segment is overrepresented in one arm, that arm's aggregate gets pulled up regardless of the true within-segment treatment effect.

Two distinct causes worth separating. (1) A randomization or logging bug that skews segment composition between arms even though the metric itself has a uniform effect. This should be treated like Sample Ratio Mismatch: a data-quality problem to fix, not a real result to interpret. (2) A genuine, real segment-mix or heterogeneous-effect situation (e.g. the arms were deliberately targeted differently, or the treatment interacts with user tenure). This is a real result that needs segment-aware reporting, not a bug.

Worked example

Two segments with very different baseline conversion, and arms that got very different mixes of each:

(All counts verified by direct computation.) Notice: treatment wins in both segments individually (45.0% > 42.2%, and 7.0% > 6.0%), but because control's traffic was mostly high-intent users (who convert well regardless of arm) and treatment's traffic was mostly low-intent users (who convert poorly regardless of arm), the aggregate makes it look like control is dramatically better. Reading only the aggregate row would lead to killing a feature that actually helped every segment it touched.

Detection. Treat segment-mix balance as a standard experiment health check alongside SRM: compare the proportion of each key segment (device, new/returning, geography) between arms, and flag if that mix itself differs significantly between arms (a chi-square test of segment distribution by arm is a direct way to check this). Separately, fit the metric with an interaction term, $\text{metric} \sim \text{arm} + \text{segment} + \text{arm}:\text{segment}$, to see whether the treatment effect actually differs by segment (heterogeneity) as opposed to just the segment mix differing (composition).

Resolution. If the segment-mix imbalance traces back to a bug (bad routing, non-random assignment), that invalidates the experiment; the fix is to correct randomization and rerun, not to reweight around the bug. If the imbalance is real and expected (e.g. the two arms intentionally target different populations, or a feature launch changes what fraction of traffic is new users), report segment-level effects explicitly rather than a single aggregate number, and if a single summary number is still needed, use a standardized/weighted average that applies one fixed reference mix (e.g. last month's traffic composition) to both arms instead of letting each arm's own live mix drive the number.

Trade-offs & pitfalls

• Don't go looking for a favorable segment cut after seeing an unfavorable aggregate result; that's post-hoc fishing and needs the same multiple-comparisons discipline as any other exploratory subgroup analysis. Segment checks belong in the pre-registered analysis plan, run regardless of which way the aggregate goes.
• Trusting an aggregate number without ever looking at segment balance is the classic wrong turn, and it's especially dangerous because the aggregate looks perfectly clean (large N, tight CI) even while hiding a completely reversed within-segment story.
• Not every segment-level difference is Simpson's paradox; if segment mix is balanced across arms and the aggregate and segment-level conclusions agree, that's just a normal heterogeneous-effect finding, worth reporting, but not a paradox to resolve.

Direct answer

The null hypothesis (H0) is the "nothing changed" default you assume true until the data gives strong enough evidence to reject it. The alternative hypothesis (H1, or Ha) is the specific claim you're trying to find evidence for. A two-tailed test's alternative allows the effect to go either direction, so an increase or a decrease both count as evidence against H0. A one-tailed test's alternative commits in advance to only one direction, and only evidence in that direction can ever reject H0, however extreme the result is in the other direction, a one-tailed test treats it as not significant.

Structured elaboration

Defining the two hypotheses

• H0: the status-quo claim, typically "no difference" or "no effect," for example mu_new equals mu_old, or p_new equals p_old. It's what you'd believe by default absent evidence otherwise.
• H1 (or Ha): the claim there is reason to suspect and want the data to support. Hypothesis testing is structured to make H0 the thing you disprove, not the thing you prove; you never "accept H0," you either reject it or fail to reject it.

One-tailed versus two-tailed

A two-sided alternative is symmetric: H1 says the parameter is simply not equal to the null value, so evidence in either direction, higher or lower, counts against H0. A one-sided alternative picks a direction in advance, for example H1 says the new value is greater than the old one, and the entire significance level (say 5%) is allocated to that one tail of the distribution. That gives more power to detect an effect in that specific direction for the same sample size, at the cost of being structurally blind to an effect in the other direction.

Worked example

Testing whether a new checkout flow changes conversion rate, where p_old is the current rate and p_new is the new flow's rate:

Testing whether it changes the metric in either direction, appropriate when you'd act on either a lift or a drop, for example rolling back a regression just as readily as shipping an improvement:

$$H_0: p_{\text{new}} = p_{\text{old}}, \qquad H_1: p_{\text{new}} \neq p_{\text{old}}$$

Testing whether it increases the metric, appropriate only when a decrease and "no effect" would be handled identically, for example neither would ship:

$$H_0: p_{\text{new}} \le p_{\text{old}}, \qquad H_1: p_{\text{new}} > p_{\text{old}}$$

Which to choose, and why

Choose the two-sided test whenever a result in either direction would change the decision, which is the common case in product work: a checkout redesign that decreases conversion is just as actionable, roll it back, as one that increases it, so a one-sided test built only to detect increases could let a real, harmful regression show up as "not significant" simply because the test structurally can't reject in that direction. Choose a one-sided test only when a result in the "wrong" direction is genuinely not actionable, and that decision was made before seeing the data. For example, a pure cost-reduction change to backend infrastructure where the only actions are "doesn't hurt the metric" versus "measurably hurts it," never "significantly helped," might reasonably justify a directional test. That's a narrow case, though, and switching to one-sided after peeking at a result that's "almost significant" two-sided is a form of p-hacking, not a legitimate use of the one-sided test.

Trade-offs and pitfalls

• One-sided tests have more power for a fixed alpha and sample size, but that power gain exactly matches giving up the ability to detect the opposite effect. It isn't a free lunch, it's a bet on direction made in advance.
• Choosing the tail after seeing which direction the data leans is a common way to quietly gain extra apparent significance without disclosing it. Always fix the direction, or commit to two-sided, before collecting data, ideally in writing.
• Default to two-sided unless there is a specific, pre-committed, defensible reason not to. When a one-sided test is used, say so explicitly in the write-up along with the pre-registered direction and rationale, so a reader can judge whether the choice was made honestly.

Correlation means two variables move together in the data; causation means changing one variable actually produces a change in the other. A correlation can arise from genuine causation, reverse causation, a shared underlying cause (a confounder), or simple coincidence, so observing that two things move together is never by itself enough to justify acting on one to change the other.

Structured elaboration

Why correlation can mislead

A business decision that assumes the first pattern, when the truth is a confounder or reverse causation, can spend money or effort changing the wrong thing.

A concrete failure mode

Suppose product analytics show that users who enable push notifications spend meaningfully more per month than users who do not. The naive read is "notifications drive spend, so force-enable them for everyone." The more likely explanation is a confounder: users who are already more engaged with the product are both more likely to opt into notifications and more likely to spend, because engagement drives both. Forcing notifications on disengaged users would not manufacture the same spend, and could instead increase churn from users who find the notifications unwanted.

Increasing causal confidence

The strongest practical fix is a randomized controlled experiment (an A/B test): randomly assign users to receive the notification prompt versus not, which breaks the link between the confounder (engagement) and treatment assignment, so any difference in downstream spend between arms can be attributed to the notification itself. When randomization is not possible (the change already shipped to everyone, or it is not feasible to withhold from a control group), quasi-experimental methods like propensity-score matching (pairing treated and untreated units that look similar on observed characteristics before comparing them) or difference-in-differences (comparing the change over time in the affected group against the change in an unaffected group) can partially control for observed confounders, at the cost of resting on assumptions (no unobserved confounding, or parallel trends, meaning the two groups would have moved the same way over time if the change had never happened) that a true randomized experiment does not need.

Trade-offs & pitfalls

• A/B tests are the gold standard for causal confidence but are not always feasible: some changes cannot ethically or practically be withheld from a subset of users, and some effects only show up over a longer horizon than a typical test window.
• Quasi-experimental methods (matching, diff-in-diff, instrumental variables) reduce but do not eliminate confounding risk; they are only as good as the confounders the team thought to measure and control for.
• "We increased causal confidence" is not the same as "we proved causation." Even a well-run experiment estimates an average effect for the population tested, under the conditions tested, not a universal law.
• The instinct to act quickly on a compelling correlation is strongest exactly when the stakes are highest, which is also when getting the causal story wrong is most expensive.

Direct answer

In a past project, I needed to know whether a new onboarding email sequence actually improved 30-day retention, not just whether it looked better in a dashboard. I framed it as a two-proportion hypothesis test on a randomized A/B split, checked the assumptions behind that test, and then translated the statistically significant result into a dollar figure before recommending a rollout, because a p-value alone doesn't tell a stakeholder whether to spend engineering time shipping it.

Structured elaboration

Hypothesis formulation. With $p_{new}$ and $p_{old}$ as the true 30-day retention rates under each sequence:

$$H_0: p_{new} - p_{old} = 0 \qquad H_1: p_{new} - p_{old} \ne 0$$

I used a two-sided alternative rather than one-sided, because a null result and a negative result both had to be plausible outcomes I'd act on differently.

Design and assumptions. Users were randomized to sequence at signup (independent units, no user sees both), the outcome is binary (retained or not at day 30), and with roughly 12,000 users per arm the sample was large enough for the normal approximation to the binomial to hold. Before running it I did an a priori power calculation targeting 80% power to detect a 2 percentage-point lift at $\alpha = 0.05$, which is what set the run duration and the minimum sample size, so I wasn't deciding "run until it looks good."

Test selection. A two-proportion z-test is the right tool here: two independent binomial samples, testing a difference in proportions, large enough $n$ per arm that the sampling distribution of the difference is well-approximated by a normal.

Worked example

Observed: $n_1 = n_2 = 12{,}000$, $x_{new} = 1{,}584$ retained ((\hat p_{new} = 0.1320)), $x_{old} = 1{,}296$ retained ((\hat p_{old} = 0.1080)). Pooled proportion under $H_0$: $\hat p = (1584+1296)/24000 = 0.1200$.

$$SE_{pooled} = \sqrt{\hat p(1-\hat p)\left(\frac{1}{n_1}+\frac{1}{n_2}\right)} = \sqrt{0.12 \times 0.88 \times \left(\frac{1}{12000}+\frac{1}{12000}\right)} \approx 0.00420$$ $$z = \frac{\hat p_{new} - \hat p_{old}}{SE_{pooled}} = \frac{0.024}{0.00420} \approx 5.72$$

That gives $p < 0.0001$ (two-sided), computed and verified with scipy.stats.norm.sf. For the effect-size CI I used the unpooled standard error (correct for a confidence interval, as opposed to the pooled SE used for the null-hypothesis test statistic):

$$SE_{unpooled} = \sqrt{\frac{\hat p_{new}(1-\hat p_{new})}{n_1} + \frac{\hat p_{old}(1-\hat p_{old})}{n_2}} = \sqrt{\frac{0.1320 \times 0.8680}{12000} + \frac{0.1080 \times 0.8920}{12000}} \approx \sqrt{0.00000955 + 0.00000803} \approx \sqrt{0.00001758} \approx 0.00419$$

giving a 95% CI on the difference of $0.024 \pm 1.96(0.00419) \approx [0.016,\ 0.032]$, or 1.6 to 3.2 percentage points.

Practical significance. A 2.4-point lift on a 12,000-user cohort translates to about 288 additional retained users in that sample alone; scaled to a monthly signup volume, I converted that into an estimated incremental-retention revenue range using the team's existing LTV-per-retained-user assumption, and paired that with the (near-zero) engineering cost of keeping the new sequence live. I presented the CI, not just the point estimate, so stakeholders could see the plausible range of the ROI rather than a single number that implied more precision than the data supported.

Trade-offs & pitfalls

• A p-value this small mostly reflects large $n$; I made a point of leading with the percentage-point lift and its CI, not the p-value, when talking to non-technical stakeholders, because "p < 0.0001" invites false confidence about the size of the effect, not just its existence.
• I checked for novelty effects: a lift that shows up in week one of a new email sequence sometimes fades as users get used to it, so I flagged that the 30-day window doesn't rule out longer-run decay and recommended a follow-up read at 90 days.
• I only had one primary metric locked in the pre-registration; if I'd been peeking at five secondary metrics along the way, I'd have needed a multiple-comparisons correction (Bonferroni or Benjamini-Hochberg) before treating any of those as confirmatory.