Statistical Inference and Hypothesis Testing Questions

Direct answer

I would treat this as a difference-in-differences (DiD) problem: compare the affected country's conversion trend before and after release to the same trend in similar unaffected countries, check that the two trends moved in parallel before the release (this is what makes DiD believable), and only conclude the release caused the drop if the affected country breaks from that parallel trend right at the release boundary and no other explanation (outage, campaign pause, holiday) lines up with the same timing.

Structured elaboration

1) Define windows and populations. Use a pre-period long enough to establish a stable trend (14-28 days) and a post-period long enough to be past launch-day noise but short enough to avoid picking up unrelated drift (7-14 days). Treatment group: users in the affected country. Control group: other countries with similar language, timezone, and historical baseline conversion, ideally 2-3 of them rather than one, so a single control country's own idiosyncratic shock does not bias the comparison.

2) Query the raw daily series per country using the schema at hand, e.g. SELECT date, country, COUNT(*) AS sessions, SUM(converted) AS conversions FROM events WHERE date BETWEEN <pre_start> AND <post_end> GROUP BY date, country, and aggregate to daily conversion rate = conversions / sessions.

3) Test parallel trends before the release. Plot the pre-period daily rates for treatment vs. control; regress rate on day index separately for each group over the pre-period and compare slopes. If the affected country's pre-period trend was already diverging from the control group, DiD is not credible and you need a different design (synthetic control, weighting multiple controls to match the pre-trend).

4) Estimate the DiD effect. did_estimate = (post_mean_treatment - pre_mean_treatment) - (post_mean_control - pre_mean_control). This differences out any country-level effect that's constant over time and any trend that's common to both groups (e.g. a global seasonal dip), isolating the change specific to the treatment group at the treatment time.

5) Rule out confounders with the same timing. Check for concurrent marketing spend changes, payment-provider outages or currency changes local to that country, a local holiday, or a client-version rollout that happened to ship unevenly by geo. Cross-reference error rates, latency, and per-step funnel drop-off (landing to add-to-cart to checkout) to see if the drop is concentrated at a step the release actually touched; if the drop shows up at a funnel step the release code never runs on, the release is unlikely to be the cause.

Worked example

Simulating a matching scenario (pinned seed, numpy.random.default_rng(1113)), 14 pre-release days and 7 post-release days, baseline daily conversion ~8.0% in the affected country and ~7.8% in the control pool, with a true injected effect of -1.2pp starting at the release date:

import numpy as np
rng = np.random.default_rng(seed=1113)
affected_pre = rng.normal(0.080, 0.003, 14)
control_pre = rng.normal(0.078, 0.003, 14)
affected_post = rng.normal(0.080 - 0.012, 0.003, 7)
control_post = rng.normal(0.078, 0.003, 7)
did = (affected_post.mean() - affected_pre.mean()) - (control_post.mean() - control_pre.mean())

This produces affected pre = 0.0792, post = 0.0654 (raw drop of -1.39pp) and control pre = 0.0778, post = 0.0763 (drop of only -0.15pp, i.e. noise), giving a DiD estimate of -1.24pp, closely recovering the -1.2pp true injected effect and confirming that most of the raw drop is attributable to the treatment group specifically, not a shared trend. The pre-period slopes (-0.00037/day for the affected country vs +0.00003/day for control) are both close to zero and close to each other, which is the parallel-trends check passing. A two-sample t-test on the post-period difference gives t = -5.55, p = 0.00013, so the post-period gap is not noise.

Trade-offs & pitfalls

DiD's validity rests entirely on the parallel-trends assumption; it is not testable in the post-period (you only ever observe one of the two counterfactuals), so a passing pre-trend check is necessary but not sufficient evidence. A common wrong turn is picking a single control country instead of a pool, which makes the estimate fragile to that one country's own idiosyncratic event. Another is choosing the post-window to end exactly where the effect looks best, which is a form of p-hacking on window choice; pre-commit to the window before looking at results. If a competing explanation (payment outage, campaign pause) has the exact same start date as the release, DiD alone cannot separate the two: you need the funnel-step and error-rate breakdown to distinguish a release-caused break from something else that broke at the same time.

Direct answer

Match the test to the outcome's data type and the sample-size regime: (a) continuous outcome, two independent groups, large enough n, use a two-sample t-test (Welch's by default); (b) binary outcome, two large groups, use a chi-square test of independence; (c) binary outcome, small samples with sparse cells (several zeros), use Fisher's exact test instead of chi-square, since the chi-square approximation breaks down when expected cell counts are small.

Structured elaboration

Worked example

(b) Chi-square, large groups. Control converts 500/10,000 (5.0%), treatment converts 640/10,000 (6.4%).

This gives $\chi^2 \approx 18.23$, $p \approx 0.00002$ (verified with scipy.stats.chi2_contingency, no continuity correction). Every expected cell is well above 5 (570 and 9,430), so the chi-square approximation is trustworthy here.

(c) Fisher's exact, small and sparse. A small pilot with $n=10$ per group, one arm with zero conversions: control 0/10 converted, treatment 4/10 converted.

The chi-square approximation's expected counts here are only 2 per cell, well under the rule-of-thumb minimum of 5, and one observed cell is zero, so chi-square is not trustworthy. Fisher's exact test computes the exact p-value directly from the hypergeometric distribution instead: $p \approx 0.087$ (verified with scipy.stats.fisher_exact), not significant at 0.05 despite the visually large 0% vs 40% gap, which is exactly the point: small samples have limited power no matter which valid test you use, and Fisher's gives you an honest, exact answer rather than an unreliable approximation that happens to look more dramatic.

Trade-offs & pitfalls

• The "expected count ≥5" rule for chi-square is a rule of thumb, not a hard law; treat it as a trigger to switch to Fisher's exact rather than agonizing over whether 4.8 versus 5.2 matters.
• Fisher's exact test is always valid for a 2x2 table regardless of sample size, so when in doubt at small-to-moderate n it's a safe default, not just an emergency fallback.
• When (b) or (c) needs to control for covariates (device, geography, prior activity), logistic regression generalizes both chi-square and Fisher's, but needs enough events per predictor (a common rule of thumb is roughly 10 events per additional variable) or it will produce unstable estimates; Firth's penalized logistic regression is the standard fix when events are too sparse for ordinary logistic regression, which is common in exactly the small/sparse scenario (c) describes.
• For (a), defaulting straight to Mann-Whitney "to be safe" throws away information if the data are actually reasonably well-behaved; check the diagnostic first rather than skipping straight to the nonparametric option.

Direct answer

A short-term activation lift alongside a longer-term retention drop is a real conflict, not necessarily a data artifact, and it needs to be investigated with the original randomization preserved (extend the same experiment's follow-up window rather than starting a fresh comparison), segmented for who's driving each signal, and probed for a causal mechanism connecting the two before deciding on rollout. The default posture should be caution: an unexplained long-term harm signal outweighs a confirmed short-term gain until you understand why.

Structured elaboration

Step 1: Rule out measurement issues first

• Confirm activation and retention are measured on comparable, correctly defined cohorts and windows (7-day activation vs. 30-day retention, using the same original randomized assignment, not a fresh comparison group).
• Check for differential attrition or instrumentation gaps between arms that could fabricate an apparent retention drop.

Step 2: Extend the original experiment's analysis window

Because randomization already balanced the arms, the most reliable next step is to look at 30/60/90-day outcomes for the same randomized population (intention-to-treat), rather than launching a new study. Compute survival/retention curves for each arm to see whether the -5% retention gap is a single-point artifact or a persistent, widening pattern.

Step 3: Segment for heterogeneity

Break down both metrics by acquisition source, user tenure, and engagement depth. A common pattern: the feature drives short-term activation disproportionately among low-intent or low-fit users (e.g. driven in by a prompt rather than genuine interest), who then churn faster than they would have otherwise, while genuinely engaged users show a real, durable lift. If so, the aggregate numbers mask two very different underlying stories.

Step 4: Look for a causal mechanism, not just a correlation

• Mediation analysis: does the activation event itself (not just being in the treatment arm) predict the later retention drop? If treated users who activate churn at the same rate as treated users who don't, the activation event itself probably isn't the mechanism, and something else about the treatment experience is.
• Downstream engagement: check whether activation in the treatment arm is "shallow" (a one-time triggered action) versus "deep" (leads to habitual use), since shallow activation with no follow-on engagement is a classic precursor to a later retention dip.

Step 5: Design follow-up experiments if the mechanism is still unclear

• A properly powered extension of the same experiment to 60-90 days, since post-hoc re-analysis alone often lacks power to distinguish a real drop from noise.
• Permanent holdout cohorts (e.g. 5-10% of eligible users held out indefinitely) to monitor whether the long-term pattern persists at scale after any rollout decision.
• Variant experiments that try to preserve the activation gain while removing the suspected harmful mechanism (e.g. delaying an onboarding prompt, or restricting a feature to users who show organic intent).

Rollout options when signals conflict

Worked example

Suppose the +10% activation lift and -5% 30-day retention drop are both measured on the same original randomized 50,000-user experiment (25,000 per arm), and a segment breakdown shows paid-channel users are 30% of the population, with organic users showing only a small residual effect on each metric (+3 points activation, 0 points retention). Solve for what the paid segment's effect must be to reconcile with the stated aggregate:

f_paid = 0.30
organic_activation, organic_retention = 3.0, 0.0
agg_activation, agg_retention = 10.0, -5.0

paid_activation = (agg_activation - (1 - f_paid) * organic_activation) / f_paid
paid_retention  = (agg_retention  - (1 - f_paid) * organic_retention)  / f_paid
# paid_activation = 26.33, paid_retention = -16.67

recon_activation = f_paid * paid_activation + (1 - f_paid) * organic_activation
recon_retention  = f_paid * paid_retention  + (1 - f_paid) * organic_retention
# recon_activation = 10.0, recon_retention = -5.0  (matches the given aggregate exactly)

Reconciling the aggregate this way implies the paid segment alone would need roughly a +26 point activation lift and a -17 point retention drop to produce the observed +10/-5 blend given a 30% population share, a far more extreme effect than the organic segment's, whose lift and drop are both small. That pattern (the harm concentrated almost entirely in the same segment driving the gain) is consistent with the feature disproportionately activating lower-intent, paid-acquired users who then churn faster than they would have without the prompt, rather than the feature being broadly harmful. It reframes the decision from "kill the feature" to "restrict the feature's trigger to organic or higher-intent users, and re-test," which preserves the real gain while addressing the plausible mechanism. This is a reverse-engineered illustration of the reasoning, not a claim about the real experiment; the actual segment split would come from re-running this exact arithmetic on the real cohort data.

Trade-offs & pitfalls

• Re-running a brand-new experiment instead of extending the original one throws away your randomization. A fresh post-hoc comparison of "activated vs. not" within the treatment arm is confounded by whatever made those users activate in the first place; always prefer intention-to-treat analysis on the original assignment.
• A segment-level story ("it's concentrated in paid-acquired users") is a hypothesis, not a proven mechanism, until validated with a targeted follow-up experiment; don't ship a segmentation-based fix without testing it.
• Waiting for more data has a real cost. Every week of delay on a genuinely beneficial feature is lost value; the goal is a fast, well-powered follow-up, not indefinite caution.
• Permanent holdouts are expensive at scale (real users denied a feature indefinitely) and need a clear sunset plan once enough long-term signal has accumulated.
• A short-term metric optimized in isolation is a known failure mode. Any team that only monitors 7-day activation as its headline metric is structurally blind to exactly this kind of delayed harm; the org-level fix is to require a longer-horizon guardrail on every launch, not just a one-off investigation after the fact.

Direct answer

Statistical power is the probability that an experiment will detect a true effect of a given size, i.e. correctly reject a false null hypothesis; it's commonly targeted at 80-90%. The minimum detectable effect (MDE) is the smallest effect size the experiment is designed to reliably detect at a chosen power and significance level. MDE should be set from the smallest effect that would actually change a business decision, not from what's convenient to detect; that choice then mechanically determines the required sample size, since a smaller MDE needs a disproportionately larger sample. Beyond sample size, the other levers for power are lowering variance, extending duration, and using a more efficient design, each with its own cost.

Structured elaboration

Choosing an MDE from business context

• Minimum actionable lift: pick the smallest effect that would be worth the engineering and rollout cost if it were real. Below that, even a confirmed effect isn't worth acting on.
• Baseline variability: high-variance metrics need a larger MDE to keep the sample size and test duration feasible; this is a real constraint, not just a preference.
• Traffic and time budget: if the team can only run the test for a fixed number of weeks at a fixed traffic volume, that budget effectively sets the smallest MDE the test can resolve. Work backward from the budget if the business's ideal MDE isn't affordable.
• Compounding effects: if small lifts stack across many surfaces or are running continuously, a smaller MDE can be worth the extra cost; a one-off feature test usually isn't.

How MDE drives sample size

For a two-proportion test, required sample size per arm scales as:

where $\Delta$ is the MDE (absolute difference) being targeted. Because $\Delta$ is squared in the denominator, halving the MDE roughly quadruples the required sample, and therefore roughly quadruples the calendar time needed at fixed daily traffic.

Levers to raise power without changing alpha

Worked example

Baseline conversion $p_0 = 0.10$, $\alpha=0.05$ two-sided, power $=0.80$. Required sample size per arm at two different MDEs, using the standard two-proportion formula:

(computed directly from the formula above with scipy.stats.norm for the critical values)

Halving the MDE from 0.02 to 0.01 multiplies the required sample by about 3.8x, consistent with the inverse-square relationship between $\Delta$ and $n$. If daily eligible traffic is 1,000 users per arm, the 0.01 MDE needs about 15 days; the 0.02 MDE needs about 4.

Trade-offs & pitfalls

• Picking the smallest MDE that "sounds rigorous" without checking the implied sample size and duration is a common planning mistake; always translate the MDE choice into a concrete number of days before committing to it.
• An MDE chosen after looking at early results (rather than fixed up front from the business case) turns the power calculation into after-the-fact justification, not a real design constraint.
• Reducing variance or adding covariates raises power "for free," but only if the adjustment is decided and validated before the experiment starts; adjusting for whatever covariate happens to make the result significant is a form of p-hacking.
• Accepting low power to launch faster is sometimes the right call for cheap, reversible changes, but it should be a stated decision (documented as "we are only powered to detect large effects"), not a silent consequence of an unexamined MDE.

Direct answer

Post-deployment drift after a positive experiment usually traces back to one of four causes: an implementation mismatch between the experiment code path and the production code path, a change in who is now exposed to the treatment (selection effects the randomized experiment didn't have), a genuine shift in the underlying population or its behavior (non-stationarity), or a metric-definition mismatch between how the experiment measured success and how production monitoring measures it. Diagnose in that order, since each has a specific, checkable signature, and only escalate to a full follow-up experiment once the cheaper checks are exhausted.

Structured elaboration

Diagnostic order and what each check looks for

flowchart TD
  A[Drift detected] --> B{Timing matches deploy exactly?}
  B -->|no| C[Look for an unrelated concurrent change]
  B -->|yes| D{Replay offline requests through prod model}
  D -->|outputs differ from experiment| E[Implementation mismatch]
  D -->|outputs match| F{Exposure population balanced vs experiment?}
  F -->|no, correlates with covariates| G[Selection bias / rollout targeting]
  F -->|yes| H{Feature or label distribution shifted?}
  H -->|yes| I[Non-stationarity / concept drift]
  H -->|no| J[Check metric definitions and ETL for mismatch]

1. Timing and scope. Confirm the drift's start date lines up with the deploy, not with something else happening at the same time (a promotion, a seasonal shift, a different team's unrelated release). Segment by cohort, geography, device, and time-of-day; a drift confined to one subgroup points toward selection or rollout targeting rather than a global implementation problem.
2. Implementation differences. Replay a sample of real production requests offline through the exact model and feature pipeline used in the experiment. If replayed outputs match what the experiment predicted, the model and features are behaving as tested, and the problem lies elsewhere; if they diverge, you likely have a preprocessing, versioning, or schema mismatch between experiment and production.
3. Selection bias / population shift. Check whether the population actually exposed in production differs from the randomized experiment population: compute a standardized mean difference or population stability index (PSI) on key covariates between the experiment cohort and the post-launch cohort, and consider training a simple classifier to predict "experiment vs. post-launch" from covariates; if it's meaningfully better than a coin flip, something about who is now in the treatment group has shifted.
4. Non-stationarity / concept drift. If the exposed population looks the same but outcomes have shifted, distinguish covariate drift ($P(X)$ changed: the mix of users or inputs you're now seeing has shifted, e.g. more mobile users than during the experiment, even though the model behaves the same given a fixed input) from concept drift ($P(Y\mid X)$ changed: the same kind of input now leads to a genuinely different outcome, e.g. users who look the same on paper now convert differently) using calibration curves and residual analysis over time.
5. Metric-definition mismatch. Verify the production monitoring metric uses the identical aggregation window, dedup rules, and definition as the experiment's primary metric; a common, boring cause of "the lift disappeared" is that it never disappeared, the two pipelines are measuring different things.
6. Logging and instrumentation check. Compare application error-log volume and event-schema version between the experiment window and the post-launch window: a spike in errors, a dropped-event rate, or a silent schema/version bump on the code path the release touched is a fast, checkable signal distinct from the PSI/replay checks above, and cheap enough to run in parallel with step 1 rather than only after steps 2-5 come back inconclusive.

Causal reasoning

Draw a causal graph with nodes for user covariates, eligibility/exposure, treatment assignment, and outcome. In the randomized experiment, assignment was independent of covariates by design; in production, rollout logic, targeting rules, or a canary schedule can reintroduce a path from covariates to exposure that didn't exist in the RCT (randomized controlled trial), which is exactly what selection-bias checks in step 3 are testing for.

Worked example

Comparing the covariate distribution of session-length buckets between the original experiment cohort and the post-launch cohort (5 buckets, proportions of total traffic):

(computed directly from the table above)

A commonly-used heuristic treats PSI under 0.1 as no meaningful shift, 0.1 to 0.25 as a moderate shift worth investigating, and above 0.25 as a significant shift. At 0.292, this population's session-length mix has shifted substantially between the experiment and post-launch windows, which is a concrete, checkable signal pointing toward selection or population change (step 3) rather than an implementation bug, and it tells you exactly which covariate to dig into next.

Trade-offs & pitfalls

• Jumping straight to "the effect wasn't real" or straight to "run a new RCT" skips the cheap, fast checks (timing, replay, PSI) that usually localize the cause in hours instead of the days a new experiment takes.
• A clean replay (implementation matches) does not by itself rule out selection bias; it only rules out one of four causes, and the checks need to be worked through in order rather than stopping at the first one that comes back clean.
• Re-running a full randomized experiment is the most conclusive fix, but it's also the slowest and most expensive; reserve it for when the offline checks are genuinely inconclusive, or when you need to prove causality (e.g. before deciding to roll back a launch that stakeholders are attached to).
• Instrumenting before the next rollout (continuous PSI monitoring, replay-based canaries, a classifier-based "is this cohort still like the experiment cohort" check) is cheaper than diagnosing after the fact, and is the right place to invest once this incident is resolved.