Statistical Inference and Hypothesis Testing Questions

Direct answer

Detect instrumentation drift by first ruling out a randomization problem (check the sample ratio between arms is still balanced), then checking whether the event-count drop is specific to the treatment arm's own behavior over time rather than a global change. Quantify the bias by comparing each arm's post-rollout event rate to its own pre-rollout baseline. Short-term, stop trusting the affected metric for the go/no-go decision and either recompute from an unaffected source or bound the result with a sensitivity range; long-term, fix the pipeline that's silently dropping events and add monitoring so this doesn't require manual detection next time.

Structured elaboration

Detect

• Sample ratio mismatch (SRM) check first. Before suspecting the event-counting logic, confirm the number of users assigned to each arm is still balanced as randomized; if the ratio itself is off, that's an assignment problem, not (only) an instrumentation one.
• Compare each arm's post-rollout event rate to its own pre-rollout baseline, not just treatment vs. control post-rollout. This distinguishes "the treatment code path is now under-logging events" from "both arms are naturally busier or quieter than before for unrelated reasons."
• Change-point detection on the event-per-user rate over time (e.g. a CUSUM, cumulative sum control chart that flags when a rolling sum of deviations from a baseline crosses a threshold) to pin down exactly when the drop started, which should line up with the rollout timestamp if the rollout caused it.

Quantify

• Fit an expected-event-count model on pre-rollout data (event count as a function of user features and time) for both arms, then compare each arm's actual post-rollout counts to what that model predicts; the gap is your estimated undercount.
• Bootstrap a confidence interval around that estimated gap rather than reporting a single point-estimate correction, since the correction itself is uncertain.

Correct (short-term mitigation for a live experiment)

• Stop using the affected metric as the basis for a ship/no-ship decision until the bias is understood.
• If a parallel, unaffected data source exists (server-side logs vs. client-side events, for instance), recompute the primary metric from that source instead.
• If no clean source exists, report the result with an explicit sensitivity range: "best case" (no bias) and "worst case" (full estimated undercount applied as a correction), and hold the decision until the range narrows or the pipeline is fixed.

Long-term platform fixes

• Add a reconciliation job that compares client-reported counts to server-side counts daily, with alerting on divergence past a set threshold.
• Add contract tests between the client SDK and the backend so a schema or field change can't silently start dropping events without a build failing.
• Add synthetic "canary" traffic (a small stream of known, scripted events) so a rollout that breaks logging is caught within minutes, not discovered later during experiment analysis.

Worked example

A rollout shipped to 50% of traffic. First, confirm the split itself is still healthy: observed 100,000 control users vs. 99,850 treatment users.

(a standard one-degree-of-freedom SRM check; computed with scipy.stats.chi2.sf)

The split is healthy (p = 0.74, no evidence of an assignment problem), so the randomization itself isn't the issue. Now compare event rates (events per user per day):

Treatment's post-rollout rate is about 18.3% lower than its own pre-rollout baseline, and about 17.9% lower than control's post-rollout rate over the same window, while control barely moved. Because the SRM check ruled out an assignment problem and control stayed essentially flat, this pattern (a rate drop confined to the treatment arm's own code path, right at the rollout point) is consistent with the new client build under-logging events specifically in the treatment path, not with a population or randomization issue.

Trade-offs & pitfalls

• Skipping the SRM check and jumping straight to "instrumentation is under-reporting" risks missing a simpler and more common explanation: the randomization itself is broken, which needs a different fix (rebalance or rebuild the assignment logic) than a logging bug does.
• A model-based expected-count correction is only as good as the pre-rollout model; if the treatment itself was expected to change user behavior (which is the whole point of the experiment), naively treating any post-rollout gap as "bias" risks correcting away part of the real treatment effect along with the real bias. Cross-check with an unaffected parallel data source whenever one exists, rather than relying on the model alone.
• Pausing the decision has a real cost (delayed launch, delayed learning), so it should be treated as a genuine trade-off against shipping on uncertain data, not a default reflex for every anomaly.
• Long-term fixes (contract tests, reconciliation jobs, canaries) only pay off if someone owns the alerting; a dashboard nobody watches is not a monitoring strategy.

Direct answer

With randomization off the table, I would pick an identification strategy based on what natural source of variation the data actually contains (a discontinuity, a staggered rollout, a plausible instrument, or just rich confounders), estimate the effect with that strategy, run the diagnostics specific to it, and report the result alongside a sensitivity analysis quantifying how large an unobserved confounder would have to be to overturn the conclusion. No single observational method is "the" answer; the choice is dictated by what assumption you're most comfortable defending.

Structured elaboration

Method comparison:

Diagnostics per method:

• PSM/IPW/DR: balance checks (standardized mean differences on covariates pre- and post-weighting should be near zero), propensity overlap (trim regions with extreme propensity scores where treated and control don't overlap).
• DiD: pre-treatment event-study plot to visually and statistically test parallel trends; placebo tests using a fake treatment date before the real one.
• RD: McCrary density test for manipulation of the running variable at the cutoff (checks for a suspicious jump in how many observations sit just above vs. just below the cutoff; a smooth density crossing the cutoff is consistent with units landing on either side as-if randomly, while a jump suggests units found a way to manipulate which side they ended up on); continuity of other covariates across the cutoff (if covariates jump too, the cutoff isn't as-if random).
• IV: first-stage F-statistic above the conventional threshold of 10 to rule out a weak instrument (the first-stage F-statistic measures how strongly the instrument actually predicts the endogenous treatment; below roughly 10, the instrument barely moves treatment, so the downstream IV estimate becomes unstable and can be badly biased even in a large sample); overidentification tests (Sargan/Hansen) if more instruments than endogenous regressors are available (these check whether the extra instruments, beyond the one strictly needed, are mutually consistent, i.e. they all point to the same causal effect, rather than one of them secretly correlating with the outcome through a channel other than the treatment); a placebo outcome the instrument shouldn't affect.
• All methods, given time series: check for autocorrelation and use clustered or Newey-West standard errors rather than assuming independent errors (both widen the usual standard-error formula to account for observations that aren't fully independent: clustered SEs correct for observations grouped together, e.g. repeated purchases from the same customer or store, and Newey-West SEs correct for observations that are near each other in time; without either correction, standard errors computed as if every observation were independent understate the true uncertainty and make the result look more precise than it is).

Which method to reach for first. Of these five, PSM/IPW and DiD are the ones worth being able to set up and diagnose cold, since a rich-covariate dataset or a staggered rollout comes up far more often in practice than a clean cutoff or a defensible instrument. RD and IV are worth recognizing and reaching for when the data structure obviously supports them, but their diagnostics (the McCrary test, the first-stage F-statistic, overidentification tests) are more specialized and fine to look up rather than memorize the exact mechanics of.

Sensitivity analysis. Because no observational diagnostic can prove the absence of unobserved confounding, report how strong a confounder would need to be to overturn the result: Rosenbaum bounds for matching estimators (a sensitivity parameter, usually called Gamma, expressing how much an unobserved confounder would need to distort the odds of treatment assignment before it could explain away the observed effect), or Oster's delta (which uses how much the estimate moves when observed controls are added, to bound how much it could move from unobserved ones) for regression-based estimates.

Worked example

A pricing team wants the causal effect of a 10% price increase rolled out to one customer segment on a specific date, with rich panel data (weekly revenue per customer, multiple pre-period weeks) and no experiment. Difference-in-differences is the natural first strategy: compare the treated segment's revenue trend before and after the rollout to a similar untreated segment's trend over the same period. Concretely, using the DiD estimator:

if pre-period weekly revenue for the treated segment averaged $48,000 and the control segment averaged $46,500, and post-period revenue is $45,000 (treated) and $46,200 (control), then:

The raw treated-group drop is $3,000/week, but $300 of that would have happened anyway (the control group's own small decline), so the DiD estimate attributes $2,700/week to the price increase specifically. That estimate is only credible if the pre-period trends were parallel; if the event-study plot showed the treated segment already declining faster than control before the price change, the $2,700 figure would be biased and a different identification strategy (or a synthetic control blending multiple comparison segments to better match the pre-trend) would be needed instead.

Trade-offs & pitfalls

Every one of these methods trades a testable diagnostic for an untestable core assumption; passing all the diagnostics (balance, parallel trends, strong first stage) increases confidence but never proves the identifying assumption holds, which is why the sensitivity analysis is not optional decoration; it's the honest statement of how much residual doubt remains. A common wrong turn is reaching for propensity-score matching by default because it's the most familiar method, when a regression discontinuity or a natural experiment in the data would give much stronger identification if one exists; always check for a usable cutoff or instrument before falling back to "control for everything we measured." Another pitfall specific to time-series pricing data is ignoring anticipation effects: if customers change behavior before the price increase takes effect (stockpiling, downgrading in advance), the "pre-period" is contaminated and both DiD and RD estimates will be biased toward understating the true effect.

Direct answer

Deciding whether a metric move is signal or noise combines a statistical test (a p-value and confidence interval against a null of "no change") with an operational monitoring layer (a control chart) that turns that test into a repeatable alerting rule, while explicitly correcting for the fact that you are testing repeatedly. The two things that break naive daily "is today significant" checks are running many implicit tests over time (multiple comparisons) and not budgeting for the power needed to actually detect the size of change that matters operationally.

Structured elaboration

Statistical building blocks. A p-value measures evidence against "no true change"; a confidence interval quantifies plausible magnitudes, which matters more operationally since a statistically significant but tiny move may not be actionable. Power is the pre-specified probability of detecting a real change of a chosen minimum size (MDE); under-powered daily checks produce both false negatives and, when they do trigger, inflated effect-size estimates (the "winner's curse" of only looking at significant results).

Control charts turn "is this one point unusual" into an operational rule:

The Shewhart chart is the one to know cold and lead with: a fixed threshold on how far today's point sits from the historical mean, easy to explain and reason about. EWMA and CUSUM are refinements for catching smaller, slower drifts once the metric's noise characteristics call for it; treat them as depth beyond the baseline, not the first thing to reach for.

Multiple testing. Monitoring $m$ metrics or checking one metric on $m$ days each at a naive $\alpha=0.05$ does not give a 5% overall false-alarm rate - it compounds. Family-wise error control (Bonferroni: $\alpha/m$ per test) is conservative but simple; false discovery rate control (Benjamini-Hochberg) is the standard default for a metrics dashboard with many tracked KPIs, since it trades a few tolerable false alarms for more power to catch real changes; Bonferroni is worth naming as the simpler, more conservative fallback when a stricter guarantee is required, not as the first tool to reach for.

Worked example

Per-point false-alarm rate of a 3-sigma Shewhart chart, and the compounding effect of naive repeated daily testing:

import numpy as np
from scipy import stats

p_false_alarm_per_point = 2 * (1 - stats.norm.cdf(3))
alpha, n_days = 0.05, 30
p_at_least_one_naive = 1 - (1 - alpha) ** n_days
alpha_bonferroni = alpha / n_days

print(f"3-sigma false-alarm rate per point                    = {p_false_alarm_per_point:.5f}  ({p_false_alarm_per_point*100:.3f}%)")
print(f"P(>=1 false alarm over 30 daily naive-alpha=0.05 checks) = {p_at_least_one_naive:.3f}   <- {p_at_least_one_naive*100:.1f}%, not 5%")
print(f"Bonferroni-corrected per-test alpha for 30 tests        = {alpha_bonferroni:.5f}")

Output:

3-sigma false-alarm rate per point                    = 0.00270  (0.270%)
P(>=1 false alarm over 30 daily naive-alpha=0.05 checks) = 0.785   <- 78.5%, not 5%
Bonferroni-corrected per-test alpha for 30 tests        = 0.00167

Now Benjamini-Hochberg on 30 metrics checked on one day, fully specified so it reproduces exactly: 27 metrics with no true effect (z-scores drawn from $N(0,1)$) and 3 metrics with a real effect (z-scores drawn from $N(3,1)$, a solidly detectable signal), pinned seed 5, target FDR $q=0.10$:

rng = np.random.default_rng(5)
m, n_null, n_alt = 30, 27, 3

z_null = rng.normal(0, 1, size=n_null)   # 27 truly null metrics
z_alt = rng.normal(3, 1, size=n_alt)     # 3 metrics with a real effect
z_all = np.concatenate([z_null, z_alt])
is_true_effect = np.array([False] * n_null + [True] * n_alt)

p_all = 2 * (1 - stats.norm.cdf(np.abs(z_all)))
order = np.argsort(p_all)
p_sorted = p_all[order]
q = 0.10
thresholds = (np.arange(1, m + 1) / m) * q
below = p_sorted <= thresholds
max_k = np.max(np.where(below)[0]) + 1 if below.any() else 0
rejected_idx = order[:max_k]
n_true_positive = is_true_effect[rejected_idx].sum()

print(f"BH at q=0.10: rejected {max_k} of {m} tests")
print(f"  of the {max_k} rejected: {n_true_positive} were true effects, {max_k - n_true_positive} were false discoveries")

Output:

BH at q=0.10: rejected 2 of 30 tests
  of the 2 rejected: 2 were true effects, 0 were false discoveries

With only naive per-test $\alpha=0.05$ and no correction, running 30 independent daily checks gives a 78.5% chance of at least one false alarm purely from repeated testing, even if nothing real ever changed. BH recovers both detectable true signals in this draw while controlling how many of the flagged metrics are expected to be false discoveries.

Trade-offs & pitfalls

The most common operational mistake is setting a single global $p<0.05$ threshold and applying it independently to every metric, every day, with no correction - this guarantees a steady stream of false alarms that erodes trust in the alerting system faster than any single missed detection would. Overcorrecting is also a real failure mode: strict Bonferroni across hundreds of dashboard metrics can suppress genuine early warnings, so tiering alerts (informational at a loose threshold, actionable at a stricter one requiring the CI to exclude the pre-specified MDE, critical requiring the signal to replicate across two consecutive windows) usually serves the business better than a single p-value gate. Control charts assume roughly stationary, non-seasonal behavior; applying a Shewhart or EWMA chart directly to a metric with strong day-of-week seasonality will fire constantly on seasonal swings unless the chart is built on seasonally-adjusted residuals or a same-day-last-week baseline instead of the raw series.

Direct answer

The p-value tells you whether the observed difference is unlikely to be pure chance under "no effect." The confidence interval (CI) tells you the range of effect sizes the data are consistent with. Effect size tells you how big the difference actually is, in units the business cares about. You need all three together: a tiny p-value with a tiny effect size is not a reason to act, and a wide confidence interval is a warning that the point estimate alone is not precise enough to bet on.

Structured elaboration

P-value

• What it is: the probability of seeing data this extreme (or more) if there were truly no difference between the groups.
• What it tells you: whether "no effect" is a poor explanation for what you observed.
• What it does NOT tell you: the probability the treatment works, or how large the effect is. A p-value of 0.001 and a p-value of 0.04 can come from effects of the same practical size, just with different sample sizes or noise.

Confidence interval

• What it is: a range of effect sizes that are plausible given the data and the model, at a chosen confidence level (typically 95%).
• What it tells you: both the size of the estimated effect and how precisely it's been measured. A narrow interval means the data pin the effect down tightly; a wide one means there's a lot of remaining uncertainty.
• What it does NOT tell you: it is not literally "a 95% probability the true value is in this specific interval." the 95% describes the long-run behavior of the procedure across repeated experiments, not a probability statement about this one realized interval.

Effect size

• What it is: the actual magnitude of the difference, absolute (percentage points) or relative (percent lift).
• What it tells you: whether the change is worth the engineering cost and rollout risk, independent of whether it's statistically significant.
• What it does NOT tell you: on its own, whether the estimate is reliable. An effect size without a confidence interval could be pure noise.

Worked example

A checkout test: baseline click-through $p_0 = 0.080$, treatment $p_1 = 0.086$, $n = 20{,}000$ per arm.

Pooled test statistic:

95% CI for the absolute difference (unpooled standard error):

(all values computed directly from these formulas with scipy.stats.norm)

Summary sentence for the PM: "Click-through rose from 8.0% to 8.6%, a 7.5% relative lift (p = 0.030). We're 95% confident the true absolute lift is somewhere between 0.06 and 1.14 percentage points. It's a real improvement, though the interval is wide enough that the low end is a modest win, not a blockbuster."

Trade-offs & pitfalls

• Presenting only the p-value invites the "significant equals big and certain" misread. Always pair it with the interval and the effect size in the business's own units.
• A p-value just under 0.05 with a confidence interval that barely excludes zero (as in this example, lower bound 0.0006) is a different story than a p-value of 0.0001 with a tight interval far from zero. Treat "significant" as a single bit of information, not the whole picture.
• Wide confidence intervals are common with realistic sample sizes and should be surfaced, not hidden. Narrowing the interval requires either more data or a less noisy metric, not a different way of describing the same data.

Direct answer

Assigning by session instead of by user breaks the core promise of randomization: users with more sessions (often a specific demographic, e.g. more engaged or mobile-heavy users) get more chances to land in whichever arm the assignment happens to favor for them, so the two arms end up systematically different on user characteristics, not just on treatment. That confound means the estimated treatment effect is contaminated by a demographic difference between arms, and no amount of additional data collection under the same buggy assignment will fix it, since the bias doesn't shrink with sample size.

Structured elaboration

Why this biases the estimate

Proper randomization guarantees that, in expectation, treatment and control are balanced on every covariate, observed or not. Session-based assignment breaks this guarantee whenever session behavior correlates with anything that also affects the outcome: for instance, if the same user is more likely to be re-assigned to the same or different arm across sessions in a way that isn't uniform, or if certain user types (heavy, frequent-session users) end up over- or under-represented in one arm because of how the hashing or bucketing interacted with session versus user identity. The result is a mix of the true treatment effect and a demographic difference between arms, and the two are not separable using only the experiment's headline outcome metric.

Diagnostics to quantify imbalance

1. Balance tests on pre-treatment covariates. For each key covariate (device type, tenure, geography, prior activity level), compute the standardized mean difference (SMD) between arms:

A common flag threshold is $|SMD| > 0.1$; run this across every covariate you have, not just the one you suspect.

2. Predictive check. Train a simple classifier (e.g. logistic regression) to predict treatment assignment from pre-treatment covariates. An AUC (Area Under the ROC Curve, a 0-to-1 score where 0.5 means the classifier does no better than a coin flip and 1.0 means it perfectly separates the two arms) meaningfully above 0.5 is direct evidence that assignment isn't independent of user characteristics, which is exactly what should be true under correct randomization.
3. Time and cohort plots. Plot the arm split over time and by device/geography to spot whether the imbalance is a constant offset or grows for higher-session users specifically, which helps confirm the mechanism (session-based hashing) rather than just detect that something is wrong.

Remediation options

Worked example

Simulated imbalance where session-based assignment happens to over-represent mobile users in the treatment arm, pinned seed:

import numpy as np
rng = np.random.default_rng(seed=55)

n_ctrl, n_treat = 3000, 3000
mobile_ctrl = rng.binomial(1, 0.45, n_ctrl)   # baseline mobile share
mobile_treat = rng.binomial(1, 0.58, n_treat) # imbalance introduced by the bug

p_ctrl, p_treat = mobile_ctrl.mean(), mobile_treat.mean()  # 0.451, 0.583
pooled_var = (p_ctrl*(1-p_ctrl) + p_treat*(1-p_treat)) / 2
smd = (p_treat - p_ctrl) / np.sqrt(pooled_var)              # 0.268

Here, mobile-user share is 45.1% in control versus 58.3% in treatment, giving $SMD = 0.268$, well above the 0.1 flag threshold. If mobile users also tend to have different baseline conversion rates than desktop users (plausible, and easy to check directly), this single covariate imbalance alone is large enough to meaningfully distort the estimated treatment effect, independent of whatever the treatment itself does. This is the kind of concrete, quantified result that justifies escalating from "we have a bug" to "we cannot trust this experiment's headline number without correction."

Trade-offs & pitfalls

• Adjustment methods only fix observed imbalance. If the same assignment bug correlates with something you didn't measure (e.g. a behavioral trait that also drives session frequency), covariate adjustment and reweighting cannot detect or correct for it; only a clean re-run removes that risk entirely.
• Large propensity weights are a warning sign, not just a nuisance. If the reweighting requires extreme weights to balance the arms, it usually means limited overlap between the imbalanced groups, and the adjusted estimate becomes unstable and hard to trust.
• A quick "re-run it" instinct isn't free. If the feature has since shipped elsewhere, seasonality shifted, or the bug affected historical data used for other decisions, a clean re-run needs its own sanity checks, not just a fresh randomization.
• The diagnostic step matters even if you plan to re-run anyway. Quantifying the imbalance (and whether it's large enough to plausibly explain the observed effect) tells you whether the original result should be treated as "wrong" or merely "less certain," which affects any decisions already made based on it.
• Prevention beats remediation. Once found, the underlying fix is an engineering one: assignment key auditing and unit tests that verify hashing is done on the intended identity field, since this exact class of bug (session vs. user key) recurs across teams that don't have that check.