Methodological Rigor and Experimental Validation Questions

Cover experimental design and validation best practices and the trade offs between novelty and reproducibility. Topics include selection of controls and baselines, primary and guardrail metrics, ablation studies, error analysis, statistical significance and confidence in results, reproducibility practices, robustness checks, and avoidance of common pitfalls and biases. Also demonstrate critical thinking by proposing alternative approaches and diagnostics when initial results are inconclusive. Interviewers will probe for concrete validation strategies and an ability to justify methodological choices.

MediumTechnical

39 practiced

Your team likes to check the experiment dashboard every day and sometimes calls a test 'done' as soon as it crosses significance. What's wrong with that, and how would you fix the process?

Sample Answer

**The Problem**Repeatedly checking a p-value and stopping as soon as it dips below 0.05 (peeking) inflates the true false positive rate well above 5%, because you're effectively giving the test many chances to cross the threshold by chance alone. Similarly, running many metrics or many segment cuts and reporting whichever one is significant is a form of multiple testing that has the same effect.

**Why It Happens**A p-value under a fixed sample size assumes you look once, at the end. Checking daily and stopping early means you're really running a sequence of tests, and the probability that at least one of them falsely crosses 0.05 climbs quickly, sometimes to 20 to 30 percent over a multi-week test.

**Fixes**- **Pre-register the sample size or duration** using a power analysis, and commit to not making a ship decision until that point is reached.- **Use sequential testing methods** (like alpha-spending or always-valid p-values) if the team genuinely needs to monitor and possibly stop early, since these methods correct for repeated looks.- **Correct for multiple comparisons** when checking many metrics or segments, for example a Bonferroni or Benjamini-Hochberg adjustment:

text

adjusted alpha = original alpha / number of comparisons   (Bonferroni)

Intuition: if you're testing 10 metrics at once, each individual test needs a stricter bar so the overall chance of a false positive across all 10 stays near 5%.- **Separate monitoring from decision-making**: dashboards can be checked daily for sanity (e.g. detecting a bug), but the significance call is only made once, at the pre-committed endpoint.

**Common Pitfall**Treating 'we saw significance on day 4' as equivalent to 'we ran a properly powered two-week test,' when the two have very different false positive guarantees.

EasyTechnical

41 practiced

You're designing an A/B test for a new checkout flow. Walk me through how you'd choose the control group.

EasyTechnical

43 practiced

Before building a fancy churn prediction model, why would you bother building a naive baseline first, and what would that baseline look like?

HardTechnical

38 practiced

The business wants to claim that a new feature caused an increase in retention, but you can't run a randomized experiment (it already shipped to everyone). How would you validate that causal claim?

Sample Answer

**Framing**Without randomization, any comparison you make is vulnerable to confounding: the group that used the feature may differ from the group that didn't in ways that also affect retention, independent of the feature itself. The goal is to approximate what a randomized experiment would have shown, using observational methods, while being explicit about the assumptions each method relies on.

**Approach**- **Start by naming the likely confounders**: users who opt into or discover a new feature are often already more engaged, which would inflate retention regardless of the feature. List these before choosing a method, since the method needs to actually address them.- **Difference-in-differences**, if you have a natural comparison group (for example, the feature rolled out to one region before another): compare the change in retention over time between the exposed and not-yet-exposed groups, rather than comparing levels. This controls for any pre-existing gap between groups and for global trends affecting everyone.

text

DiD estimate = (Retention_after - Retention_before)_exposed - (Retention_after - Retention_before)_unexposed

Intuition: this isolates the change that's unique to the exposed group, netting out both the pre-existing gap and any trend that would have happened anyway.- **Propensity score matching**, if there's no clean regional or time-based split: model the probability of feature adoption based on observed pre-feature characteristics (engagement level, tenure, plan type), then compare retention between adopters and non-adopters who had similar predicted adoption propensity. This narrows, but does not eliminate, confounding, since it only controls for confounders you actually measured.- **Instrumental variables or a regression discontinuity**, if one exists: for example, if the feature became available only above some usage threshold, comparing users just above and just below that threshold approximates randomization near the cutoff.- **Sensitivity analysis**: after picking a method, explicitly test how much unmeasured confounding it would take to explain away the result, so the claim comes with an honest confidence level rather than false precision.

**Communicating the Result**Present the finding as 'consistent with a causal effect under [these assumptions]' rather than 'the feature caused X%,' and flag the confounders you could not fully control for. If leadership needs a stronger causal claim, propose a targeted follow-up: a small randomized holdout for the next related feature, so future claims don't rely entirely on observational methods.

**Common Pitfall**Comparing retention between adopters and non-adopters directly and calling the gap the 'effect' of the feature, when the gap is at least partly explained by the fact that more engaged users were the ones who adopted it in the first place.

MediumTechnical

39 practiced

A model passed offline validation with strong metrics but is underperforming once it's live. Walk me through how you'd do the error analysis.

Sample Answer

**Framing**The goal is to find where the offline evaluation and production reality diverge, rather than re-running the same offline checks and hoping for a different answer.

**Approach**- **Compare the input distributions**: pull a sample of live production inputs and compare feature distributions against the training/validation set. A shift here (new user segment, seasonal behavior, a changed upstream field) is one of the most common causes of an offline-to-online gap.- **Check the label definition consistency**: confirm the production label (if you can measure it, e.g. through delayed feedback) matches how the offline label was defined. A subtle mismatch, like counting a 'conversion' differently online versus in the historical training extract, can silently break things.- **Audit the serving pipeline versus the training pipeline**: verify that features are computed identically at serving time as at training time (same aggregation windows, same missing-value handling, same units). Training/serving skew here is a very common silent failure.- **Slice performance by segment and time**: rather than looking at one aggregate live metric, break it down by cohort, geography, device, or time since launch, to see if the drop is broad or concentrated in a specific slice, which points to a specific root cause.- **Read actual misclassified or mispredicted examples**: sample individual cases the model got wrong in production and eyeball them; patterns that jump out from real examples (a specific input format, a language, a rare category) are often faster to spot than aggregate metrics reveal.

**Key Points**- Error analysis should move from broad (is it distribution shift, a pipeline bug, or a genuine model limitation) to narrow (this specific slice, this specific feature).- Keep a hypothesis log: what you checked, what you ruled out, and what you found, so the investigation doesn't repeat itself.

**Common Pitfall**Jumping straight to retraining the model as a fix before confirming whether the root cause is even a model problem versus a pipeline bug or a data quality issue, which wastes a retraining cycle without addressing the real cause.

Unlock Full Question Bank

Get access to hundreds of Methodological Rigor and Experimental Validation interview questions and detailed answers.

Join thousands of developers preparing for their dream job.