Online Experimentation and Model Validation Questions

Running experiments in production to validate model changes and measure business impact. Topics include splitting traffic across model variants canary deployments and champion challenger testing selecting metrics that capture both model performance and business outcomes performing sample size and test duration calculations accounting for statistical power and multiple testing adjustments and handling instrumentation and novelty bias. Candidates should be able to analyze heterogeneous treatment effects monitor experiments in real time and design ramping plans and rollback guardrails to protect user experience and business metrics. The topic also covers decision rules for when to rely on offline evaluation versus online experiments and how to interpret differences between offline model metrics and live user outcomes as part of model validation and deployment strategy.

EasyTechnical

31 practiced

Two weeks into an experiment you notice the control group has 48,000 users and treatment has 52,000, when the intended split was 50/50. Walk me through how you'd check whether that's a real sample ratio mismatch (SRM) or just noise, and show me the code.

Sample Answer

**Approach**A sample ratio mismatch (SRM) means the observed traffic split deviates from the intended allocation by more than chance. Test it with a chi-square goodness-of-fit test comparing observed counts to expected counts under the intended ratio.

python

from scipy.stats import chisquare

def detect_srm(control_count, treatment_count, expected_ratio=1.0, alpha=0.001):
    total = control_count + treatment_count
    expected_control = total * (expected_ratio / (1 + expected_ratio))
    expected_treatment = total - expected_control
    chi2_stat, p_value = chisquare(
        f_obs=[control_count, treatment_count],
        f_exp=[expected_control, expected_treatment],
    )
    return {"p_value": p_value, "srm_detected": p_value < alpha}

result = detect_srm(48000, 52000)
print(result)

Running this on 48,000 vs 52,000 out of 100,000 total gives a chi-square statistic of 160 (an extremely large deviation for that sample size), so the p-value is effectively zero and SRM is clearly flagged, this imbalance is far too large to be sampling noise.

**Key points**- Use a strict alpha (often 0.001, not 0.05) because SRM checks run on every experiment and you want very few false alarms.- SRM is often caused by bugs in assignment, bot filtering that hits one arm harder, or a redirect/logging bug that drops events asymmetrically.

**Edge cases**- Very small total counts can make the test underpowered to detect real SRM; combine with a manual sanity check.- If the intended ratio isn't 1:1 (e.g., a 90/10 canary), pass the correct expected_ratio, not 1.0.

MediumBehavioral

26 practiced

Tell me about a time an experiment came back statistically significant and everyone wanted to ship, but you argued against it. What happened?

MediumTechnical

27 practiced

A fraud model's AUC looks stable in your weekly offline evaluation, but the operations team says the 'risk score' thresholds they rely on have started feeling off, flagging either too many or too few cases. What would you check?

Sample Answer

**Why AUC can hide this problem**AUC only measures ranking quality, whether fraudulent cases are correctly ordered above legitimate ones, it says nothing about whether a predicted score of 0.8 actually corresponds to an 80% chance of fraud. Operations teams act on the raw score against a fixed threshold, so what they're feeling is a calibration problem, not a ranking problem, and AUC can stay flat while calibration silently drifts.

**What to check**- **Reliability diagram**: bucket predictions into score deciles and compare the average predicted probability in each bucket against the observed fraud rate in that bucket, on recent live data. A well-calibrated model has points near the diagonal; drift shows up as a systematic gap.- **Brier score over time**: track mean squared error between predicted probability and actual outcome week over week; a rising trend indicates calibration decay even if AUC hasn't moved.- **Population and feature drift**: check whether input feature distributions, transaction size, device mix, geography, have shifted since training, since calibration is far more sensitive to distribution shift than pure ranking ability is.- **Base rate shift**: if the true fraud rate itself changed, for example a new fraud pattern emerged, a model trained on the old base rate will systematically over- or under-predict even with unchanged ranking ability.

**Fix**- Short-term: recalibrate the score-to-probability mapping, for example Platt scaling or isotonic regression fit on recent labeled data, without retraining the whole model.- Longer-term: retrain on more recent data, and add calibration monitoring (Brier score, reliability diagrams) as a standing part of the online validation dashboard, not just AUC.

**Why this belongs in online validation, not just offline**Calibration drift is fundamentally about the live population diverging from the training population, so it can only be caught by continuously comparing model outputs to real-time labeled outcomes, an online monitoring concern rather than a one-time offline check.

HardTechnical

30 practiced

A competitor just launched a feature copying your product's core value prop, and leadership wants your new ranking model shipped to all users within 48 hours, no time for a normal multi-week A/B test. How do you approach validation under that constraint?

Sample Answer

**Reframe the ask, don't just comply or refuse**The real requirement isn't "skip validation," it's "get acceptable confidence fast." The job is to find the fastest validation plan that keeps the risk of a bad outcome bounded, and to be explicit with leadership about exactly what confidence is being traded away, not to silently ship unvalidated or to stall with a timeline that isn't actually necessary either.

**A staged, compressed plan**

1. **Hours 0-4, offline gate**: run the model against the existing offline evaluation suite on held-out historical data as a fast sanity filter; this doesn't validate business impact, but it catches an obviously broken model before it ever sees a real user.

2. **Hours 4-12, shadow mode**: run the new model in production on live traffic without acting on its output, just logging what it would have shown, compared against what the current model actually served. This catches serving-side bugs such as latency, errors, or degenerate outputs with zero user risk, and is the fastest way to get real-traffic confidence.

3. **Hours 12-24, small live canary with tight guardrails**: expose 1 to 5% of traffic from a random low-risk cohort, and monitor a small number of fast-moving proxy guardrails, error rate, latency, an early engagement signal like immediate bounce rate, at short intervals such as every 15 to 30 minutes, with an automated rollback trigger on any guardrail breach. This isn't a properly powered read on the primary business metric in this timeframe, and that should be stated explicitly.

4. **Hours 24-48, ramp with continuous monitoring**: if the canary shows no red flags, ramp in stages, 5% to 25% to 100%, watching the same guardrails continuously, accepting that there won't be a statistically clean primary-metric read before full rollout, only a guardrail-based safety check.

**What you tell leadership explicitly**We can get you live in 48 hours with strong guardrails against an obviously bad outcome, but we will not have statistical confidence on the actual business impact until roughly a week after full rollout, when we can look back at the natural experiment created by the staged ramp. If that trade is acceptable given the competitive urgency, here's the plan, and here's exactly how fast we can detect and roll back if we're wrong.

**Risk you're accepting and how you bound it**You're trading a properly powered causal estimate for a fast, guardrail-gated rollout; the bound on downside risk comes from automated rollback triggers and short monitoring windows, not from statistical certainty. This is very different from skipping validation, it's compressing it into checks that are fast but still real.

**When you'd push back harder**If the model touches a domain with irreversible harm, such as financial transactions or safety-critical decisions, the answer changes: the 48-hour timeline isn't achievable safely regardless of business pressure, and the fastest safe alternative should be proposed instead.

MediumTechnical

24 practiced

You're validating a new dynamic pricing model for a ride-sharing marketplace, but rider and driver behavior in a city are tightly coupled, so a normal user-level A/B split would leak treatment effects between arms. How would you design the experiment instead?

Unlock Full Question Bank

Get access to hundreds of Online Experimentation and Model Validation interview questions and detailed answers.

Join thousands of developers preparing for their dream job.