Baseline and Comparative Analysis Questions

Designing and executing rigorous comparative evaluations that demonstrate whether and why a new method provides meaningful benefit. Topics include selecting simple and strong baselines, ensuring hyperparameter parity, designing ablation studies, choosing appropriate evaluation metrics for the problem, conducting error analysis, running statistical tests and power analysis, correcting for multiple comparisons, avoiding improper data dredging, and building reproducible evaluation pipelines that produce results interpretable for product and research stakeholders.

EasyTechnical

34 practiced

If you tune your new model's hyperparameters for a week but leave the baseline at its default settings, is that a fair comparison? Walk me through your reasoning.

MediumTechnical

34 practiced

How would you build an evaluation pipeline for comparing model versions over time that both engineers and product stakeholders can actually trust?

EasyTechnical

57 practiced

Two models are being compared: one has higher accuracy, the other has higher F1 score. Which one do you tell the team to ship, and how do you decide which metric to trust?

MediumTechnical

36 practiced

Someone asks 'how much test data do we actually need to reliably tell if model B beats model A?' How do you answer that with a power analysis?

Sample Answer

**What power analysis answers**Given an expected effect size (how big a true improvement you want to be able to detect), the variability of your metric, and a target significance level, power analysis tells you the minimum sample size needed to detect that effect with a chosen probability (power), typically 80%.

**The intuition**Smaller true effects and noisier metrics both require more data to detect reliably; if you don't have enough data, even a real improvement can look statistically insignificant just because your test lacked the sensitivity to find it.

**Formula for comparing two proportions (e.g. two conversion or accuracy rates)**

text

n ~ 2 * (z_alpha/2 + z_beta)^2 * p*(1-p) / delta^2

Here `delta` is the minimum effect size you care about detecting, `p` is the baseline rate, and `z_alpha/2`, `z_beta` come from your chosen significance level and power (e.g. 1.96 and 0.84 for alpha=0.05, power=80%). Intuition: the required sample size grows with the square of how small an effect you want to detect, so trying to detect a 0.5% lift needs roughly 4x the data of detecting a 1% lift.

**Worked example**Baseline accuracy p = 0.80, we want to detect a delta = 0.02 (2 point) improvement at alpha=0.05, power=80%:

python

from scipy.stats import norm

p, delta = 0.80, 0.02
z_alpha = norm.ppf(1 - 0.05/2)
z_beta = norm.ppf(0.80)
n = 2 * (z_alpha + z_beta) ** 2 * p * (1 - p) / delta ** 2
print(round(n))  # ~2458 examples per group

**How I'd use this**If our current test set is far smaller than the required n, I'd tell the team either to collect more evaluation data, or to accept that we can only reliably detect larger effects, and reframe the decision criteria accordingly rather than over-interpreting a small-sample result.

EasyTechnical

34 practiced

A teammate says 'the new model scored higher on our test set once, so it's better, let's ship it.' What's wrong with that reasoning?

Unlock Full Question Bank

Get access to hundreds of Baseline and Comparative Analysis interview questions and detailed answers.

Join thousands of developers preparing for their dream job.