HardTechnical

65 practiced

Discuss the risks of sequential peeking in A/B tests (checking results repeatedly) and describe statistical techniques to control Type I error under optional stopping: alpha-spending functions, Pocock and O'Brien-Fleming boundaries, and Bayesian sequential testing. Which approaches are practical for product teams and why?

Sample Answer

Risk summary: repeatedly "peeking" at A/B test results without a stopping rule inflates the Type I error — each look multiplies the chance of a false positive so a nominal 5% alpha can become much larger. Optional stopping without adjustment undermines p-values and confidence intervals, leading to wrong product decisions, wasted rollout effort, and biased effect estimates.

Statistical techniques to control Type I error under optional stopping:

- Alpha-spending functions (group-sequential framework) - Idea: pre-allocate overall alpha across planned looks using a spending function α(t) that maps information fraction to cumulative alpha spent. - Benefit: flexible schedule (can be deterministic or information-driven) and maintains overall Type I error. - Practical note: requires pre-specification of maximum looks or maximum information; works well when you can estimate information accrual.

- Pocock boundaries - Equal critical values at each interim look (more liberal early, moderately conservative overall). - Good when you expect several interim looks and want simple, constant thresholds. - Trade-off: more alpha spent early → smaller final-sample power than more conservative designs.

- O’Brien–Fleming boundaries - Very conservative early thresholds, almost nominal at the final look. - Good when you want strong protection against false positives early and allow decisive evidence later. - Trade-off: hard to stop early because early thresholds are very strict.

- Bayesian sequential testing - Uses posterior probabilities or Bayes factors; allows continuous monitoring without inflating Type I error in the Bayesian sense (optional stopping is coherent if priors and decision rules are specified). - Practical caveats: decisions depend on chosen prior and loss function; regulators/businesses often expect frequentist-style error control, so calibration via simulation is recommended. - Advantages: intuitive decision metrics (P(effect>0) > threshold), easier to communicate to product teams.

Practical recommendations for product teams- Pre-specify a stopping plan. Never ad-hoc peek.- If you need frequent monitoring but want familiar frequentist guarantees: use alpha-spending with either Pocock (if you want a simple constant threshold) or O’Brien–Fleming (if you want strict early control). Implement using gsDesign/seqDesign or packaged A/B testing libraries.- If you prefer flexible, decision-focused rules and can simulate/calibrate: use a Bayesian stopping rule with transparent priors and thresholds; run simulations to map Bayesian thresholds to frequentist error rates for stakeholders.- For most product teams: an alpha-spending group-sequential design (few planned looks) or a well-calibrated Bayesian approach are practical — both are implementable, explainable, and prevent false positives while enabling early stopping when evidence is strong.

MediumTechnical

63 practiced

For a search ranking problem with graded relevance labels on a 5-point scale (0..4), explain NDCG@k, MAP, MRR, and Recall@k. Describe how discounting in NDCG works and compute NDCG@3 for a sample ranked list with relevance grades [3, 2, 0, 4] (assume ideal DCG computed on sorted grades).

Sample Answer

NDCG@k, MAP, MRR, Recall@k — brief definitions and when to use them:

- NDCG@k (Normalized Discounted Cumulative Gain): Designed for graded relevance. DCG@k = sum_{i=1..k} (gain(rel_i) / log2(i+1)). Common gain choices: gain(rel)=2^{rel}-1 (exponential) to emphasize higher grades, or gain=rel (linear). NDCG@k = DCG@k / IDCG@k where IDCG@k is DCG@k for the ideal (sorted-by-relevance) ranking. Discounting (division by log2(i+1)) reduces contribution of lower-ranked items — earlier correct, highly relevant items matter most.

- MAP (Mean Average Precision): Average of precision@r over ranks r where relevant items occur, then averaged over queries. MAP assumes binary relevance; for graded labels you typically binarize (e.g., rel>=threshold) or use variants (graded AP). Good when you care about precision across all relevant items.

- MRR (Mean Reciprocal Rank): For each query take 1 / rank_of_first_relevant_item, average across queries. Simple, focuses on getting one relevant item early. For graded relevance you must define what counts as “relevant” (threshold).

- Recall@k: Fraction of all relevant items retrieved in top-k. Useful when completeness (finding all relevant docs) matters. Also requires a relevance threshold for graded labels.

Discounting in NDCG: log2(i+1) gives diminishing weight to lower ranks; using exponential gain 2^{rel}-1 amplifies differences at high grades so a grade-4 at rank 5 can outweigh many lower-grade items at top.

Compute NDCG@3 for ranked list with relevance grades [3, 2, 0, 4] (use gain=2^{rel}-1):

Step 1 — DCG@3 using top-3 relevances [3,2,0]:- gains: g1=2^3-1=7, g2=2^2-1=3, g3=2^0-1=0- DCG@3 = 7 / log2(1+1) + 3 / log2(2+1) + 0 / log2(3+1) = 7/1 + 3/log2(3) + 0 ≈ 7 + 3/1.585 = 7 + 1.8928 = 8.8928

Step 2 — IDCG@3 (ideal top-3 grades sorted descending: [4,3,2]):- gains: 2^4-1=15, 2^3-1=7, 2^2-1=3- IDCG@3 = 15/1 + 7/log2(3) + 3/log2(4) ≈ 15 + 7/1.585 + 3/2 ≈ 15 + 4.418 + 1.5 = 20.918

Step 3 — NDCG@3 = DCG@3 / IDCG@3 ≈ 8.8928 / 20.918 ≈ 0.425 (42.5%)

Notes:- If you use linear gains (gain=rel) numbers differ, but interpretation is the same.- MAP and MRR require choosing a relevance cutoff for graded labels (e.g., >=2). Recall@k similarly needs that threshold.

MediumTechnical

75 practiced

Metric gaming: provide concrete strategies to detect and prevent teams from 'optimizing the metric' in ways that harm product quality (e.g., artificially inflating session time). Give examples of gaming, detection signals (distributional anomalies, unnatural quantization), and governance policies (multi-metric decision making, guardrail metrics, audits).

Sample Answer

Situation: In my role as an AI engineer supporting product metrics, I’ve seen teams unintentionally (and sometimes intentionally) optimize for single KPIs in ways that degrade product quality—e.g., driving up "session time" by auto-refreshing content or adding infinite scroll that inflates engagement.

Concrete examples of gaming:- Artificial session inflation: background pings, invisible DOM activity, or auto-play that keeps sessions open.- Click-farming: adding misleading CTAs that generate clicks without value.- Funnel padding: inserting dummy steps to reclassify drop-offs as "completed".

Detection signals (how I’d spot them):- Distributional anomalies: sudden shift in session length mean/median, heavy tails, or step-changes confined to specific cohorts or experiments.- Unnatural quantization: spikes at round numbers (e.g., many sessions at exactly 300s) or repeated identical event timestamps.- Behavioral mismatches: high session time combined with low downstream actions (conversion, retention).- Cohort divergence: new release cohorts diverging from historical behavior only for the targeted metric.- Instrumentation artifacts: high server-side heartbeats without correlated client events.

Concrete detection tactics:- Run statistical drift tests (KS, AD) on metric distributions per cohort and per release.- Monitor joint distributions: session_time vs conversions; flag low correlation periods.- Time-series change point detection and entropy measures on event timing.- Blackbox fuzz testing of instrumentation to detect heartbeat-only sessions.- Synthetic users and canary cohorts to validate true engagement.

Prevention & governance policies:- Multi-metric decision rules: require improvement in primary metric plus at least one quality guardrail (e.g., conversion rate, retention, NPS) before rollout.- Define guardrail metrics for every KPI (e.g., session_time → task_completion_rate, bounce rate).- Rate-limit instrumentation changes: any metric-definition change requires review and automated tests.- Audits & experiment review: regular metric audits, mandatory experiment manifests describing how metrics are collected and potential gaming risks.- Alerting & escalation: anomaly alerts route to product + SRE + analytics for triage.- Incentive alignment: tie team goals to balanced scorecards, not single metrics.- Transparency: publish metric definitions, collection code, and event schemas; maintain a metrics catalog with owners.

Result / rationale: These combined detection and governance steps catch obvious gaming, reduce false-positive product changes, and align teams toward genuine user value rather than short-term metric wins.

MediumBehavioral

63 practiced

Behavioral: Tell me about a time when you had to convince product stakeholders to accept a more complex evaluation metric or experiment design because the simpler metric would be gamed or misleading. Describe the Situation, Task, Actions you took, and the Result (STAR). What did you learn?

HardTechnical

114 practiced

Explain cluster-randomized experiments (e.g., randomizing at user, household, or region level) and why clustering is necessary in contexts with spillover or correlated behavior. Define intra-cluster correlation coefficient (ICC) and describe how ICC affects required sample size and variance estimation.

ML System Evaluation and Metrics Questions

Sample Answer

Sample Answer

Sample Answer

Sample Answer

Sample Answer

Unlock Full Question Bank