MediumTechnical

77 practiced

In Python, implement a function that computes a 95% bootstrap confidence interval for AUC given arrays y_true and y_scores. You may use numpy and scikit-learn. Describe assumptions and how you'd handle stratified resampling to preserve class balance. Return lower and upper percentiles and the point estimate.

Sample Answer

**Approach (brief)**

Use stratified bootstrap: resample indices within each class with replacement to preserve class balance, compute AUC per replicate using sklearn.metrics. Return point estimate (AUC on full data) and 2.5 / 97.5 percentiles of bootstrap AUCs as the 95% CI.

**Assumptions**- Observations are independent and identically distributed.- Enough positive/negative samples to resample meaningfully (if very small, CI may be unstable).- AUC handles score ties; if many ties consider alternative metrics or tie-breaking.

**Implementation (Python)**

python

import numpy as np
from sklearn.metrics import roc_auc_score

def auc_bootstrap_ci(y_true, y_scores, n_boot=1000, seed=None):
    rng = np.random.RandomState(seed)
    y_true = np.asarray(y_true)
    y_scores = np.asarray(y_scores)
    if y_true.shape[0] != y_scores.shape[0]:
        raise ValueError("y_true and y_scores must have same length")
    # point estimate
    point = roc_auc_score(y_true, y_scores)
    # indices by class
    pos_idx = np.where(y_true == 1)[0]
    neg_idx = np.where(y_true == 0)[0]
    if len(pos_idx) == 0 or len(neg_idx) == 0:
        raise ValueError("Need both positive and negative samples to compute AUC")
    aucs = []
    for _ in range(n_boot):
        # stratified resample indices
        samp_pos = rng.choice(pos_idx, size=len(pos_idx), replace=True)
        samp_neg = rng.choice(neg_idx, size=len(neg_idx), replace=True)
        samp_idx = np.concatenate([samp_pos, samp_neg])
        # preserve original labels for roc_auc_score
        bt_y = y_true[samp_idx]
        bt_scores = y_scores[samp_idx]
        try:
            aucs.append(roc_auc_score(bt_y, bt_scores))
        except ValueError:
            # in degenerate bootstrap (all same class) skip
            continue
    lower = np.percentile(aucs, 2.5)
    upper = np.percentile(aucs, 97.5)
    return lower, upper, point

**Notes / alternatives**- For small sample sizes consider stratified bootstrap with paired sampling or use DeLong for analytic CI.- Report number of effective bootstrap replicates if some were skipped.

MediumTechnical

65 practiced

Explain how AUC-ROC can be misleading when the positive class is rare. Describe alternative metrics and evaluation approaches (e.g., precision-recall, average precision, expected precision at operational thresholds) that better capture performance for rare events. Discuss threshold selection strategies tied to business costs.

Sample Answer

**Brief critique of AUC-ROC for rare positives**

AUC-ROC measures separability across all thresholds by plotting TPR vs FPR. When positives are rare, FPR can be very small yet still produce many false alarms in absolute terms (because negatives >> positives). A high AUC-ROC can mask poor precision: model may rank positives above many negatives but still generate unacceptable false positives in production.

**Better metrics and why**

- Precision-Recall (PR) curve: focuses on precision (positive predictive value) vs recall; more sensitive to performance on the positive class.- Average Precision (AP): area under PR curve; summarizes precision across recalls and correlates with actual positive retrieval quality.- Expected Precision at operational thresholds: report precision at the thresholds you’d actually use (e.g., top-k, recall target), since business cares about specific operating points.- F-beta (e.g., F1 or F0.5): combines precision and recall with tunable trade-off.

**Evaluation approaches**

- Use stratified cross-validation preserving rare-event distribution; use confidence intervals or bootstrap for AP/precision at threshold.- Calibrate scores (Platt, isotonic) so probabilities reflect true risk — important for cost-based thresholding.- Evaluate absolute counts: false positives per day, true positives captured, and lift over baseline.

**Threshold selection tied to business costs**

- Define cost matrix: C_fp and C_fn (or revenue per TP). Compute expected cost per threshold: ExpectedCost = C_fp * FP_rate * N_neg + C_fn * FN_rate * N_pos.- Choose threshold minimizing expected cost or maximizing expected profit. If costs uncertain, perform sensitivity analysis and present precision/recall at candidate thresholds.- When immediate action capacity is limited (e.g., human review budget), optimize for precision at fixed review budget (precision@k).

This approach aligns modeling choices with operational constraints and gives stakeholders actionable metrics rather than a single AUC number.

EasyTechnical

86 practiced

For a recommendation or search system, explain precision@k, recall@k and NDCG (Normalized Discounted Cumulative Gain). Discuss when each metric is appropriate, how rank position and relevance grading affect them, and how you would collect labeled data to compute these metrics offline.

Sample Answer

**Definition — Precision@k and Recall@k**- Precision@k: fraction of top-k items that are relevant.

text

precision@k = (# relevant items in top k) / k

Plain English: measures correctness of the top-k set.- Recall@k: fraction of all relevant items retrieved in top-k.

text

recall@k = (# relevant items in top k) / (total # relevant items for query)

Plain English: measures coverage of true relevant items.

**Definition — DCG / NDCG**- Discounted Cumulative Gain (DCG) sums graded relevance with position discount.

text

DCG@k = sum_{i=1..k} (2^{rel_i} - 1) / log2(i + 1)

Plain English: higher gain for higher relevance, penalized by lower rank.- Normalized DCG (NDCG) divides by ideal DCG.

text

NDCG@k = DCG@k / IDCG@k

Plain English: scales DCG to [0,1] by comparing to perfect ordering.

**When to use each**- Precision@k: when top-k quality matters (e.g., homepage recommendations, short lists).- Recall@k: when covering all relevant items is critical (e.g., legal/medical search, discovery tasks).- NDCG: when graded relevance and rank positions both matter (e.g., web search with dwell-time-based labels).

**Effect of rank and grading**- Precision@k ignores order within top-k; rank position beyond k is irrelevant.- Recall@k depends on total relevant count; sensitive when relevant pool varies across queries.- NDCG explicitly rewards putting highly relevant items earlier; graded labels (0/1/2+) are supported.

**Collecting labeled data offline**- Use a mix: human relevance judgments (graded labels), implicit feedback (clicks, dwell time) with bias correction, and interleaved A/B logs.- Build query/item pools using sampling (stratify by popularity), collect multiple raters, compute consensus labels and confidence.- Apply propensity weighting or counterfactual techniques to debias logged clicks before computing metrics.

HardSystem Design

82 practiced

Design an end-to-end ML evaluation pipeline for a personalization model serving millions of users daily. Define offline validation checks, experiment integration with the platform, canary and ramp strategies, metric collection schemas, SLO definitions and automated rollback criteria. Explain how to ensure metric integrity (e.g., reconciliation between offline and online metrics) across the pipeline.

Sample Answer

**Overview / goals**Design a robust evaluation pipeline ensuring safe rollouts, reproducible experiments, and tight reconciliation between offline and online metrics for a personalization model at scale.

**1) Offline validation checks**- Data checks: distribution drift, missingness, label leakage, schema validation (use Great Expectations).- Model checks: calibration, per-segment AUC/CTR lift, fairness slices, resource/latency profiling.- Shadow evaluation: score live traffic in parallel, compute online-like metrics (exposure-weighted CTR, position bias adjusted).

**2) Experiment integration**- Use experimentation platform (feature gates + bucketing) to expose variants; log deterministic user-id, cohort, timestamp, model-version, exposure metadata.- Define primary/secondary metrics and guardrails in experiment config; auto-collect deltas with statistical significance (sequential testing / fixed-horizon with corrections).

**3) Canary & ramp strategy**- Canary: 0.5–1% internal users for 24–48h with strict safety checks.- Progressive ramp: 1% → 5% → 20% → 100% with minimum dwell time and automated health checks.- Traffic shadowing for additional validation without affecting users.

**4) Metric schema & collection**- Event schema: user_id, session_id, timestamp, model_version, treatment, exposure_context, action (impression/click/purchase), value, weight.- Use batch-aggregates + time-series store (BigQuery/Redshift + Prometheus/Influx for latency/SLO).- Store raw events for replay/reconciliation.

**5) SLOs & rollback criteria**- SLOs: latency p99 < X ms, negative impact on baseline engagement < Y% (e.g., -1%), crash/error rate < Z.- Automated rollback triggers: - Safety: any severe error (>threshold) or data pipeline failure. - Metric-based: statistically significant degradation on primary metric beyond allowed delta for a sustained window (e.g., 95% CI below -0.5% for 2 consecutive windows). - Drift: input feature drift beyond control limits.

**6) Ensuring metric integrity / reconciliation**- One source-of-truth: reconcile SDK event logs with aggregated DB daily. Compute joins on deterministic keys to check missingness and sampling bias.- Offline vs online alignment: map offline predicted scores to online exposures; compute exposure-weighted offline metrics to compare with observed online outcomes; use uplift models to explain gaps.- Monitoring: automated alerts for replay mismatch rate, sampling ratio deviations, timestamp skew, duplicate events.- Root-cause flow: if mismatch detected, run replay from raw events to reproduce metrics, compare cohorts/segments, and apply corrective actions (fix logging, re-run aggregates, rollback if needed).

**Example scenario**- New personalization ranker passes offline checks and shadow tests; canary to 1% internal users; after stable metrics and reconciled offline/online lifts, ramp to 20% then full. Automated monitors catch a 1.2% drop in retention at 5% — pipeline auto-pauses ramp, runs reconcilation, finds logging bug; patch + re-run canary before continuing.

This pipeline balances statistical rigor, safety, rapid iteration, and traceable reconciliation required for large-scale personalization.

MediumTechnical

61 practiced

A feature increases user engagement but also slightly increases latency for requests. You're asked to choose a primary metric and guardrails for rollout. Design a decision framework that quantifies trade-offs between engagement improvements, latency SLOs, and potential downstream costs. Explain how you'd instrument experiments and report multi-metric results to stakeholders.

ML System Evaluation and Metrics Questions

Sample Answer

Sample Answer

Sample Answer

Sample Answer

Sample Answer

Unlock Full Question Bank