EasyTechnical

80 practiced

For a binary fraud-detection classifier with 0.1% prevalence of fraud, discuss the advantages and disadvantages of optimizing for accuracy versus precision, recall, F1, AUC-ROC, and precision@k. Which metric(s) would you recommend as the primary metric and which as guardrails in deployment, and why?

Sample Answer

**Brief framing**

Prevalence = 0.1% (highly imbalanced). Metric choice should reflect business cost of missed fraud (false negatives) vs cost of false alarms (false positives) and operational constraints (investigator capacity).

**Metric pros/cons**

- Accuracy - Pros: Intuitive. - Cons: Miserleading — a trivial model predicting “not fraud” yields 99.9% accuracy; not useful.

- Precision - Pros: Measures purity of flagged cases (important if investigator budget is limited). - Cons: Can ignore recall; model can achieve high precision by flagging very few cases.

- Recall (sensitivity) - Pros: Captures ability to catch fraud (important if miss cost is high). - Cons: Can be increased by many false positives; may overwhelm operations.

- F1 - Pros: Balances precision and recall (harmonic mean). - Cons: Assumes equal importance of precision and recall — may not match asymmetric costs.

- AUC-ROC - Pros: Threshold-agnostic, measures ranking quality across class imbalance to some extent. - Cons: Can be optimistic in extreme imbalance; AUC-PR is more informative for rare positives.

- Precision@k (or Precision@top decile) - Pros: Directly measures precision among top-scored k cases — matches fixed investigator capacity and prioritization. - Cons: Needs choice of k; ignores recall outside top-k.

**Recommendation (Research-scientist perspective)**

- Primary metric: Precision@k (where k = operational capacity) combined with Recall at that k or AUC-PR for global ranking. These reflect real-world throughput and rare-positive evaluation.- Guardrails: Track AUC-PR (ranking quality), overall recall, calibration (score vs true prob.), and false positive rate. Also monitor precision over time and per-segment fairness.

**Why**- Precision@k aligns model evaluation with the operational constraint (investigator slots) and business ROI.- AUC-PR complements by measuring ranking ability across thresholds; recall/calibration guard against blind spots and distribution shifts.

MediumSystem Design

62 practiced

Outline the architecture for an offline evaluation pipeline that computes research metrics at scale: ingestion from event logs, preprocessing, metric computation, caching, reproducibility/versioning, and data lineage. Specify technologies you might choose, APIs for reproducible experiments, and how you'd validate metric correctness and detect silent bugs.

Sample Answer

**Clarify goals & constraints**- Offline batch pipeline that ingests event logs, computes research metrics at scale, supports reproducible experiments, caching for fast reads, and strong lineage/versioning for audits.

**High-level architecture**- Ingestion: Kafka or Kinesis → raw event lake (S3 / GCS) with partitioning by date/event-type.- Preprocessing: Spark/Beam jobs (run via Airflow or Beam Dataflow) that clean, dedupe, enrich; write canonical parquet/Delta tables.- Metric computation: Spark/Dataproc or Flink batch jobs operating over Delta Lake / Iceberg. Metrics written into a versioned metrics store (Delta table + Postgres for indices).- Caching & query: Materialized metric tables, pre-aggregations, and a serving layer (Presto/Trino) for ad-hoc queries; Redis for low-latency cached results.- Orchestration & lineage: Airflow with OpenLineage integration; data catalog (DataHub/Amundsen) tracking datasets, schemas, owners.

**Reproducibility / APIs**- Data-versioned artifacts: store canonical datasets in Delta/Iceberg with table snapshots and S3 URIs; record commit IDs and input snapshot IDs in runs.- Experiment API (HTTP / Python): - submit_experiment(experiment_id, code_git_sha, pipeline_config, input_data_snapshot_id, params) -> run_id - get_run_metadata(run_id) -> {git_sha, input_snapshots, containers, Docker image, env} - fetch_metrics(run_id) -> metrics table reference- Use MLflow / DVC for code/artifact versioning; containerize compute jobs (Docker) and store images in registry.

**Validation & detecting silent bugs**- Unit-test metric functions with synthetic and edge-case fixtures.- Golden datasets: maintain small, human-verified inputs with expected metric vectors; run CI that compares outputs within tolerances.- Statistical shadow runs: parallel old/new implementation on sampled data; compare distributions using KS-test, mean-diff, alert on drift.- Schema & data quality: Great Expectations checks and anomaly detectors on event rates, key cardinalities, null rates.- End-to-end checksums: compare row counts and partition checksums between stages.- CI gating: pipeline cannot promote metric snapshots without passing tests; alerting + auto-rollback of materialized metrics.- Monitoring: track metric time-series for sudden jumps; use explainable provenance to root-cause via lineage.

**Trade-offs**- Delta/Iceberg + Spark scales and gives ACID snapshots but increases infra complexity; simpler parquet + strict bookkeeping can work for smaller scale.

This design emphasizes deterministic runs (snapshots + git SHAs), automated validation to catch silent bugs, and explicit lineage so research metrics are auditable and reproducible.

EasyTechnical

71 practiced

As a research scientist at an e-commerce company designing recommendation experiments, explain the differences between primary, secondary, and guardrail metrics. For each category provide 2–3 concrete example metrics relevant to a recommender (e.g., CTR, time-on-site, retention, abusive-content-rate), justify why each example fits its category, and identify the stakeholders who care about each metric.

MediumTechnical

79 practiced

For an object-detection research experiment, explain how mean Average Precision (mAP) is computed, how IoU thresholds affect AP and mAP, and what practical choices (IoU thresholds, matching rules, per-class averaging, NMS settings) matter for fair and reproducible evaluation across models.

MediumTechnical

57 practiced

Many production metrics suffer from missing or censored data (for example, user sessions that end before a conversion is observed). Explain methods to handle missingness and censoring when computing experiment metrics, including survival analysis, imputation, inverse probability weighting, and truncation corrections. Discuss pros and cons and when each approach is appropriate.

Sample Answer

**Brief framing** In experiments where outcomes (e.g., time-to-conversion) are unobserved because sessions end early or data collection stops, treating those as simple missing values biases metric estimates. Below are principled methods, pros/cons, and when to use them.

**1) Survival analysis (Kaplan–Meier, Cox PH)** - What: Model time-to-event with right-censoring; estimate survival or hazard functions. - Pros: Uses all available follow-up, unbiased under noninformative censoring, can compare arms (log-rank, Cox). - Cons: Assumes censoring independent of future event given covariates; Cox requires proportional hazards (can be tested/relaxed). - Use when: primary metric is time-to-event or conversion rate over time.

**2) Inverse probability weighting (IPCW / IPW)** - What: Weight observed outcomes by inverse probability of being uncensored (or of selection) estimated from a model. - Pros: Can correct informative censoring if censoring model conditions on enough covariates; flexible for ATE estimation. - Cons: Weights can be unstable (high variance); requires correct censoring-probability model. - Use when: censoring depends on observed covariates and you need marginal effect estimates.

**3) Imputation (single or multiple)** - What: Fill missing outcomes using predictive models (multiple imputation preferred). - Pros: Multiple imputation captures uncertainty; easy to plug into standard estimators. - Cons: Relies on missing-at-random (MAR) assumption; model misspecification biases results; tricky for time-to-event without censoring-aware imputation. - Use when: missingness is plausibly MAR and you can build a good predictive model; combine with survival-aware imputation for times.

**4) Truncation corrections (e.g., left truncation, administrative censoring adjustments)** - What: Explicitly model observation window truncation (e.g., only users observed long enough). Use methods like conditional likelihoods, Aalen–Johansen for competing risks. - Pros: Corrects bias from design-imposed truncation. - Cons: More complex; needs careful definition of risk sets. - Use when: experiment window or enrollment causes systematic truncation.

**Practical guidance / diagnostics** - Always visualize Kaplan–Meier curves and censoring distributions by arm. - Check whether censoring correlates with baseline covariates; if yes, favor IPCW or model-based approaches. - Use sensitivity analyses: vary censoring models, perform worst/best-case imputations. - Report estimands clearly (e.g., 14-day conversion probability vs. conditional on 7-day observed).

Choosing a method depends on the estimand, censoring mechanism, sample size, and tolerance for model assumptions. Combining methods (e.g., IPCW + Cox, multiple imputation with survival model) often gives robustness.

Metrics, Guardrails, and Evaluation Criteria Questions

Sample Answer

Sample Answer

Sample Answer

Sample Answer

Sample Answer

Unlock Full Question Bank