Advanced ML Techniques & Research Application Questions

Advanced machine learning techniques, architectures, training methods, evaluation strategies, and the application of research insights to production ML systems. Covers bridging research findings to practical deployment, scalable training and serving, model governance, experiment design, and responsible AI practices.

HardTechnical

49 practiced

You have a strict budget for expensive labeled data. Design an active learning strategy combined with sequential testing to compare two candidate models while minimizing labeling cost. Specify acquisition functions, stopping criteria, how to control type I/II errors, corrections for multiple comparisons, and how to model noisy annotators.

Sample Answer

**Brief framing**You must compare two candidate models A and B under a strict labeling budget. Combine pool-based active learning to acquire informative labels with sequential hypothesis testing to stop early while controlling errors.

**Acquisition functions (cost-aware)**- Uncertainty sampling on the model-difference score s(x)=P_A(y|x)−P_B(y|x): pick x with |s(x)| near 0. - Query-by-committee / disagreement: ensemble predictions from A,B and small perturbations; score by vote entropy. - Expected model-change or expected information gain (EIG) on the model-difference metric, normalized by label cost c(x). - If annotators vary in quality/cost, add value-of-information per annotator: select (x,annotator) maximizing EIG / c.

**Sequential testing & stopping rule**- Form streaming paired observations z_i = 1{A correct} − 1{B correct} on queried labeled items. Perform a Sequential Probability Ratio Test (SPRT) or group-sequential test on the mean µ = E[z]. H0: µ ≤ δ (no meaningful improvement) vs H1: µ ≥ Δ (practical effect). Stop when log likelihood ratio crosses upper/lower boundaries.- LLR per sample: use estimated per-sample likelihoods; for Bernoulli paired outcomes you can compute the incremental LLR.

text

LLR_n = sum_{i=1}^n log ( p1(z_i) / p0(z_i) )

Plain-English: accumulate evidence favoring H1 over H0; stop when LLR > A (accept H1) or LLR < B (accept H0).

**Controlling Type I/II & power**- Set boundaries A = log((1−β)/α) and B = log(β/(1−α)) for target α, β in SPRT. If nuisance parameters unknown, use conservative estimates from a labeled pilot or use alpha-spending functions (O’Brien-Fleming or Pocock) for group-sequential sampling.- Precompute sample size needed for desired power under expected effect size and annotator noise; use sequential design to often need fewer labels.

**Multiple comparisons**- If comparing >2 models or many metrics: use hierarchical testing — test global null first. For multiple pairwise tests, use Holm-Bonferroni sequential correction or alpha-spending across hypotheses. For many adaptive queries, control FDR with Benjamini-Hochberg on p-values from interim analyses or use alpha-investing procedures that permit data-dependent sequential testing.

**Modeling noisy annotators**- Use Dawid–Skene or hierarchical Bayesian confusion-matrix models to infer true labels and annotator reliabilities; treat inferred true labels as latent for SPRT likelihoods. Prefer joint inference: update model parameters and annotator reliabilities online (EM or variational). - Actively select annotators via expected reduction in posterior entropy (labeler value) vs cost; request repeat labels when disagreement or high uncertainty.

**Practical recipe**1. Collect small pilot to estimate effect size and annotator confusion. 2. Build acquisition scoring s(x) = EIG(model-diff)/cost; select (x,annotator) per step. 3. After each labeled batch (or sample), update model predictions, annotator posteriors, compute LLR and apply SPRT/group-sequential boundaries with alpha-spending. 4. If multiple hypotheses, apply Holm or alpha-spending allocation. 5. Stop early when evidence crosses thresholds or budget exhausted; report adjusted p-values and posterior effect estimates with annotator uncertainty.

This design minimizes labels by targeting high-value examples, models annotator noise, and provides rigorous sequential control of Type I/II errors.

MediumTechnical

39 practiced

Explain model calibration and why it matters for decision systems. Describe methods to calibrate probabilities (temperature scaling, isotonic regression), how to compute Expected Calibration Error (ECE), and how you would monitor calibration drift over time in production.

Sample Answer

**What is calibration & why it matters** Model calibration means predicted probabilities match observed frequencies (e.g., among items predicted 0.7, ~70% are positive). For decision systems—risk scoring, resource allocation, thresholding—miscalibrated probabilities lead to systematic over/under-treatment, wrong cost trade-offs, and poor downstream optimization.

**Calibration methods** - Temperature scaling (parametric): learn scalar T on validation set to minimize NLL; softmax(logits / T). Simple, keeps ranking intact. Good for deep nets. - Isotonic regression (non‑parametric): learn a monotonic piecewise-constant mapping from raw scores to probabilities. Flexible, can overfit with small val sets. - (Also mention Platt scaling: sigmoid fit for binary probs.)

**Compute Expected Calibration Error (ECE)** - Partition predictions into M bins by score; for each bin b compute gap = |accuracy(b) - avg_score(b)| and weight by bin size.

text

ECE = sum_{b=1..M} (|B_b| / N) * | acc(B_b) - conf(B_b) |

This gives a summary miscalibration; complement with reliability diagram and classwise ECE.

**Monitoring calibration drift in production** - Continuous telemetry: compute ECE, per-class ECE, Brier score on rolling windows (e.g., last 7/30 days) and stratify by segment (region, cohort). - Statistical tests: population stability index or shifted-label tests; CIs for bin accuracies to detect significant change. - Alerts & mitigation: alert if ECE or Brier increases > threshold; automated retrain or recalibrate pipeline (recompute temperature scaling or isotonic map) using recent labeled data; holdout A/B to validate. - Best practice: store intermediate logits/scores, maintain labeled feedback loops, and record calibration metadata in model registry for reproducibility.

HardTechnical

51 practiced

Design a controlled experiment to measure the generalization gap between pretraining and downstream fine-tuning across multiple domains. Specify datasets and selection criteria, controlled variables (model size, compute budget), evaluation metrics, sample sizes, and statistical tests you would use to determine significance while isolating the effect of pretraining data size and domain mismatch.

Sample Answer

**Objective & hypothesis** Measure how pretraining data size and domain mismatch affect the generalization gap: gap = downstream fine-tuned performance − zero-shot/pretrained performance. Hypotheses: (H1) larger pretraining reduces gap; (H2) greater domain mismatch increases gap.

**Datasets & selection** - Pretraining corpora (synthetic control): curated subsets of CommonCrawl and domain-specific corpora (medical PubMed, legal, code). Select by token count: 10M, 100M, 1B, 10B tokens. - Downstream domains: image–text (COCO captions), medical (MIMIC-CXR), legal (ECtHR), code (HumanEval). Criteria: publicly available, diverse modalities, standardized metrics, enough examples for statistical power.

**Controlled variables** - Model architecture and size: fixed transformer family with 3 sizes (base, medium, large) but comparisons only within a size. - Compute budget: identical FLOPs for pretraining and fine-tuning per run. - Optimization/hyperparams: fixed schedules; run small grid search uniformly. - Random seeds: 10 seeds per condition.

**Experiment matrix** - Factors: pretraining size (4 levels) × pretraining domain (in-domain, related, unrelated, mixed) × model size (fixed per stratum). - For each cell: pretrain from scratch, evaluate zero-shot, fine-tune on downstream training split (n_train), evaluate on held-out test.

**Sample sizes & splits** - Downstream train/val/test: use standard splits; ensure test size ≥ 1000 examples per domain. - Power analysis: aim to detect effect size d=0.2 with 80% power → ~200 samples per cell for metric variance; we use 10 seeds to estimate variance and bootstrap test statistics.

**Metrics** - Primary: task-specific accuracy/F1/ROUGE/BLEU or EM; compute delta = fine-tune − zero-shot. - Secondary: calibration (ECE), sample efficiency curve (performance vs. n_train).

**Statistical tests & significance** - Use mixed-effects ANOVA with fixed effects: pretraining size, domain mismatch, their interaction; random effect: seed run. - Post-hoc: Tukey HSD for pairwise comparisons. - Complement with bootstrap CIs (95%) on deltas. - Correct for multiple comparisons (Benjamini-Hochberg).

**Isolation strategy** - Vary only pretraining size or only domain composition while holding other factors constant. Use matched-pair contrasts (same seed, same model init) between conditions to control run-level noise.

**Outcome interpretation & robustness** - Report effect sizes, CIs, and p-values. Visualize with interaction plots and learning curves. Run sensitivity analyses: different fine-tune data sizes, alternative metrics, and transfer direction checks (swap domains).

EasyTechnical

49 practiced

What is a feature store and why is it important in production ML? Describe the main components (feature registry, offline store, online store, materialization), how to maintain consistency between offline and online features, and give a simple example schema and access pattern for serving real-time features.

Sample Answer

**What is a feature store & why it matters (brief)** A feature store is a central system that standardizes feature definitions, storage, and access for ML. It ensures reproducibility, reduces training/serving skew, and accelerates model deployment by providing consistent offline (batch) and online (low-latency) feature access.

**Main components**- Feature registry: canonical feature definitions, versioning, metadata (owner, transform code, freshness SLA). - Offline store: large, cost-efficient store (e.g., BigQuery, S3 + parquet) for batch training and historical joins. - Online store: low-latency key-value DB (e.g., Redis, DynamoDB) for real-time inference. - Materialization pipeline: moves features from offline compute to online store, handles incremental updates, backfills, and TTL.

**Maintaining consistency**- Use the same transformation code (or serialized transformation graph) in registry for training and serving. - Include event timestamp / event-time semantics on features; join by event time during offline extracts. - Materialize using streaming or minibatch jobs that apply identical transforms and enforce idempotency. - Validate: CI tests, end-to-end feature parity checks (statistics, sample compares), automated alerts on freshness/skew.

**Example schema**- Entity: user_id (primary key) - Feature set: user_engagement - user_id: string - last_seen_ts: timestamp (event time) - avg_session_length_7d: float - purchases_30d: int - feature_version: string

**Real-time access pattern (example)** - Client sends user_id to prediction service. Prediction service calls online store: get(user_id) → returns latest features (ensuring TTL/freshness). If missing, fallback to on-demand computation or recent batch read.

Example call:

python

# simple pseudo-code for serving
features = feature_store.online.get(entity="user", key="user_123")
if not features:
    features = feature_store.on_demand.compute(user_id="user_123")
model_input = preprocess(features)
prediction = model.predict(model_input)

MediumTechnical

44 practiced

As an applied scientist leading a small team, describe how you would balance exploratory research experiments with short-term product deliverables. Include decision criteria, how you estimate experiment ROI, how you align priorities with stakeholders, and how you measure progress and outcomes.

Sample Answer

**Situation & high-level approach**I split team capacity into three lanes: 60% delivery (short-term product work), 25% exploratory research, 15% infrastructure/ops and skill development. I adjust the split quarterly based on product needs.

**Decision criteria**- Impact: expected improvement in core metric (CTR, revenue, latency).- Risk & uncertainty: technical feasibility and data readiness.- Time-to-value: how long to a usable prototype.- Strategic learning: unique IP or long-term differentiation.

**Estimating experiment ROI**I estimate ROI with a simple expected-value model:

text

Expected ROI = Probability(success) * Net benefit per unit time - Cost per unit time

Intuition: multiply likelihood of success by benefit (e.g., revenue uplift or latency reduction), subtract resource costs. I quantify benefits in product KPIs or engineering savings.

**Aligning priorities with stakeholders**- Monthly reviews with PMs/engineers to map experiments to KPI owners.- Define success criteria and fallback plans before kickoff.- Use a lightweight RACI and show expected ROI and timeline to inform trade-offs.

**Measuring progress & outcomes**- Short checkpoints: weekly status + quantitative metrics (validation loss, offline A/B estimates).- Gate reviews: prototype -> pilot A/B -> full rollout with predefined metrics and significance thresholds.- Post-mortem: document learnings, negative results, reproducible code and next steps.

Example: For a recommender improvement, I ran a 6-week probe (25% capacity), estimated 10% uplift with 40% success probability, ran an A/B pilot, and only promoted models that passed 95% CI on CTR and retention.

Unlock Full Question Bank

Get access to hundreds of Advanced ML Techniques & Research Application interview questions and detailed answers.

Join thousands of developers preparing for their dream job.