ML Fundamentals Assessment Questions
Assessment of foundational machine learning concepts and techniques, including supervised and unsupervised learning, common algorithms, evaluation metrics, bias-variance trade-offs, overfitting, feature engineering, and model selection. Used for screening ML knowledge in interviews, training, or onboarding within ML & AI contexts.
HardSystem Design
29 practiced
Design a production ML system for near-real-time fraud detection with these constraints: 5,000 inference QPS, P50 latency <50ms, access to both streaming real-time features and daily batch features, online updates to model once per day, and strict monitoring and rollback requirements. Provide architecture components, storage choices, feature serving strategy, and trade-offs.
Sample Answer
Requirements:- 5k QPS, P50 <50ms- Real-time streaming features + daily batch features- Model updated once/day online- Strict monitoring + fast rollbackHigh-level architecture:- Ingress (API Gateway / load balancer) → Feature Fetcher (hybrid: online feature store + batch feature accessor) → Model Serving cluster (stateless gRPC/TensorFlow Serving) → Decision API → Async Logging → Monitoring & RollbackComponents and choices:1. Model serving- Kubernetes autoscaled pods running a lightweight, optimized model (TF-Serve / TorchServe or custom C++ gRPC) behind a low-latency LB.- Use model sharding and replicated instances to handle 5k QPS. With avg 2ms inference, 5k QPS needs ~10-20 pods depending on concurrency.- Use batching at server side with tight latency budgets (micro-batch 1–10ms) if model allows.2. Feature storage & serving- Real-time (streaming) features: use an online feature store (Feast or custom) backed by low-latency KV store like Redis or DynamoDB (DAX) for sub-ms reads. Stream processors (Kafka + Flink) materialize aggregates (e.g., rolling counts) into Redis.- Daily batch features: stored in a columnar store (BigQuery / Snowflake / Parquet on S3). Pre-materialize and push to online store at daily update time for joins.- Hybrid join: request path queries online store only. If a needed batch feature is large, pre-populate into online store during nightly job.3. Feature pipeline- Ingest events into Kafka. Stream jobs compute session/rolling features and update Redis. Nightly ETL computes heavy features and upserts to online store. Use consistent schemas and feature registry.4. Online learning / model update- Daily retrain in batch. Validate with shadow traffic and backtesting. Deploy new model as canary (small %), then ramp to 100% if metrics pass. Use model registry (MLflow or Sagemaker) and automated CI checks. Use blue-green or traffic-splitting for zero-downtime swaps.5. Monitoring & rollback- Extensive telemetry: latency, QPS, feature drift, prediction distribution, false positive/negative rates, business metrics. Use Prometheus + Grafana + alerts.- Automated rollback triggers: threshold breaches on latency, prediction shifts, or degradation in key business metrics. Rollback via model registry to previous stable version (instant routing change).Trade-offs:- Redis gives ultra-low latency at cost of memory; keep hot features only. Cold/batch features stored in cheaper columnar stores.- Strict P50 <50ms forbids expensive on-request joins; precompute/denormalize features.- Micro-batching improves throughput but risks latency—keep batch sizes tiny or disable if latency increases.- Canarying adds deployment complexity but reduces risk.- Online feature store consistency: eventual consistency accepted for some aggregates; for strict correctness consider transactional writes at higher cost.Scalability & cost:- Autoscale serving by CPU/GPU and use instance pools. Optimize model (quantization, pruning) to reduce footprint.- Use TTL eviction for Redis to control memory.Edge cases:- Missing features: use safe defaults and monitor missing-rate metric.- Backpressure: graceful degradation to cheaper rules-based model or queued processing.This design balances sub-50ms median latency with hybrid feature availability, safe daily model updates, and strict monitoring + rollback.
MediumTechnical
25 practiced
Write a Python function that takes validation-set predicted probabilities and true labels and returns the threshold that maximizes F1 score. Describe how you would avoid overfitting by selecting a threshold and how to apply thresholding in production safely.
Sample Answer
Approach: sweep candidate thresholds on validation predicted probabilities, compute F1 for each, and pick the threshold with maximum F1. To avoid overfitting, tune threshold on a hold-out validation set (or via cross-validation / nested CV) separate from training and final test. In production, apply calibrated probabilities, guard with margins, monitor data drift, and log decisions.Key concepts:- Use a high-resolution sweep (or search on unique probs) to find maximum F1.- Complexity: O(n * m) where n = number of samples, m = thresholds (or O(n log n) if scanning unique probs).- Edge cases: no positive labels (return threshold=1.0 or handle separately), constant probs (tie-breaker), highly imbalanced data (F1 may be unstable).Avoiding overfitting when selecting threshold:- Reserve a separate tuning/validation set distinct from training and final test.- Prefer nested CV or repeated stratified CV to estimate threshold stability.- Use bootstrapping to estimate variance of the chosen threshold; prefer thresholds that generalize (stable across folds).- If model outputs are poorly calibrated, calibrate probabilities (Platt scaling/isotonic) before threshold selection.Production safety:- Calibrate model probabilities in deployment or calibrate periodically using recent labeled data.- Use a margin (e.g., only accept positive if prob > best_thr + delta) for high-cost mistakes or add a "defer to human" region near the threshold.- Log scores, decisions, and label feedback; monitor metrics and data distribution for drift.- If threshold is updated, perform A/B test or canary rollout; keep deterministic decision logic and feature pipelines identical between validation and production.
python
import numpy as np
from sklearn.metrics import f1_score
from sklearn.calibration import CalibratedClassifierCV
def best_threshold_by_f1(probs, y_true, num_thresholds=1000):
"""
probs: 1D array of predicted probabilities for positive class
y_true: 1D array of binary true labels (0/1)
Returns: best_threshold (float), best_f1 (float)
"""
probs = np.asarray(probs)
y_true = np.asarray(y_true)
assert probs.shape == y_true.shape
# Candidate thresholds from 0 to 1 (include extremes)
thresholds = np.linspace(0.0, 1.0, num_thresholds)
best_f1 = -1.0
best_thr = 0.5
for t in thresholds:
preds = (probs >= t).astype(int)
# handle degenerate case where predictions are all one class
try:
f1 = f1_score(y_true, preds)
except ValueError:
f1 = 0.0
if f1 > best_f1:
best_f1 = f1
best_thr = t
return best_thr, best_f1MediumTechnical
31 practiced
Implement the forward pass of batch normalization (training mode and inference mode) in PyTorch-like pseudocode. Your implementation should update running mean/variance in training and use them in evaluation. Explain the role of gamma (scale) and beta (shift) parameters and numerical stability considerations.
Sample Answer
Approach: compute per-channel mean/variance across batch (and spatial dims for conv), normalize, scale+shift with gamma/beta. In training update running estimates with momentum; in eval use running stats. Add eps for numerical stability.Key points:- Gamma scales normalized activations (learns channel-wise variance adjustment); beta shifts (learns mean offset).- eps prevents divide-by-zero and stabilizes sqrt for tiny variance.- Use population variance (unbiased=False) to match common frameworks.- Momentum controls running estimate responsiveness.- Complexity O(N*C*spatial). Edge cases: small batch sizes (use Group/LayerNorm alternatives), channels with zero variance.
python
def batch_norm_forward(x, gamma, beta, running_mean, running_var, momentum=0.1, eps=1e-5, training=True):
"""
x: shape (N, C, ...) (for FC: (N, C), for conv: (N, C, H, W))
gamma, beta: shape (C,)
running_mean, running_var: shape (C,)
"""
# reshape to (N, C, -1) to compute per-channel stats
N, C = x.shape[0], x.shape[1]
x_flat = x.reshape(N, C, -1)
if training:
# compute batch mean/var over N * spatial
mean = x_flat.mean(axis=(0,2)) # shape (C,)
var = x_flat.var(axis=(0,2), unbiased=False) # population var
# normalize
x_hat = (x - mean.reshape(1,C,*(1,)*(x.ndim-2))) / np.sqrt(var.reshape(1,C,*(1,)*(x.ndim-2)) + eps)
# update running stats
running_mean[:] = (1 - momentum) * running_mean + momentum * mean
running_var[:] = (1 - momentum) * running_var + momentum * var
else:
# use running stats in eval
x_hat = (x - running_mean.reshape(1,C,*(1,)*(x.ndim-2))) / np.sqrt(running_var.reshape(1,C,*(1,)*(x.ndim-2)) + eps)
out = gamma.reshape(1,C,*(1,)*(x.ndim-2)) * x_hat + beta.reshape(1,C,*(1,)*(x.ndim-2))
cache = (x, x_hat, mean if training else running_mean, var if training else running_var, gamma, eps)
return out, running_mean, running_var, cacheHardTechnical
31 practiced
Compare uncertainty estimation approaches for deep learning models: Bayesian neural networks, MC-dropout, deep ensembles, and temperature scaling/Platt scaling for calibration. Distinguish aleatoric and epistemic uncertainty, compare computational costs, and recommend practical approaches for production.
Sample Answer
Short answer: aleatoric uncertainty is irreducible noise in the data (measurement noise, label ambiguity); epistemic uncertainty is reducible uncertainty in the model (lack of data, OOD inputs). Good production practice is to capture both: model aleatoric directly (heteroscedastic loss / predictive variance) and capture epistemic with ensembles or approximate Bayesian methods; calibrate probabilities post-hoc (temperature / Platt) to improve decision thresholds.Comparison:- Bayesian Neural Networks (BNNs) - What: place distributions over weights; principled posterior predictive captures epistemic. - Pros: theoretically grounded; captures full posterior uncertainty. - Cons: high implementation and compute complexity (MCMC or VI), tricky tuning, slow training/inference. - Cost: high training + inference.- MC-Dropout - What: use dropout at inference as approximate Bayesian sampling. - Pros: easy to add to existing models; cheap to implement. - Cons: approximation quality varies; may underestimate uncertainty; requires multiple forward passes. - Cost: moderate — inference cost = T forward passes.- Deep Ensembles - What: train K independently initialized models; aggregate predictions. - Pros: strong empirical performance for epistemic uncertainty and robustness; simple and reliable. - Cons: K× train cost and inference cost; storage and orchestration overhead. - Cost: high (but embarrassingly parallelizable).- Temperature Scaling / Platt Scaling (Calibration) - What: post-hoc scalar (or small model) to adjust softmax logits to better match observed probabilities. - Pros: extremely cheap, preserves accuracy, reduces Expected Calibration Error (ECE). - Cons: does not change ranking or epistemic behavior, won’t fix OOD under-confidence/over-confidence by itself. - Cost: negligible.Practical recommendations for production (ML engineer perspective):- Start simple: train a single model with heteroscedastic output if aleatoric uncertainty matters (regression or classification with label noise).- For epistemic uncertainty and robustness, prefer deep ensembles (3–5 members) if compute allows — best empirical trade-off. Use model-parallel training or checkpoint ensembling to reduce costs.- If constrained on resources, use MC-dropout (20–50 samples) or last-layer Bayesian approximations (e.g., Laplace or SWAG) as a lighter alternative.- Always calibrate predicted probabilities with temperature scaling on held-out val data; evaluate with NLL, ECE, Brier, and OOD/AUROC.- For extreme resource/latency constraints, consider distilling ensemble predictions into a single student model (ensemble distillation) to approximate uncertainty at lower cost.- Monitor production: track calibration drift, input covariate shift, and flag high epistemic uncertainty for human review or data collection.Metrics & ops tips:- Evaluate predictive mean/variance, NLL, ECE, and OOD detection (AUROC).- Use thresholds on epistemic uncertainty to gate actions (defer to human), and retrain when epistemic uncertainty concentrates on regions of real inputs.- Prefer pragmatic, testable solutions: ensembles + temperature scaling + heteroscedastic loss is a reliable baseline for production. BNNs are best for research or when you must adhere strictly to Bayesian posterior semantics.
HardTechnical
26 practiced
Explain VC dimension and its implications for sample complexity and generalization. Give concrete examples: VC dimension of linear classifiers in d dimensions, and the intuition for VC dimension of decision trees. Discuss limitations of VC dimension theory for modern over-parameterized neural networks.
Sample Answer
VC dimension (VCdim) measures the capacity of a hypothesis class H: it is the largest number n such that there exists a set of n points that H can label in all 2^n possible ways (i.e., H shatters the set). It links to generalization via uniform convergence: with high probability, the true risk and empirical risk are close for all h ∈ H once sample size m = O((VCdim + log(1/δ))/ε^2). So larger VCdim → more samples needed to guarantee small generalization gap.Concrete examples- Linear classifiers in R^d: For affine hyperplanes (with bias) the VC dimension is d+1. Intuition: you can place d+1 points in general position and realize any labeling by choosing an appropriate separating hyperplane; you cannot shatter d+2 in general. For homogeneous hyperplanes through origin (no bias) VCdim = d.- Decision trees: VCdim grows with tree size. A binary decision tree with s internal nodes (or L leaves) can realize very complex partitions; for axis-aligned trees on continuous features a depth-k tree can shatter up to about 2^k points (so VCdim is O(#leaves) and grows exponentially with depth). Intuition: each split multiplies the number of distinct regions, enabling many labelings.Limitations for modern over-parameterized neural networks- Classical VC bounds are often vacuous for deep nets: parameter counts (and hence worst-case VCdim) are huge, predicting enormous sample needs which contradicts empirical generalization.- They are distribution-free and worst-case: they ignore implicit regularization from SGD, architecture, margin effects, or data structure.- Better modern explanations are data-dependent (norms, margins, PAC-Bayes, compression bounds) and consider implicit biases, stability, and effective capacity. Phenomena like double descent and interpolation show that capacity measured by parameter count/VCdim doesn’t fully capture real generalization behavior in over-parameterized regimes.Takeaway: VC dimension gives a clean, foundational link between model complexity and sample complexity, and provides useful intuition (e.g., linear classifiers scale with dimension; tree capacity grows with depth), but for modern deep learning you need refined, data-dependent theories to explain observed generalization.
Unlock Full Question Bank
Get access to hundreds of ML Fundamentals Assessment interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.