Assesses the candidate's ability to choose and justify statistical and machine learning algorithms for prediction and inference tasks and to compare model families across multiple dimensions. Candidates should know the strengths and weaknesses of common approaches including linear and logistic regression, decision trees, random forests, gradient boosting machines, support vector machines, nearest neighbor methods, and neural networks, and be able to explain when each is appropriate. Key comparison dimensions include interpretability, data and feature requirements, training and inference computational cost, memory footprint, scalability to production, sample complexity, and susceptibility to overfitting and underfitting. The topic covers evaluation metrics appropriate to the problem such as accuracy, precision, recall, F1, area under the receiver operating characteristic curve, mean squared error, mean absolute error, and R squared, along with validation strategies including cross validation, hold out sets, and bootstrapping. Candidates should discuss regularization techniques, early stopping, hyperparameter tuning, feature engineering and dimensionality reduction, and ensemble methods as tools to manage the complexity versus generalization trade off. Operational and robustness considerations are also important, including model calibration, monitoring, retraining frequency, latency and throughput constraints, model size, handling distribution shift and outliers, and stakeholder requirements for explainability and fairness. Interviewers may probe concrete decision making trade offs and expect candidates to justify preferring simpler interpretable models versus more complex models based on dataset characteristics, problem constraints, resource limits, and business needs.
MediumSystem Design
64 practiced
Discuss practical trade-offs between training on GPUs and performing inference on CPUs for deep learning models. Include considerations around model selection (smaller CNN vs large transformer), batching, latency, throughput, and cost. How would these trade-offs influence your algorithm selection for a latency-sensitive product feature?
Sample Answer
Clarify requirements:- Latency target (p99), throughput (requests/sec), request pattern (sporadic vs steady), and cost constraints. Also consider model type (vision vs text), input size, and acceptable accuracy/quality drop.High-level trade-offs- Training: GPUs are essential for fast iteration, large-batch SGD, mixed precision, and large models (transformers). CPUs are impractical for training large models due to slow matrix ops.- Inference: CPUs can be cost‑effective for small models and sparse traffic; GPUs shine for large models, heavy batching, or when low-latency on large inputs is required.Model selection impact- Small CNN (mobile/quantized): fast on CPU with vectorized BLAS, AVX, or edge NPUs. Lower memory, enables sub-10ms latency per image without GPU.- Large transformer: high FLOPs and memory bandwidth — CPU inference often violates latency or throughput SLAs. Requires GPU, TPU, or model compression (distillation, pruning, quantization) to run on CPU.Batching, latency, throughput- Batching increases throughput but adds queueing delay. For latency-sensitive features, micro-batches or batch size=1 preferred.- GPUs favor larger batches to amortize kernel overhead; CPUs can handle many small concurrent requests but saturate at core/IPC limits.Cost considerations- Per-inference cost: CPU instances cheaper hourly but slower per request; GPU instances expensive but may reduce per-inference time for heavy models.- For sporadic traffic, serverless CPU/auto-scaling may be cheaper. For predictable high QPS with large models, pooled GPUs/FP16 kernels reduce cost per request.Practical algorithm selection for latency-sensitive product1. Aim for smallest model meeting accuracy: distill large transformer into a compact transformer/CNN; prefer architectures designed for efficiency (MobileNet, EfficientNet-lite, DistilBERT).2. Apply quantization-aware training and pruning to allow CPU deployment.3. Use model sharding: run lightweight front-end on CPU for filtering, escalate to GPU for complex cases.4. Optimize inference path: ONNX/TensorRT for GPU, OpenVINO or XNNPACK for CPU, fuse ops, use async I/O and warm pools.5. Choose batching policy: batch size=1 with request coalescing window tuned to latency budget.Example decision:- 50ms p99 image classification on mobile: choose MobileNetV3 quantized, serve on CPU edge or optimized server CPU.- 200ms p99 contextual QA needing a transformer: use distilled/quantized transformer on GPU with small micro-batches and a CPU front-end cache for common queries.Trade-offs summary:- Prioritize model compression and CPU-optimized runtimes for tight latency and low cost.- Use GPUs when model complexity or throughput demands exceed CPU capability; balance batch strategy to meet latency SLA.
EasyTechnical
49 practiced
Explain L1 (lasso) and L2 (ridge) regularization for linear models. Describe their mathematical effect on coefficients, their influence on feature selection and sparsity, and situations where you would prefer one over the other in model selection.
Sample Answer
L1 (lasso) and L2 (ridge) regularization add penalties to the linear regression loss to prevent overfitting and control coefficient size.Math:- Ridge (L2): minimize ||y - Xw||^2 + λ||w||_2^2 where ||w||_2^2 = Σ w_j^2. The penalty is quadratic.- Lasso (L1): minimize ||y - Xw||^2 + λ||w||_1 where ||w||_1 = Σ |w_j|. The penalty is linear in absolute value.Effect on coefficients:- L2 shrinks coefficients continuously toward zero; solutions remain dense. It favors small weights distributed across correlated features (reduces variance).- L1 promotes exact zeros because the L1 penalty has sharp corners at zero; this produces sparse solutions and performs implicit feature selection.Feature selection and sparsity:- L1 is useful when you expect many irrelevant features and want automatic feature selection.- L2 keeps correlated predictors with shared weight rather than dropping them; better when all features carry some signal.When to prefer which:- Prefer L1 (lasso) for interpretability, model simplicity, or when p >> n and many features are irrelevant.- Prefer L2 (ridge) when multicollinearity exists, when you want stable predictions rather than sparse interpretation, or in neural nets (L2 = weight decay) to regularize weights without forcing sparsity.- Elastic Net (combination of L1 and L2) is a good compromise when you want sparsity but also stability with correlated features.Tuning: choose λ by cross-validation; analyze coefficient paths vs λ to understand regularization behavior.
EasyTechnical
93 practiced
You have a very small labeled dataset: n = 200 examples and p = 1,000 features (sparse). Which families of models would you consider first and why? Discuss data-level and model-level strategies (feature selection, regularization, linear vs tree vs neural nets) to get good generalization under this sample-complexity constraint.
Sample Answer
Start with models that have low sample complexity and exploit sparsity: regularized linear models and simple tree/ensemble variants.Model families & why- Linear: logistic/linear regression with L1 (Lasso) or Elastic Net — handles p≫n, yields sparse solutions and interpretable coefficients; fast with sparse matrices (liblinear/SGD).- Linear SVM (or SGD-SVM): robust to high-dim sparse data.- Tree-based: shallow decision trees or regularized gradient-boosted trees (max depth small, heavy L2, subsample). Trees can capture nonlinearity but require strong regularization to avoid overfit.- Neural nets: avoid training deep nets from scratch. Only consider tiny networks or transfer learning / pretrained embeddings (feature extractor), then train a simple classifier on top.Data-level strategies- Feature selection: start with univariate filters (chi², mutual information), then model-based selection (L1, recursive feature elimination, stability selection). Keep features few — aim for O(n / 5–10).- Dimensionality reduction: PCA/TruncatedSVD or domain-specific embeddings (e.g., TF-IDF + SVD for text).- Augmentation / semi-supervised: if possible, leverage unlabeled data (self-training, pseudo-labeling) or weak supervision.- Feature engineering: combine sparse signals, bin continuous features, normalize.Model-level strategies- Strong regularization: L1/L2 or Elastic Net, early stopping for boosting/SGD.- Nested cross-validation for hyperparameter tuning to avoid optimistic bias (e.g., 5×2 CV).- Use sparse-aware solvers and class weighting if imbalanced.- Ensembles: shallow ensembles (bagging or modest GBM) after careful validation; stacking only if enough hold-out data.- Uncertainty estimation: calibrate probabilities (Platt/Isotonic) and report confidence intervals.Practical workflow1. Exploratory filter to remove obvious noise/features.2. Train L1-regularized logistic and linear SVM as baselines.3. If nonlinearity suspected, try shallow GBM with aggressive regularization.4. Use nested CV, report validation variance, and prefer simpler model if performance similar.Rationale: with n=200 and p=1000, bias toward simpler, regularized models reduces variance and leverages sparsity; data-level reduction and transfer learning can effectively increase sample size.
MediumTechnical
66 practiced
List regularization techniques commonly used with gradient-boosted trees and describe how each helps reduce overfitting (e.g., shrinkage/learning-rate, subsampling rows/features, max-depth, min_child_weight, column-subsample, L1/L2 penalties). Describe diagnostics you would use to detect overfitting in boosting models.
Sample Answer
Common regularization techniques for gradient-boosted trees and how they reduce overfitting:- Learning rate (shrinkage): scale each tree’s contribution (e.g., 0.01–0.1). Slower learning requires more trees but reduces chance of fitting noise.- Subsample (row sampling): train each tree on a random subset of rows (e.g., 0.5–0.8). Adds stochasticity, reduces variance and overfitting.- Colsample_bytree / colsample_bylevel (feature subsampling): use random subsets of features per tree/level. Prevents reliance on a few strong predictors, reduces correlation between trees.- Max_depth / max_leaf_nodes: limit tree complexity. Shallower trees capture simpler patterns and generalize better.- min_child_weight / min_samples_leaf: require minimum data in leaves to create splits; prevents fitting tiny noisy partitions.- L1 (alpha) and L2 (lambda) regularization on leaf weights: shrink coefficients, encourage sparsity (L1) or small weights (L2).- gamma (min_split_loss): require minimum loss reduction to make a split, discouraging weak splits.- Early stopping: monitor validation metric and stop when no improvement; prevents training past optimum.Diagnostics to detect overfitting:- Train vs validation learning curves: diverging training loss much lower than validation loss indicates overfit.- Validation metrics (AUC, RMSE) plateau or degrade while train improves.- Feature importance and SHAP: overly concentrated importance on few features or high-magnitude SHAP on training-only examples suggests memorization.- Calibration plots and prediction distribution: unrealistically confident predictions on train vs val.- Cross-validation stability: large variance across folds suggests model is fitting fold-specific noise.- Permutation importance / leave-one-feature-out: check robustness of feature effects.Typical mitigation: reduce complexity (depth/min_child_weight), increase subsampling, lower learning rate with more trees, add L1/L2, and use early stopping with proper CV.
HardTechnical
47 practiced
In a regulated domain like healthcare, how do you balance interpretability and predictive accuracy during model selection? Propose a framework that includes technical choices, validation requirements, documentation, stakeholder communication, and fallback plans if a black-box model is needed but regulators request interpretability.
Sample Answer
Situation: Building predictive models for clinical risk stratification where regulators require explainability and clinicians need high accuracy.Framework — decision process and artifacts:1. Requirements & constraints- List functional (accuracy, latency), regulatory (GDPR, FDA guidance), clinical (acceptable risk thresholds), and stakeholder needs (clinician trust, patient consent).- Define minimum explainability requirement (local vs global) and measurable fairness metrics.2. Technical choices (tiered)- Start with interpretable baselines: logistic regression with engineered features, decision trees, generalized additive models (GAMs).- If performance insufficient, escalate to constrained-complexity models: monotonic GBMs, attention-based shallow networks with feature attribution regularization.- As last resort, black-box (deep networks) with strong post-hoc explainers (SHAP, integrated gradients), concept bottlenecks, or surrogate interpretable models.3. Validation & evaluation- Hold-out and nested cross-validation, performance on clinically relevant metrics (AUROC, calibration, decision-curve analysis).- Robustness: adversarial tests, distribution-shift evaluation, subgroup fairness, and clinical-scenario simulations.- Explainability tests: stability of explanations, fidelity of surrogate models, and human-grounded evaluation with clinicians (does explanation change trust/decision?).4. Documentation & compliance artifacts- Model card, data provenance log, training pipeline, hyperparameters, validation reports, fairness and risk assessments, and deployment runbooks.- Record limitations and use-cases. Prepare audit-ready notebooks and reproducible CI pipelines.5. Stakeholder communication- Structured briefings: technical summary, risks/benefits, explanation examples mapped to clinical concepts.- Interactive sessions with clinicians/regulators showing counterfactuals, per-patient explanations, and failure modes.- Consent and patient-facing materials explaining automated decisions.6. Fallback & remediation if regulator demands interpretability- Offer interpretable surrogate model with fidelity threshold: deploy surrogate in parallel for triage while black-box handles complex cases.- Implement “human-in-the-loop” gating: require clinician sign-off for high-risk predictions.- Reduce scope: deploy black-box only for non-decision-critical tasks or internal research; keep any regulated decision-making to interpretable models.- Propose a phased plan: pilot with extensive monitoring and independent audit; commit to additional constraints (feature limits, monotonicity) to increase transparency.Why this works:- Prioritizes safety and trust by defaulting to interpretable models, escalates only when measurable benefit outweighs risks, and provides auditability and human oversight. This aligns technical choices with regulatory and clinical decision-making needs while preserving pathways to leverage advanced models responsibly.
Unlock Full Question Bank
Get access to hundreds of Model and Algorithm Selection interview questions and detailed answers.
