Assesses the candidate's ability to choose and justify statistical and machine learning algorithms for prediction and inference tasks and to compare model families across multiple dimensions. Candidates should know the strengths and weaknesses of common approaches including linear and logistic regression, decision trees, random forests, gradient boosting machines, support vector machines, nearest neighbor methods, and neural networks, and be able to explain when each is appropriate. Key comparison dimensions include interpretability, data and feature requirements, training and inference computational cost, memory footprint, scalability to production, sample complexity, and susceptibility to overfitting and underfitting. The topic covers evaluation metrics appropriate to the problem such as accuracy, precision, recall, F1, area under the receiver operating characteristic curve, mean squared error, mean absolute error, and R squared, along with validation strategies including cross validation, hold out sets, and bootstrapping. Candidates should discuss regularization techniques, early stopping, hyperparameter tuning, feature engineering and dimensionality reduction, and ensemble methods as tools to manage the complexity versus generalization trade off. Operational and robustness considerations are also important, including model calibration, monitoring, retraining frequency, latency and throughput constraints, model size, handling distribution shift and outliers, and stakeholder requirements for explainability and fairness. Interviewers may probe concrete decision making trade offs and expect candidates to justify preferring simpler interpretable models versus more complex models based on dataset characteristics, problem constraints, resource limits, and business needs.
EasyTechnical
49 practiced
A dataset contains 5% missing values and several extreme numeric outliers. Describe preprocessing strategies for missing values and outliers and explain how different missingness mechanisms (MCAR, MAR, MNAR) influence both imputation choices and model selection. Discuss when tree models might require less preprocessing.
Sample Answer
**High-level approach**- With only 5% missing and extreme numeric outliers, combine pragmatic and principled steps: quantify patterns, inspect distributions, then apply robust imputation/outlier handling with sensitivity checks.**Missing-value strategies**- Quick options: if missing rows <5% and MCAR, listwise deletion is acceptable.- Simple imputations: median (robust) or KNN for small feature sets.- Model-based: MICE / iterative imputer when relationships exist.- Missingness indicator: add a binary flag to capture informative missingness.- Validation: use cross-validation and, when possible, multiple imputation to reflect uncertainty.**Outlier strategies**- Robust scaling (median + IQR), winsorization (1–99%), or log transform for heavy right tails.- For extreme but correct values, prefer robust models (median-based) over truncation.- If outliers are errors, correct or remove after domain checks.**How missingness mechanism influences choices**- MCAR: missingness independent — unbiased listwise deletion or simple imputation okay.- MAR: missing depends on observed data — conditional/model-based imputation (MICE, chained models) recommended; include predictors of missingness in imputation models.- MNAR: missing depends on unobserved values — standard imputations biased; consider joint models that model missingness process, sensitivity analysis, or collect more data / use pattern-mixture models.**Model selection & trees**- Tree-based models (random forest, gradient boosting) tolerate outliers and can handle missingness natively (surrogate splits, built-in missing handling). They often need less scaling or winsorization.- However, for MNAR or when uncertainty quantification matters, still perform careful imputation and sensitivity checks even with trees.**Practical applied-scientist note**- Run experiments comparing pipelines (delete, median, MICE, indicator) using downstream metrics and calibration; document assumptions about missingness and report sensitivity to imputation/outlier choices.
HardTechnical
60 practiced
You evaluated two models' AUCs using a time-series cross-validation where folds are temporally correlated. Propose a statistical test and experimental design to compare their AUCs that accounts for temporal dependence and for multiple comparisons across time windows. Explain how to construct confidence intervals and report practical significance.
Sample Answer
**Clarify goal & constraints**I want a paired comparison of AUCs across temporally correlated CV folds, controlling family-wise / false-discovery across many time windows and producing CIs that respect dependence.**Recommended test & design**- Use a paired block bootstrap (moving-block or stationary bootstrap) on the *fold-level* paired AUC differences d_t = AUC_model1(t) − AUC_model2(t). Resample contiguous blocks of folds to preserve temporal dependence and recompute the mean difference Δ* per bootstrap replicate.- For hypothesis testing, compute the bootstrap distribution of Δ* and derive a two-sided p-value (proportion of replicates ≤0 or ≥0 as appropriate).**Multiple comparisons**- If comparing across multiple distinct time windows, apply a dependency-aware correction: - Prefer cluster-based permutation / bootstrap correction: identify contiguous time clusters where observed d_t exceed a threshold, compute max-cluster-statistic distribution under bootstrap, and control family-wise error. - Alternatively use Benjamini–Yekutieli (FDR under dependence) if FDR is acceptable.**Confidence intervals**- Construct bias-corrected and accelerated (BCa) bootstrap CIs from the block bootstrap replicates for the mean Δ and for each window; for cluster-CIs report intervals for cluster-level effect.- Report both CI and bootstrap-based p-value.**Practical significance**- Report absolute Δ (with CI), Cohen’s d on fold differences, and translate Δ into business metric (e.g., expected change in precision at operating threshold, revenue or false-alert reduction).- Provide decision rule: e.g., require lower bound of 95% CI > 0.01 AUC and positive business uplift to declare meaningful improvement.**Notes & diagnostics**- Choose block length by rule-of-thumb (≈ fold autocorrelation decay) or optimal selectors; verify stationarity within evaluation period.- Check bootstrap stability (≥5k replicates), show autocorrelation plots of d_t, and report sensitivity to block size.
EasyTechnical
63 practiced
Summarize bagging, boosting, and stacking ensemble approaches. For each, state typical failure modes, training and inference computational costs, and in which applied scenarios you would prefer one ensemble type over another (e.g., noisy labels, small data, need for interpretability).
Sample Answer
**Bagging (Bootstrap Aggregation)**- Summary: Train many independent base learners on bootstrap samples and average/vote. Reduces variance.- Failure modes: Correlated base learners (low diversity), biased weak models, limited benefit for very small datasets.- Costs: Training — parallelizable, O(m · T) where T is cost per model and m number of models; Inference — O(m · predict_time) (can be parallelized or pruned).- Best when: High-variance models (trees), noisy features but labels relatively clean, medium-to-large data; good when interpretability less critical.**Boosting**- Summary: Sequentially train models where each focuses on previous errors (e.g., AdaBoost, Gradient Boosting).- Failure modes: Overfitting with noise and mislabeled data, sensitivity to outliers, longer to tune.- Costs: Training — sequential and often expensive (O(m · T) but not parallel across m); Inference — O(m · predict_time).- Best when: Strong predictive performance needed, structured/tabular data, small-to-medium datasets; avoid if many label errors or when fast training is required.**Stacking (Stacked Ensembles)**- Summary: Combine heterogeneous models by training a meta-learner on out-of-fold predictions.- Failure modes: Data leakage if stacking improperly, overfitting meta-learner, increased complexity.- Costs: Training — high (train base models + meta-learner with cross-validation); Inference — depends on base set size (can be heavy).- Best when: Diverse model families available, limited single-model performance, competition-style setups; use when interpretability less critical and compute budget allows.Practical guidance: For noisy labels prefer bagging or robust boosting variants (e.g., robust loss); for very small data prefer simple models or careful boosting with regularization; for interpretability favor single transparent models or small bagged ensembles with feature importance.
EasyTechnical
60 practiced
Describe model interpretability techniques for tabular models: model coefficients, feature importances (Gini and permutation), partial dependence plots, ICE plots, SHAP, LIME, and counterfactual explanations. Explain their strengths, limitations, and how interpretability requirements affect algorithm selection and deployment.
Sample Answer
**Overview — why interpretability matters**As an applied scientist I treat interpretability as a functional requirement: compliance, debugging, user trust, and feedback for feature engineering. Different techniques provide local vs global, model-agnostic vs model-specific, and causal vs associative perspectives.**Techniques (strengths / limitations)**- Model coefficients (linear / logistic) - Strengths: direct global attribution, sign and magnitude interpretable, fast. - Limitations: only for linear models or GLMs; confounding and collinearity distort meaning.- Feature importances: Gini (tree-based) - Strengths: built-in, fast, global ranking. - Limitations: biased to high-cardinality features; not causal; scale-dependent.- Feature importances: Permutation - Strengths: model-agnostic, measures performance impact. - Limitations: expensive; correlated features can mask importance.- Partial dependence plots (PDP) - Strengths: show average marginal effect of a feature; good global view. - Limitations: assumes feature independence; hides heterogeneity.- Individual Conditional Expectation (ICE) plots - Strengths: reveals per-sample heterogeneity and interactions. - Limitations: can be noisy for many samples; interpretation still associative.- SHAP - Strengths: solid theoretical grounding (Shapley values), consistent local + global explanations, handles interactions (particularly TreeSHAP fast). - Limitations: compute cost for large models, background distribution choice affects attributions, can be misread as causal.- LIME - Strengths: flexible local surrogate explanations, model-agnostic, fast for single predictions. - Limitations: unstable across runs, sensitive to sampling and kernel; local linear surrogate may misrepresent complex decision boundaries.- Counterfactual explanations - Strengths: actionable: shows minimal changes to flip a prediction; useful for fairness/regulatory use-cases. - Limitations: may be infeasible or unrealistic without constraints; multiple solutions and no single ground truth.**How interpretability affects algorithm selection & deployment**- If transparency is required (regulation, stakeholders), prefer inherently interpretable models (GLM, decision rules) or use TreeSHAP for tree ensembles. - For latency-sensitive production, prefer fast explainers (coefficients, tree SHAP) or precompute attributions.- For debugging and feature engineering, combine global (PDP, permutation) and local (ICE, SHAP) views.- Always validate explanations: test stability, check correlated features, and enforce domain constraints for counterfactuals before exposing to users.Example practice: for credit scoring I used logistic regression with monotonic constraints for baseline compliance, and SHAP to audit nonlinear uplift models — balancing accuracy and explainability while operationalizing precomputed SHAP summaries for fast decisioning.
HardSystem Design
92 practiced
Design an automated model selection framework that recommends model families and hyperparameter budgets based on dataset metadata (size, feature types, sparsity, class balance), compute constraints, and explainability requirements. Describe candidate meta-features, a meta-learning approach (for example performance prediction or ranking), how to incorporate business constraints, evaluation loop, and handling of cold-start datasets.
Sample Answer
**Problem clarification & goals**Recommend model families (tree, linear, gradient-boosting, neural, sparse linear, embedding-based) and a hyperparameter budget given: dataset meta (n, p, types, sparsity, imbalance), compute limits, and explainability SLA. Optimize for expected downstream performance, latency, cost, and interpretability.**Candidate meta-features**- Size: n, p, n/p ratio, effective sample size (after dedup/NA)- Feature types: % numeric / categorical / text / image- Sparsity: average nonzero per row, density matrix- Label properties: class entropy, imbalance ratio, cardinality for multiclass- Statistical: skew/kurtosis, missingness patterns, correlation / multicollinearity (VIF)- Task: regression/classification, metric preference (AUC, F1, latency)- Compute: max GPU hours, memory, max latency- Explainability requirement: score 0–5 (post-hoc allowed, local vs global)**Meta-learning approach**- Two-stage: 1) Candidate family ranking using a learned ranker (pairwise ranking or list-wise LambdaMART) trained on historical runs with meta-features; 2) For top K families, performance prediction model (Gaussian Process or ensemble tree regression) that predicts validation metric as function of hyperparameter budget and dataset meta-features — outputting an expected curve (metric vs CPU/GPU-hours).- Use censored regression to handle truncated budgets. Incorporate uncertainty to trade off exploration.**Incorporating business constraints**- Constraint layer filters out families that violate explainability threshold (e.g., deep nets banned if global interpretability required) or compute ceilings.- Multi-objective utility: combine predicted performance, cost (compute*time), and explainability penalty into a utility function; allow stakeholder-weighted coefficients.**Evaluation loop & orchestration**- Offline: cross-validate meta-models via leave-one-dataset-out; measure ranking NDCG and regret.- Online/autotune: run short low-fidelity probes (subsample, fewer trees, fewer epochs) to refine predictions, update posterior (Bayesian optimization with meta-prior).- Feedback: archive run traces to retrain meta-models periodically.**Cold-start handling**- Use feature-based nearest-neighbor retrieval of similar datasets; transfer priors from clusters.- Default robust pipeline: start with proven families (lightgbm, logistic with regularization) with conservative budgets.- Active probing: small-budget multi-family probes to quickly gather informative signals, then re-rank.**Trade-offs & pitfalls**- Beware dataset shift between historical corpus and new dataset (use uncertainty-aware selection).- Meta-training bias: ensure diversity in historical runs and include low-resource examples.This framework balances meta-learning, constrained optimization, and online probing to recommend families and budget with interpretable constraints and measurable regret.
Unlock Full Question Bank
Get access to hundreds of Model and Algorithm Selection interview questions and detailed answers.
