Supervised Learning Algorithm Selection Questions

Covers selecting and comparing supervised machine learning algorithms for classification and regression tasks. Includes core algorithm families such as logistic regression, decision trees, random forests, gradient boosting implementations like XGBoost and LightGBM, support vector machines, and neural networks, as well as linear regression and regularization techniques including lasso, ridge, and elastic net. Candidates should be able to explain when each algorithm is appropriate based on interpretability requirements, dataset size, feature characteristics, computational constraints, training and inference latency, and robustness to noise and missing data. Also covers hyperparameter tuning, cross validation, bias variance trade offs, feature engineering impact, handling imbalanced classes, evaluation metrics for classification and regression, model calibration, and how to reason about theoretical foundations and senior level trade offs between algorithmic choices.

MediumTechnical

88 practiced

Design a small experiment to compare candidate supervised models on fairness and calibration. Specify dataset splits, metrics for fairness (e.g., demographic parity, equalized odds), calibration metrics, and statistical tests you would run to determine if differences are significant. Also describe how you'd report results to stakeholders.

Sample Answer

Situation & goal: Compare several supervised candidate models (e.g., baseline logistic, finetuned transformer, light GBM) on predictive performance while evaluating fairness and calibration to inform model selection and mitigation.

Experiment design- Data splits: single randomized split with stratification by label and protected attribute(s): Train 60%, Validation 20%, Test 20%. Ensure test set is held-out and untouched; if data limited, use nested CV with stratified folds and a final held-out test. Preserve subgroup representation (minimum n per group).- Preprocessing: same pipeline applied to all models; log any group-specific missingness.

Metrics- Predictive: accuracy / F1, ROC AUC.- Fairness: demographic parity difference and ratio (P(ŷ=1|A=a) differences), equalized odds gaps (max |TPR_a - TPR_b| and |FPR_a - FPR_b| across groups), and subgroup calibration (see below). Report direction and magnitude.- Calibration: Expected Calibration Error (ECE) with 10 bins, Maximum Calibration Error (MCE), Brier score, and reliability diagrams per subgroup.

Statistical testing & uncertainty- Use bootstrap (e.g., 1,000 resamples stratified by group and label) on the test set to compute 95% CIs for each metric and for pairwise differences between models.- For paired classification differences (same test examples), use McNemar’s test for accuracy; DeLong test for AUC differences.- For calibration/fairness gaps, use bootstrap hypothesis testing: compute distribution of gap under resamples; derive p-values for whether gap > threshold (practical significance). For multiple model comparisons, control FDR with Benjamini–Hochberg.- If sample sizes small per subgroup, run permutation tests to avoid distributional assumptions.

Analysis & reporting to stakeholders- Present a concise executive summary: recommended model, key trade-offs (e.g., +1.5% AUC but larger equalized odds gap).- Visuals: table of metrics with 95% CIs and significance annotations; reliability diagrams and calibration curves per subgroup; bar charts for TPR/FPR per group; distributional plots of predicted probabilities by group.- Decision criteria: show which models meet pre-defined fairness thresholds (e.g., demographic parity ratio within [0.8,1.25] or equalized odds gap < delta). Include costs/benefits and suggested mitigations (reweighing, post-hoc calibration, threshold adjustment, or constrained training).- Attach methodology appendix: dataset splits, seeding, number of bootstraps, statistical procedures to ensure reproducibility.

Takeaway: quantify uncertainty, report both statistical and practical significance, and provide actionable mitigation paths tied to business constraints.

EasyTechnical

80 practiced

Explain what model calibration means and why it's important for downstream decision-making. Compare Platt scaling, isotonic regression, and temperature scaling for binary and multiclass problems, and explain how you would evaluate calibration (e.g., calibration curves, Brier score).

Sample Answer

Model calibration means that a model’s predicted probabilities reflect true outcome frequencies — e.g., among samples predicted 0.8, roughly 80% are positive. Calibration is critical for downstream decisions (thresholding, risk scoring, resource allocation, ensembling, and any cost-sensitive action) because miscalibrated probabilities lead to wrong trade-offs even if ranking (AUC) is good.

Comparison of methods- Platt scaling (binary): Fit a logistic regression on model scores/logits: sigmoid(a·logit + b). Simple, parametric, corrects systematic bias (over/underconfidence). Works well when miscalibration is roughly sigmoid-shaped. Can extend to multiclass with one-vs-rest or vectorized variants but less common.- Isotonic regression (binary & multiclass via one-vs-rest): Non-parametric, fits a monotonic piecewise-constant mapping from scores to probabilities. Very flexible — can capture arbitrary monotonic distortions — but risks overfitting if calibration data is small.- Temperature scaling (primarily multiclass softmax models): Divide logits by a single temperature T learned on validation set, then apply softmax. Extremely simple, preserves class ranking, effective for common neural-network overconfidence, low risk of overfitting. For more flexibility you can learn class-wise temperatures or full matrix transforms (but risk overfitting).

How to evaluate calibration- Calibration curve / reliability diagram: Bin predicted probabilities and plot observed frequency vs predicted probability; ideal is y=x.- Metrics: Brier score (mean squared error between predicted probability and outcome) — lower is better and decomposes into calibration + refinement. Expected Calibration Error (ECE) and Maximum Calibration Error (MCE) summarize average or worst-bin deviation. Negative log-likelihood (cross-entropy) also penalizes miscalibration.- Use proper hold-out calibration set or cross-validation; report both discrimination (AUC) and calibration metrics; inspect sample counts per bin to avoid noisy curve interpretation.

Practical guidance: use temperature scaling for large neural nets as first try; if more complex shape needed and you have sufficient data, use isotonic; use Platt for simple binary corrections. Always validate on held-out data and report Brier/ECE plus reliability diagrams.

MediumTechnical

82 practiced

Explain when to create interaction or polynomial features for supervised learning and when it is unnecessary because of model choice (e.g., tree-based vs linear models). Provide practical steps to generate, select, and validate interaction features and discuss computational trade-offs.

Sample Answer

When to create interaction or polynomial features- If your model is linear (linear/logistic/GLM, linear SVM) it cannot capture feature multiplicative or nonlinear relationships, so explicitly add interaction (x1*x2) or polynomial (x^2, x^3) features when you suspect nonlinearity or domain knowledge suggests interactions.- If you use models that can learn interactions implicitly (decision trees, random forests, gradient-boosted trees, many neural nets), explicit engineered interactions are often unnecessary—unless data is small, interpretability is required, or you want to inject strong domain priors.

Practical steps to generate, select, and validate1. Hypothesis & scope: list candidate interactions from domain knowledge; limit degree (pairwise or up to degree 2/3).2. Generate features: use pipeline tools (PolynomialFeatures, FeatureUnion) with sparse handling and scaling for continuous variables; create categorical interactions via crossed categories or target encoding if high-cardinality.3. Pre-filter: remove near-constant combos, low-variance, or collinear engineered cols.4. Selection: use regularization (L1/Lasso, ElasticNet) with cross-validation for linear models; use tree-based feature importance, SHAP interaction values, or forward-stepwise selection for interpretable picks.5. Validation: nested CV to avoid selection bias; evaluate lift on held-out test set and check calibration; ensure improvements are robust across slices.

Computational trade-offs- Feature explosion: pairwise of p features → O(p^2) new columns; memory and compute grow quickly. Use hashing/truncation, sample-based screening, or interaction-aware models instead.- Sparsity: many interactions are sparse; use sparse matrices and algorithms that support them.- Training cost vs. model complexity: sometimes a nonlinear model (e.g., GBDT or small neural net) is cheaper than generating and selecting thousands of features for a linear model.

Rule of thumb- Start with simple models + domain-driven interactions. If interactions become numerous, prefer nonlinear learners or automatic interaction detection (trees, neural nets) and use rigorous CV and interpretability tools (SHAP) to confirm meaningful gains.

MediumTechnical

93 practiced

You are given a tabular dataset with 10,000 rows and 200 columns: 120 numeric, 60 categorical (many with missing values), and 20 text-derived sparse features (TF-IDF). The business needs a moderately interpretable model, training should finish within a few hours, and inference latency must be <200ms. As an AI Engineer, recommend a shortlist of 3 candidate supervised algorithms, justify each choice, and outline a quick evaluation plan.

Sample Answer

Recommended shortlist (3 models) with justification:

1) Gradient Boosted Trees (e.g., LightGBM / XGBoost)- Why: Handles mixed numeric/categorical and sparse inputs well, robust to missing values, strong baseline performance on tabular data. Supports feature importance for interpretability (SHAP). Training can be parallelized to meet a few-hour budget; inference is typically <200ms with reasonable tree count/depth.- Caveat: Tune number of trees and max_depth to balance latency.

2) Regularized Logistic Regression with Sparse-aware preprocessing- Why: Very interpretable coefficients, fast training/inference, naturally low-latency. Use one-hot / target / frequency encoding for categories, and keep TF-IDF features (sparse). Apply L1/L2 or elastic-net to handle high-dimensional sparse features and select features.- Caveat: May underperform if relationships are highly nonlinear.

3) Explainable Neural Network (wide & shallow MLP or TabNet-lite)- Why: Captures nonlinearities missed by linear models while controlling complexity for interpretability and latency. Use embedding layers for categorical vars, dense for numeric, and a small head for TF-IDF. Use feature attribution (integrated gradients, attention) for explanations.- Caveat: Requires careful regularization and early stopping.

Quick evaluation plan:- Preprocessing: consistent splits (stratified), impute missing, encode categories (compare target vs frequency), keep TF-IDF as-is or reduce via TruncatedSVD.- Metrics: business metric (AUC, RMSE, etc.), calibration, inference latency (p95), model size.- Workflow: 1. Baseline: simple logistic + TF-IDF + freq-encoding. 2. Grid/optuna tuning for LightGBM (early stopping) and regularized LR. 3. Train compact NN with small epochs, monitor latency. 4. Use 5-fold CV for performance and stability; record inference time on target hardware. 5. Interpretability: compute SHAP for tree and NN, present top features and partial dependence for key features.- Decision criteria: meets business metric within CI, inference <200ms p95, and acceptable interpretability.

MediumTechnical

76 practiced

You must choose supervised algorithms and evaluation strategies for a time-series forecasting classification problem where data is ordered and non-stationary. Explain appropriate validation strategies (rolling window, expanding window), how to prevent leakage, feature engineering considerations (lags, rolling stats), and which algorithms you would prefer or avoid.

Sample Answer

Start by framing goals: because data is ordered and non‑stationary you must respect time ordering in every step, validate with forward-looking splits, and avoid any form of “peeking” into the future.

Validation strategies- Rolling window (sliding): train on [t−W,t), validate on [t,t+V). Slide the window forward. Good when you expect recent behavior to be most relevant and want fixed training size.- Expanding window (walk‑forward): train on [0,t), validate on [t,t+V). Increase training set over time. Good when more historical data generally helps.- Implement a full walk‑forward cross‑validation: repeat many sequential train→validate steps, aggregate metrics (mean + CI). Use stratified time buckets if class balance varies.

Prevent leakage- Never shuffle time. All splits must be strictly earlier→later.- Compute features inside the CV loop: for each train/val fold, recompute lags/rolling stats using only past values from that fold (no global precomputed features).- Fit scalers/encoders on train fold and apply to val fold only.- Beware of label leakage: avoid features that use future target information (e.g., forward-looking aggregates).- For derived calendar features ensure they’re computable at prediction time.

Feature engineering- Lags: include several past values (t−1, t−2, …) tuned per problem. Use domain knowledge to pick relevant horizons.- Rolling stats: mean, std, min/max, skew over multiple windows (short/medium/long). Compute with causal windows (lookback only).- Differencing and percent change to remove trends and stabilize variance.- Seasonal and calendar features: hour/day/week/month, holidays, cyclical encodings (sin/cos).- Event flags, external covariates (weather, promotions) aligned and lagged appropriately.- Feature selection/regularization: avoid explosion of correlated lags; use importance-based selection, PCA on historical windows, or regularized models.

Algorithms: prefer vs avoid- Prefer: gradient-boosted trees (LightGBM/XGBoost/CatBoost) with time-respecting folds — robust with engineered features, handle missing values and non‑stationarity well. Sequence models (Temporal CNNs/TCN, LSTM, Transformer variants like Informer) when raw sequential patterns matter and you have enough data; combine with feature-based models in ensembles.- Avoid: standard k‑fold CV, methods that assume iid/stationarity without adaptation. Classic ARIMA/ETS are for stationary numeric forecasting and less direct for classification unless adapted.- Use online-learning or periodically retrained models for concept drift detection (ADWIN, DDM) and scheduled re-training.

Evaluation metrics and operationalization- Choose metrics aligned to business (precision/recall, AUC, F1, Brier for probability calibration). Evaluate per-fold and inspect temporal performance degradation.- Monitor model drift and set up retrain triggers; use backtests that simulate retraining cadence (daily/weekly).

This approach ensures honest performance estimation, prevents leakage, and balances engineered features with sequence models depending on data size and non‑stationarity.

Unlock Full Question Bank

Get access to hundreds of Supervised Learning Algorithm Selection interview questions and detailed answers.

Join thousands of developers preparing for their dream job.