Core supervised and unsupervised machine learning algorithms and the theoretical principles that guide their selection and use. Covers linear regression, logistic regression, decision trees, random forests, gradient boosting, support vector machines, k means clustering, hierarchical clustering, principal component analysis, and anomaly detection. Topics include model selection, bias variance trade off, regularization, overfitting and underfitting, ensemble methods and why they reduce variance, computational complexity and scaling considerations, interpretability versus predictive power, common hyperparameters and tuning strategies, and practical guidance on when each algorithm is appropriate given data size, feature types, noise, and explainability requirements.
MediumTechnical
21 practiced
Compare Support Vector Machines (SVM) and logistic regression for binary classification. Cover the differences in objective (hinge vs log-loss), margin interpretation, the role of regularization, use and cost of kernels, typical complexity in training and memory, and practical advice on when to use linear SVMs, kernel SVMs, or logistic regression in production.
Sample Answer
Objective:- Logistic regression minimizes log-loss (cross-entropy): it models P(y=1|x) via a sigmoid and optimizes likelihood — probabilistic outputs and well-calibrated scores.- SVM (binary) minimizes hinge loss with an explicit margin term: it finds a separating hyperplane maximizing margin (with slack via C or soft margin). The hinge loss focuses on points near the decision boundary (support vectors).Margin interpretation:- SVM explicitly maximizes the geometric margin; solutions are determined by support vectors.- Logistic has no explicit geometric margin maximization; margin is implicit via regularization but model weights trade off fit vs. complexity.Regularization:- Both use L2 (or L1) regularization. In logistic, regularization controls overfitting and calibration; in SVM, the C parameter trades off margin size vs. misclassification (equivalently weight regularizer). L1 gives sparsity in both but affects optimization differently.Kernels and cost:- Linear logistic and linear SVM scale to large sparse problems; kernel SVMs (RBF, polynomial) map data implicitly—powerful for nonlinear decision boundaries but require computing/storing an n×n Gram matrix (or approximations). Kernel SVM training is typically O(n^2)–O(n^3) time and O(n^2) memory, so impractical for large n without approximations (SMO helps but still scales poorly).- Logistic regression can be kernelized too (kernel logistic), but it's less common due to similar scaling and higher optimization cost.Training complexity & memory:- Linear solvers (LBFGS/SGD/coordinate descent/liblinear) are O(nd) per pass and memory O(d) (or O(nnz) for sparse).- Kernel SVMs scale poorly with n; runtime and memory blow up for large datasets.- SVMs (nonlinear) often require quadratic programming; logistic via convex optimization (fast stochastic methods) is usually faster and easier to parallelize.Practical advice for production:- Use logistic regression when you need calibrated probabilities, interpretability, fast training/prediction, and scale to large datasets. Good baseline and for downstream probabilistic decisions.- Use linear SVM when you want a robust max-margin classifier for high-dimensional sparse data (text, TF-IDF) and you care about margin-driven generalization; performance similar to linear logistic often—training choice depends on available solvers and tuning (C).- Use kernel SVM only for smaller datasets where nonlinear boundaries are essential and n is modest (thousands). For larger nonlinearity needs, prefer kernel approximations (random Fourier features, Nystrom) or move to nonlinear models (tree ensembles, neural nets) that scale better.- Always cross-validate regularization and kernel hyperparams; prefer logistic if probability calibration is required; scale/normalize features for both; monitor support vector count for SVM prediction latency.
EasyTechnical
23 practiced
Compare L1 (Lasso) and L2 (Ridge) regularization for linear models. Explain how each penalty affects coefficients (sparsity, shrinkage), behavior with correlated features, numerical/optimization considerations, and give guidance for choosing between L1, L2, and Elastic Net on a high-dimensional correlated dataset in production.
Sample Answer
L1 (Lasso) vs L2 (Ridge) — short, practical comparison for linear models:Sparsity and shrinkage- L1 (Lasso): Adds |β| penalty. Encourages exact zeros → feature selection; yields sparse models. Shrinkage is uneven (some coefficients zeroed, others less penalized).- L2 (Ridge): Adds β^2 penalty. Shrinks all coefficients continuously toward zero but rarely to exactly zero → no sparsity, more stable small coefficients.Behavior with correlated features- Lasso: Tends to pick one feature from a correlated group and zero the rest (unstable which one). Useful if you want a compact model but can be inconsistent.- Ridge: Distributes weights across correlated features (grouping effect), more stable when predictors are collinear.Numerical / optimization considerations- Both are convex; Ridge has closed-form (normal equations with shrinkage) and smooth gradients; numerically stable after feature standardization. Lasso is non-differentiable at 0 — solved via coordinate descent, ISTA/FISTA or LARS; needs careful regularization path and warm starts. Always standardize features; tune regularization strength by CV. For very high-dimensional data, Lasso can be more computationally efficient due to sparsity but may struggle with many correlated predictors.Guidance for high-dimensional, correlated datasets in production- Use Elastic Net (combines L1+L2) as default: L1 gives sparsity for interpretability and L2 stabilizes selection among correlated features. Cross-validate both overall strength (alpha) and L1 ratio. If you require pure variable selection and interpretability and correlations are weak, Lasso may suffice. If prediction accuracy and stability matter more (especially with multicollinearity), prefer Ridge or Elastic Net with higher L2 component.- Practical tips: standardize inputs, use cross-validation, monitor coefficient stability across resamples, use regularization paths and warm starts, and consider downstream costs (inference speed, maintainability) when selecting sparsity vs dense models.
HardTechnical
21 practiced
You have an ensemble of heterogeneous models (tree-based, neural nets, linear) and limited validation data. Design an automated pipeline to learn stacking/ensembling weights in production that copes with concept drift, maintains low inference latency, and prevents leakage. Justify choices for meta-learner, regularization on weights, retraining cadence, and online vs batch updates.
Sample Answer
Situation & goals: we must learn stacking weights for heterogeneous base models in production under limited validation data, avoid leakage, handle concept drift, keep latency low.Pipeline (high-level):1. Data plumbing: strict time-based ingestion, assign examples to streaming buffer + rolling holdout using anchored timestamps (no randomized mixing).2. Drift detector: monitor population/stability metrics (PSI, feature distribution KS, label rate) and model performance. Trigger update when drift detected or when periodic trigger hits.3. Meta-learner & training: - Use a constrained linear meta-learner (weighted convex combination of base model outputs). Constrain weights to be non-negative and sum-to-one (softmax or solve a constrained least-squares / quadratic program). Reason: few parameters → low risk of overfitting on limited val data and deterministic inference. - Learn on model-level calibrated scores (e.g., logit or isotonic/Platt-calibrated probabilities) to make outputs comparable. - Regularize with L2 (shrinkage) plus an entropy/Dirichlet prior to avoid weight collapse to single model. Optionally apply sparse L1 if you need model pruning for latency.4. Learning algorithm: - Hybrid updates: lightweight online updates for fast adaptation (Exponentiated Gradient / Online Newton Step on convex loss, with learning-rate schedule) using recent streaming examples; and periodic batch re-fit using a rolling time-window (e.g., last N days) with nested time-series cross-validation to select regularization. - Maintain a small, time-anchored validation holdout not used in online updates for unbiased evaluation.5. Leakage prevention: - All fitting uses strictly past data relative to prediction time (no peeking). - When computing stacking features, compute base model outputs using models frozen at the time the sample would have been predicted (use cached historical inferences or replay pipelines). - Use forward-chaining (time series) CV when tuning hyperparameters.6. Latency & production constraints: - Precompute base-model scores where possible; cache ensemble inputs. - Keep meta-learner as a single vector multiply + softmax — O(M) operations (M = #models). If sparsity needed, prune low-weight models periodically. - If extreme low-latency required, convert weights to integer-multiplier lookup or compile into model graph.7. Monitoring & safety: - Track weight trajectories, guardrails (min/max per-model weights), rollback to last stable weights on performance regressions. - Store training artifacts and seeds for auditability.Justification summary:- Linear constrained meta-learner minimizes parameters and leakage risk, is fast at inference, and robust with limited data.- L2 + entropy/Dirichlet encourages stable, diversified ensembles while L1 optional for latency pruning.- Hybrid online + batch cadence balances quick adaptation to drift (online) and more robust, less noisy re-training (batch). Always preserve a time-anchored holdout for honest evaluation.- Strict time-based data handling and frozen-model replay prevent leakage.Example algorithms/tools: use Vowpal Wabbit or scikit-learn's Ridge with simplex projection for batch; use Exponentiated Gradient (EG) or online logistic regression for streaming updates; use ADWIN/EDDM for drift detection.
MediumTechnical
26 practiced
Explain gradient boosting with decision trees: describe the stage-wise additive model, how gradients (pseudo-residuals) are computed and fitted by new trees, the roles of learning rate (shrinkage), number of leaves/depth, row/feature subsampling, and regularization terms. Contrast XGBoost, LightGBM, and CatBoost on algorithmic choices and production trade-offs.
Sample Answer
Stage-wise additive model: Gradient boosting builds an ensemble Fm(x)=Fm-1(x)+ν · hm(x) where each hm is a new regression tree and ν is the learning rate. At each stage we fit a tree to the negative gradient (pseudo-residuals) of the loss L w.r.t. current predictions: for example for squared error r_i = -∂L/∂F(x_i)=y_i−Fm-1(x_i); for logistic loss r_i = y_i−p_i. The new tree approximates these residuals (often via least-squares leaf values) so adding it reduces the loss.How gradients are computed & fitted:- Compute pseudo-residuals g_i = -∂L(y_i, Fm-1(x_i))/∂F.- Fit a regression tree hm(x) to targets g_i (using weights sometimes with second-order info).- Optionally use Newton step: leaf value = -Σg_i / (Σh_i + λ) where h_i is second derivative, giving faster convergence and built-in regularization.Hyperparameters and their roles:- Learning rate (shrinkage, ν): scales each tree’s contribution. Smaller ν requires more trees but improves generalization and stability.- Number of leaves / depth: controls tree complexity and interactions. More leaves capture higher-order interactions but risk overfitting. Use early stopping.- Row/feature subsampling: stochastic gradient boosting (subsample rows, colsample_bytree/level) reduces variance and speeds training; feature subsampling decorrelates trees.- Regularization terms: L2 on leaf weights (λ), L1 (α), minimum child weight, max depth, and leaf count; they control leaf values and prune unstable splits.Contrast: XGBoost vs LightGBM vs CatBoost- XGBoost: level-wise tree growth by default, strong second-order optimization, many regularization knobs (λ, α, min_child_weight). Stable, reproducible, slightly slower and more memory-heavy than LightGBM. Good default for robustness and interpretability.- LightGBM: histogram-based, leaf-wise (best-first) growth with max_leaves — faster and often higher accuracy but more prone to overfitting on small data; optimized for large datasets and low memory; excellent GPU support and very fast training.- CatBoost: handles categorical features natively via ordered target statistics and permutations to avoid target leakage, uses symmetric trees (balanced) and ordered boosting to reduce prediction shift; slower to train but gives better out-of-the-box quality with categorical data and is more robust to overfitting without heavy tuning.Production trade-offs:- Training speed vs generalization: LightGBM fastest; XGBoost slower but stable; CatBoost slower but reduces feature engineering for categoricals.- Memory & latency: histogram-based implementations reduce memory; deeper/leaf-wise models may increase model size and inference cost—consider limiting leaves or using model compression.- Reproducibility: XGBoost deterministic given seed; LightGBM leaf-wise can be less stable across versions/hardware; CatBoost’s ordered mode adds complexity but reduces target leakage.- Feature handling: CatBoost reduces preprocessing for categoricals; otherwise encode categorical features and ensure consistent preprocessing in production.- Monitoring: watch for concept drift, calibration (especially for probabilistic outputs), and latency/throughput constraints; prefer simpler trees or fewer leaves for low-latency inference.Typical recipe: start with modest depth (6–8) or limited leaves, low learning rate (0.01–0.1) with early stopping, subsample rows/features, tune regularization, and pick the framework based on dataset size and categorical prevalence.
EasyTechnical
27 practiced
Explain how decision trees choose splits for classification and regression. Describe impurity measures (Gini, entropy, MSE), how thresholds for continuous features are selected, handling of missing/categorical values, and common stopping/pruning strategies (max depth, min samples leaf, cost-complexity pruning) used in production systems.
Sample Answer
Decision trees choose splits by greedily selecting the feature and threshold that produce the largest reduction in impurity (information gain). For classification, common impurity measures are:- Gini impurity: G = 1 − Σ p_i^2. Measures probability of misclassification; fast to compute and favored in CART.- Entropy (information gain): H = −Σ p_i log p_i. More sensitive to class probabilities and sometimes yields different splits; used in ID3/C4.5.For regression, impurity is usually variance-based:- Mean Squared Error (MSE): reduction in node variance (sum of squared residuals). The split chosen minimizes weighted MSE of children.Threshold selection for continuous features:- Sort unique values and evaluate candidate thresholds at midpoints between consecutive distinct feature values (or use quantile-based heuristics for large data). For each candidate compute impurity reduction and pick best. This is O(n log n) per feature if sorting once.Handling missing/categorical values:- Categorical: one-hot or ordinal encoding; CART can handle categorical by evaluating subsets of categories (binary split) or using target-based encodings. Practical approach: treat small-cardinality categories via subset search; for high cardinality use frequency/target encoding.- Missing values: options include imputation (mean/mode), adding a “missing” category, or using surrogate splits (learn alternative features that mimic the primary split) so missing records follow surrogate paths.Stopping and pruning strategies used in production:- Pre-pruning: max_depth, min_samples_split, min_samples_leaf, min_impurity_decrease to prevent overfitting and control latency.- Post-pruning: cost-complexity pruning (alpha-penalized) — collapse subtrees that don’t sufficiently reduce impurity relative to model complexity; implemented in CART (ccp_alpha).- Cross-validation to choose hyperparameters (depth, min samples, ccp_alpha) balances bias/variance for production reliability.In production, combine simple pre-pruning limits with cross-validated cost-complexity pruning, and robust handling of missing/categorical values to ensure stability and reproducibility.
Unlock Full Question Bank
Get access to hundreds of Machine Learning Algorithms and Theory interview questions and detailed answers.
