Supervised Learning Algorithm Selection Questions

Covers selecting and comparing supervised machine learning algorithms for classification and regression tasks. Includes core algorithm families such as logistic regression, decision trees, random forests, gradient boosting implementations like XGBoost and LightGBM, support vector machines, and neural networks, as well as linear regression and regularization techniques including lasso, ridge, and elastic net. Candidates should be able to explain when each algorithm is appropriate based on interpretability requirements, dataset size, feature characteristics, computational constraints, training and inference latency, and robustness to noise and missing data. Also covers hyperparameter tuning, cross validation, bias variance trade offs, feature engineering impact, handling imbalanced classes, evaluation metrics for classification and regression, model calibration, and how to reason about theoretical foundations and senior level trade offs between algorithmic choices.

MediumTechnical

80 practiced

You have a high-dimensional sparse dataset (text TF-IDF with 100k features) for classification. Compare the suitability of logistic regression with L2, linear SVM, naive Bayes, and a shallow neural network. Discuss training time, memory, regularization, and expected performance.

Sample Answer

For a 100k-dimensional sparse TF‑IDF text classification task, here's a practical comparison:

Logistic Regression (L2)- Training time: Fast with sparse-aware solvers (liblinear, saga); scales linearly in nonzeros. Converges quickly for convex loss.- Memory: Low if you use sparse matrix formats; stores weight vector dense (100k), negligible.- Regularization: L2 helps control overfitting and handles correlated features well.- Expected performance: Strong baseline; well-calibrated probabilities and good generalization for linearly separable-ish text.

Linear SVM- Training time: Comparable to logistic regression for linear kernels (e.g., liblinear, LinearSVC). May be slightly slower depending on hinge optimization.- Memory: Similar to LR.- Regularization: L2 SVM provides margin-based regularization; less sensitive to class probability calibration.- Expected performance: Often equal or slightly better than LR on margin-sensitive tasks; similar feature importance.

Naive Bayes (Multinomial/Bernoulli)- Training time: Extremely fast (single pass to accumulate counts), trivial memory.- Memory: Low — stores class priors and feature likelihoods.- Regularization: Implicit via smoothing (Laplace); less flexible.- Expected performance: Surprisingly strong for text with conditional independence approximations, especially on small data; typically worse than LR/SVM on larger datasets where feature dependencies matter.

Shallow Neural Network- Training time: Slower — many epochs, dense parameter updates unless using sparse embeddings and optimizers that exploit sparsity.- Memory: Higher — hidden layer weights (100k × h) can be huge unless using feature hashing, embedding, or very small h.- Regularization: Needs dropout, weight decay; more hyperparameters to tune.- Expected performance: Can match or exceed linear models only if there’s enough data and interactions; otherwise risk of overfitting and long training.

Recommendations:- Start with Logistic Regression or Linear SVM (L2) using sparse inputs — best tradeoff of speed, memory, and performance.- Use Naive Bayes for quick baselines or low-data regimes.- Consider shallow NN only if you can reduce dimensionality (feature hashing, embedding, pretrained representations) or have large labeled data and compute budget.

MediumTechnical

91 practiced

Implement in Python a simple k-nearest neighbors (kNN) classifier predict function that supports Euclidean distance and handles ties by choosing the class with smallest average distance. Assume inputs: X_train (n x d), y_train (n,), X_test (m x d), k integer. Do not use scikit-learn.

Sample Answer

Approach:Compute Euclidean distances between each test point and all train points, find k nearest neighbors, then predict by majority vote. If there's a tie in vote counts, break it by computing the average distance of the tied classes among those k neighbors and select the class with the smallest average distance. Use numpy for vectorized operations.

python

import numpy as np

def knn_predict(X_train, y_train, X_test, k):
    """
    Simple k-NN classifier (Euclidean distance).
    Tie-breaking: if multiple classes have same vote count, choose the class
    with the smallest average distance among its occurrences in the k neighbors.
    
    Inputs:
    - X_train: (n, d) numpy array of training features
    - y_train: (n,) numpy array of training labels (can be numeric or strings)
    - X_test:  (m, d) numpy array of test features
    - k: integer, number of neighbors (1 <= k <= n)
    
    Returns:
    - preds: (m,) numpy array of predicted labels (same dtype as y_train)
    """
    X_train = np.asarray(X_train)
    y_train = np.asarray(y_train)
    X_test = np.asarray(X_test)
    n = X_train.shape[0]
    if k < 1 or k > n:
        raise ValueError("k must be between 1 and number of training samples")

    # Compute squared Euclidean distances: shape (m, n)
    # Using (a-b)^2 = a^2 + b^2 - 2ab for efficiency
    X_test_sq = np.sum(X_test**2, axis=1)[:, None]   # (m,1)
    X_train_sq = np.sum(X_train**2, axis=1)[None, :] # (1,n)
    cross = X_test @ X_train.T                       # (m,n)
    dists_sq = X_test_sq + X_train_sq - 2 * cross
    # numerical stability
    dists_sq = np.maximum(dists_sq, 0.0)
    dists = np.sqrt(dists_sq)

    m = X_test.shape[0]
    preds = np.empty(m, dtype=y_train.dtype)

    for i in range(m):
        # indices of k nearest neighbors
        nn_idx = np.argpartition(dists[i], k-1)[:k]
        # sort those k by distance to have deterministic ordering
        nn_idx = nn_idx[np.argsort(dists[i][nn_idx])]
        nn_labels = y_train[nn_idx]
        nn_dists = dists[i][nn_idx]

        # Count votes
        unique, counts = np.unique(nn_labels, return_counts=True)
        max_count = counts.max()
        # candidate labels with max votes
        candidates = unique[counts == max_count]

        if candidates.size == 1:
            preds[i] = candidates[0]
        else:
            # tie-breaker: compute average distance among occurrences in the k neighbors
            best_label = None
            best_avg_dist = np.inf
            for label in candidates:
                mask = (nn_labels == label)
                avg_dist = nn_dists[mask].mean()
                # choose smaller avg distance; deterministic tie-breaker: smaller label if avg equal
                if (avg_dist < best_avg_dist) or (avg_dist == best_avg_dist and label < best_label):
                    best_avg_dist = avg_dist
                    best_label = label
            preds[i] = best_label

    return preds

Key points:- Vectorized distance computation for speed.- argpartition finds k nearest efficiently; then sort those k for determinism.- Tie-breaking uses average neighbor distance per tied class; final deterministic tie-breaker uses label ordering.Time complexity: O(m * n * d) for distance computation (dominant), plus O(m * k log k) for per-sample neighbor processing. Space: O(m * n) for distance matrix; can be reduced by processing in batches.Edge cases: k out of range, identical points, non-numeric labels (works if comparable), very large n (batch distances to save memory). Alternative: KD-tree or BallTree for large datasets / high performance.

MediumTechnical

126 practiced

Outline a practical hyperparameter tuning workflow for LightGBM on a medium-sized dataset (200k rows) where compute is limited to a single GPU with 16GB memory. Include search strategy, resource-aware settings, and early-stopping choices.

Sample Answer

Situation: You need a practical, resource-aware LightGBM tuning workflow on 200k rows with a single 16GB GPU.

Workflow (stepwise):1. Data and baseline- Prepare light, memory-efficient inputs: float32, categorical as int/category, drop unnecessary cols.- Fit a quick baseline model (CPU or GPU) with conservative defaults to get a baseline metric and training time.

2. Two-stage search strategy (cheap → refined)- Stage A — fast, coarse search on a subsample (30–50k rows) or fewer folds: - Use random search or low-budget Bayesian (10–50 trials) to explore broad ranges: - learning_rate: 0.01–0.3 - num_leaves: 16–512 (start smaller for memory) - max_bin: 63–255 (lower memory with 63–127) - min_data_in_leaf: 20–200 - feature_fraction: 0.6–1.0 - bagging_fraction: 0.6–1.0, bagging_freq: 1–5 - Use device=gpu to mirror production; set force_col_wise=true to reduce memory peaks. - Use 3-fold CV or validation split; early_stopping_rounds=50 on the subsample.

- Stage B — refined search on full data: - Use Bayesian optimization (Optuna) with pruning (median/pruner) for 50–200 trials. - Narrow parameter ranges from Stage A; focus on num_leaves, learning_rate, min_child_samples, lambda_l1/l2. - Use 4–5 fold CV (or stratified) but enable LightGBM’s built-in cv with device=gpu. - early_stopping_rounds=100 and a time/iteration budget per trial to avoid long runs.

3. Resource-aware settings (GPU and memory)- device = 'gpu', gpu_platform_id/gpu_device_id if needed.- force_col_wise=true and max_bin lower (e.g., 127) to reduce memory.- Limit num_threads to available CPU threads to avoid oversubscription.- Reduce categorical cardinality via hashing/target-encoding if needed.- Lower num_leaves and increase min_data_in_leaf to reduce tree size.

4. Early-stopping and pruning policy- Use a validation set (holdout or CV) separate from training folds.- early_stopping_rounds: 50–100 for subsample, 100 for full-data trials.- Use optimization framework pruning (Optuna pruners) to kill unpromising trials early.- Monitor both validation metric and training memory/time; kill trials nearing memory limit.

5. Final model and validation- Refit final model on full training set using best hyperparameters with a small learning_rate (e.g., halved) and more boosting rounds; use early_stopping_rounds with a separate test fold.- Generate calibration checks, feature importance, and one last time memory/profile check on GPU.

Notes and trade-offs:- Subsampling accelerates search but may bias optimal hyperparams (stage B mitigates that).- Lower max_bin and num_leaves reduce accuracy ceilings but are necessary with 16GB GPU.- Pruning + early stopping saves compute; prioritize robust validation to avoid overfitting.

MediumTechnical

88 practiced

You have a tabular dataset with many categorical features, some with high cardinality (thousands of distinct values). For a gradient boosting model (e.g., LightGBM), describe encoding strategies you'd consider and trade-offs in terms of performance, memory, and risk of leakage.

Sample Answer

Situation: For LightGBM with many categorical features including high-cardinality ones, I prioritize encodings that balance predictive power, memory, and leakage risk.

Approach / strategies:- Native categorical (LightGBM): let the model handle categories via histogram splitting. Pros: memory efficient, no leakage. Cons: can be slow with thousands of levels and may underperform vs engineered encodings.- Frequency / count encoding: replace each category with its count or frequency. Pros: cheap, low-memory, often strong signal. Cons: loses finer target relationships.- Target (mean) encoding with regularization: replace category by smoothed target mean using global prior and count-based smoothing: encoded = (count * cat_mean + prior_weight * global_mean) / (count + prior_weight) Use out-of-fold (K-fold) / leave-one-out or nested CV to avoid target leakage, add small Gaussian noise and clip. Pros: very predictive for categorical signal. Cons: high leakage risk if improperly cross-validated; needs careful implementation.- Hashing trick: map categories to fixed-size integer buckets (e.g., 2^12). Pros: fixed memory, fast, handles new categories. Cons: collisions can dilute signal.- Entity embeddings (neural): learn dense representations (e.g., small feedforward or autoencoder). Pros: capture complex relations, compact. Cons: heavier training, more engineering.- Group/statistical features: e.g., category -> aggregated target stats, time-based decay for temporal data.

Trade-offs summary:- Performance: Target encoding > frequency > hashing/native for pure predictive power if leakage prevented. Entity embeddings can beat others when ample data.- Memory: Hashing and frequency are smallest; one-hot explodes memory and is unusable for high cardinality.- Leakage risk: Target encoding highest risk unless strict OOF/K-fold or temporal splits used. Native and frequency have low risk.Practical rules I use:- One-hot only for low-cardinality (<20).- Use frequency + hashed buckets as baseline for thousands of levels.- Use target encoding with K-fold or time-based folds and smoothing when signal is needed; validate with nested CV and monitor for overfit.

MediumTechnical

92 practiced

You're building a credit scoring model where regulatory requirements demand explainability and monotonic behavior with respect to income (higher income should not decrease score). Which supervised algorithms and techniques would you consider to meet these constraints?

Sample Answer

Recommended approach: combine inherently interpretable models and constrained ML methods so you satisfy explainability and enforce monotonicity on income.

Algorithms / techniques:- Logistic regression on engineered monotonic features (e.g., monotonic bins or WOE): fully transparent, easy to explain and audit.- Generalized Additive Models (GAMs) with monotonic spline constraints (e.g., pygam, interpretML’s Explainable Boosting Machine with shape constraints): flexible but still interpretable per-feature.- Monotonic tree ensembles: LightGBM, XGBoost, CatBoost support monotone constraints (set sign for income feature). They keep performance while guaranteeing monotonicity.- Isotonic regression for post-hoc monotonic calibration if raw model violates monotonicity.

Explainability & validation:- Use per-feature partial dependence plots / ALE to show monotonic behavior.- Use SHAP but restrict explanation to monotonic features; verify SHAP values for income never reduce score with higher income.- Produce simple scorecard translation (bins + points) for regulator consumption.- Unit tests: enforce monotonicity on validation sets and adversarial checks (pairwise income higher -> score not lower).

Example (LightGBM monotone constraint):

python

import lightgbm as lgb
params = {"objective":"binary","monotone_constraints":"(0,1)"}  # assume feature 1 is income (+1 = monotone increasing)
gbm = lgb.train(params, lgb.Dataset(X_train, y_train), valid_sets=[lgb.Dataset(X_val, y_val)])

Trade-offs:- Monotonic constraints can reduce model flexibility and marginally lower AUC; mitigate with better features and GAMs.- Prefer simple, audited pipelines (binning, WOE) when regulators need human-readable decisions.

Unlock Full Question Bank

Get access to hundreds of Supervised Learning Algorithm Selection interview questions and detailed answers.

Join thousands of developers preparing for their dream job.