MediumTechnical

44 practiced

Design a hyperparameter tuning strategy for a random forest classifier under a fixed compute budget (e.g., 24 CPU-hours). List which hyperparameters to tune (n_estimators, max_depth, max_features, min_samples_leaf, class_weight), propose a search strategy (random vs Bayesian), evaluation method (OOB vs CV), how to parallelize trials, and stopping criteria to maximize model quality within the budget.

Sample Answer

Situation: You have 24 CPU-hours and must get the best RandomForest classifier under that fixed compute budget. Below is a concrete, pragmatic tuning strategy that balances exploration, exploitation, parallelism and reliable evaluation.

Hyperparameters to tune (recommended ranges/prior):- n_estimators: [50, 2000] (log-uniform). Treat as budgeted resource; can early-stop.- max_depth: [3, 50] (int, uniform or log for wide range). None allowed.- max_features: [0.1, 1.0] or {"sqrt","log2","auto"} (continuous fraction or categorical).- min_samples_leaf: [1, 50] (int, log/linear).- class_weight: {None, "balanced", "balanced_subsample", custom weights} (categorical).

Search strategy:- Hybrid: start with an initial random/sobol design (e.g., 30–50 trials) to map broad response surface, then switch to Bayesian optimization (e.g., Gaussian Process or Tree-structured Parzen Estimator) for the remainder. Bayesian is more sample-efficient than pure random and fits a tight compute budget.- Treat n_estimators as a multi-fidelity knob: use successive halving or Hyperband ideas (see BOHB) to allocate more trees only to promising configs.

Evaluation method:- Prefer OOB score when: dataset is large (>100k rows) and RF is used (fast, no data leakage from CV), because OOB gives near-instant estimates and saves compute.- Prefer stratified K-fold CV (k=3) when dataset is smaller, imbalanced, or requires robust validation; use parallel CV folds.- If using CV, use grouped CV when needed to prevent leakage.

Parallelization:- Two levels: 1) Intra-trial: each RandomForest uses n_jobs across cores to build trees in parallel. Choose per-trial cores small (e.g., 2–4) to allow multiple concurrent trials. 2) Inter-trial: run multiple trials in parallel across workers (e.g., 6–8 trials simultaneously) using a scheduler (Ray Tune, Optuna, SMAC, or a Kubernetes job queue). Use asynchronous Bayesian optimization (ASHA/BOHB or asynchronous TPE) so workers don’t idle waiting for synchronous updates.- Example: On a 16‑core host, set per-trial n_jobs=2 and run 7 trials in parallel.

Stopping criteria and budget management:- Hard wall: 24 CPU-hours total; track consumed CPU-seconds and stop scheduling new trials when >= budget.- Trial-level early stopping: use multi-fidelity (partial n_estimators) and promote only top X% after small budgets (successive halving). If using CV, stop trials whose partial CV folds are poor vs incumbent by margin.- Convergence stop: stop BO when no improvement > epsilon (e.g., 0.001 AUC) for N consecutive best-trial updates or after a maximum number of trials estimated to fit budget.- Time-per-trial cap: enforce max runtime per trial (e.g., 1 hour).

Concrete allocation example for 24 CPU-hours (single 16-core machine):- Reserve overhead: 1 CPU-hour.- Initial random phase: 40 trials × 2 cores × avg 5 min = ~6.7 CPU-hrs.- Bayesian phase + multi-fidelity: run asynchronous BO for remaining ~16 CPU-hrs, using successive halving to focus on promising configs; expect 100–200 candidate evaluations at low fidelity and 10–20 full evaluations.- Always keep a final evaluation step: re-evaluate top 3 configs with full CV (k=5) and more trees to pick the final model.

Other practical notes:- Log everything (MLflow/Weights & Biases) and fix random seeds.- Normalize/scale features only if necessary; preserve preprocessing pipeline in CV.- If class imbalance is critical, tune class_weight and evaluation metric (AUC, F1, PR-AUC).- Prefer metric-based checkpointing: if a trial’s partial OOB or fold average is hopeless, terminate early.

Why this works:- Random initialization prevents BO from local optima; BO then exploits efficiently.- Multi-fidelity and early stopping concentrate compute on promising areas and let you evaluate many configurations cheaply.- Asynchronous parallel trials maximize hardware utilization under the fixed CPU-hour budget.

EasyTechnical

54 practiced

Define precision, recall, specificity, accuracy, and F1 score for binary classification and explain how they relate to the confusion matrix. Give two business examples where precision is more important than recall and vice versa. Explain why accuracy can be misleading on imbalanced datasets and what metrics you would prefer instead.

EasyTechnical

45 practiced

Explain how random forests reduce overfitting compared to a single decision tree. Discuss the roles of bootstrap sampling, random feature selection at splits (feature bagging), tree depth, and averaging. Also explain how out-of-bag (OOB) error estimation works and when you would rely on OOB vs cross-validation.

Sample Answer

Random forests reduce overfitting primarily by reducing variance through ensembling and decorrelation of individual trees, while keeping bias roughly similar to a single deep tree.

- Bootstrap sampling (bagging): each tree is trained on a bootstrap sample (sample with replacement). On average ~63.2% of examples appear in a given bootstrap sample; the rest are "out‑of‑bag" for that tree. Bootstrapping produces different training sets so trees see different data and make different errors — averaging these predictions reduces variance compared to one tree.

- Random feature selection at splits (feature bagging): at each split a random subset of features is considered. This prevents strong predictors from dominating every tree and forces different trees to use different features, further decorrelating trees. Lower correlation among tree errors leads to more effective variance reduction when averaging.

- Tree depth: individual trees in a random forest are often grown deep (low bias, high variance). The forest combats the high variance via bagging and feature randomness. You can still regularize by limiting depth or minimum samples per leaf if overfitting persists or for interpretability/latency reasons.

- Averaging: final prediction is the average (regression) or majority vote (classification) across trees. Averaging reduces variance by roughly factor of 1/n if errors are independent; decorrelation makes this reduction practical.

Out‑of‑bag (OOB) error:- For each training example, aggregate predictions from trees that did not include that example in their bootstrap sample (OOB trees). Compare aggregated OOB predictions to true labels to compute an unbiased estimate of generalization error.- OOB is efficient (no extra holdout or CV folds) and reliable when you have many trees and balanced data. Use OOB for quick model evaluation and some hyperparameter tuning.

When to prefer OOB vs cross‑validation:- Use OOB for fast, reasonably accurate estimates during development and when dataset is large.- Use cross‑validation when doing thorough hyperparameter optimization, with small datasets, or when you want more stable estimates for model selection (especially if you tune many hyperparameters or use models that break OOB assumptions).

HardTechnical

43 practiced

You must train an SVM on 10 million samples with 10,000 features and kernels are infeasible. Describe scalable alternatives and their trade-offs: linear SVM with SGD or liblinear, approximate kernel methods (random Fourier features, Nystrom), dimensionality reduction, distributed training, and streaming/online algorithms. Outline an end-to-end training and validation plan including normalization and checkpoints.

Sample Answer

Approach summary: with 10M samples × 10k features kernels are infeasible. I’d evaluate a spectrum: linear methods (fast, baseline), approximate kernels (explicit feature maps), dimensionality reduction, distributed/streaming training. Choose by accuracy vs latency/resource trade-offs.

Alternatives & trade-offs:- Linear SVM (SGD or liblinear): Very scalable, low memory, can handle sparse inputs. SGD (e.g., SGDClassifier with hinge loss) gives streaming/online updates and is easy to parallelize; liblinear gives optimized batch coordinate descent for dense/sparse but is memory-bound. Trade-off: may underfit if data is nonlinearly separable.- Random Fourier Features (RFF): Map inputs to D' explicit features to approximate RBF kernel; complexity O(n·D'·d). Good when nonlinear structure exists and D' is moderate (e.g., 1k–10k). Trade-off: approximation error vs computational cost; requires tuning D'.- Nystrom: Low-rank kernel approximation using m landmark points; good for moderate m, works well when kernel matrix low-rank. Trade-off: choosing landmarks and memory for intermediate matrices.- Dimensionality reduction: feature selection (sparse models, L1), PCA, or randomized PCA to reduce d from 10k to k (e.g., 200–1000). Pros: reduces compute and noise. Cons: risk of losing predictive signals; PCA is linear and unsupervised.- Distributed training: Use minibatch SGD or Hogwild on Spark/MPI or frameworks (TF, PyTorch with DDP). Pros: handle dataset across nodes; Trade-off: network IO, synchronization, complexity.- Streaming/online algorithms: Passive-Aggressive, Pegasos, or Vowpal Wabbit – extremely fast, low memory, ideal for very large datasets and sparse features. Trade-off: potentially more tuning, less stable than batch opt.

End-to-end plan:1. Data prep: validate labels, handle missing values, encode categoricals.2. Normalization: per-feature mean/variance scaling for dense features; use global streaming stats (Welford) to avoid full passes. For sparse binary counts, use TF-IDF or log-scaling.3. Baseline: train linear SVM via SGD (with learning rate schedule, L2/L1 regularization) on a random 1% sample to get baseline metrics (AUC/accuracy, precision/recall).4. Feature engineering: test L1 feature selection or mutual information to reduce features; evaluate PCA/randomized SVD on sample.5. Approximate kernel experiment: implement RFF with multiple D' and Nystrom with different m on sample; compare to baseline for lift vs cost.6. Scale-up strategy: - If linear suffices: full-data SGD in minibatches with shuffling; use multiple workers and asynchronous updates or parameter server; save checkpoints every epoch. - If RFF/Nystrom needed: precompute mapped features in distributed store (parquet) then train linear solver on mapped features (still linear). - For streaming/production: consider Vowpal Wabbit for single-pass training and online updates.7. Validation & monitoring: - Hold out temporal validation set (or stratified 10%) and an unseen test set. Track loss, AUC, calibration, and fairness metrics. - Use early stopping based on validation AUC and checkpoint best models.8. Checkpoints & reproducibility: - Periodic model checkpoints (per N minibatches or epochs) with metadata: hyperparams, data hash, normalization stats, training loss, validation metrics. - Store preprocessing pipeline (scalers, feature maps) serialized so inference matches training.9. Hyperparameter tuning: - Use random/ASHA on sample then apply to full training. For distributed, use small-budget trials then scale best configs.10. Resource considerations: - Use sparse data structures, mixed precision if dense, and monitor IO. For very large D' consider streaming feature maps to avoid materializing full matrix.

Key practical tips:- Start small (1–5% sample) to pick approach; only scale once gains justify cost.- Prefer pipelines where nonlinear approximation produces explicit linear features — simplifies distributed training.- Use robust normalization computed in streaming fashion and always checkpoint those stats for inference.

HardTechnical

54 practiced

Explain ElasticNet regularization: write the objective combining L1 and L2 penalties, explain the roles of alpha and l1_ratio, and discuss why ElasticNet is useful when features are correlated. Provide practical guidance for tuning alpha and l1_ratio and how ElasticNet relates to Lasso and Ridge as edge cases.

ML Fundamentals: Supervised Learning Algorithms Questions

Sample Answer

Sample Answer

Sample Answer

Sample Answer

Sample Answer

Unlock Full Question Bank