Machine Learning Algorithms and Theory Questions

Core supervised and unsupervised machine learning algorithms and the theoretical principles that guide their selection and use. Covers linear regression, logistic regression, decision trees, random forests, gradient boosting, support vector machines, k means clustering, hierarchical clustering, principal component analysis, and anomaly detection. Topics include model selection, bias variance trade off, regularization, overfitting and underfitting, ensemble methods and why they reduce variance, computational complexity and scaling considerations, interpretability versus predictive power, common hyperparameters and tuning strategies, and practical guidance on when each algorithm is appropriate given data size, feature types, noise, and explainability requirements.

HardTechnical

45 practiced

Research problem: boosting algorithms can overfit noisy labels. Propose algorithmic modifications to gradient boosting to improve robustness to label noise, justify your choices theoretically or intuitively, and outline an experimental protocol to validate robustness on synthetic and real noisy-label datasets.

HardTechnical

29 practiced

Design methods to provide calibrated probability estimates and meaningful uncertainty estimates for an ensemble model used in decision-making. Compare temperature scaling, isotonic regression, Platt scaling, deep ensembles, and Bayesian model averaging. Describe evaluation metrics and experiments you would run to compare calibration and uncertainty quality.

Sample Answer

**Approach overview**I’d treat calibration and uncertainty as complementary: calibration fixes predicted probabilities to match empirical frequencies; uncertainty quantification (UQ) assesses epistemic/aleatoric sources to support decisions. I would implement and compare post-hoc calibrators (temperature scaling, Platt, isotonic) and ensemble/Bayesian methods (deep ensembles, Bayesian model averaging).

**Methods (brief)**- Temperature scaling: single-parameter softmax temperature; simple, preserves rank, good for neural nets.- Platt scaling: logistic regression on logits; works for binary/multiclass via one-vs-rest.- Isotonic regression: nonparametric monotonic mapping; flexible but risks overfitting; needs more calibration data.- Deep ensembles: multiple independently trained nets; captures epistemic uncertainty via predictive variance; strong empirical performance.- Bayesian model averaging (BMA): weight models by posterior; principled uncertainty, expensive and dependent on prior/approximation quality.

**Evaluation metrics**- Calibration: Expected Calibration Error (ECE), Classwise ECE, Brier score, reliability diagrams.- Sharpness & UQ quality: predictive entropy, negative log-likelihood (NLL), log score, predictive interval coverage (PIW/Coverage), AUROC for OOD detection, calibration under dataset shift.- Decision-aware: cost-weighted calibration error, utility/regret in downstream decision policy.

**Experiments**- In-domain: split holdout calibration set; compare ECE/NLL/Brier and reliability curves.- Small-data/noise regimes: test isotonic overfit vs. temperature robustness.- OOD & covariate shift: measure entropy separation, OOD AUROC, and coverage drop.- Ablations: ensemble size vs. calibration/UQ; BMA using different approximations (MC-dropout, Laplace, VI).- Statistical tests: bootstrap confidence intervals for ECE/NLL; compare calibration maps.

**Recommendation**Start with temperature scaling for baseline; use deep ensembles for best UQ and decision robustness; apply isotonic only with ample calibration data; consider BMA when priors and compute allow—report metrics plus decision-level impact.

MediumTechnical

29 practiced

You observe that k-means performs poorly on your dataset because clusters have varying density and non-globular shapes. Propose at least three alternative clustering algorithms or strategies and justify each choice. Discuss scalability and parameter sensitivity for each alternative.

MediumTechnical

22 practiced

Explain gradient boosting as function-space gradient descent: derive the notion of pseudo-residuals for L2 and logistic loss, explain how regression trees are used as base learners to approximate the negative gradient, and describe the role of learning rate and shrinkage in stability and generalization.

Sample Answer

**Overview — function-space gradient descent**Gradient boosting views model F(x) as a function in hypothesis space and fits it by iteratively taking steps opposite to the functional gradient of empirical loss L over training points. At iteration m we add a base learner h_m(x) to reduce loss: F_m = F_{m-1} + v * h_m, where v is the learning rate.

**Pseudo-residuals (negative gradients)**- L2 (squared) loss: l(y,F)= (y−F)^2/2. Negative gradient per sample:

text

r_i = - d/dF l(y_i,F)|_{F=F_{m-1}(x_i)} = y_i - F_{m-1}(x_i)

Intuition: residual equals error; fit regression of targets minus current predictions.

- Logistic (binary cross-entropy) for y∈{0,1}, l = -y log p - (1-y) log(1-p) with p = σ(F):

text

r_i = - d/dF l(y_i,F)|_{F=F_{m-1}} = y_i - p_i

where

text

p_i = 1 / (1 + exp(-F_{m-1}(x_i)))

Intuition: pseudo-residual = label minus predicted probability (the standard gradient for log-loss).

**Regression trees as base learners**- At each step fit a regression tree h_m(x) to the pseudo-residuals {r_i}. Trees partition input space into leaves R_{jm} and predict constant leaf values.- After fitting structure, compute optimal leaf updates γ_{jm} by minimizing loss on leaf samples:

text

γ_{jm} = argmin_γ sum_{x_i in R_{jm}} l(y_i, F_{m-1}(x_i) + γ)

This is effectively a local line-search per leaf; for L2 γ equals mean residual in leaf, for logistic it's a Newton/closed-form step.

**Learning rate / shrinkage**- v (0< v ≤1) scales the step: F_m = F_{m-1} + v * γ_{jm} * I(x∈R_{jm}).- Small v improves stability and generalization by taking smaller, more regularized steps => reduces overfitting, allows many weak learners to combine smoothly.- Trade-off: lower v needs more iterations and compute; interact with tree depth and subsampling (stochastic gradient boosting) for regularization.

**Research notes**- View as gradient descent in function space clarifies extensions: second-order (Newton) variants use Hessian information for faster convergence; alternatives modify loss, base learner family, or add regularizers on leaf values/structure.

HardTechnical

29 practiced

Case study: you must compare several clustering algorithms on single-cell RNA-seq data where labels are partial, noisy and batch effects are present. Design a complete evaluation framework: preprocessing (normalization, batch correction), algorithms to compare, metrics (internal and external), statistical tests, and how to present these results in a research manuscript.

Sample Answer

**Clarify goals & constraints**- Objective: robustly compare clustering methods on scRNA-seq where ground truth labels are partial/noisy and batches exist. Emphasize reproducibility, sensitivity to batch, and biological interpretability.

**Preprocessing pipeline**- Quality control: filter cells/genes (mitochondrial %, gene counts), doublet removal.- Normalization: scran deconvolution or SCTransform for per-cell depth + variance stabilization.- Feature selection: highly variable genes per batch and intersect/union strategies.- Batch correction (compare alternatives): Harmony, Scanorama, scVI (latent-batch model), Combat-Seq. Keep an uncorrected baseline.- Dimensionality reduction: PCA on corrected/uncorrected features; for methods using raw counts, feed appropriately.

**Algorithms to compare**- Graph-based: Leiden/Walktrap on kNN graph.- Model-based: Gaussian Mixture Models (GMM) on PCA, scVI latent clustering.- Density-based: HDBSCAN.- Hierarchical: Agglomerative with different linkage.- Single-cell specialized: Seurat pipeline, SIMLR.- Include ensemble clustering (consensus) as robustness check.

**Evaluation metrics**- External (use partial/noisy labels): Adjusted Rand Index (ARI), Normalized Mutual Information (NMI), pairwise precision/recall on known markers. Use label propagation to expand partial labels and report sensitivity.- Internal: Silhouette (on corrected embedding), Davies-Bouldin, cluster stability via subsampling/bootstrap (ARI across runs), entropy of batch mixing (kBET, LISI).- Biological validation: enrichment of marker genes, differential expression consistency, cell-cycle and known pathway coherence.

**Statistical testing**- Compare methods across multiple datasets/bootstraps. Use paired tests (Wilcoxon signed-rank) on metric distributions.- Multiple hypothesis correction (Benjamini-Hochberg).- Effect sizes (Cliff’s delta) and confidence intervals via bootstrap.- Ablation: test impact of batch correction choice using two-way ANOVA (method × batch-correction) on metrics.

**Presentation in manuscript**- Figure 1: pipeline schematic.- Main results: heatmap/table of metrics (methods × datasets) with significance annotations; boxplots of metric distributions.- UMAP/PCA overlays for representative datasets: colored by true labels, predicted clusters, batch.- Stability plots: ARI vs subsampling fraction.- Biological evidence: marker gene heatmaps, GO enrichment tables.- Methods section: exact preprocessing, hyperparameters, seeds; link to code and notebooks.- Discussion: trade-offs, when each method fails, recommendations.

This framework emphasizes reproducibility, rigorous statistics, and biological interpretability—suitable for research-grade evaluation.

Unlock Full Question Bank

Get access to hundreds of Machine Learning Algorithms and Theory interview questions and detailed answers.

Join thousands of developers preparing for their dream job.