Classical Machine Learning Algorithms Questions

Direct answer

Averaging independent estimates cancels out uncorrelated errors, so an ensemble's variance shrinks roughly in proportion to how independent its base learners' mistakes are. If the base learners are highly correlated (they tend to make the same mistakes on the same examples), averaging barely helps, because there's nothing uncorrelated left to cancel. Bagging and random forests work by deliberately reducing that correlation, through resampled training data and, for random forests, randomized feature subsets at each split.

Structured elaboration

The variance-reduction mechanism. For $M$ estimators each with variance $\sigma^2$ and pairwise correlation $\rho$ between any two of them, the variance of their average is:

As $M \to \infty$, the first term vanishes but the second converges to $\rho\sigma^2$, a floor set entirely by the correlation between base learners. This is the core intuition: adding more trees keeps shrinking the $\sigma^2/M$ term, but if $\rho$ is large, the ensemble variance plateaus well above zero no matter how many learners you add. Diversity (low $\rho$) is what lets averaging actually pay off.

Why bagging and random forests reduce correlation.

• Bagging trains each base learner on a different bootstrap resample of the training data, so each tree sees a slightly different, overlapping-but-not-identical dataset; their errors become less correlated because they're fit to different noise realizations of the same underlying signal.
• Random forests add a second layer: at each split, only a random subset of features is considered as candidates. This deliberately prevents every tree from all discovering and splitting on the single strongest feature first, which would otherwise make the trees structurally similar (and their errors correlated) even with different bootstrap samples.

Other ways to increase diversity. Varying the base learner's hyperparameters (depth, minimum leaf size) across the ensemble; using different model families entirely (stacking a tree, a linear model, and a neural net); injecting randomness into training (different random seeds, dropout-like row/column sampling); or training on different feature subspaces (the random subspace method). All of these push $\rho$ down, which is where most of the practical ensemble gain comes from once you already have reasonably accurate individual learners.

Worked example

Fix $\sigma^2 = 1$ and $M = 100$ base learners, and compare two correlation regimes:

Going from $\rho=0.30$ to $\rho=0.05$, holding $M$ and $\sigma^2$ fixed, drops the ensemble variance from $0.307$ to $0.0595$, a reduction of about $81\%$, even though the number of trees and each individual tree's own variance didn't change at all. That's the diversity mechanism in isolation: the same $M$, the same per-tree quality, but a five-fold better ensemble outcome purely from decorrelating the errors, which is exactly what random forests' extra feature-subsampling step is designed to do relative to plain bagging.

Trade-offs & pitfalls

• Diversity is not free: features or hyperparameter choices that increase diversity often do so by making individual base learners slightly weaker (e.g., a very small feature subset per split can starve a tree of its best signal). The formula above assumes $\sigma^2$ is held fixed, but in practice pushing $\rho$ down can push $\sigma^2$ up if taken too far, and the net effect on ensemble variance can reverse.
• The floor $\rho\sigma^2$ means that beyond a certain point, adding more trees has rapidly diminishing returns; the marginal value of tree number 500 versus tree number 100 is small once $M$ is already large enough that $\sigma^2/M$ is small relative to $\rho\sigma^2$.
• This formula addresses variance only; it says nothing about bias, so an ensemble of biased base learners stays biased no matter how diverse they are.
• The independence/correlation framing is intuitive but approximate for classification (0/1 loss) rather than squared-error regression; the qualitative conclusion, correlated errors limit ensemble gains, carries over, but the exact algebraic form above is derived for a variance-of-the-mean setting.

Direct answer

Algorithms that make decisions based on distances or gradients need feature scaling, algorithms that split on the raw order of values do not. SVMs, KNN, k-means, PCA, and neural networks are all in the first group because a large-range feature would dominate a distance calculation or a gradient step purely due to its units. Decision trees, random forests, and gradient-boosted trees are in the second group, because a split threshold like "feature > 5" only depends on the ordering of values, not their scale.

Structured elaboration

Interaction with regularization. L1/L2 penalties are applied directly to coefficient values, so if two features are equally predictive but on very different scales, the smaller-scale feature needs a numerically larger coefficient to have the same effect on predictions, and that larger coefficient gets penalized more heavily by the regularizer, purely as an artifact of units, not predictive value.

Interaction with interpretability. Once you scale, raw coefficient values are in "standard deviations of the feature" rather than the feature's original units. To report an interpretable, original-units effect, you rescale back: $\beta_{\text{original}} = \beta_{\text{scaled}} / \sigma_{\text{feature}}$, and adjust the intercept accordingly.

Worked example

Two features with equal true predictive contribution to the target, but very different scales: $x_1 \in [0, 1]$ needs a coefficient of $w_1 = 1$ to contribute a typical effect of about 1, while $x_2 \in [0, 10^6]$ needs only $w_2 = 10^{-6}$ to contribute that same typical effect of about 1.

$w_1 = 1 \Rightarrow w_1^2 = 1, \qquad w_2 = 10^{-6} \Rightarrow w_2^2 = 10^{-12}$

An L2 penalty $R(w) = \lambda \sum_j w_j^2$ adds $\lambda \times 1$ from $w_1$ but only $\lambda \times 10^{-12}$ from $w_2$, twelve orders of magnitude less, even though both features contribute equally to predictions. The regularizer is effectively punishing $x_1$ almost entirely and leaving $x_2$'s coefficient nearly free, purely because of scale, not because $x_1$ is actually less trustworthy or more prone to overfitting.

Trade-offs & pitfalls

• Always fit the scaler on the training set only, then apply the same learned mean/std (or min/max) to validation, test, and inference data, fitting on the full dataset leaks test-set statistics into the training pipeline.
• Standardization (z-score) suits roughly Gaussian features and pairs naturally with L2 regularization; min-max scaling suits cases needing a bounded range (e.g., inputs to a sigmoid activation) but is more sensitive to outliers.
• Scaling erases direct interpretability of raw coefficients, always rescale back to original units before presenting coefficients to a non-technical stakeholder.
• Pitfall: assuming gradient-boosted trees never benefit from any numeric transform because splits don't need scaling. Skew, extreme outliers, or wildly different feature cardinalities can still affect split-finding efficiency and interaction discovery even though scale itself is irrelevant to tree splits.

Direct answer

For a 100k-row, 200-feature tabular dataset with high-cardinality categoricals, I'd default to gradient boosting (LightGBM/CatBoost) over a neural network with embeddings, because GBMs typically reach strong accuracy on data of this size with less tuning, handle missingness and mixed scales natively, and have native categorical handling that avoids the embedding-design and data-volume concerns a neural net brings. A neural net becomes the better choice mainly when the dataset is substantially larger, the categoricals carry rich learnable semantics worth sharing across tasks, or the model needs to plug into an end-to-end differentiable pipeline.

Structured elaboration

Why sample size at 100k matters. Neural nets, and embeddings for high-cardinality categoricals in particular, need enough examples per category to learn a meaningful representation. With 200 features and 100k rows, a high-cardinality categorical (say, 5,000 distinct values) may only have on the order of 20 examples per category on average, not much for a network to learn a useful embedding vector from scratch, whereas a tree-based split on that same categorical (using target statistics) is a much lower-variance way to extract signal from a similarly sparse per-category sample.

When the neural net does start to win. As data grows well past 100k, embeddings get enough examples per category to learn genuinely useful representations, sometimes capturing similarity structure between categories (e.g. two product SKUs behaving similarly) that a GBM's per-feature splits don't naturally exploit. A neural net is also the better choice when there's a reason to want a differentiable end-to-end pipeline (e.g. joint training with another neural component, or transfer of learned embeddings to a related task), independent of which model wins on raw accuracy.

Worked example

Suppose one of the 200 features is a "merchant ID" categorical with 5,000 distinct values across the 100k rows. With even distribution across categories, that's $100{,}000 / 5{,}000 = 20$ rows per merchant on average. A tree-based split using target statistics (e.g. the historical target rate for that merchant, computed with appropriate regularization/smoothing to avoid leakage) reduces this to a single informative numeric feature per split candidate, requiring only enough examples per category to estimate a stable rate, not enough to fit a multi-dimensional embedding vector. An embedding of size, say, 16 for that same categorical has 16 free parameters per category to learn, i.e. $5{,}000 \times 16 = 80{,}000$ parameters for that one feature alone, competing for signal against only 20 rows per category; that parameter count relative to available data per category is the concrete reason embeddings are data-hungry in exactly the regime this question describes.

Trade-offs and pitfalls

• GBMs can plateau in accuracy once real interaction structure gets very deep or the categorical semantics matter (e.g. two rarely co-occurring categories that behave similarly); a neural net with shared embeddings can sometimes capture that structure a tree-based split cannot.
• Ensembling a GBM and a neural net sometimes captures complementary error patterns and outperforms either alone, but that's a meaningfully larger engineering and maintenance surface for a modest accuracy gain, worth it only when the accuracy gain is validated and the maintenance cost is acceptable.
• A common wrong turn is reaching for embeddings because "neural nets are more powerful" in the abstract, without checking whether the per-category sample size actually supports learning a useful embedding; at 100k rows with high-cardinality categoricals, that assumption often doesn't hold.
• The right first step in almost every case is to fit the GBM as a fast baseline; if it saturates and the use case justifies the added complexity (larger expected future data volume, need for a differentiable pipeline), only then invest in the neural net path.

Direct answer

Yes, PCA is a reasonable default here: strong pairwise correlation among thousands of features is exactly the regime where a handful of components can absorb most of the variance, and reducing dimensionality before modeling helps when samples are scarce relative to features, because it shrinks the effective number of parameters the downstream model has to estimate. The decision is not PCA-or-nothing, though: it trades interpretability and some potentially label-relevant signal for variance compression, dimensionality reduction, and (indirectly) variance reduction in whatever model comes after it, so it is worth validating against alternatives rather than reaching for it reflexively.

Structured elaboration

Why correlated features plus scarce samples favors PCA. With $n$ samples and $p$ features where $p \gg n$, the covariance matrix is rank-deficient and any model with $p$ free parameters is in a severe small-sample regime, prone to high variance and overfitting. There is also a hard structural fact worth knowing: after centering, the $n$ centered rows of $X$ sum to zero, which forces them to lie in an $(n-1)$-dimensional subspace regardless of how large $p$ is. So no matter how many thousands of features you have, PCA can produce at most $n-1$ components with nonzero variance; strong correlation among the original features is what determines whether most of that variance concentrates in a handful of components rather than being spread thinly across all $n-1$.

Decision checklist:

1. Check the correlation structure first, either a correlation heatmap or the eigenvalue spectrum of the covariance matrix (the sorted list of eigenvalues, each one representing how much variance the corresponding principal component captures). If eigenvalues drop off sharply after the first few, PCA will compress well; if they decay slowly, PCA buys little.
2. Decide whether the downstream model is linear or tree-based. Linear/distance-based models (linear/logistic regression, k-NN, SVM) tend to benefit most from PCA's decorrelation and dimensionality reduction; tree-based models (random forests, gradient boosting) usually do fine on raw correlated features and may lose useful axis-aligned splits after rotation.
3. Decide whether you need interpretability. If a stakeholder needs to know "which original features drove this prediction," PCA's linear-combination components make that harder to answer directly (the loadings, the per-original-feature weights that make up each component, tell you which original features contribute most to a given component, but that's still an indirect read compared to a model built directly on the original features).
4. Always fit the PCA transform (mean and components) on the training fold only, then apply it to validation/test, to avoid leaking information about the held-out set into the projection.
5. Treat the number of components as a hyperparameter tuned by cross-validated downstream performance, not just a fixed variance threshold, when a supervised task is the actual goal.

What you gain: fewer, decorrelated features for the downstream model to fit, which reduces variance and the risk of overfitting when $n$ is small; faster training and smaller storage; and, as a side effect, PCA acts as a form of denoising if the discarded low-variance directions are mostly noise.

What you lose: direct interpretability of individual input features; any predictive signal that happens to sit in a low-variance direction, since PCA is unsupervised and has no way to know a direction is small-variance-but-label-relevant; and a small amount of information whenever you keep $k < n-1$ components (the reconstruction error grows as you discard more).

Worked example

A small, fully reproducible illustration of the concentration effect. Simulate data from 3 shared latent factors plus feature-specific noise, with $n=50$ samples, $p=200$ features, a fixed seed, so anyone re-running this gets the same numbers:

import numpy as np

rng = np.random.default_rng(42)
n, p, k_true = 50, 200, 3
latent = rng.normal(size=(n, k_true))
loadings = rng.normal(size=(k_true, p))
X = latent @ loadings + rng.normal(scale=0.5, size=(n, p))

Xc = X - X.mean(axis=0)
_, S, _ = np.linalg.svd(Xc, full_matrices=False)
explained = (S ** 2) / (S ** 2).sum()
print("rank of Xc:", np.linalg.matrix_rank(Xc))          # 49, capped at n-1
print("explained variance, top 3:", explained[:3].round(4))
print("cumulative, top 3:", explained[:3].sum().round(4))

Running this gives explained variance ratios of approximately [0.4239, 0.3201, 0.1646] for the first three components, summing to about 0.9086, and a matrix rank of 49 (which is exactly $n-1$, confirming the structural cap above). Despite 200 features, three components recover over 90% of the variance because the features were generated from only 3 latent factors, which is the same mechanism that makes real strongly-correlated feature sets compress well.

Trade-offs & pitfalls

• Fitting PCA on the full dataset (train + test) before splitting is a common leakage mistake; always fit on train only.
• Standardize before PCA if the thousands of features are not on comparable scales, or the components will be dominated by whichever features happen to have the largest raw variance.
• Don't treat a variance-based cutoff as the final answer when there is a supervised target; cross-validate the number of components against the actual downstream metric, since maximum-variance directions and maximum-label-information directions are not guaranteed to coincide.
• If interpretability or nonlinear structure genuinely matters more than compression, compare PCA against supervised feature selection (L1-regularized models, mutual information) or a nonlinear reduction, rather than assuming PCA is the only lever available.

Direct answer

Gradient boosting can be viewed as gradient descent, not in the usual space of model parameters, but in the space of functions: at each round it computes the negative gradient of the loss with respect to the current prediction (the "pseudo-residual") and fits a new tree to approximate that direction. XGBoost goes a step further and uses a second-order Taylor expansion of the loss, incorporating the Hessian alongside the gradient, which gives a closed-form optimal weight for each leaf and lets the model take better-calibrated steps, converging faster and more accurately than a purely first-order approach.

Structured elaboration

Functional gradient descent. Ordinary gradient descent updates a parameter vector $\theta$ by moving against the gradient of a loss: $\theta \leftarrow \theta - \eta \nabla_\theta L$. Gradient boosting treats the entire prediction function $F$ as the thing being optimized, and takes steps in function space:

where $h_m$ is chosen to approximate the negative gradient of the loss with respect to the current predictions, evaluated at each training point:

Since $h_m$ can only be drawn from a restricted hypothesis class (trees of a given depth), fitting $h_m$ to these pseudo-residuals is a projection of the true negative-gradient direction onto whatever the tree class can represent, which is why the fit at each step is approximate rather than exact.

Pseudo-residuals for common losses. For squared-error loss $L = \frac{1}{2}(y-F)^2$:

the pseudo-residual is exactly the ordinary residual, which is why "fit to residuals" is the usual first intuition for gradient boosting. For logistic loss with $F$ the log-odds and $p = \sigma(F)$:

so each tree fits the gap between the true label and the current predicted probability, not the raw label itself.

Second-order (Hessian) information. A first-order step only tells you a direction; it doesn't tell you how far to move or how confidently. XGBoost expands the loss to second order around the current prediction using a Taylor series:

where $g_i = \partial L/\partial F$ is the gradient (same as the first-order pseudo-residual, up to sign) and $h_i = \partial^2 L/\partial F^2$ is the Hessian. For a fixed tree structure with leaf $j$ containing the set of examples $I_j$, and a regularization term $\lambda$ on the leaf weight $w_j$, the per-leaf objective is:

Taking the derivative with respect to $w_j$ and setting it to zero gives the closed-form optimal leaf weight:

This is a Newton step: instead of just moving the leaf output in the gradient's direction by a fixed learning rate, the update is automatically scaled by the local curvature $H_j$ of the loss in that leaf, taking a larger step where the loss is flatter (small $H_j$) and a smaller, more cautious step where it's sharply curved (large $H_j$). This is why second-order boosting tends to converge in fewer rounds than a purely gradient-fit (first-order) approach: each leaf's weight is close to the actual loss-minimizing value for that leaf's structure, rather than an approximate direction that still needs a separately-tuned shrinkage factor to behave well.

Worked example

Take logistic loss with a single leaf containing 4 training examples, current predicted probabilities $p = [0.6, 0.3, 0.8, 0.4]$ and true labels $y = [1, 0, 1, 0]$. Per-example gradients $g_i = p_i - y_i$: $g = [0.6-1,\ 0.3-0,\ 0.8-1,\ 0.4-0] = [-0.4,\ 0.3,\ -0.2,\ 0.4]$, summing to $G = -0.4+0.3-0.2+0.4 = 0.1$. Per-example Hessians for logistic loss are $h_i = p_i(1-p_i)$: $h = [0.6(0.4),\ 0.3(0.7),\ 0.8(0.2),\ 0.4(0.6)] = [0.24,\ 0.21,\ 0.16,\ 0.24]$, summing to $H = 0.24+0.21+0.16+0.24 = 0.85$. With regularization $\lambda = 1$:

This small negative weight is added (scaled by the learning rate) to the log-odds prediction for every example in this leaf, nudging predictions slightly downward, consistent with this leaf's examples being, on net, very slightly over-predicted ($G>0$ means the gradient sum favors decreasing $F$). Note that the Hessian sum $H=0.85$ directly dampens the step size in the denominator: a leaf with more confident (further from 0.5) predictions has smaller Hessians, so a Hessian-aware update takes a comparatively larger step there than a purely gradient-based rule (which would just use $G$ scaled by a fixed learning rate) would.

Trade-offs & pitfalls

• The Newton-step leaf weight is only optimal for the current tree structure; XGBoost also uses $G_j, H_j$ to score candidate splits themselves (the "gain" formula), so the second-order information shapes both how the tree is built and how its leaves are weighted, not just the leaf weights alone.
• Second-order boosting still needs learning_rate as an external shrinkage factor on top of the Newton step; the closed-form $w^*$ tells you the loss-minimizing weight for a fixed tree structure, not how aggressively to trust that structure round over round, so overfitting is still a real risk without shrinkage and regularization ($\lambda$).
• For losses where the Hessian is constant (squared error, $h_i \equiv 1$), the second-order update reduces to the same leaf weight a first-order approach would give (mean of residuals in the leaf, up to the $\lambda$ regularization term); the practical benefit of second-order information is largest for losses with input-dependent curvature, like logistic loss, where $h_i = p_i(1-p_i)$ varies across examples.
• A common misconception is that "using the Hessian" means literally forming a Hessian matrix; in gradient boosting frameworks it's a per-example scalar (the second derivative of a scalar loss with respect to a scalar prediction), which is why this stays computationally cheap even at large scale.