Machine Learning Fundamentals Questions

Ways to compute feature importance:

1. Gain (split importance) — sum of impurity decrease for splits using each feature (fast, built-in). Pros: efficient, reflects tree training objective. Cons: biased toward features with many splits or high cardinality and not causal.
2. Permutation importance — measure drop in metric when permuting a feature in validation set. Pros: model-agnostic, reflects actual impact on predictions. Cons: expensive, affected by feature collinearity (can underestimate importance of correlated features).
3. SHAP values — local additive attributions based on game theory approximations (TreeSHAP is efficient for trees). Pros: consistent, decomposes individual predictions, handles interactions. Cons: computational cost, requires careful aggregation for global importance and can be misinterpreted.
   Which to present:

• Regulatory reporting: use permutation importance with stability intervals and documented methodology (transparent, interpretable global effect) plus simple gain for reference. Include caveats on correlated features.
• Internal debugging: provide SHAP (per-case explanations) combined with gain to quickly identify model behavior and interactions; use permutation tests for confirmatory analysis.
  Always include confidence intervals, correlation-aware analyses (conditional permutation), and clear caveats about causality.

Conceptual comparison: L1 (lasso) adds absolute-value penalty driving many weights to zero — it performs implicit feature selection. L2 (ridge) adds squared penalty that shrinks weights continuously but rarely zeroes them, preferring distributed small weights.
In high-dimensional sparse scenario: choose L1 because it yields sparse models, easier interpretation, and reduced variance by selecting only informative features. Often Elastic Net (combining L1 and L2) is practical to handle correlated features.
Impact on feature selection & interpretability:

• L1: produces sparse coefficients, simpler models, easier to explain which features matter. But when features are highly correlated it may arbitrarily pick one.
• L2: retains correlated features with small weights, better numeric stability, but less interpretable since many small nonzero coefficients remain.
  Regularization trade-off: stronger penalty reduces variance but increases bias; choose via cross-validation and consider domain needs for interpretability vs predictive performance.

High-level: k-NN classifies a query by finding the k closest labeled examples and aggregating their labels (majority vote or weighted). It’s simple, non-parametric, and adapts to data without training a global model.
Practical limitations:

• Latency/throughput: brute-force distance to all points is O(n·d), too slow for large n.
• Memory: storing full dataset can be large.
• Curse of dimensionality: distances become less meaningful as d grows; neighbors may be noisy.
  Engineering mitigations:
• Approximate nearest neighbor (ANN) libraries (FAISS, Annoy, HNSW) reduce query time with sublinear lookup and controlled recall/precision trade-offs.
• Dimensionality reduction (PCA, autoencoders) or feature selection to reduce d.
• Vector quantization / product quantization to reduce memory and speed queries (used in FAISS).
• Sharding and caching hot queries; use GPU-accelerated ANN for high throughput.
  Trade-offs: ANN and compression improve latency/scale but can slightly reduce accuracy; dimensionality reduction may lose discriminative signal, so validate end-to-end.

Direct answer
Bias is the error a model makes because it's too simple to capture the true pattern in the data; variance is the error it makes because it's too sensitive to the specific noise in the training set it happened to see. There's a tradeoff because, as you increase a model's complexity, bias tends to go down (it can represent more complicated patterns) while variance tends to go up (it has more freedom to fit noise), and a model's total expected error is driven by both.

Structured elaboration
One way to make this concrete: the expected squared prediction error of a model decomposes into three pieces.

$\text{Expected error} = \text{Bias}^2 + \text{Variance} + \text{Irreducible noise}$

• Bias is how far off the model's average prediction is from the true value, if you imagine retraining it on many different samples of training data. A high-bias model (like a straight line fit to a curved relationship) systematically misses the pattern regardless of which specific training set it sees; that's underfitting.
• Variance is how much the model's prediction would change if you retrained it on a different sample of training data. A high-variance model fits the particular quirks and noise of the training set it happened to get, so it looks great on that data but performs inconsistently, and usually worse, on new data; that's overfitting.
• Irreducible noise is randomness inherent in the problem itself that no model, however good, can predict away.
• What shifts the balance: increasing model complexity (more parameters, deeper trees, higher-degree polynomials) generally trades bias for variance. Increasing the amount of training data mainly reduces variance (a complex model has less room to fit noise when there's more real signal to constrain it), without directly changing bias. Increasing label noise makes variance more costly, because a flexible model has more noise to latch onto.

Worked example
Imagine fitting polynomials of increasing degree to noisy data generated from a true quadratic relationship. A degree-1 (straight-line) fit has high bias: no matter how much data you give it or which sample you train on, it can't capture the curve, so it consistently underfits both the training data and new data in a similar, systematic way. A degree-15 fit has low bias (it's flexible enough to pass close to almost any set of points) but high variance: fit it to one noisy sample and it wiggles through the noise one way, fit it to a different noisy sample drawn from the same true relationship and it wiggles a different way. Its training error looks excellent, but its error on new data is inconsistent and, on average, worse than a well-chosen degree-2 fit, which matches the true relationship's complexity and rides squarely between the two extremes. If you plotted training error and validation error against polynomial degree, training error would keep falling as degree increases, while validation error would fall, bottom out near degree 2, and then rise again as the higher-degree fits start memorizing noise; that divergence between the two curves is the signature of moving from the underfitting regime into the overfitting regime.

Trade-offs and pitfalls
The most common mistake is treating "more complex model" as an unambiguous improvement; it only reduces bias, and can make things worse overall if variance grows faster than bias shrinks. Another common confusion is treating bias and variance as literally opposite ends of a single knob you can read off directly from a training curve; they're both properties of the model's error, and you generally infer the regime you're in indirectly, from the gap between training and validation performance, rather than measuring bias and variance directly.

Direct answer
Overfitting is when a model has essentially memorized quirks of its specific training data rather than learning the underlying pattern, so it performs well on training data but noticeably worse on new data. Underfitting is the opposite: the model is too simple or undertrained to capture the real pattern, so it performs poorly on both training and new data.

Structured elaboration

• Diagnostic sign 1, the training-validation gap: overfitting shows up as a large and often growing gap between training performance (very good) and validation performance (noticeably worse). Underfitting shows up as both training and validation performance being poor and close together, because the model hasn't captured enough signal to do well anywhere.
• Diagnostic sign 2, learning curves over training time: plotting training and validation loss against training epoch or iteration, overfitting shows training loss continuing to fall while validation loss flattens out and then starts rising. Underfitting shows both curves plateauing early at a high loss, with more training not helping much.
• Diagnostic sign 3, learning curves over data size: plotting performance against the amount of training data used, a high-variance (overfitting) model shows a persistent, slowly-closing gap between training and validation performance as data grows. A high-bias (underfitting) model shows training and validation performance converging quickly to the same, disappointingly-low level, meaning more data alone won't fix it.
• General remedies for overfitting: reduce model complexity, add regularization, gather more training data, or use techniques like cross-validation to select a less complex model in the first place. The trade-off is that most of these remedies also risk pushing the model toward underfitting if applied too aggressively, so they're usually tuned rather than applied at maximum strength.
• General remedies for underfitting: increase model complexity or capacity, add more informative features, reduce excessive regularization, or train for longer if the model hasn't converged. The trade-off here is the mirror image: push too far and you're back to overfitting, and added complexity often costs more compute and makes the model harder to interpret.

Worked example
Suppose you're building a classifier on tabular customer data and see training accuracy of 98 percent but validation accuracy of 72 percent. That large gap, with training performance far outpacing validation, is the classic overfitting signature: the model has likely fit noise or overly specific patterns present only in the training rows. If instead you saw training accuracy of 74 percent and validation accuracy of 71 percent, both mediocre and close together, that points to underfitting: the model hasn't found enough signal even in the data it was trained on, so the fix is more likely to be a richer feature set or a more expressive model rather than more regularization.

Trade-offs and pitfalls
A common mistake is applying an overfitting remedy (like adding heavy regularization) to a model that's actually underfitting, which makes the underfitting worse rather than helping; the diagnostic step (check whether training performance itself is also poor) has to come before picking a remedy. Another pitfall is treating a single evaluation as diagnostic; a genuinely useful read on which regime you're in generally needs the shape of a curve, either over training time or over training-set size, not just one number.