Machine Learning Fundamentals Questions

Supervised learning: models trained on labeled input→output pairs to predict outputs for new inputs. Example (classification): Email spam filter using labeled emails (spam/ham) with a logistic regression or gradient-boosted trees; supervised learning is appropriate because ground-truth labels exist and goal is to predict discrete classes. Example (regression): Predicting house prices from features (size, location) with a random forest or linear regression; labels are numeric sale prices, so supervised regression fits.

Unsupervised learning: models discover structure in unlabeled data. Example (clustering): Customer segmentation via k-means on purchase behavior to create marketing cohorts—no labels, goal is grouping. Example (dimensionality reduction): PCA to compress high-dimensional sensor data before feeding to downstream models—reduces noise and storage while preserving variance. Each paradigm aligns with whether labels exist and the desired outcome (predict vs discover).

Interpretability: Linear models are generally stronger — coefficients map directly to feature importance and sign. Decision trees are interpretable at small depth but become opaque when large or ensemble.
Feature interactions: Decision trees are stronger — they capture non-linear interactions via splits; linear models need explicit interaction terms.
Robustness to outliers: Decision trees are stronger — splits are based on thresholds and less sensitive to extreme values; linear models (OLS) are sensitive unless regularized or robustified.
Latency at prediction time: Linear models are stronger — dot product is O(#features) with small constant; single tree is similar but ensembles (random forest, gradient boosting) increase latency.
Ease of calibration: Linear models tend to be stronger — outputs (logistic regression) are probabilistic and well-calibrated; tree-based models often require Platt scaling or isotonic regression to calibrate probabilities.
Summary: choose linear models when interpretability, low-latency and calibrated probabilities matter; choose trees when interactions and nonlinearity dominate.

Three production-friendly encodings:

1. Hashing trick (feature hashing): map high-cardinality IDs into a fixed-size vector (e.g., 2^20 buckets). Pros: constant memory, simple, fast to serve. Cons: collisions may reduce signal; lacks learned similarity.
2. Learned embeddings (embedding table): train low-dim embeddings per ID (embedding size 32–128) used by models (FFN or MLP). Pros: captures semantics, high accuracy. Cons: large memory for millions of IDs—requires sharding, memory store (Redis/Key-Value), or parameter servers; increased serving latency unless cached.
3. Target encoding / frequency encoding: replace ID with statistics (mean label, count, smoothed) computed from historical data. Pros: compact numeric features, low latency. Cons: leakage risk—must use out-of-fold encoding and smoothing; may not capture complex relations.
   Trade-offs:

• Model complexity: embeddings and target encodings enable richer models (neural nets) while hashing fits linear/GBDT with less complexity.
• Serving latency: hashing and target encodings are fastest. Embeddings need an efficient store or on-device cache to meet low latency; consider quantized embeddings or approximate lookups.
• Memory and operational cost: embeddings require memory/ops to manage updates and cold-starts; hashing is low maintenance.
  Choose based on accuracy vs operational constraints; hybrid patterns (hashing for rare IDs, embeddings for top-K heavy hitters, target stats for others) often work well.

Ways to compute feature importance:

1. Gain (split importance) — sum of impurity decrease for splits using each feature (fast, built-in). Pros: efficient, reflects tree training objective. Cons: biased toward features with many splits or high cardinality and not causal.
2. Permutation importance — measure drop in metric when permuting a feature in validation set. Pros: model-agnostic, reflects actual impact on predictions. Cons: expensive, affected by feature collinearity (can underestimate importance of correlated features).
3. SHAP values — local additive attributions based on game theory approximations (TreeSHAP is efficient for trees). Pros: consistent, decomposes individual predictions, handles interactions. Cons: computational cost, requires careful aggregation for global importance and can be misinterpreted.
   Which to present:

• Regulatory reporting: use permutation importance with stability intervals and documented methodology (transparent, interpretable global effect) plus simple gain for reference. Include caveats on correlated features.
• Internal debugging: provide SHAP (per-case explanations) combined with gain to quickly identify model behavior and interactions; use permutation tests for confirmatory analysis.
  Always include confidence intervals, correlation-aware analyses (conditional permutation), and clear caveats about causality.

Direct answer
Overfitting is when a model has essentially memorized quirks of its specific training data rather than learning the underlying pattern, so it performs well on training data but noticeably worse on new data. Underfitting is the opposite: the model is too simple or undertrained to capture the real pattern, so it performs poorly on both training and new data.

Structured elaboration

• Diagnostic sign 1, the training-validation gap: overfitting shows up as a large and often growing gap between training performance (very good) and validation performance (noticeably worse). Underfitting shows up as both training and validation performance being poor and close together, because the model hasn't captured enough signal to do well anywhere.
• Diagnostic sign 2, learning curves over training time: plotting training and validation loss against training epoch or iteration, overfitting shows training loss continuing to fall while validation loss flattens out and then starts rising. Underfitting shows both curves plateauing early at a high loss, with more training not helping much.
• Diagnostic sign 3, learning curves over data size: plotting performance against the amount of training data used, a high-variance (overfitting) model shows a persistent, slowly-closing gap between training and validation performance as data grows. A high-bias (underfitting) model shows training and validation performance converging quickly to the same, disappointingly-low level, meaning more data alone won't fix it.
• General remedies for overfitting: reduce model complexity, add regularization, gather more training data, or use techniques like cross-validation to select a less complex model in the first place. The trade-off is that most of these remedies also risk pushing the model toward underfitting if applied too aggressively, so they're usually tuned rather than applied at maximum strength.
• General remedies for underfitting: increase model complexity or capacity, add more informative features, reduce excessive regularization, or train for longer if the model hasn't converged. The trade-off here is the mirror image: push too far and you're back to overfitting, and added complexity often costs more compute and makes the model harder to interpret.

Worked example
Suppose you're building a classifier on tabular customer data and see training accuracy of 98 percent but validation accuracy of 72 percent. That large gap, with training performance far outpacing validation, is the classic overfitting signature: the model has likely fit noise or overly specific patterns present only in the training rows. If instead you saw training accuracy of 74 percent and validation accuracy of 71 percent, both mediocre and close together, that points to underfitting: the model hasn't found enough signal even in the data it was trained on, so the fix is more likely to be a richer feature set or a more expressive model rather than more regularization.

Trade-offs and pitfalls
A common mistake is applying an overfitting remedy (like adding heavy regularization) to a model that's actually underfitting, which makes the underfitting worse rather than helping; the diagnostic step (check whether training performance itself is also poor) has to come before picking a remedy. Another pitfall is treating a single evaluation as diagnostic; a genuinely useful read on which regime you're in generally needs the shape of a curve, either over training time or over training-set size, not just one number.