Machine Learning Fundamentals Questions

Early stopping: monitor validation metric during training and stop when it stops improving to prevent overfitting. It acts as implicit regularization by limiting training time, preventing the model from fitting noise.
Implementation with noisy validation:

• Use a patience parameter: require N consecutive non-improving evaluations before stopping.
• Smooth metrics using rolling average or exponential moving average to reduce noise-driven stops.
• Use relative improvement threshold (delta) so tiny fluctuations don’t count as improvement.
• Check multiple metrics (validation loss + primary metric) and stop only when both plateau.
• For production training jobs: checkpoint the best model and continue training but keep periodic evaluation on larger/cleaner holdout to confirm improvements.
• Optionally employ ensemble of top-K checkpoints instead of single best.
  Example config: patience=10 evals, delta=0.001, EMA smoothing with alpha=0.2, checkpoint best model. This prevents premature stopping while still constraining overfitting.

Direct answer: Feature engineering for supervised learning is shaped by having a known target to optimize against, while for unsupervised tasks (clustering, similarity-based retrieval) there's no target to guide feature selection or interaction discovery, so the emphasis shifts toward representations that capture meaningful STRUCTURE in the data itself, and scaling/imputation choices matter differently across the two paradigms too.

Structured elaboration:
For supervised tasks: transformations can be validated against the target directly (does this transform improve held-out predictive performance), tree-based models can discover interactions relative to the target through their split structure, and feature selection has a clear objective function (predictive performance) to optimize.
For unsupervised tasks (like clustering): there's no target to validate against, so feature scaling matters MUCH more directly, since most clustering algorithms rely on a distance metric, and an unscaled feature with a much larger numeric range will dominate the distance calculation regardless of its actual real-world importance; imputation choices for clustering also need extra care, since an imputed value can distort the geometric structure the clustering algorithm relies on, in a way that a supervised model's regularization might otherwise partially compensate for.
For similarity-based retrieval specifically: the feature representation IS the thing being directly compared (via a distance or similarity metric), so the representation's quality is the entire ballgame, unlike a supervised model where a somewhat-suboptimal representation can still be partially compensated for by the model's own fitting process.

Worked example: A customer-segmentation clustering task on features including both a 0-1 engagement score and a raw dollar lifetime-value figure (potentially in the tens of thousands) would, without scaling, have the clustering algorithm's distance metric almost entirely dominated by the dollar figure, effectively ignoring the engagement score regardless of its real segmentation value; scaling both features to a comparable range before clustering is what actually lets both signals contribute meaningfully to the resulting segments.

For a concrete tree-based-supervised example: a gradient-boosted-tree fraud model benefits from an explicit ratio feature like transaction_amount / account_average_transaction_amount even though trees can discover some interactions on their own, because a single split threshold on the raw ratio captures a relationship that would otherwise take the tree many splits across the two raw features (amount, average) to approximate, and only if enough data supports learning that structure; the target is directly available to validate whether adding the ratio actually improves held-out AUC.

For a concrete similarity-retrieval example: a product-search system embedding product titles and images for nearest-neighbor retrieval needs the embeddings L2-normalized before computing cosine or dot-product similarity, since two embeddings that point in the same semantic direction but have different raw magnitudes (for example, from a longer product title producing a larger-norm vector) would otherwise be scored as less similar than they should be, purely as an artifact of vector length rather than actual semantic closeness. There's no label here to compensate for that distortion, since the representation itself is what gets compared.

Trade-offs and pitfalls: The common mistake is carrying supervised-learning feature-engineering habits ("don't worry about scale, the model will compensate") directly into an unsupervised task, where that assumption often simply doesn't hold, particularly for any distance-based method.

Overfitting: model fits training data (including noise) too well and performs poorly on new data. Underfitting: model is too simple to capture underlying patterns, performing poorly on both train and validation. Two diagnostic checks on learning curves: 1) Compare training vs validation error over epochs — large gap (low train error, high val error) indicates overfitting; both high indicates underfitting. 2) Observe error trend as model complexity increases — if validation error increases while training error decreases, that's overfitting; if both decrease and plateau at high error, that's underfitting.

Accuracy is misleading with 99.9% because a naive model predicting 'not fraud' always would be 99.9% accurate while missing all frauds. Better metrics: 1) Precision — fraction of flagged transactions that are truly fraud; measures correctness of positive predictions and helps limit false alarms. 2) Recall (true positive rate) — fraction of actual frauds detected; measures coverage of fraud detection, critical for risk reduction. 3) Precision-Recall AUC or F1-score — balances precision and recall (F1 is harmonic mean), useful when both false positives and false negatives matter. Additionally, use ROC-AUC for overall separability but PR-AUC is preferred under heavy class imbalance.

Direct answer
A dataset is split into training, validation, and test sets so that a model can be fit, tuned, and honestly evaluated using three separate pools of data, none of which contaminate each other. The training set is what the model actually learns its parameters from. The validation set is used to make decisions about the model (comparing hyperparameters, comparing candidate models, deciding when to stop training) while it's still being developed. The test set is held back and used exactly once, at the very end, to get an unbiased estimate of how the finished model will perform on new data.

Structured elaboration

• Why not just use the training set to evaluate: a model's performance on the exact data it was fit to is optimistic by construction, especially for flexible models, so it doesn't tell you how the model will do on data it hasn't seen.
• Why the validation set isn't enough by itself: if you use the validation set to make many decisions (try model A, try model B, try ten hyperparameter settings, pick the best one on validation performance), you're implicitly fitting to the validation set too, just at a higher level than parameter fitting. The best-on-validation choice is itself a form of overfitting to that particular validation set, so its own validation score is an optimistic estimate of true future performance.
• Why the test set has to be held back and touched once: it exists specifically to give an estimate that hasn't been used, even indirectly, to make any modeling decision. As soon as you look at test performance and then go back and change something in response, it stops serving that purpose and becomes a second validation set in disguise.
• Typical split ratios: for a large dataset there's plenty of data to go around, so a common starting point is roughly 70 percent training, 15 percent validation, 15 percent test, though the exact split matters less once each piece is large enough to give a stable estimate. For a small dataset, holding out 15 percent for validation and 15 percent for test might leave too little data to train on or to get a stable estimate from either held-out set, so techniques like k-fold cross-validation (using different rotating subsets of the training data as validation across several training runs) are often preferred over a single fixed validation split, to make more efficient use of limited data. Splitting strategy also needs to adapt when data isn't simple independent rows, for example, time-ordered data (where the split should respect time order, not be randomized) or imbalanced or grouped data (where naive random splitting can leak related rows across the split boundary or produce splits with badly skewed class proportions).

Worked example
For a dataset of 100,000 rows using a 70/15/15 split, that's 70,000 rows for training, 15,000 for validation, and 15,000 for test. During development, you might try several model configurations, each trained on the 70,000-row training set and scored on the 15,000-row validation set, and pick whichever configuration scores best there. Only once that choice is final do you run the chosen model on the 15,000-row test set a single time, and that number, not the validation number, is what you report as the honest estimate of how the model will perform on new data.

Trade-offs and pitfalls
The most common mistake is treating the test set as just another validation set and checking it repeatedly during development; even a small amount of this quietly turns the test score into an optimistic estimate again. Another common mistake is splitting the data randomly regardless of its structure; for time-series data especially, a random split will leak future information into training and make validation performance look far better than the model will actually achieve in production, where it only ever has access to the past.