Machine Learning Fundamentals Questions

Training/validation/test splits: training fits model, validation tunes hyperparameters, test estimates final generalization. For 100k examples: common split 70/15/15 (70k train, 15k val, 15k test). If time-series: use chronological splits (e.g., first 80% train, next 10% val, final 10% test) to avoid leakage and mimic production. For class imbalance: ensure stratified splits so class proportions are preserved across sets; if minority class is tiny, consider oversampling/SMOTE on training only or use larger validation/test sets for reliable estimates. Also use cross-validation or repeated stratified folds when data is limited.

Factors influencing retraining frequency: data volume (how fast new labeled data accumulates), detected distribution or label drift, business cost and latency for retraining, SLA for model freshness, and regulatory requirements.
Hybrid schedule proposal:

• Periodic: monthly lightweight retrain using recent rolling window to capture slow drift.
• Event-driven: trigger full retrain when automated drift detectors exceed thresholds (feature distribution shift, label distribution change, or model performance drop) or when a significant data milestone (e.g., +30% new labeled data) is reached.
  Validation before swap-in:
• Run A/B or shadow evaluation: compare candidate vs production on holdout test and online shadow traffic.
• Pre-deploy checks: unit tests, fairness and slice performance, calibration, latency and memory benchmarks, and canary rollout with monitored KPIs.
• Statistical acceptance: require non-inferior primary metric with p-value/confidence interval and no regressions on critical slices.
  If checks pass, stage rollout with gradual traffic ramp and automated rollback if anomalies appear.

Direct answer
Overfitting is when a model gets extremely good at the exact examples it practiced on, but hasn't actually learned the general pattern, so it struggles on anything new. A useful analogy: imagine a student who memorizes the answers to last year's practice exam questions word for word, rather than learning the underlying concepts. They'd score perfectly if given that exact practice exam again, but do poorly on this year's real exam, which asks the same concepts in slightly different ways.

Structured elaboration
That's exactly what's happening when a model shows excellent training performance but poor performance on new data: it has effectively memorized quirks and noise specific to the training examples, rather than learning the broader pattern that would actually transfer to new, similar situations. The training metrics look great because the model is being tested on the same "practice questions" it studied; the real-world performance is poor because new data is like a slightly different, real exam.

Worked example
A simple, concrete step: hold back some data the model never gets to train on (this is the equivalent of a genuinely new practice exam the student hasn't seen), and check performance there instead of only on the training data. If performance on that held-back data is much worse than on the training data, that gap is itself evidence of overfitting, similar to how the student's real exam score being much lower than their practice-exam score would be the tell that memorization, not real understanding, was happening.

Trade-offs and pitfalls
A second simple, concrete step: make the model somewhat "simpler" or less free to memorize fine details, similar to how you might encourage a student to focus on understanding a few core concepts well rather than memorizing every specific example, which usually generalizes better to questions they haven't seen before. The trade-off worth flagging honestly to a non-technical stakeholder is that a small amount of "forgetting" specific training examples is actually the price of better real-world performance, and chasing perfect training performance is often exactly the wrong goal.

Direct answer
A dataset is split into training, validation, and test sets so that a model can be fit, tuned, and honestly evaluated using three separate pools of data, none of which contaminate each other. The training set is what the model actually learns its parameters from. The validation set is used to make decisions about the model (comparing hyperparameters, comparing candidate models, deciding when to stop training) while it's still being developed. The test set is held back and used exactly once, at the very end, to get an unbiased estimate of how the finished model will perform on new data.

Structured elaboration

• Why not just use the training set to evaluate: a model's performance on the exact data it was fit to is optimistic by construction, especially for flexible models, so it doesn't tell you how the model will do on data it hasn't seen.
• Why the validation set isn't enough by itself: if you use the validation set to make many decisions (try model A, try model B, try ten hyperparameter settings, pick the best one on validation performance), you're implicitly fitting to the validation set too, just at a higher level than parameter fitting. The best-on-validation choice is itself a form of overfitting to that particular validation set, so its own validation score is an optimistic estimate of true future performance.
• Why the test set has to be held back and touched once: it exists specifically to give an estimate that hasn't been used, even indirectly, to make any modeling decision. As soon as you look at test performance and then go back and change something in response, it stops serving that purpose and becomes a second validation set in disguise.
• Typical split ratios: for a large dataset there's plenty of data to go around, so a common starting point is roughly 70 percent training, 15 percent validation, 15 percent test, though the exact split matters less once each piece is large enough to give a stable estimate. For a small dataset, holding out 15 percent for validation and 15 percent for test might leave too little data to train on or to get a stable estimate from either held-out set, so techniques like k-fold cross-validation (using different rotating subsets of the training data as validation across several training runs) are often preferred over a single fixed validation split, to make more efficient use of limited data. Splitting strategy also needs to adapt when data isn't simple independent rows, for example, time-ordered data (where the split should respect time order, not be randomized) or imbalanced or grouped data (where naive random splitting can leak related rows across the split boundary or produce splits with badly skewed class proportions).

Worked example
For a dataset of 100,000 rows using a 70/15/15 split, that's 70,000 rows for training, 15,000 for validation, and 15,000 for test. During development, you might try several model configurations, each trained on the 70,000-row training set and scored on the 15,000-row validation set, and pick whichever configuration scores best there. Only once that choice is final do you run the chosen model on the 15,000-row test set a single time, and that number, not the validation number, is what you report as the honest estimate of how the model will perform on new data.

Trade-offs and pitfalls
The most common mistake is treating the test set as just another validation set and checking it repeatedly during development; even a small amount of this quietly turns the test score into an optimistic estimate again. Another common mistake is splitting the data randomly regardless of its structure; for time-series data especially, a random split will leak future information into training and make validation performance look far better than the model will actually achieve in production, where it only ever has access to the past.

Definition: Bias is error from incorrect assumptions underfitting the true pattern; variance is error from sensitivity to training data overfitting.
Concrete examples:

• High-bias model: linear regression on nonlinear data (e.g., quadratic relationship). Remedy: increase model capacity — add polynomial features or switch to a nonlinear model (tree-based or neural net). In production, implement feature engineering or deploy a higher-capacity model and validate with cross-validation and holdout tests.
• High-variance model: deep neural network with small dataset causing overfitting. Remedy: regularization — L2/dropout, data augmentation, early stopping, or collect more labeled data. In production, add regularization hyperparameters, implement robust validation (k-fold), and use monitoring for generalization drift.
  Practical pipeline steps:
• For bias: add features, choose richer model family, monitor train vs validation error gap to confirm reduced bias.
• For variance: apply regularization, early stopping, ensembling, and expand dataset; track validation stability and reduce model complexity until variance is controlled.