Machine Learning Fundamentals Questions

Direct answer
Overfitting is when a model gets extremely good at the exact examples it practiced on, but hasn't actually learned the general pattern, so it struggles on anything new. A useful analogy: imagine a student who memorizes the answers to last year's practice exam questions word for word, rather than learning the underlying concepts. They'd score perfectly if given that exact practice exam again, but do poorly on this year's real exam, which asks the same concepts in slightly different ways.

Structured elaboration
That's exactly what's happening when a model shows excellent training performance but poor performance on new data: it has effectively memorized quirks and noise specific to the training examples, rather than learning the broader pattern that would actually transfer to new, similar situations. The training metrics look great because the model is being tested on the same "practice questions" it studied; the real-world performance is poor because new data is like a slightly different, real exam.

Worked example
A simple, concrete step: hold back some data the model never gets to train on (this is the equivalent of a genuinely new practice exam the student hasn't seen), and check performance there instead of only on the training data. If performance on that held-back data is much worse than on the training data, that gap is itself evidence of overfitting, similar to how the student's real exam score being much lower than their practice-exam score would be the tell that memorization, not real understanding, was happening.

Trade-offs and pitfalls
A second simple, concrete step: make the model somewhat "simpler" or less free to memorize fine details, similar to how you might encourage a student to focus on understanding a few core concepts well rather than memorizing every specific example, which usually generalizes better to questions they haven't seen before. The trade-off worth flagging honestly to a non-technical stakeholder is that a small amount of "forgetting" specific training examples is actually the price of better real-world performance, and chasing perfect training performance is often exactly the wrong goal.

Situation: Highly skewed 10-class problem with one class 70%.
Data-level strategies:

• Resampling: upsample minority classes (SMOTE variants for multiclass) or downsample majority. Trade-off: upsampling risks overfitting; downsampling loses data and may reduce overall accuracy.
• Synthetic augmentation: generate realistic samples for rare classes if domain permits.
• Stratified batch sampling during training to ensure class exposure.
  Model-level strategies:
• Class-weighting or focal loss to penalize mistakes on minority classes.
• One-vs-rest classifiers or hierarchical classification if classes have structure.
• Ensemble of specialist models focusing on minority groups.
  Precision vs recall trade-offs in production:
• If minority-class false negatives are costly, prioritize recall (increase sensitivity) using lower decision thresholds, accept reduced precision and more false positives; add downstream verification to mitigate.
• If false positives cost more, tune for precision (higher threshold), accept missed positives. Monitor per-class metrics, confusion matrices, and business KPIs post-deployment and consider cost-sensitive learning or calibration to align model behavior with operational cost.

Direct answer
With only 1 percent of examples labeled, a semi-supervised approach is worth trying, since there's likely a large amount of untapped structure in the 99 percent unlabeled data that a purely supervised model trained on the 1 percent alone would never see. But whether it actually helps has to be checked, not assumed, because a poorly-chosen semi-supervised technique can just as easily add confident noise as it can add useful signal.

Structured elaboration

• Why semi-supervised is worth trying here: a model trained on only 1 percent of the data has very little to learn from directly, and is likely to be both high-variance (unstable across different random 1 percent samples) and missing patterns that only show up with more examples. Techniques such as self-training (using the model's own confident predictions on unlabeled data as if they were labels, then retraining on the enlarged set) or consistency-regularization let the model extract additional signal from the unlabeled 99 percent, without needing new human labels, which is exactly the constraint here (getting more labels is expensive).
• What would make you confident it's actually helping: the honest test is a controlled comparison, not just checking whether the semi-supervised model's training-time metrics look good (a self-training approach in particular can look confidently good on the pseudo-labels it generated for itself, which is a weak signal since it's grading its own homework, while actually being wrong). Train a baseline model on only the labeled 1 percent, train the semi-supervised model using the same labeled 1 percent plus the unlabeled 99 percent, and compare both on the same held-out labeled evaluation set that neither model trained on. If the semi-supervised model beats the baseline there, by a margin larger than the run-to-run noise you'd see just from re-training either model with a different random seed, that's real evidence it helped.
• Setting up a fair comparison: use the identical held-out evaluation set for both models, sourced entirely from labeled data that neither model touched during training. Since 1 percent of the data is a small sample, also check that the comparison isn't just noise from that particular small labeled split, for example by repeating the comparison across a few different random 1 percent labeled subsets, or by looking at whether the improvement is consistent rather than a one-off.

Worked example
Suppose the full dataset has 500,000 examples, so 1 percent labeled means roughly 5,000 labeled examples. You'd hold out a separate labeled evaluation set, say 1,000 of those 5,000, and train the supervised baseline on the remaining 4,000. The semi-supervised model would also train on those same 4,000 labeled examples, plus the roughly 495,000 unlabeled ones, using a technique like self-training. Both models get evaluated on the same 1,000 held-out labeled examples, and only if the semi-supervised model's score there is meaningfully and consistently higher than the baseline's would you conclude the extra unlabeled data was actually worth using.

Trade-offs and pitfalls
The main trap is judging "it's helping" from the semi-supervised model's own training process rather than an independent, held-out comparison; a self-training loop, in particular, can report improving performance on its own pseudo-labels while actually drifting away from correct behavior, because it's partly grading its own homework. Getting more labels, if it's feasible even at a slower pace, is also worth weighing against the engineering cost and added complexity of a semi-supervised pipeline; sometimes a modest amount of additional targeted labeling (focused on the examples the current model is least confident about) is a more sample-efficient investment than a more elaborate semi-supervised setup.

Direct answer
Overfitting is when a model has essentially memorized quirks of its specific training data rather than learning the underlying pattern, so it performs well on training data but noticeably worse on new data. Underfitting is the opposite: the model is too simple or undertrained to capture the real pattern, so it performs poorly on both training and new data.

Structured elaboration

• Diagnostic sign 1, the training-validation gap: overfitting shows up as a large and often growing gap between training performance (very good) and validation performance (noticeably worse). Underfitting shows up as both training and validation performance being poor and close together, because the model hasn't captured enough signal to do well anywhere.
• Diagnostic sign 2, learning curves over training time: plotting training and validation loss against training epoch or iteration, overfitting shows training loss continuing to fall while validation loss flattens out and then starts rising. Underfitting shows both curves plateauing early at a high loss, with more training not helping much.
• Diagnostic sign 3, learning curves over data size: plotting performance against the amount of training data used, a high-variance (overfitting) model shows a persistent, slowly-closing gap between training and validation performance as data grows. A high-bias (underfitting) model shows training and validation performance converging quickly to the same, disappointingly-low level, meaning more data alone won't fix it.
• General remedies for overfitting: reduce model complexity, add regularization, gather more training data, or use techniques like cross-validation to select a less complex model in the first place. The trade-off is that most of these remedies also risk pushing the model toward underfitting if applied too aggressively, so they're usually tuned rather than applied at maximum strength.
• General remedies for underfitting: increase model complexity or capacity, add more informative features, reduce excessive regularization, or train for longer if the model hasn't converged. The trade-off here is the mirror image: push too far and you're back to overfitting, and added complexity often costs more compute and makes the model harder to interpret.

Worked example
Suppose you're building a classifier on tabular customer data and see training accuracy of 98 percent but validation accuracy of 72 percent. That large gap, with training performance far outpacing validation, is the classic overfitting signature: the model has likely fit noise or overly specific patterns present only in the training rows. If instead you saw training accuracy of 74 percent and validation accuracy of 71 percent, both mediocre and close together, that points to underfitting: the model hasn't found enough signal even in the data it was trained on, so the fix is more likely to be a richer feature set or a more expressive model rather than more regularization.

Trade-offs and pitfalls
A common mistake is applying an overfitting remedy (like adding heavy regularization) to a model that's actually underfitting, which makes the underfitting worse rather than helping; the diagnostic step (check whether training performance itself is also poor) has to come before picking a remedy. Another pitfall is treating a single evaluation as diagnostic; a genuinely useful read on which regime you're in generally needs the shape of a curve, either over training time or over training-set size, not just one number.

Three regularization techniques: 1) L2 (weight decay) — adds penalty proportional to squared weights, discouraging large weights and smoothing the function. 2) L1 (Lasso) — penalizes absolute weights, encouraging sparsity and feature selection, which reduces overfitting. 3) Dropout — randomly zeroes neuron activations during training, preventing co-adaptation and forcing redundant representations that generalize better. (Also early stopping: stop when validation error rises, preventing overfitting.)