Machine Learning Fundamentals Questions

Direct answer
Overfitting is when a model gets extremely good at the exact examples it practiced on, but hasn't actually learned the general pattern, so it struggles on anything new. A useful analogy: imagine a student who memorizes the answers to last year's practice exam questions word for word, rather than learning the underlying concepts. They'd score perfectly if given that exact practice exam again, but do poorly on this year's real exam, which asks the same concepts in slightly different ways.

Structured elaboration
That's exactly what's happening when a model shows excellent training performance but poor performance on new data: it has effectively memorized quirks and noise specific to the training examples, rather than learning the broader pattern that would actually transfer to new, similar situations. The training metrics look great because the model is being tested on the same "practice questions" it studied; the real-world performance is poor because new data is like a slightly different, real exam.

Worked example
A simple, concrete step: hold back some data the model never gets to train on (this is the equivalent of a genuinely new practice exam the student hasn't seen), and check performance there instead of only on the training data. If performance on that held-back data is much worse than on the training data, that gap is itself evidence of overfitting, similar to how the student's real exam score being much lower than their practice-exam score would be the tell that memorization, not real understanding, was happening.

Trade-offs and pitfalls
A second simple, concrete step: make the model somewhat "simpler" or less free to memorize fine details, similar to how you might encourage a student to focus on understanding a few core concepts well rather than memorizing every specific example, which usually generalizes better to questions they haven't seen before. The trade-off worth flagging honestly to a non-technical stakeholder is that a small amount of "forgetting" specific training examples is actually the price of better real-world performance, and chasing perfect training performance is often exactly the wrong goal.

Conceptual comparison: L1 (lasso) adds absolute-value penalty driving many weights to zero — it performs implicit feature selection. L2 (ridge) adds squared penalty that shrinks weights continuously but rarely zeroes them, preferring distributed small weights.
In high-dimensional sparse scenario: choose L1 because it yields sparse models, easier interpretation, and reduced variance by selecting only informative features. Often Elastic Net (combining L1 and L2) is practical to handle correlated features.
Impact on feature selection & interpretability:

• L1: produces sparse coefficients, simpler models, easier to explain which features matter. But when features are highly correlated it may arbitrarily pick one.
• L2: retains correlated features with small weights, better numeric stability, but less interpretable since many small nonzero coefficients remain.
  Regularization trade-off: stronger penalty reduces variance but increases bias; choose via cross-validation and consider domain needs for interpretability vs predictive performance.

Linear models: Advantage — fast, interpretable, easy to serve; Limitation — can't capture complex non-linear relationships. Decision trees: Advantage — interpretable and handle mixed feature types; Limitation — single trees can overfit and be unstable; need ensembles for robustness. k-NN: Advantage — simple, no training phase; Limitation — prediction is slow and memory-heavy at inference, problematic at scale. Feedforward neural networks: Advantage — can model complex non-linear patterns; Limitation — higher compute and maintenance cost, harder to interpret and tune in production.

Direct answer
With only 1 percent of examples labeled, a semi-supervised approach is worth trying, since there's likely a large amount of untapped structure in the 99 percent unlabeled data that a purely supervised model trained on the 1 percent alone would never see. But whether it actually helps has to be checked, not assumed, because a poorly-chosen semi-supervised technique can just as easily add confident noise as it can add useful signal.

Structured elaboration

• Why semi-supervised is worth trying here: a model trained on only 1 percent of the data has very little to learn from directly, and is likely to be both high-variance (unstable across different random 1 percent samples) and missing patterns that only show up with more examples. Techniques such as self-training (using the model's own confident predictions on unlabeled data as if they were labels, then retraining on the enlarged set) or consistency-regularization let the model extract additional signal from the unlabeled 99 percent, without needing new human labels, which is exactly the constraint here (getting more labels is expensive).
• What would make you confident it's actually helping: the honest test is a controlled comparison, not just checking whether the semi-supervised model's training-time metrics look good (a self-training approach in particular can look confidently good on the pseudo-labels it generated for itself, which is a weak signal since it's grading its own homework, while actually being wrong). Train a baseline model on only the labeled 1 percent, train the semi-supervised model using the same labeled 1 percent plus the unlabeled 99 percent, and compare both on the same held-out labeled evaluation set that neither model trained on. If the semi-supervised model beats the baseline there, by a margin larger than the run-to-run noise you'd see just from re-training either model with a different random seed, that's real evidence it helped.
• Setting up a fair comparison: use the identical held-out evaluation set for both models, sourced entirely from labeled data that neither model touched during training. Since 1 percent of the data is a small sample, also check that the comparison isn't just noise from that particular small labeled split, for example by repeating the comparison across a few different random 1 percent labeled subsets, or by looking at whether the improvement is consistent rather than a one-off.

Worked example
Suppose the full dataset has 500,000 examples, so 1 percent labeled means roughly 5,000 labeled examples. You'd hold out a separate labeled evaluation set, say 1,000 of those 5,000, and train the supervised baseline on the remaining 4,000. The semi-supervised model would also train on those same 4,000 labeled examples, plus the roughly 495,000 unlabeled ones, using a technique like self-training. Both models get evaluated on the same 1,000 held-out labeled examples, and only if the semi-supervised model's score there is meaningfully and consistently higher than the baseline's would you conclude the extra unlabeled data was actually worth using.

Trade-offs and pitfalls
The main trap is judging "it's helping" from the semi-supervised model's own training process rather than an independent, held-out comparison; a self-training loop, in particular, can report improving performance on its own pseudo-labels while actually drifting away from correct behavior, because it's partly grading its own homework. Getting more labels, if it's feasible even at a slower pace, is also worth weighing against the engineering cost and added complexity of a semi-supervised pipeline; sometimes a modest amount of additional targeted labeling (focused on the examples the current model is least confident about) is a more sample-efficient investment than a more elaborate semi-supervised setup.

High-level: k-NN classifies a query by finding the k closest labeled examples and aggregating their labels (majority vote or weighted). It’s simple, non-parametric, and adapts to data without training a global model.
Practical limitations:

• Latency/throughput: brute-force distance to all points is O(n·d), too slow for large n.
• Memory: storing full dataset can be large.
• Curse of dimensionality: distances become less meaningful as d grows; neighbors may be noisy.
  Engineering mitigations:
• Approximate nearest neighbor (ANN) libraries (FAISS, Annoy, HNSW) reduce query time with sublinear lookup and controlled recall/precision trade-offs.
• Dimensionality reduction (PCA, autoencoders) or feature selection to reduce d.
• Vector quantization / product quantization to reduce memory and speed queries (used in FAISS).
• Sharding and caching hot queries; use GPU-accelerated ANN for high throughput.
  Trade-offs: ANN and compression improve latency/scale but can slightly reduce accuracy; dimensionality reduction may lose discriminative signal, so validate end-to-end.