Machine Learning Fundamentals Questions

Concept: Model distillation transfers knowledge from a large teacher to a smaller student by training the student to match teacher outputs (soft targets) and/or intermediate representations, producing a compact model with similar behavior and lower latency.
Example workflow:

1. Train teacher offline (large transformer/CNN) to high accuracy; keep validation/test sets.
2. Generate soft labels: run teacher on large unlabeled or labeled dataset to collect logits/soft probabilities (temperature >1).
3. Train student model to minimize combined loss: cross-entropy with hard labels + KL divergence to teacher soft targets; optionally mimic intermediate features.
4. Validate student on held-out test; iterate hyperparameters (temperature, loss weights).
5. Package student for low-latency serving (smaller architecture, optimized runtime, pruning/quantization), deploy behind existing endpoints.
   Primary caveats:

• Student may not capture rare behaviors or adversarial robustness of teacher; validate on edge cases.
• Distillation can propagate teacher biases—audit fairness and calibration.
• Teacher-student mismatch: if student capacity is too low, distilled performance will be limited.
• Maintenance complexity: need pipelines to regenerate distillation data when teacher updates; monitor drift and retrain student periodically.
• Debugging: errors can be opaque; keep teacher available for fallback comparisons and unit tests.

Direct answer
With only 1 percent of examples labeled, a semi-supervised approach is worth trying, since there's likely a large amount of untapped structure in the 99 percent unlabeled data that a purely supervised model trained on the 1 percent alone would never see. But whether it actually helps has to be checked, not assumed, because a poorly-chosen semi-supervised technique can just as easily add confident noise as it can add useful signal.

Structured elaboration

• Why semi-supervised is worth trying here: a model trained on only 1 percent of the data has very little to learn from directly, and is likely to be both high-variance (unstable across different random 1 percent samples) and missing patterns that only show up with more examples. Techniques such as self-training (using the model's own confident predictions on unlabeled data as if they were labels, then retraining on the enlarged set) or consistency-regularization let the model extract additional signal from the unlabeled 99 percent, without needing new human labels, which is exactly the constraint here (getting more labels is expensive).
• What would make you confident it's actually helping: the honest test is a controlled comparison, not just checking whether the semi-supervised model's training-time metrics look good (a self-training approach in particular can look confidently good on the pseudo-labels it generated for itself, which is a weak signal since it's grading its own homework, while actually being wrong). Train a baseline model on only the labeled 1 percent, train the semi-supervised model using the same labeled 1 percent plus the unlabeled 99 percent, and compare both on the same held-out labeled evaluation set that neither model trained on. If the semi-supervised model beats the baseline there, by a margin larger than the run-to-run noise you'd see just from re-training either model with a different random seed, that's real evidence it helped.
• Setting up a fair comparison: use the identical held-out evaluation set for both models, sourced entirely from labeled data that neither model touched during training. Since 1 percent of the data is a small sample, also check that the comparison isn't just noise from that particular small labeled split, for example by repeating the comparison across a few different random 1 percent labeled subsets, or by looking at whether the improvement is consistent rather than a one-off.

Worked example
Suppose the full dataset has 500,000 examples, so 1 percent labeled means roughly 5,000 labeled examples. You'd hold out a separate labeled evaluation set, say 1,000 of those 5,000, and train the supervised baseline on the remaining 4,000. The semi-supervised model would also train on those same 4,000 labeled examples, plus the roughly 495,000 unlabeled ones, using a technique like self-training. Both models get evaluated on the same 1,000 held-out labeled examples, and only if the semi-supervised model's score there is meaningfully and consistently higher than the baseline's would you conclude the extra unlabeled data was actually worth using.

Trade-offs and pitfalls
The main trap is judging "it's helping" from the semi-supervised model's own training process rather than an independent, held-out comparison; a self-training loop, in particular, can report improving performance on its own pseudo-labels while actually drifting away from correct behavior, because it's partly grading its own homework. Getting more labels, if it's feasible even at a slower pace, is also worth weighing against the engineering cost and added complexity of a semi-supervised pipeline; sometimes a modest amount of additional targeted labeling (focused on the examples the current model is least confident about) is a more sample-efficient investment than a more elaborate semi-supervised setup.

Decision framework:

• Map offline metrics to business goals: precision@10 aligns to immediate click/conversion; coverage/novelty align to long-term engagement and discovery.
• If short-term conversion revenue matters, prefer model A; if retention and catalog health matter, prefer model B or hybrid.
  Offline to online experiments:

1. A/B test with user-level randomization: measure CTR, conversion, session length, retention (7/30-day), and downstream metrics (e.g., LTV).
2. Track diversity/serendipity metrics and content consumption distribution for churn signals.
3. Run interleaving or bucket tests to measure user satisfaction directly (surveys, thumbs up/down).
4. Explore a blended approach: route a fraction of traffic to a hybrid policy that mixes A and B (e.g., top-7 from A, 3 from B) and measure combined KPIs.
   Validation metrics to monitor online: immediate CTR@10, revenue per user, dwell time, repeat visits, retention, and negative signals (unfavorites, skips). Use cohort analysis to see long-term effects; choose the model that optimizes prioritized business metrics rather than a single offline metric.

Linear models: Advantage — fast, interpretable, easy to serve; Limitation — can't capture complex non-linear relationships. Decision trees: Advantage — interpretable and handle mixed feature types; Limitation — single trees can overfit and be unstable; need ensembles for robustness. k-NN: Advantage — simple, no training phase; Limitation — prediction is slow and memory-heavy at inference, problematic at scale. Feedforward neural networks: Advantage — can model complex non-linear patterns; Limitation — higher compute and maintenance cost, harder to interpret and tune in production.

Direct answer
Bias is the error a model makes because it's too simple to capture the true pattern in the data; variance is the error it makes because it's too sensitive to the specific noise in the training set it happened to see. There's a tradeoff because, as you increase a model's complexity, bias tends to go down (it can represent more complicated patterns) while variance tends to go up (it has more freedom to fit noise), and a model's total expected error is driven by both.

Structured elaboration
One way to make this concrete: the expected squared prediction error of a model decomposes into three pieces.

$\text{Expected error} = \text{Bias}^2 + \text{Variance} + \text{Irreducible noise}$

• Bias is how far off the model's average prediction is from the true value, if you imagine retraining it on many different samples of training data. A high-bias model (like a straight line fit to a curved relationship) systematically misses the pattern regardless of which specific training set it sees; that's underfitting.
• Variance is how much the model's prediction would change if you retrained it on a different sample of training data. A high-variance model fits the particular quirks and noise of the training set it happened to get, so it looks great on that data but performs inconsistently, and usually worse, on new data; that's overfitting.
• Irreducible noise is randomness inherent in the problem itself that no model, however good, can predict away.
• What shifts the balance: increasing model complexity (more parameters, deeper trees, higher-degree polynomials) generally trades bias for variance. Increasing the amount of training data mainly reduces variance (a complex model has less room to fit noise when there's more real signal to constrain it), without directly changing bias. Increasing label noise makes variance more costly, because a flexible model has more noise to latch onto.

Worked example
Imagine fitting polynomials of increasing degree to noisy data generated from a true quadratic relationship. A degree-1 (straight-line) fit has high bias: no matter how much data you give it or which sample you train on, it can't capture the curve, so it consistently underfits both the training data and new data in a similar, systematic way. A degree-15 fit has low bias (it's flexible enough to pass close to almost any set of points) but high variance: fit it to one noisy sample and it wiggles through the noise one way, fit it to a different noisy sample drawn from the same true relationship and it wiggles a different way. Its training error looks excellent, but its error on new data is inconsistent and, on average, worse than a well-chosen degree-2 fit, which matches the true relationship's complexity and rides squarely between the two extremes. If you plotted training error and validation error against polynomial degree, training error would keep falling as degree increases, while validation error would fall, bottom out near degree 2, and then rise again as the higher-degree fits start memorizing noise; that divergence between the two curves is the signature of moving from the underfitting regime into the overfitting regime.

Trade-offs and pitfalls
The most common mistake is treating "more complex model" as an unambiguous improvement; it only reduces bias, and can make things worse overall if variance grows faster than bias shrinks. Another common confusion is treating bias and variance as literally opposite ends of a single knob you can read off directly from a training curve; they're both properties of the model's error, and you generally infer the regime you're in indirectly, from the gap between training and validation performance, rather than measuring bias and variance directly.