Machine Learning Fundamentals Questions

Concept: Model distillation transfers knowledge from a large teacher to a smaller student by training the student to match teacher outputs (soft targets) and/or intermediate representations, producing a compact model with similar behavior and lower latency.
Example workflow:

1. Train teacher offline (large transformer/CNN) to high accuracy; keep validation/test sets.
2. Generate soft labels: run teacher on large unlabeled or labeled dataset to collect logits/soft probabilities (temperature >1).
3. Train student model to minimize combined loss: cross-entropy with hard labels + KL divergence to teacher soft targets; optionally mimic intermediate features.
4. Validate student on held-out test; iterate hyperparameters (temperature, loss weights).
5. Package student for low-latency serving (smaller architecture, optimized runtime, pruning/quantization), deploy behind existing endpoints.
   Primary caveats:

• Student may not capture rare behaviors or adversarial robustness of teacher; validate on edge cases.
• Distillation can propagate teacher biases—audit fairness and calibration.
• Teacher-student mismatch: if student capacity is too low, distilled performance will be limited.
• Maintenance complexity: need pipelines to regenerate distillation data when teacher updates; monitor drift and retrain student periodically.
• Debugging: errors can be opaque; keep teacher available for fallback comparisons and unit tests.

Regression predicts continuous values; classification assigns discrete categories. Framing mistakes cause operational issues: 1) Temperature alarm wrongly as classification: converting continuous temperature to 'normal/warning' bins loses nuance—small but important trends get dropped, causing missed gradual failures or noisy alerts. 2) Loan default probability treated as classification (yes/no) when decisions need risk scores: using binary outputs prevents fine-grained risk-based pricing and regulatory reporting, leading to poor user offers and revenue loss. Correct framing preserves information needed for business decisions and downstream thresholds.

Supervised learning: models trained on labeled input→output pairs to predict outputs for new inputs. Example (classification): Email spam filter using labeled emails (spam/ham) with a logistic regression or gradient-boosted trees; supervised learning is appropriate because ground-truth labels exist and goal is to predict discrete classes. Example (regression): Predicting house prices from features (size, location) with a random forest or linear regression; labels are numeric sale prices, so supervised regression fits.

Unsupervised learning: models discover structure in unlabeled data. Example (clustering): Customer segmentation via k-means on purchase behavior to create marketing cohorts—no labels, goal is grouping. Example (dimensionality reduction): PCA to compress high-dimensional sensor data before feeding to downstream models—reduces noise and storage while preserving variance. Each paradigm aligns with whether labels exist and the desired outcome (predict vs discover).

Use STAR-like minimal practices list with one implementation step each:

1. Fairness: practice — group-level metric monitoring. Implementation: compute and alert on parity metrics (e.g., equalized odds, demographic parity) per sensitive group each deployment; block deploy if gap > threshold.
2. Privacy: practice — data minimization and access control. Implementation: enforce role-based access, store only required features, and apply field-level encryption; log and audit accesses.
3. Explainability: practice — model explanations for decisions. Implementation: integrate SHAP or counterfactuals into prediction logs and provide human-readable reasoning for flagged cases.
4. Robustness & testing: practice — pre-deployment tests for bias and performance across slices. Implementation: add CI tests that run fairness, distribution-shift, and residual analyses on holdout slices.
5. Governance: practice — decision record and signoff. Implementation: require a responsible-AI checklist and stakeholder signoff (legal/ethics/product) before production rollout.
   Each practice ties to operational controls (metrics, automation, audits) so engineering teams can embed checks into pipelines.

Direct answer
Overfitting is when a model has essentially memorized quirks of its specific training data rather than learning the underlying pattern, so it performs well on training data but noticeably worse on new data. Underfitting is the opposite: the model is too simple or undertrained to capture the real pattern, so it performs poorly on both training and new data.

Structured elaboration

• Diagnostic sign 1, the training-validation gap: overfitting shows up as a large and often growing gap between training performance (very good) and validation performance (noticeably worse). Underfitting shows up as both training and validation performance being poor and close together, because the model hasn't captured enough signal to do well anywhere.
• Diagnostic sign 2, learning curves over training time: plotting training and validation loss against training epoch or iteration, overfitting shows training loss continuing to fall while validation loss flattens out and then starts rising. Underfitting shows both curves plateauing early at a high loss, with more training not helping much.
• Diagnostic sign 3, learning curves over data size: plotting performance against the amount of training data used, a high-variance (overfitting) model shows a persistent, slowly-closing gap between training and validation performance as data grows. A high-bias (underfitting) model shows training and validation performance converging quickly to the same, disappointingly-low level, meaning more data alone won't fix it.
• General remedies for overfitting: reduce model complexity, add regularization, gather more training data, or use techniques like cross-validation to select a less complex model in the first place. The trade-off is that most of these remedies also risk pushing the model toward underfitting if applied too aggressively, so they're usually tuned rather than applied at maximum strength.
• General remedies for underfitting: increase model complexity or capacity, add more informative features, reduce excessive regularization, or train for longer if the model hasn't converged. The trade-off here is the mirror image: push too far and you're back to overfitting, and added complexity often costs more compute and makes the model harder to interpret.

Worked example
Suppose you're building a classifier on tabular customer data and see training accuracy of 98 percent but validation accuracy of 72 percent. That large gap, with training performance far outpacing validation, is the classic overfitting signature: the model has likely fit noise or overly specific patterns present only in the training rows. If instead you saw training accuracy of 74 percent and validation accuracy of 71 percent, both mediocre and close together, that points to underfitting: the model hasn't found enough signal even in the data it was trained on, so the fix is more likely to be a richer feature set or a more expressive model rather than more regularization.

Trade-offs and pitfalls
A common mistake is applying an overfitting remedy (like adding heavy regularization) to a model that's actually underfitting, which makes the underfitting worse rather than helping; the diagnostic step (check whether training performance itself is also poor) has to come before picking a remedy. Another pitfall is treating a single evaluation as diagnostic; a genuinely useful read on which regime you're in generally needs the shape of a curve, either over training time or over training-set size, not just one number.