Machine Learning Fundamentals Questions

Early stopping: monitor validation metric during training and stop when it stops improving to prevent overfitting. It acts as implicit regularization by limiting training time, preventing the model from fitting noise.
Implementation with noisy validation:

• Use a patience parameter: require N consecutive non-improving evaluations before stopping.
• Smooth metrics using rolling average or exponential moving average to reduce noise-driven stops.
• Use relative improvement threshold (delta) so tiny fluctuations don’t count as improvement.
• Check multiple metrics (validation loss + primary metric) and stop only when both plateau.
• For production training jobs: checkpoint the best model and continue training but keep periodic evaluation on larger/cleaner holdout to confirm improvements.
• Optionally employ ensemble of top-K checkpoints instead of single best.
  Example config: patience=10 evals, delta=0.001, EMA smoothing with alpha=0.2, checkpoint best model. This prevents premature stopping while still constraining overfitting.

Framework: create reproducible splits, metric choices, and anti-leakage steps.
Data splits:

• Shuffle dataset, create train (60%), validation (20%), test (20%) with stratification on label.
• Keep test isolated and only used once for final comparison.
  Training & tuning:
• Use train to fit each model; use validation for hyperparameter tuning (grid/regularization depth for tree, C for LR).
  Metrics:
• Primary: ROC-AUC (ranking) and F1 at operational threshold; also precision/recall and calibration.
  Fairness/robustness:
• Compute metrics per important slices.
  Avoid leakage:
• Perform preprocessing pipeline (imputation, scaling, encoding) fit only on train and applied to val/test.
• Ensure temporal or group-wise splits if data has time or user groups.
  Experiment steps:

1. Fix random seeds and cross-validation folds.
2. Tune each model identically (nested CV or same validation folds).
3. Compare on test set with statistical test (e.g., paired bootstrap) and report confidence intervals.
   This ensures a fair, reproducible comparison without leakage.

Definition: Bias is error from incorrect assumptions underfitting the true pattern; variance is error from sensitivity to training data overfitting.
Concrete examples:

• High-bias model: linear regression on nonlinear data (e.g., quadratic relationship). Remedy: increase model capacity — add polynomial features or switch to a nonlinear model (tree-based or neural net). In production, implement feature engineering or deploy a higher-capacity model and validate with cross-validation and holdout tests.
• High-variance model: deep neural network with small dataset causing overfitting. Remedy: regularization — L2/dropout, data augmentation, early stopping, or collect more labeled data. In production, add regularization hyperparameters, implement robust validation (k-fold), and use monitoring for generalization drift.
  Practical pipeline steps:
• For bias: add features, choose richer model family, monitor train vs validation error gap to confirm reduced bias.
• For variance: apply regularization, early stopping, ensembling, and expand dataset; track validation stability and reduce model complexity until variance is controlled.

Regression predicts continuous values; classification assigns discrete categories. Framing mistakes cause operational issues: 1) Temperature alarm wrongly as classification: converting continuous temperature to 'normal/warning' bins loses nuance—small but important trends get dropped, causing missed gradual failures or noisy alerts. 2) Loan default probability treated as classification (yes/no) when decisions need risk scores: using binary outputs prevents fine-grained risk-based pricing and regulatory reporting, leading to poor user offers and revenue loss. Correct framing preserves information needed for business decisions and downstream thresholds.

Ways to compute feature importance:

1. Gain (split importance) — sum of impurity decrease for splits using each feature (fast, built-in). Pros: efficient, reflects tree training objective. Cons: biased toward features with many splits or high cardinality and not causal.
2. Permutation importance — measure drop in metric when permuting a feature in validation set. Pros: model-agnostic, reflects actual impact on predictions. Cons: expensive, affected by feature collinearity (can underestimate importance of correlated features).
3. SHAP values — local additive attributions based on game theory approximations (TreeSHAP is efficient for trees). Pros: consistent, decomposes individual predictions, handles interactions. Cons: computational cost, requires careful aggregation for global importance and can be misinterpreted.
   Which to present:

• Regulatory reporting: use permutation importance with stability intervals and documented methodology (transparent, interpretable global effect) plus simple gain for reference. Include caveats on correlated features.
• Internal debugging: provide SHAP (per-case explanations) combined with gain to quickly identify model behavior and interactions; use permutation tests for confirmatory analysis.
  Always include confidence intervals, correlation-aware analyses (conditional permutation), and clear caveats about causality.