Machine Learning Fundamentals Questions

1. Data distribution drift: input features changed over time (new user behavior). Detection: monitor per-feature drift (KS/JS) and alert. Mitigation: retrain with recent data or add covariate shift correction.
2. Label distribution / concept drift: relationship between features and labels shifts. Detection: monitor model performance on labeled stream and performance by cohort. Mitigation: frequent labeling, online learning or periodic retrain with recent labels.
3. Upstream instrumentation bugs: missing events or incorrect parsing. Detection: sudden drops in event counts or schema validity errors. Mitigation: circuit-breaker to fallback, fix ETL, and replay logs.
4. Data leakage or leakage removal effects: training contained signals no longer present in production. Detection: feature importance changes and abnormal online errors. Mitigation: audit features, remove leaky features, retrain.
5. Model degradation due to stale model or concept complexity: model capacity insufficient for new patterns. Detection: rising error despite stable data and increased residuals. Mitigation: increase model capacity, change architecture, or introduce ensemble and A/B test.
6. Serving issues (numeric precision, feature normalization mismatch): Detection: compare online vs offline predictions on same inputs. Mitigation: add predictive checks and reproducible inference pipeline tests.

Regression losses: 1) Mean Squared Error (MSE) — penalizes larger errors heavily, good when large errors are especially bad. 2) Mean Absolute Error (MAE) — robust to outliers, measures median-like error. 3) Huber Loss — combines MSE and MAE: quadratic near zero errors, linear for large errors to reduce outlier impact.
Classification losses: 1) Binary Cross-Entropy (log loss) — standard for probabilistic binary classification. 2) Categorical Cross-Entropy — for multi-class softmax outputs. 3) Hinge Loss — used for SVMs, focuses on maximizing margin between classes.

Regression predicts continuous values; classification assigns discrete categories. Framing mistakes cause operational issues: 1) Temperature alarm wrongly as classification: converting continuous temperature to 'normal/warning' bins loses nuance—small but important trends get dropped, causing missed gradual failures or noisy alerts. 2) Loan default probability treated as classification (yes/no) when decisions need risk scores: using binary outputs prevents fine-grained risk-based pricing and regulatory reporting, leading to poor user offers and revenue loss. Correct framing preserves information needed for business decisions and downstream thresholds.

Linear models: Advantage — fast, interpretable, easy to serve; Limitation — can't capture complex non-linear relationships. Decision trees: Advantage — interpretable and handle mixed feature types; Limitation — single trees can overfit and be unstable; need ensembles for robustness. k-NN: Advantage — simple, no training phase; Limitation — prediction is slow and memory-heavy at inference, problematic at scale. Feedforward neural networks: Advantage — can model complex non-linear patterns; Limitation — higher compute and maintenance cost, harder to interpret and tune in production.

Definition: Bias is error from incorrect assumptions underfitting the true pattern; variance is error from sensitivity to training data overfitting.
Concrete examples:

• High-bias model: linear regression on nonlinear data (e.g., quadratic relationship). Remedy: increase model capacity — add polynomial features or switch to a nonlinear model (tree-based or neural net). In production, implement feature engineering or deploy a higher-capacity model and validate with cross-validation and holdout tests.
• High-variance model: deep neural network with small dataset causing overfitting. Remedy: regularization — L2/dropout, data augmentation, early stopping, or collect more labeled data. In production, add regularization hyperparameters, implement robust validation (k-fold), and use monitoring for generalization drift.
  Practical pipeline steps:
• For bias: add features, choose richer model family, monitor train vs validation error gap to confirm reduced bias.
• For variance: apply regularization, early stopping, ensembling, and expand dataset; track validation stability and reduce model complexity until variance is controlled.