Machine Learning Fundamentals Questions

Interpretability: Linear models are generally stronger — coefficients map directly to feature importance and sign. Decision trees are interpretable at small depth but become opaque when large or ensemble.
Feature interactions: Decision trees are stronger — they capture non-linear interactions via splits; linear models need explicit interaction terms.
Robustness to outliers: Decision trees are stronger — splits are based on thresholds and less sensitive to extreme values; linear models (OLS) are sensitive unless regularized or robustified.
Latency at prediction time: Linear models are stronger — dot product is O(#features) with small constant; single tree is similar but ensembles (random forest, gradient boosting) increase latency.
Ease of calibration: Linear models tend to be stronger — outputs (logistic regression) are probabilistic and well-calibrated; tree-based models often require Platt scaling or isotonic regression to calibrate probabilities.
Summary: choose linear models when interpretability, low-latency and calibrated probabilities matter; choose trees when interactions and nonlinearity dominate.

1. Semi-supervised learning (consistency regularization / self-training): train model on labeled data, generate pseudo-labels for confident unlabeled examples, and iteratively retrain. Trade-offs: low labeling cost, moderate complexity, good gains if model confidence is reliable; risk of confirmation bias.
2. Active learning: select most informative unlabeled examples (uncertainty or query-by-committee) to label with human annotators. Trade-offs: higher labeling cost per example but maximizes information gain; complexity in orchestration and requires labeling pipeline; best when labeling budget limited.
3. Transfer learning / pretraining: pretrain on related large datasets or self-supervised objectives, then fine-tune on limited labels. Trade-offs: low labeling cost, moderate complexity, often large gains especially in vision/NLP; requires relevant pretraining data and compute.
   Combine approaches (pretrain + active learning + self-training) for complementary benefits while managing labeling budget and system complexity.

Accuracy is misleading with 99.9% because a naive model predicting 'not fraud' always would be 99.9% accurate while missing all frauds. Better metrics: 1) Precision — fraction of flagged transactions that are truly fraud; measures correctness of positive predictions and helps limit false alarms. 2) Recall (true positive rate) — fraction of actual frauds detected; measures coverage of fraud detection, critical for risk reduction. 3) Precision-Recall AUC or F1-score — balances precision and recall (F1 is harmonic mean), useful when both false positives and false negatives matter. Additionally, use ROC-AUC for overall separability but PR-AUC is preferred under heavy class imbalance.

Trade-offs: More complex models (deep nets, ensembles) usually give higher accuracy but lower interpretability; simpler models (logistic regression, decision trees) are explainable but may underperform.
Regulated-domain architecture balancing performance and interpretability:

• Two-stage system:
  1. Interpretable core model (logistic regression / monotonic GBM like LightGBM with monotonic constraints) used for final decisions and compliance reporting. Features are engineered, documented, and limited to auditable inputs.
  2. Performance-boosting black-box model (neural net or ensemble) runs in parallel as an advisory scorer.
• Decision policy: use the interpretable model for automated approvals/denials; use black-box output to flag cases for manual review or for scoring buckets where higher accuracy is allowed with human oversight.
• Explainability tools: SHAP/Integrated Gradients for local explanations, and global feature importances. Keep feature transformations invertible and logged for audit.
• Governance: model cards, input validation, bias/fairness checks, and a human-in-the-loop for edge/override cases. Maintain immutable logs of predictions and explanations for compliance.
• Operational choices: restrict features to legally allowed inputs, use monotonic constraints on sensitive features, and version control models/pipelines. Periodically retrain and re-evaluate fairness and calibration.
  This balances predictive quality (via advisory model) while ensuring the production decision is explainable, auditable, and compliant.

Definition: Bias is error from incorrect assumptions underfitting the true pattern; variance is error from sensitivity to training data overfitting.
Concrete examples:

• High-bias model: linear regression on nonlinear data (e.g., quadratic relationship). Remedy: increase model capacity — add polynomial features or switch to a nonlinear model (tree-based or neural net). In production, implement feature engineering or deploy a higher-capacity model and validate with cross-validation and holdout tests.
• High-variance model: deep neural network with small dataset causing overfitting. Remedy: regularization — L2/dropout, data augmentation, early stopping, or collect more labeled data. In production, add regularization hyperparameters, implement robust validation (k-fold), and use monitoring for generalization drift.
  Practical pipeline steps:
• For bias: add features, choose richer model family, monitor train vs validation error gap to confirm reduced bias.
• For variance: apply regularization, early stopping, ensembling, and expand dataset; track validation stability and reduce model complexity until variance is controlled.