Machine Learning Fundamentals Questions

High-level pipeline for a small team:

1. Data ingestion: stream logs to a message bus (Kafka) and store raw events in object storage (S3) with partitioning by date.
2. ETL / Feature engineering: nightly Spark job to parse logs, join user/context tables, compute aggregates, and produce a feature parquet dataset; store feature specs and lineage.
3. Training: use a scheduled job (Airflow) to pull latest features, train a binary classifier (e.g., XGBoost) with cross-validation, log metrics and artifacts to model registry (MLflow).
4. Validation: run holdout evaluation, fairness and calibration checks, and unit tests. Require approval if metrics worsen.
5. Deployment: package model as a REST/gRPC microservice behind feature adapter that computes features online (real-time store + lookups). Use canary rollout and health checks.
6. Monitoring: collect prediction distributions, feature drift, data quality alerts, and business KPIs; log inputs and outputs for debugging.
7. Retraining triggers: automated retrain when (a) data drift exceeds threshold, (b) model performance degrades on recent holdout, or (c) periodic schedule (weekly/monthly). Use CI for retrain with automatic evaluation gating and human review for production promotion.

Ways to compute feature importance:

1. Gain (split importance) — sum of impurity decrease for splits using each feature (fast, built-in). Pros: efficient, reflects tree training objective. Cons: biased toward features with many splits or high cardinality and not causal.
2. Permutation importance — measure drop in metric when permuting a feature in validation set. Pros: model-agnostic, reflects actual impact on predictions. Cons: expensive, affected by feature collinearity (can underestimate importance of correlated features).
3. SHAP values — local additive attributions based on game theory approximations (TreeSHAP is efficient for trees). Pros: consistent, decomposes individual predictions, handles interactions. Cons: computational cost, requires careful aggregation for global importance and can be misinterpreted.
   Which to present:

• Regulatory reporting: use permutation importance with stability intervals and documented methodology (transparent, interpretable global effect) plus simple gain for reference. Include caveats on correlated features.
• Internal debugging: provide SHAP (per-case explanations) combined with gain to quickly identify model behavior and interactions; use permutation tests for confirmatory analysis.
  Always include confidence intervals, correlation-aware analyses (conditional permutation), and clear caveats about causality.

Interpretability: Linear models are generally stronger — coefficients map directly to feature importance and sign. Decision trees are interpretable at small depth but become opaque when large or ensemble.
Feature interactions: Decision trees are stronger — they capture non-linear interactions via splits; linear models need explicit interaction terms.
Robustness to outliers: Decision trees are stronger — splits are based on thresholds and less sensitive to extreme values; linear models (OLS) are sensitive unless regularized or robustified.
Latency at prediction time: Linear models are stronger — dot product is O(#features) with small constant; single tree is similar but ensembles (random forest, gradient boosting) increase latency.
Ease of calibration: Linear models tend to be stronger — outputs (logistic regression) are probabilistic and well-calibrated; tree-based models often require Platt scaling or isotonic regression to calibrate probabilities.
Summary: choose linear models when interpretability, low-latency and calibrated probabilities matter; choose trees when interactions and nonlinearity dominate.

Factors influencing retraining frequency: data volume (how fast new labeled data accumulates), detected distribution or label drift, business cost and latency for retraining, SLA for model freshness, and regulatory requirements.
Hybrid schedule proposal:

• Periodic: monthly lightweight retrain using recent rolling window to capture slow drift.
• Event-driven: trigger full retrain when automated drift detectors exceed thresholds (feature distribution shift, label distribution change, or model performance drop) or when a significant data milestone (e.g., +30% new labeled data) is reached.
  Validation before swap-in:
• Run A/B or shadow evaluation: compare candidate vs production on holdout test and online shadow traffic.
• Pre-deploy checks: unit tests, fairness and slice performance, calibration, latency and memory benchmarks, and canary rollout with monitored KPIs.
• Statistical acceptance: require non-inferior primary metric with p-value/confidence interval and no regressions on critical slices.
  If checks pass, stage rollout with gradual traffic ramp and automated rollback if anomalies appear.

Direct answer
A dataset is split into training, validation, and test sets so that a model can be fit, tuned, and honestly evaluated using three separate pools of data, none of which contaminate each other. The training set is what the model actually learns its parameters from. The validation set is used to make decisions about the model (comparing hyperparameters, comparing candidate models, deciding when to stop training) while it's still being developed. The test set is held back and used exactly once, at the very end, to get an unbiased estimate of how the finished model will perform on new data.

Structured elaboration

• Why not just use the training set to evaluate: a model's performance on the exact data it was fit to is optimistic by construction, especially for flexible models, so it doesn't tell you how the model will do on data it hasn't seen.
• Why the validation set isn't enough by itself: if you use the validation set to make many decisions (try model A, try model B, try ten hyperparameter settings, pick the best one on validation performance), you're implicitly fitting to the validation set too, just at a higher level than parameter fitting. The best-on-validation choice is itself a form of overfitting to that particular validation set, so its own validation score is an optimistic estimate of true future performance.
• Why the test set has to be held back and touched once: it exists specifically to give an estimate that hasn't been used, even indirectly, to make any modeling decision. As soon as you look at test performance and then go back and change something in response, it stops serving that purpose and becomes a second validation set in disguise.
• Typical split ratios: for a large dataset there's plenty of data to go around, so a common starting point is roughly 70 percent training, 15 percent validation, 15 percent test, though the exact split matters less once each piece is large enough to give a stable estimate. For a small dataset, holding out 15 percent for validation and 15 percent for test might leave too little data to train on or to get a stable estimate from either held-out set, so techniques like k-fold cross-validation (using different rotating subsets of the training data as validation across several training runs) are often preferred over a single fixed validation split, to make more efficient use of limited data. Splitting strategy also needs to adapt when data isn't simple independent rows, for example, time-ordered data (where the split should respect time order, not be randomized) or imbalanced or grouped data (where naive random splitting can leak related rows across the split boundary or produce splits with badly skewed class proportions).

Worked example
For a dataset of 100,000 rows using a 70/15/15 split, that's 70,000 rows for training, 15,000 for validation, and 15,000 for test. During development, you might try several model configurations, each trained on the 70,000-row training set and scored on the 15,000-row validation set, and pick whichever configuration scores best there. Only once that choice is final do you run the chosen model on the 15,000-row test set a single time, and that number, not the validation number, is what you report as the honest estimate of how the model will perform on new data.

Trade-offs and pitfalls
The most common mistake is treating the test set as just another validation set and checking it repeatedly during development; even a small amount of this quietly turns the test score into an optimistic estimate again. Another common mistake is splitting the data randomly regardless of its structure; for time-series data especially, a random split will leak future information into training and make validation performance look far better than the model will actually achieve in production, where it only ever has access to the past.