Machine Learning Fundamentals Questions

Definition: Calibration means predicted probabilities reflect true outcome frequencies (e.g., among examples predicted 0.8, ~80% are positive). Miscalibration causes: class imbalance, overfitting, regularization, surrogate losses, model complexity, and dataset shift.
Two post-processing techniques:

1. Platt scaling: fit a sigmoid (logistic regression) on model scores using a held-out calibration set; good when relationship between score and probability is monotonic.
2. Isotonic regression: fit a non-parametric monotonic mapping on calibration set; more flexible, better for complex miscalibration but risks overfitting with small calibration sets.
   Evaluation method:

• Reliability diagram (calibration curve) and Brier score: plot predicted probability bins vs observed frequency; Brier score summarizes squared error of probabilities. Use calibration error metrics (ECE/MCE) and bootstrap for confidence intervals.

Direct answer
A parametric model has a fixed number of parameters decided in advance, regardless of how much training data you give it; a non-parametric model's effective complexity can grow as you feed it more data, because it doesn't commit to a fixed functional form ahead of time. Linear regression is parametric (a fixed set of coefficients, one per feature, no matter how many training rows you have); k-nearest neighbors and decision trees are non-parametric (their effective structure, the tree's shape or the set of neighbors considered, can grow more complex as more data arrives).

Structured elaboration

• Capacity: a parametric model's capacity, the range of functions it can represent, is capped by its fixed parameter count; a straight line fit with 2 coefficients can only ever represent straight lines, however much data you give it. A non-parametric model's capacity effectively grows with the data; a decision tree can keep splitting into more, finer-grained regions as more examples justify it, and a k-nearest-neighbors model implicitly keeps every training point around, so its complexity scales directly with dataset size.
• Sample efficiency: because a parametric model makes a strong assumption about the shape of the relationship, it can often learn a reasonable fit from relatively few examples, provided that assumption is roughly correct. A non-parametric model makes fewer upfront assumptions, so it typically needs more data before its estimates stabilize, since it's effectively learning the shape of the relationship from the data itself rather than assuming it.
• How complexity scales with data: for a parametric model, adding more data mainly makes the same fixed number of parameter estimates more precise; it doesn't let the model represent fundamentally more complex relationships. For a non-parametric model, adding more data can genuinely let it represent more complex relationships (a tree can grow deeper and more nuanced, k-nearest-neighbors effectively gets a finer-grained map of the space), which is powerful but also means non-parametric models are generally more prone to overfitting if not otherwise constrained (for example, by limiting tree depth or choosing a larger k).

Worked example
If the true relationship between a feature and the target is a straight line plus some noise, a parametric linear regression will do well with a modest amount of data, since its built-in assumption matches reality; giving it far more data mostly just tightens its coefficient estimates rather than changing what it can represent. If instead the true relationship is a complicated, wiggly curve that no simple formula captures cleanly, a small decision tree or a k-nearest-neighbors model with a small amount of data might approximate it crudely, but given much more data, either could approximate that wiggly shape increasingly closely, something the fixed-form linear regression structurally cannot do regardless of how much data it sees.

Trade-offs and pitfalls
The common mistake is treating "non-parametric" as meaning "model with no parameters at all"; it actually means the number of effective parameters isn't fixed in advance and can grow with the data, not that there are literally none. Another pitfall is assuming non-parametric models are always the safer default because they make fewer assumptions; with limited data, that flexibility often costs more in variance (a higher risk of overfitting) than it gains in reduced bias, so the parametric-versus-non-parametric choice still has to be weighed against how much data is actually available.

Training/validation/test splits: training fits model, validation tunes hyperparameters, test estimates final generalization. For 100k examples: common split 70/15/15 (70k train, 15k val, 15k test). If time-series: use chronological splits (e.g., first 80% train, next 10% val, final 10% test) to avoid leakage and mimic production. For class imbalance: ensure stratified splits so class proportions are preserved across sets; if minority class is tiny, consider oversampling/SMOTE on training only or use larger validation/test sets for reliable estimates. Also use cross-validation or repeated stratified folds when data is limited.

Trade-offs: More complex models (deep nets, ensembles) usually give higher accuracy but lower interpretability; simpler models (logistic regression, decision trees) are explainable but may underperform.
Regulated-domain architecture balancing performance and interpretability:

• Two-stage system:
  1. Interpretable core model (logistic regression / monotonic GBM like LightGBM with monotonic constraints) used for final decisions and compliance reporting. Features are engineered, documented, and limited to auditable inputs.
  2. Performance-boosting black-box model (neural net or ensemble) runs in parallel as an advisory scorer.
• Decision policy: use the interpretable model for automated approvals/denials; use black-box output to flag cases for manual review or for scoring buckets where higher accuracy is allowed with human oversight.
• Explainability tools: SHAP/Integrated Gradients for local explanations, and global feature importances. Keep feature transformations invertible and logged for audit.
• Governance: model cards, input validation, bias/fairness checks, and a human-in-the-loop for edge/override cases. Maintain immutable logs of predictions and explanations for compliance.
• Operational choices: restrict features to legally allowed inputs, use monotonic constraints on sensitive features, and version control models/pipelines. Periodically retrain and re-evaluate fairness and calibration.
  This balances predictive quality (via advisory model) while ensuring the production decision is explainable, auditable, and compliant.

Use STAR-like minimal practices list with one implementation step each:

1. Fairness: practice — group-level metric monitoring. Implementation: compute and alert on parity metrics (e.g., equalized odds, demographic parity) per sensitive group each deployment; block deploy if gap > threshold.
2. Privacy: practice — data minimization and access control. Implementation: enforce role-based access, store only required features, and apply field-level encryption; log and audit accesses.
3. Explainability: practice — model explanations for decisions. Implementation: integrate SHAP or counterfactuals into prediction logs and provide human-readable reasoning for flagged cases.
4. Robustness & testing: practice — pre-deployment tests for bias and performance across slices. Implementation: add CI tests that run fairness, distribution-shift, and residual analyses on holdout slices.
5. Governance: practice — decision record and signoff. Implementation: require a responsible-AI checklist and stakeholder signoff (legal/ethics/product) before production rollout.
   Each practice ties to operational controls (metrics, automation, audits) so engineering teams can embed checks into pipelines.