Machine Learning Fundamentals Questions

Direct answer
A parametric model has a fixed number of parameters decided in advance, regardless of how much training data you give it; a non-parametric model's effective complexity can grow as you feed it more data, because it doesn't commit to a fixed functional form ahead of time. Linear regression is parametric (a fixed set of coefficients, one per feature, no matter how many training rows you have); k-nearest neighbors and decision trees are non-parametric (their effective structure, the tree's shape or the set of neighbors considered, can grow more complex as more data arrives).

Structured elaboration

• Capacity: a parametric model's capacity, the range of functions it can represent, is capped by its fixed parameter count; a straight line fit with 2 coefficients can only ever represent straight lines, however much data you give it. A non-parametric model's capacity effectively grows with the data; a decision tree can keep splitting into more, finer-grained regions as more examples justify it, and a k-nearest-neighbors model implicitly keeps every training point around, so its complexity scales directly with dataset size.
• Sample efficiency: because a parametric model makes a strong assumption about the shape of the relationship, it can often learn a reasonable fit from relatively few examples, provided that assumption is roughly correct. A non-parametric model makes fewer upfront assumptions, so it typically needs more data before its estimates stabilize, since it's effectively learning the shape of the relationship from the data itself rather than assuming it.
• How complexity scales with data: for a parametric model, adding more data mainly makes the same fixed number of parameter estimates more precise; it doesn't let the model represent fundamentally more complex relationships. For a non-parametric model, adding more data can genuinely let it represent more complex relationships (a tree can grow deeper and more nuanced, k-nearest-neighbors effectively gets a finer-grained map of the space), which is powerful but also means non-parametric models are generally more prone to overfitting if not otherwise constrained (for example, by limiting tree depth or choosing a larger k).

Worked example
If the true relationship between a feature and the target is a straight line plus some noise, a parametric linear regression will do well with a modest amount of data, since its built-in assumption matches reality; giving it far more data mostly just tightens its coefficient estimates rather than changing what it can represent. If instead the true relationship is a complicated, wiggly curve that no simple formula captures cleanly, a small decision tree or a k-nearest-neighbors model with a small amount of data might approximate it crudely, but given much more data, either could approximate that wiggly shape increasingly closely, something the fixed-form linear regression structurally cannot do regardless of how much data it sees.

Trade-offs and pitfalls
The common mistake is treating "non-parametric" as meaning "model with no parameters at all"; it actually means the number of effective parameters isn't fixed in advance and can grow with the data, not that there are literally none. Another pitfall is assuming non-parametric models are always the safer default because they make fewer assumptions; with limited data, that flexibility often costs more in variance (a higher risk of overfitting) than it gains in reduced bias, so the parametric-versus-non-parametric choice still has to be weighed against how much data is actually available.

1. Data distribution drift: input features changed over time (new user behavior). Detection: monitor per-feature drift (KS/JS) and alert. Mitigation: retrain with recent data or add covariate shift correction.
2. Label distribution / concept drift: relationship between features and labels shifts. Detection: monitor model performance on labeled stream and performance by cohort. Mitigation: frequent labeling, online learning or periodic retrain with recent labels.
3. Upstream instrumentation bugs: missing events or incorrect parsing. Detection: sudden drops in event counts or schema validity errors. Mitigation: circuit-breaker to fallback, fix ETL, and replay logs.
4. Data leakage or leakage removal effects: training contained signals no longer present in production. Detection: feature importance changes and abnormal online errors. Mitigation: audit features, remove leaky features, retrain.
5. Model degradation due to stale model or concept complexity: model capacity insufficient for new patterns. Detection: rising error despite stable data and increased residuals. Mitigation: increase model capacity, change architecture, or introduce ensemble and A/B test.
6. Serving issues (numeric precision, feature normalization mismatch): Detection: compare online vs offline predictions on same inputs. Mitigation: add predictive checks and reproducible inference pipeline tests.

1. Semi-supervised learning (consistency regularization / self-training): train model on labeled data, generate pseudo-labels for confident unlabeled examples, and iteratively retrain. Trade-offs: low labeling cost, moderate complexity, good gains if model confidence is reliable; risk of confirmation bias.
2. Active learning: select most informative unlabeled examples (uncertainty or query-by-committee) to label with human annotators. Trade-offs: higher labeling cost per example but maximizes information gain; complexity in orchestration and requires labeling pipeline; best when labeling budget limited.
3. Transfer learning / pretraining: pretrain on related large datasets or self-supervised objectives, then fine-tune on limited labels. Trade-offs: low labeling cost, moderate complexity, often large gains especially in vision/NLP; requires relevant pretraining data and compute.
   Combine approaches (pretrain + active learning + self-training) for complementary benefits while managing labeling budget and system complexity.

Training/validation/test splits: training fits model, validation tunes hyperparameters, test estimates final generalization. For 100k examples: common split 70/15/15 (70k train, 15k val, 15k test). If time-series: use chronological splits (e.g., first 80% train, next 10% val, final 10% test) to avoid leakage and mimic production. For class imbalance: ensure stratified splits so class proportions are preserved across sets; if minority class is tiny, consider oversampling/SMOTE on training only or use larger validation/test sets for reliable estimates. Also use cross-validation or repeated stratified folds when data is limited.

Direct answer
Classification and regression are both supervised learning tasks; they differ in what kind of thing the model predicts. Classification predicts a category from a fixed set of possible labels (binary, like spam or not-spam, or multi-class, like which of five product categories an item belongs to). Regression predicts a continuous numeric quantity, like a price or a duration, where the space of possible answers is effectively infinite and ordered.

Structured elaboration

• Target variable: classification targets are discrete categories with no inherent numeric ordering (unless you deliberately treat an ordinal target that way); regression targets are continuous numbers where the distance between two predictions is meaningful (predicting $105 when the truth is $100 is a smaller error than predicting $200).
• Preparing the target: for classification, you typically map each category to an integer class index (and to a one-hot or similar representation if the model needs it), and for a binary target that's just 0 or 1. For regression, the target is usually left as its natural numeric scale, though you might apply a transform (like a log transform for a right-skewed target such as price) to make the error distribution better behaved for the model you're using.
• Typical algorithm families: many algorithm families come in both a classification and a regression flavor, because they share the same underlying model structure and just swap the loss function and output layer (linear/logistic regression, decision trees and tree ensembles, k-nearest neighbors, neural networks). The choice of family is mostly independent of classification-versus-regression; it's driven by other factors like data size, interpretability needs, and feature types.
• Evaluation goals differ substantially: classification is usually evaluated on how often the predicted category is right, and on the specific costs of different kinds of mistakes (a false negative and a false positive are rarely equally costly). Regression is evaluated on how far off the numeric predictions are, using an error measure appropriate to how you want to penalize large misses versus small ones.

Worked example
Predicting a customer's lifetime value (a dollar amount) is a natural regression problem: the answer is a continuous quantity, and being off by $50 versus $500 clearly matters differently. But the same business need is sometimes reframed as classification: "will this customer be high-value (above some threshold) or not." That's a legitimate choice when the downstream decision is itself binary (for example, whether to enroll the customer in a premium retention program), because then you only ever act on the threshold anyway. The trade-off is information loss: collapsing a continuous quantity into a binary label throws away the magnitude, and the threshold itself becomes an extra design decision (a customer just below the cutoff is treated identically to one far below it, even though they're not equally "not valuable"). For a business decision that's genuinely a yes/no call, I'd choose the classification framing for simplicity and direct interpretability; if the magnitude itself will be used downstream (for example, to size a marketing budget per customer), I'd keep it as regression and threshold later if needed, rather than throwing away the magnitude up front.

Trade-offs and pitfalls
A common mistake is picking the framing based on which one is easier to model rather than what the business decision actually needs; regression-to-classification-by-thresholding is a one-way door; you can always derive a classification decision from a regression prediction after the fact, but you can't recover the discarded magnitude if you trained a classifier directly. Another pitfall is forgetting that the choice of evaluation metric has to match the framing: reporting classification accuracy on what was really a coarsened regression problem can hide the fact that a model is only good near the decision boundary and badly miscalibrated elsewhere.