Data Preparation and Class Imbalance for ML Questions

Direct answer

Storing intermediate preprocessing artifacts trades storage cost and staleness risk for speed and reproducibility; computing on the fly trades a lower storage footprint and always-fresh results for repeated computation cost, and the right default leans toward materializing whatever is expensive to recompute and stable, while computing cheap or highly context-dependent transforms on the fly.

Structured elaboration

When to materialize: an expensive transform (a large embedding computation, a complex multi-table join) that's reused across many downstream consumers or many training runs is worth storing once rather than recomputing repeatedly; when to compute on the fly: a cheap transform, or one whose correct behavior depends heavily on parameters that change per-use (a filter or aggregation window that differs by consumer), is often simpler and safer to compute fresh each time than to maintain many slightly-different stored variants.

Worked example

Precomputed sentence embeddings for a large text corpus (expensive to compute, reused across many downstream models) are a clear case for storing the intermediate artifact; a simple one-hot encoding of a low-cardinality categorical feature (cheap, deterministic, unlikely to benefit meaningfully from caching) is a clear case where computing on the fly, inside the model pipeline itself, is simpler and avoids an extra artifact to manage and keep in sync.

Trade-offs and pitfalls

The staleness risk specifically deserves more attention than it usually gets: a stored artifact computed from an older version of the upstream data or transformation code, silently reused by a LATER training run that assumes it's current, is a subtle bug class that's easy to introduce (nobody remembered to regenerate the artifact after an upstream change) and hard to detect (the pipeline runs successfully, just on stale inputs), which is why any decision to materialize should come with an explicit invalidation or versioning strategy, not just a one-time "compute it and forget it" step.

Direct answer

Data leakage in preprocessing means information from outside the training fold, most commonly from the validation or test set, influences a transform the model then learns from. Fitting a scaler, encoder, or imputer on the full dataset before splitting is the most common way this happens, because the fitted statistics (mean, category frequencies, encoding values) end up quietly informed by data the model should never see until evaluation.

Structured elaboration

Two concrete examples: (1) fitting a StandardScaler on the combined train+test set means the scaled training values are shifted very slightly by the test set's own mean and variance, an effect that's small per-feature but systematically biases every downstream metric optimistic; (2) using a feature like "days until cancellation" to predict churn is target leakage in a starker form, since that feature literally cannot be known at prediction time for a customer who hasn't churned yet, it's derived from the very outcome you're trying to predict.

The structural fix is to make the leakage-safe order the ONLY possible order, not just the intended one:

from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LogisticRegression

preprocess = ColumnTransformer([
    ("num", Pipeline([("impute", SimpleImputer(strategy="median")),
                       ("scale", StandardScaler())]), numeric_cols),
    ("cat", Pipeline([("impute", SimpleImputer(strategy="most_frequent")),
                       ("encode", OneHotEncoder(handle_unknown="ignore"))]), categorical_cols),
])
pipeline = Pipeline([("preprocess", preprocess), ("model", LogisticRegression(max_iter=1000))])
scores = cross_val_score(pipeline, X, y, cv=5, scoring="roc_auc")

Because preprocess lives INSIDE the Pipeline, cross_val_score refits it fresh on each fold's training portion only; computing means, encodings, or imputation values on the whole dataset up front, before this pipeline ever sees a fold boundary, is the mistake this structure makes structurally impossible.

Worked example

If you instead ran preprocess.fit_transform(X) once on the full dataset and only THEN passed the transformed X into cross_val_score, every fold's "held-out" validation rows would have already contributed to the scaler's mean and the encoder's category vocabulary before training even begins, silently violating the fold boundary the cross-validation was supposed to enforce.

Trade-offs and pitfalls

Keeping the SAME imputation logic consistent between training and inference matters as much as getting the fold boundary right: if training imputes missing values with a median computed at fit time but the production serving code recomputes a fresh median from whatever data happens to be available at request time, the model sees systematically different feature values in production than it did during training, a subtler leakage-adjacent bug with the same root cause (statistics computed at the wrong time).

Direct answer

Tomek links, Edited Nearest Neighbors (ENN), and NearMiss all remove majority-class points strategically rather than randomly, targeting points near the class boundary or in ambiguous regions, which sharpens the decision boundary instead of just shrinking the majority class uniformly.

Structured elaboration

• Tomek links: identify pairs of one majority and one minority point that are each other's nearest neighbor (an ambiguous, boundary-straddling pair) and remove the majority point from each such pair, cleaning up the boundary without touching majority points that are clearly, unambiguously majority.
• Edited Nearest Neighbors (ENN): for each majority point, check whether its own k nearest neighbors mostly agree with its label; if a majority point's neighborhood is mostly minority-class, remove it, treating it as noisy or boundary-region.
• NearMiss: several variants that keep majority points selected specifically by their distance to minority points (closest to minority points, in the most common variant), deliberately concentrating the reduced majority set right at the boundary rather than sampling it uniformly.

Typical use cases: Tomek links and ENN are best used as a CLEANING step after (or combined with) oversampling, since they remove ambiguous points that SMOTE-style synthesis can otherwise leave near the boundary; NearMiss is more often used as a primary undersampling strategy on its own, when you want an aggressively boundary-focused reduced dataset.

Effect on class-boundary cleanliness and bias: all three tend to produce a SHARPER, more separable boundary in the retained data, which can genuinely help a classifier learn a cleaner decision rule, but this comes at the cost of systematically REMOVING the hardest, most boundary-adjacent examples, which can make the model overconfident, since it never sees the truly ambiguous cases it will still encounter in production.

Worked example

With a majority point $p$ whose single nearest neighbor overall is a minority point $q$, and $q$'s nearest neighbor is also $p$, that's a Tomek link; removing $p$ eliminates one specific source of boundary ambiguity without touching the thousands of majority points that are clearly, unambiguously far from any minority point.

Trade-offs and pitfalls

Undersampling hurts more than it helps when the boundary region itself carries genuinely important signal you need the model to still handle at inference time (a fraud model that will keep encountering exactly the ambiguous cases these methods remove from training loses its only exposure to them); in that situation, informed undersampling can leave the model brittle on precisely the hardest, most consequential cases, and a validation set drawn from the ORIGINAL (unedited) distribution is essential to catch this before shipping.

Direct answer

Oversampling duplicates or synthesizes more minority examples so the classes are balanced; undersampling drops majority examples instead. Oversampling keeps all your information but risks overfitting to repeated or synthetic minority points; undersampling avoids that risk but throws away real majority-class signal.

Structured elaboration

A rule of thumb: reach for undersampling first when you have so much majority-class data that even after cutting it down to match the minority class you still have plenty to train on; reach for oversampling when the majority class isn't actually that large and dropping examples would hurt.

A hybrid like SMOTE plus Tomek-link cleaning is worth the extra complexity when neither pure approach is working well on its own: SMOTE adds minority signal, and the Tomek-link cleanup step removes the majority-class points that are now ambiguous neighbors of the newly synthesized minority points, sharpening the boundary that plain oversampling alone tends to blur.

Worked example

A dataset with 100,000 majority examples and 500 minority examples: undersampling to a 1:1 ratio drops you to 500 vs 500 (1,000 total, discarding 99,500 majority examples), which is often still plenty of majority signal and trains fast. The same 1:1 ratio via oversampling gives you 100,000 vs 100,000 (200,000 total), where the minority side is either 200x duplication of 500 original points or SMOTE-synthesized interpolations of them, either way a much thinner underlying signal stretched much further.

Trade-offs and pitfalls

Neither is free: undersampling that's too aggressive can genuinely remove informative majority-class structure (rare but real majority-class patterns near the boundary), and oversampling that's too aggressive can make the model overconfident about regions of feature space it has only seen via duplication or interpolation, not real data. The common mistake is picking a ratio (like a rigid 1:1) by convention rather than by validating a few ratios against a held-out set and a business-relevant metric.

Direct answer

Train AUC of 0.98 with validation AUC of 0.75 on a pipeline that SMOTEs before splitting is the textbook leakage signature: synthetic points generated from BOTH what becomes the training set and what becomes the validation set let information about the validation examples leak into training through their nearest neighbors, so the model looks far better than it will ever perform on genuinely unseen data.

Structured elaboration

Debugging checklist, roughly in order of how cheap each check is:

1. Confirm the order of operations. Check the code path directly: does the SMOTE call happen before or after train_test_split? If before, that alone is very likely the whole story.
2. Reproduce with the fix and compare. Move SMOTE inside the training fold only (fit and oversample after splitting, apply nothing to validation/test) and rerun. If validation AUC jumps toward the training number, leakage is confirmed as the dominant cause.
3. Check SMOTE's neighbor computation directly. If it's genuinely computed only within the training set post-fix, verify no validation indices appear among the neighbor indices used to generate synthetic points.
4. Rule out compounding causes. Even after fixing the order, check whether other preprocessing (scaling, encoding) is ALSO fit on the combined data, since leakage bugs frequently travel in pairs within the same pipeline.
5. Look at declining minority recall in production separately. This is a distinct, second-order symptom: even after fixing the leakage, a validation set that itself doesn't represent the true production distribution (for example, still class-balanced from an earlier bad step) will overstate real-world performance, so re-check that validation and production both reflect the true class ratio.

Worked example

If SMOTE runs before the split, a validation example $x_v$ can be selected as a nearest neighbor when generating a synthetic point $x_s$ that ends up in the training set; the model then trains on $x_s$, which is a near-copy of $x_v$'s own feature values, so when the model is evaluated on $x_v$ it is effectively grading itself on a variant of a question it already saw the answer to.

Trade-offs and pitfalls

Short-term remediation is the pipeline reorder above, verified by rerunning and confirming the AUC gap narrows to something explainable by genuine model variance rather than leakage. Long-term remediation is structural: wrap the entire imputation-plus-resampling-plus-model sequence in a single scikit-learn Pipeline (with imblearn.pipeline.Pipeline specifically, since resampling steps must live inside cross-validation folds, unlike a plain sklearn Pipeline) so the ordering bug becomes structurally impossible to reintroduce, and add a CI check that fails if any resampling call appears before a split call in the training code.