Data Preparation and Class Imbalance for ML Questions

Direct answer

Walk it as a sequence of concrete decisions, not a checklist recited abstractly: how the data was sourced and why, how labeling was designed and its cost estimated, what quality checks caught problems before they reached training, how splits were built to reflect real deployment, what augmentation earned its place, and how imbalance was diagnosed and handled, each tied to a specific project rather than generic principles.

Structured elaboration

A concrete narrative structure: sourcing (where did the raw data come from, and what gaps or biases did that source introduce that had to be corrected for later); labeling methodology and cost (who labeled it, what guidelines they used, how much it cost per label and in total, and how that cost shaped how much data was realistically available); quality checks (what you measured before trusting the data: missingness, label agreement between annotators, outlier scans, a sanity check against domain expectations); splits (stratified, group-aware, or time-aware, chosen specifically for how the model would actually be deployed and evaluated); augmentation (what techniques were used and why, tied to the actual data scarcity or imbalance problem being solved, not applied by default); class imbalance (how it was diagnosed, what was tried, and what actually worked).

Label schema and inter-annotator agreement are worth being genuinely specific about: state the actual label categories used, the guideline document annotators worked from, the agreement metric computed (Cohen's kappa or similar) and its value, and what threshold of agreement you required before trusting a label as reliable, since a vague "we had good labels" answer is much weaker than a concrete number and a concrete process.

Worked example

A believable version of this story: "We sourced 50,000 support tickets from the last 18 months, discovered the earliest 6 months used a different tagging taxonomy that made those labels unreliable and excluded them, spent $8,000 labeling a 5,000-ticket gold set with two independent annotators achieving a Cohen's kappa of 0.82, used that gold set to validate a cheaper single-annotator pass on the remaining data, split by ticket creation date (not randomly) since we needed the model to generalize forward in time, and found the imbalance (5% escalation rate) was better handled with class weighting than SMOTE once we validated both against the held-out time-based split."

Trade-offs and pitfalls

The single biggest signal of a shallow answer to this question is genericness: reciting the five steps in the abstract without a single concrete number, decision, or trade-off specific to an actual project is exactly what a prepared-but-inexperienced candidate produces, while a senior candidate's answer is dense with specific numbers, specific mistakes caught, and specific trade-offs actually made.

Direct answer

Favor an unsupervised anomaly detector when you have little or no reliable labeled data for the rare class and are trying to catch NOVEL patterns you haven't seen before; favor a supervised classifier when you have enough labeled examples of the specific pattern you're targeting and want to optimize directly for catching more of exactly that pattern.

Structured elaboration

Decision criteria:

• Label availability and quality: supervised classification needs enough labeled positives to learn a real decision boundary; below some threshold (a handful to a few dozen confirmed examples), a supervised model has too little signal, and an unsupervised "what does normal look like" approach makes better use of the abundant unlabeled negative data instead.
• Novelty of the pattern: if attackers, fraudsters, or failure modes evolve and you need to catch things you've never labeled before, unsupervised approaches generalize to novel patterns better, since they're not fit to any specific known positive shape; a purely supervised classifier trained only on past labeled fraud will systematically miss fraud that looks different from what it learned.
• Operational constraints: a supervised classifier gives you a single, directly-optimized precision/recall trade-off you can tune; an unsupervised anomaly score requires you to separately calibrate what threshold on "anomalousness" corresponds to an acceptable alert volume, which is less direct but doesn't require labels to set up in the first place.

In production, these are rarely either/or: run both and combine, using the unsupervised score as an additional FEATURE fed into the supervised model (so the supervised model gets to learn how much to trust the anomaly signal), or run the unsupervised detector as a first-pass filter over a huge volume of events before a more expensive supervised model or human reviewer looks at the smaller flagged subset.

Worked example

For a new fraud pattern with only 3 confirmed cases so far, an unsupervised Isolation Forest trained on the abundant "normal" transaction data can flag transactions that look statistically unusual along any dimension, without needing to have seen this exact pattern before; a supervised classifier trained on those same 3 examples would essentially be memorizing 3 points, with almost no ability to generalize to a fourth, differently-shaped case.

Trade-offs and pitfalls

The common mistake is picking one approach permanently rather than revisiting the choice as labels accumulate: what starts as an unsupervised-first system, appropriate when labels are scarce, should graduate toward a supervised or hybrid approach once enough confirmed labels exist to make supervised learning worthwhile, since a well-tuned supervised classifier will generally outperform an unsupervised one on the SPECIFIC pattern it has plenty of labels for.

Direct answer

Train AUC of 0.98 with validation AUC of 0.75 on a pipeline that SMOTEs before splitting is the textbook leakage signature: synthetic points generated from BOTH what becomes the training set and what becomes the validation set let information about the validation examples leak into training through their nearest neighbors, so the model looks far better than it will ever perform on genuinely unseen data.

Structured elaboration

Debugging checklist, roughly in order of how cheap each check is:

1. Confirm the order of operations. Check the code path directly: does the SMOTE call happen before or after train_test_split? If before, that alone is very likely the whole story.
2. Reproduce with the fix and compare. Move SMOTE inside the training fold only (fit and oversample after splitting, apply nothing to validation/test) and rerun. If validation AUC jumps toward the training number, leakage is confirmed as the dominant cause.
3. Check SMOTE's neighbor computation directly. If it's genuinely computed only within the training set post-fix, verify no validation indices appear among the neighbor indices used to generate synthetic points.
4. Rule out compounding causes. Even after fixing the order, check whether other preprocessing (scaling, encoding) is ALSO fit on the combined data, since leakage bugs frequently travel in pairs within the same pipeline.
5. Look at declining minority recall in production separately. This is a distinct, second-order symptom: even after fixing the leakage, a validation set that itself doesn't represent the true production distribution (for example, still class-balanced from an earlier bad step) will overstate real-world performance, so re-check that validation and production both reflect the true class ratio.

Worked example

If SMOTE runs before the split, a validation example $x_v$ can be selected as a nearest neighbor when generating a synthetic point $x_s$ that ends up in the training set; the model then trains on $x_s$, which is a near-copy of $x_v$'s own feature values, so when the model is evaluated on $x_v$ it is effectively grading itself on a variant of a question it already saw the answer to.

Trade-offs and pitfalls

Short-term remediation is the pipeline reorder above, verified by rerunning and confirming the AUC gap narrows to something explainable by genuine model variance rather than leakage. Long-term remediation is structural: wrap the entire imputation-plus-resampling-plus-model sequence in a single scikit-learn Pipeline (with imblearn.pipeline.Pipeline specifically, since resampling steps must live inside cross-validation folds, unlike a plain sklearn Pipeline) so the ordering bug becomes structurally impossible to reintroduce, and add a CI check that fails if any resampling call appears before a split call in the training code.

Direct answer

SMOTE fails on high-dimensional sparse data because nearest-neighbor distance stops being a meaningful signal once dimensionality is very high (the "curse of dimensionality" makes most points roughly equidistant from each other), and interpolating between two nearly-arbitrary "nearest" neighbors in sparse text or embedding space can produce synthetic points that don't correspond to anything realistic.

Structured elaboration

The mechanism: in high-dimensional sparse spaces (bag-of-words vectors with tens of thousands of dimensions, most of them zero for any given document), the ratio between the nearest and farthest neighbor distances shrinks toward 1 as dimensionality grows, meaning the "nearest neighbor" SMOTE picks is often only marginally closer than a randomly-chosen point, undermining the whole premise that interpolating toward it produces a meaningfully realistic synthetic example. Interpolating between two sparse bag-of-words vectors can also produce a synthetic vector with FRACTIONAL word-count-like values that don't correspond to any real document (a "0.4 occurrences" of a word), a kind of synthetic point that's not just noisy but structurally nonsensical for the representation.

Three alternative strategies for creating informative minority examples in this setting:

1. Contextual augmentation: use a pretrained language model to suggest contextually-plausible word substitutions or insertions in real minority-class text, producing genuinely valid new documents rather than interpolated vectors.
2. Back-translation: translate minority-class text to another language and back, producing a natural paraphrase that's a real, well-formed document, sidestepping the vector-interpolation problem entirely by operating in TEXT space rather than embedding space.
3. Latent-space interpolation with a language model: rather than interpolating in the raw sparse bag-of-words space, interpolate in a dense, pretrained embedding space (like sentence embeddings) where distances are far more meaningful, then either use the interpolated embedding directly as a feature (if your downstream model consumes embeddings) or decode it back toward realistic text using a generative model, addressing the meaningless-distance problem SMOTE has directly.

Worked example

Two minority-class documents represented as sparse 20,000-dimensional bag-of-words vectors, each with maybe 50 non-zero entries: SMOTE's nearest-neighbor search over this space is dominated by which few of those thousands of dimensions happen to overlap, a fairly arbitrary basis for declaring two documents "similar," and the interpolated synthetic vector ends up with values like 0.3 for a word that appeared once in one document and not the other, a value with no clear interpretation as a word count.

Trade-offs and pitfalls

Each alternative trades away something SMOTE's raw simplicity provided: contextual augmentation and back-translation both need an external pretrained model as a dependency (added complexity and a new failure mode if that model produces low-quality or off-topic substitutions), and latent-space interpolation requires you to trust that the embedding space itself captures the right notion of similarity for your task, an assumption worth validating rather than taking for granted just because it's dense rather than sparse.

Direct answer

Pick the threshold by sweeping candidate cutoffs against validation-set predicted probabilities, choosing the smallest threshold that still keeps recall at or above your target (or symmetrically, the largest that keeps precision at or above a floor), rather than trusting the model's default 0.5 cutoff.

Structured elaboration

A reproducible procedure:

1. Get out-of-fold or held-out validation predicted probabilities (never threshold on training-set predictions, they're optimistic).
2. Sort unique predicted probabilities as threshold candidates, compute recall and precision at each.
3. Filter to thresholds where recall meets your floor (or precision meets your floor, in the symmetric version of the problem), then pick the one that best satisfies the OTHER metric among the survivors, typically the one minimizing false positives when recall is the hard constraint.
4. Re-validate the chosen threshold on a separate fold or later time window before shipping, since a threshold chosen and evaluated on the exact same validation set can be slightly optimistic.

To estimate the expected false-positive volume at a chosen threshold given a daily event count: multiply the false-positive RATE at that threshold (from the validation confusion matrix) by the expected daily negative volume.

Label lag matters because your "recall" measured today only reflects the outcomes you've already confirmed; positives that haven't been labeled yet (fraud not yet confirmed, churn not yet observed) are invisible in the count, so recall computed too soon after a threshold change is biased optimistic and should be revisited once enough time has passed for labels to mature.

Worked example

With 1,000,000 daily events and a threshold whose validation false-positive rate is 2%, expected daily false positives are $1{,}000{,}000 \times 0.02 = 20{,}000$. If review capacity is only 5,000 alerts a day, that threshold is operationally infeasible even if it hits the recall target, and the conversation needs to go back to the stakeholder about a lower recall floor or added review capacity.

Trade-offs and pitfalls

The symmetric version (maximize recall subject to a minimum precision) uses the exact same sweep, just filtering on precision first. The common mistake is choosing a threshold purely from the validation confusion matrix without translating it into an operational volume (alerts per day) that someone downstream actually has to act on; a technically correct threshold that produces an unworkable alert volume isn't actually a solution.