Data Preparation and Class Imbalance for ML Questions

Direct answer

Pick the threshold by sweeping candidate cutoffs against validation-set predicted probabilities, choosing the smallest threshold that still keeps recall at or above your target (or symmetrically, the largest that keeps precision at or above a floor), rather than trusting the model's default 0.5 cutoff.

Structured elaboration

A reproducible procedure:

1. Get out-of-fold or held-out validation predicted probabilities (never threshold on training-set predictions, they're optimistic).
2. Sort unique predicted probabilities as threshold candidates, compute recall and precision at each.
3. Filter to thresholds where recall meets your floor (or precision meets your floor, in the symmetric version of the problem), then pick the one that best satisfies the OTHER metric among the survivors, typically the one minimizing false positives when recall is the hard constraint.
4. Re-validate the chosen threshold on a separate fold or later time window before shipping, since a threshold chosen and evaluated on the exact same validation set can be slightly optimistic.

To estimate the expected false-positive volume at a chosen threshold given a daily event count: multiply the false-positive RATE at that threshold (from the validation confusion matrix) by the expected daily negative volume.

Label lag matters because your "recall" measured today only reflects the outcomes you've already confirmed; positives that haven't been labeled yet (fraud not yet confirmed, churn not yet observed) are invisible in the count, so recall computed too soon after a threshold change is biased optimistic and should be revisited once enough time has passed for labels to mature.

Worked example

With 1,000,000 daily events and a threshold whose validation false-positive rate is 2%, expected daily false positives are $1{,}000{,}000 \times 0.02 = 20{,}000$. If review capacity is only 5,000 alerts a day, that threshold is operationally infeasible even if it hits the recall target, and the conversation needs to go back to the stakeholder about a lower recall floor or added review capacity.

Trade-offs and pitfalls

The symmetric version (maximize recall subject to a minimum precision) uses the exact same sweep, just filtering on precision first. The common mistake is choosing a threshold purely from the validation confusion matrix without translating it into an operational volume (alerts per day) that someone downstream actually has to act on; a technically correct threshold that produces an unworkable alert volume isn't actually a solution.

Direct answer

At roughly 20% label noise, favor robust training over trying to perfectly clean every label first: use a robust loss or label smoothing to reduce sensitivity to individual bad labels, combine it with a noise-detection method like confident learning or ensemble-disagreement scoring to prioritize a limited relabeling budget, and iterate rather than expecting one pass to fully solve it.

Structured elaboration

Robust training strategies: label smoothing softens hard 0/1 targets toward a small uniform distribution, reducing how confidently the model is pushed to match any single (possibly wrong) label; co-teaching trains two networks simultaneously, each selecting the lower-loss (likely cleaner) examples to teach the other, which empirically resists a meaningful fraction of label noise; confident learning estimates the joint distribution of noisy-versus-true labels using the model's own predicted probabilities, flagging examples where the model consistently, confidently disagrees with the given label as likely mislabeled; noisy-student-style approaches train a model, use it to generate pseudo-labels on a larger unlabeled or lower-confidence set, and iteratively retrain, which can average out some label noise across iterations if done carefully.

A combined approach for a 20%-noise dataset: (1) train an initial model with a robust loss (label smoothing, or a robust loss function less sensitive to outlier-loss examples than plain cross-entropy) as a first pass; (2) use that model's predictions to run confident-learning-style noise detection, ranking examples by how strongly the model disagrees with their given label; (3) prioritize the highest-disagreement examples for human relabeling within whatever budget exists, rather than relabeling randomly; (4) retrain on the corrected-plus-remaining data, and repeat if time and budget allow.

Worked example

If confident learning flags the top 5% most-suspicious labels (by model-disagreement score) out of a 20%-noisy dataset, and a spot-check confirms 70% of those flagged examples are indeed mislabeled, that's a far more efficient use of a limited relabeling budget than randomly sampling 5% of the dataset, where you'd expect only about 20% to actually be wrong.

Trade-offs and pitfalls

For the compounding case where the NOISY labels are concentrated in the MINORITY class (a common real pattern, since minority-class labeling often comes from a noisier or more ad-hoc process than majority-class labeling), lean more heavily on class weighting than on synthetic oversampling techniques like SMOTE, since interpolating between minority points amplifies whatever noise is already present by spreading it into the synthetic points too; and treat any single minority label you're uncertain about with outsized scrutiny, since with a genuinely small minority count, one bad label has disproportionate influence on both training and your evaluation of how well the noise-handling approach worked. The same principle extends to a suspected deliberate, adversarial labeling campaign targeting the minority class specifically: pair the statistical noise-detection approach above with a manual audit of the SOURCE of minority labels (were they all logged through the same suspicious channel or reviewer), since adversarially-planted noise can be designed to evade a purely statistical detector that assumes noise is random rather than deliberately crafted.

Direct answer

Five common strategies: dropping rows or columns, mean/median/mode imputation, K-Nearest-Neighbors or other model-based imputation, and adding a missingness indicator column. Each fits a different scenario, and picking the wrong one for the wrong reason (usually "just drop it" out of convenience) can quietly bias your model.

Structured elaboration

• Dropping rows: appropriate when missingness is rare (a small percentage of rows) and appears close to random; inappropriate when it would discard a meaningful fraction of your data or when missingness itself carries signal.
• Dropping columns: appropriate when a feature is missing so often (say, over 70-80%) that imputing it would be mostly fabrication; inappropriate for a feature the business considers important, where a proxy or targeted imputation is worth the extra effort instead.
• Mean/median/mode imputation: appropriate as a fast, robust baseline for numeric (median, robust to skew) or categorical (mode) features with a modest missing rate; inappropriate when missingness is informative (MNAR) or the feature has strong subgroup structure that a single global statistic would flatten.
• Model-based imputation (KNN, iterative/MICE): appropriate when you have enough correlated features to predict the missing one reasonably well and can afford the extra computation; inappropriate on very large datasets where the added cost isn't worth a marginal accuracy gain over simple imputation.
• Missingness indicator: appropriate alongside any of the above, whenever missingness itself might correlate with the outcome (a customer who declined to answer an income question might behave differently than one who answered); inappropriate as a standalone strategy since it doesn't actually fill in a usable feature value.

Keeping the SAME imputation logic between training and inference (the same fitted median or the same fitted imputer object, not a value recomputed fresh in production) is what makes any of these approaches actually reproducible in a live system.

Worked example

For a numeric "annual income" feature missing in 4% of rows, roughly at random: dropping those rows loses 4% of your data for likely no benefit; median imputation is fast, safe, and probably good enough. For a categorical "referral source" feature missing in 60% of rows specifically for customers who signed up through a channel that doesn't capture it: that's not random, an indicator column (missing/not-missing) is likely more informative than any imputed value could be.

Trade-offs and pitfalls

The recurring judgment call across all five strategies is distinguishing "missing at random, safe to impute simply" from "missing for a reason that itself matters," which is exactly the MCAR/MAR/MNAR distinction covered elsewhere; getting that classification wrong is the single most common way a reasonable-sounding imputation choice turns out to bias the model.

Direct answer

Focal loss adds a modulating factor to standard cross-entropy that automatically down-weights easy, already-well-classified examples, letting hard and minority-class examples dominate the gradient more than they would under plain weighted cross-entropy alone.

Structured elaboration

$$FL(p_t) = -\alpha_t (1 - p_t)^{\gamma} \log(p_t)$$

where $p_t$ is the model's predicted probability for the TRUE class of a given example. The $(1-p_t)^{\gamma}$ term is the modulating factor: when the model is already confident and correct ($p_t$ close to 1), $(1-p_t)^{\gamma}$ is close to 0, shrinking that example's contribution to the loss almost to nothing; when the model is wrong or uncertain ($p_t$ small), $(1-p_t)^{\gamma}$ stays close to 1, leaving that example's loss largely unshrunk.

gamma controls how AGGRESSIVELY easy examples get down-weighted: $\gamma=0$ recovers plain (optionally alpha-weighted) cross-entropy exactly; larger gamma (commonly 2) increasingly focuses training on hard examples. alpha is a more conventional class-balancing weight (like inverse-frequency weighting), applied alongside the modulating factor rather than instead of it.

Focal loss tends to outperform simple class weighting specifically when the majority class contains a large number of EASY examples that a plain weighted loss would still spend a lot of gradient budget on (correctly classifying an obvious majority example over and over contributes little useful signal but still adds up in the total loss); focal loss's modulating factor suppresses exactly that wasted signal, concentrating training on the genuinely informative hard and minority-class cases.

Worked example

For an easy, correctly-classified example with $p_t = 0.95$ and $\gamma=2$: the modulating factor is $(1-0.95)^2 = 0.0025$, so its contribution to the loss is scaled down to a quarter of one percent of what plain cross-entropy would assign it. For a hard example with $p_t = 0.4$: the modulating factor is $(1-0.4)^2 = 0.36$, 144 times larger relative weight than the easy example's factor, concentrating the effective gradient budget heavily on the harder case.

Trade-offs and pitfalls

Focal loss adds two hyperparameters (alpha and gamma) that need tuning, unlike a single class-weight ratio, and an overly large gamma can destabilize training early on, when most examples are still poorly classified and the modulating factor barely shrinks anything, making the effective learning signal noisier than plain weighted cross-entropy until the model starts to separate the classes.

Direct answer

SMOTE builds new minority-class examples by interpolating between real minority points and their nearest minority neighbors, instead of just copying existing points. That keeps the model from simply memorizing duplicates while still giving it more minority-region training signal.

Structured elaboration

For each minority sample $x_i$, SMOTE picks one of its k nearest minority-class neighbors $x_{nn}$ and generates a synthetic point along the line segment between them:

$$x_{\text{synthetic}} = x_i + \lambda \cdot (x_{nn} - x_i), \quad \lambda \sim \text{Uniform}(0, 1)$$

Repeated across many minority points and neighbors, this fills in the minority region of feature space rather than stacking duplicate points on top of each other.

Main variants worth knowing:

• Borderline-SMOTE only generates synthetic points near the boundary between classes, where the extra signal matters most for the decision boundary, rather than uniformly across the whole minority region.
• ADASYN generates more synthetic points for minority examples that are harder to classify (surrounded by more majority neighbors), adaptively focusing effort where the model is currently weakest.
• SMOTEENN / SMOTETomek combine SMOTE with a cleaning step (Edited Nearest Neighbors or Tomek-link removal) that deletes noisy or overlapping points after oversampling, producing a cleaner boundary.

Pitfalls: SMOTE can generate synthetic points that fall inside the majority-class region if the minority class is noisy or the classes overlap; it does nothing to fix a fundamentally uninformative feature set; and it assumes a meaningful Euclidean (or similar) distance in your feature space, which breaks down for one-hot-encoded categorical features (SMOTE-NC and SMOTEN exist specifically to handle mixed and pure categorical inputs, using a modified distance and majority-vote-style category assignment). A dataset characteristic that makes SMOTE a poor choice: very high-dimensional sparse data (bag-of-words text, high-dimensional embeddings), where nearest-neighbor distances stop being meaningful and interpolated points can land in nonsensical regions of the space.

Worked example

With two minority points $x_1=[2,4]$ and its nearest neighbor $x_{nn}=[6,8]$, and $\lambda=0.3$: $x_{\text{synthetic}} = [2,4] + 0.3\cdot([6,8]-[2,4]) = [2,4] + 0.3\cdot[4,4] = [3.2, 5.2]$, a new point strictly between the two originals rather than a copy of either.

Trade-offs and pitfalls

After applying SMOTE in a production pipeline, worthwhile sanity checks are: confirming the synthetic-to-real ratio isn't so high that the model is mostly learning from interpolated points, spot-checking that synthetic points don't fall in obviously wrong regions of feature space, and re-validating on the ORIGINAL (non-oversampled) distribution so your reported performance reflects reality, not the oversampled training set. For mixed numeric-and-categorical data specifically, use SMOTE-NC (numeric interpolation, majority-vote category assignment) rather than plain SMOTE, and be aware that categories which appear rarely in the minority class can end up over-represented in the synthetic points.