Data Preparation and Class Imbalance for ML Questions

Direct answer

Focal loss and class-balanced loss both modify the loss for imbalance directly, but via different signals: focal loss down-weights based on how confidently CORRECT the model currently is on an example, while class-balanced loss (via the effective number of samples) down-weights based on how much MARGINAL new information a class's additional examples actually provide, given how much overlap already exists among that class's samples; distribution-balanced loss further adds an explicit correction for label co-occurrence bias in multi-label settings.

Structured elaboration

• Focal loss: modulates per-EXAMPLE, based on the model's current confidence on that specific example; theoretical motivation is that easy, already-correct examples contribute diminishing useful gradient signal, so shrinking their contribution focuses training on what's still hard.
• Class-balanced loss (effective number of samples): modulates per-CLASS, using $E_n = \dfrac{1-\beta^n}{1-\beta}$ as an estimate of the effective (non-redundant) sample count for a class with $n$ raw examples; theoretical motivation is that as a class accumulates more samples, the MARGINAL benefit of each additional one diminishes (due to overlap/redundancy among similar examples), so weighting by $1/E_n$ rather than raw inverse frequency more accurately reflects each class's true information content.
• Distribution-balanced loss: extends this further for MULTI-LABEL settings specifically, additionally correcting for the fact that label CO-OCCURRENCE patterns bias which negative examples a given label effectively sees (a rare label that always co-occurs with a common one gets an implicitly different negative sampling distribution than one that doesn't), which neither focal loss nor plain class-balanced loss accounts for.

Expected empirical behavior under extreme imbalance: focal loss tends to help most when the imbalance manifests as many EASY majority examples diluting gradient signal; class-balanced loss tends to help most when class counts span several orders of magnitude and raw inverse-frequency weighting would over- or under-correct due to within-class redundancy; distribution-balanced loss's specific advantage only shows up in multi-label settings with meaningful label co-occurrence structure, and is unlikely to outperform the simpler alternatives on a single-label problem where its extra correction term has nothing to correct.

A rigorous experimental setup to evaluate them at production scale: hold the model architecture, optimizer, and all other hyperparameters fixed, vary only the loss function, run each across multiple seeds with a matched hyperparameter search budget per loss (since each has its own tunable parameters), and evaluate on a metric appropriate to the label structure (per-class or macro F1 for single-label, a label-correlation-aware metric for multi-label), reporting variance across seeds alongside the point estimate.

Worked example

For a class with 10,000 raw examples but very high redundancy ($\beta$ close to 1 in the effective-number formula, meaning many of those 10,000 are near-duplicates of each other), the effective number $E_n$ can be far smaller than 10,000, so class-balanced loss weights that class's LOSS contribution more heavily than raw inverse-frequency weighting would, correctly recognizing that the class's true information content is lower than its raw count suggests.

Trade-offs and pitfalls

The most common mistake comparing these three is reaching for the most sophisticated one (distribution-balanced loss) by default, when its extra correction term specifically targets a multi-label co-occurrence problem that may not even exist in a given single-label dataset, in which case it reduces to doing extra, unhelpful work relative to the simpler class-balanced or focal-loss alternatives.

Direct answer: Combining stability selection with permutation importance across cross-validation folds gives a feature-selection procedure that's robust to both a single unlucky data split and to a single noisy importance estimate, aggregating evidence across many folds rather than trusting any one run, as long as the tuning of that procedure itself is kept separate from the fold used to report its final performance.

Structured elaboration:
The algorithm: for each of several cross-validation folds, fit a model on the fold's training portion and compute permutation importance on its held-out portion; separately (or combined), run a stability-selection-style resampling within each fold to get a selection frequency per feature. Aggregate across all folds: a feature that's both consistently selected across resampling AND consistently shows a real permutation-importance drop across folds is a much stronger candidate than one that only clears the bar in a single fold or by a single method.

Pitfalls to guard against: correlated features can distort BOTH stability selection (as discussed elsewhere, Lasso arbitrarily picks one of a correlated group) and permutation importance (shuffling one of a correlated pair understates its importance since the model partially recovers signal from its correlated partner); aggregating across folds helps average out some of this instability but doesn't eliminate the underlying correlated-feature distortion, which needs to be checked for separately (a correlation-clustering step) if it's suspected.

Selection bias from reusing the same folds for tuning and evaluation is a separate, easy-to-miss pitfall: if the stability-selection threshold (what selection frequency counts as "stable") or the permutation-importance cutoff is chosen by looking at how well the resulting feature set performs on the same folds later reported as the final cross-validated score, that score is optimistically biased, since the threshold was implicitly tuned against the very data used to report performance. The fix is to nest the procedure: an outer set of folds is held out purely for final, unbiased evaluation, while an inner loop (further splits of only the remaining training data) is used to tune the stability threshold, the permutation-importance cutoff, and any other selection hyperparameter; only after those choices are fixed does the already-frozen procedure get scored on the outer fold's held-out data, which never influenced any tuning decision.

Worked example: A feature selected in 90% of stability-selection resamples across folds, AND showing a consistently large permutation-importance drop in every fold, is strong, well-corroborated evidence of genuine importance; a feature selected inconsistently across folds (40% in one, 85% in another) or with permutation-importance drops that vary wildly fold-to-fold is a signal of instability worth investigating before trusting it. Separately, if the 90%-selection-frequency threshold itself was chosen by scanning several candidate thresholds and picking whichever gave the best cross-validated accuracy on those same folds, that reported accuracy is inflated; nesting an inner tuning loop inside each outer fold (tune the threshold only on the inner splits, then apply the frozen threshold to the outer fold's held-out data) removes that inflation.

Trade-offs and pitfalls: This combined approach costs meaningfully more compute than either method alone (multiple folds times multiple resamples times multiple model fits), and nesting nested cross-validation to avoid the fold-reuse bias multiplies that cost further; for a very large candidate pool or an expensive-to-fit model, this cost needs to be weighed against a cheaper, single-pass alternative, accepting that the cheaper alternative's reported performance is likely somewhat optimistic if it skips the nested structure.

Direct answer

Standard k-fold shuffles rows randomly across folds, which for time-series data means training on the future to predict the past, information a real deployment would never have. Walk-forward (rolling-origin) validation and an expanding window both fix this by always keeping training data strictly earlier in time than validation data.

Structured elaboration

Walk-forward / rolling-origin: train on a fixed-size window of the most recent past, validate on the next period, then slide the whole window forward and repeat. Expanding window: keep the training start fixed but grow the training window forward each round, so each successive fold trains on more history than the last while validation always stays just ahead of it.

def walk_forward_splits(n, n_splits, val_size):
    """Yield (train_idx, val_idx) using an expanding training window."""
    fold_size = (n - val_size) // n_splits
    for i in range(n_splits):
        train_end = fold_size * (i + 1)
        val_start, val_end = train_end, min(train_end + val_size, n)
        if val_end <= val_start:
            break
        yield list(range(0, train_end)), list(range(val_start, val_end))

for train_idx, val_idx in walk_forward_splits(n=1000, n_splits=5, val_size=100):
    print(f"train=[0,{train_idx[-1]}]  val=[{val_idx[0]},{val_idx[-1]}]")

If the target is ALSO rare within this time-ordered setting (equipment failures, fraud), preserving an approximate positive-class ratio per fold matters too: rather than a purely chronological cut that might leave one window with almost no positives, you can widen the training window for that fold until it captures a reasonable minimum count of positive examples, trading strict recency for a usable minority sample.

Worked example

Running the code above prints five expanding windows: train=[0,179] val=[180,279], train=[0,359] val=[360,459], and so on through five folds, each one training on strictly earlier data than the validation window that follows it, with training set size growing each round while validation size stays fixed at 100.

Trade-offs and pitfalls

Common pitfalls: allowing ANY feature that uses future information (a rolling average computed with a centered window instead of a trailing one, or a label-derived aggregate) to leak across the train/validation boundary even though the ROW split itself is chronologically correct; and forgetting that walk-forward validation naturally produces fewer, larger folds than standard k-fold for the same dataset, so per-fold metric variance needs to be interpreted with that in mind.

Direct answer

Tomek links, Edited Nearest Neighbors (ENN), and NearMiss all remove majority-class points strategically rather than randomly, targeting points near the class boundary or in ambiguous regions, which sharpens the decision boundary instead of just shrinking the majority class uniformly.

Structured elaboration

• Tomek links: identify pairs of one majority and one minority point that are each other's nearest neighbor (an ambiguous, boundary-straddling pair) and remove the majority point from each such pair, cleaning up the boundary without touching majority points that are clearly, unambiguously majority.
• Edited Nearest Neighbors (ENN): for each majority point, check whether its own k nearest neighbors mostly agree with its label; if a majority point's neighborhood is mostly minority-class, remove it, treating it as noisy or boundary-region.
• NearMiss: several variants that keep majority points selected specifically by their distance to minority points (closest to minority points, in the most common variant), deliberately concentrating the reduced majority set right at the boundary rather than sampling it uniformly.

Typical use cases: Tomek links and ENN are best used as a CLEANING step after (or combined with) oversampling, since they remove ambiguous points that SMOTE-style synthesis can otherwise leave near the boundary; NearMiss is more often used as a primary undersampling strategy on its own, when you want an aggressively boundary-focused reduced dataset.

Effect on class-boundary cleanliness and bias: all three tend to produce a SHARPER, more separable boundary in the retained data, which can genuinely help a classifier learn a cleaner decision rule, but this comes at the cost of systematically REMOVING the hardest, most boundary-adjacent examples, which can make the model overconfident, since it never sees the truly ambiguous cases it will still encounter in production.

Worked example

With a majority point $p$ whose single nearest neighbor overall is a minority point $q$, and $q$'s nearest neighbor is also $p$, that's a Tomek link; removing $p$ eliminates one specific source of boundary ambiguity without touching the thousands of majority points that are clearly, unambiguously far from any minority point.

Trade-offs and pitfalls

Undersampling hurts more than it helps when the boundary region itself carries genuinely important signal you need the model to still handle at inference time (a fraud model that will keep encountering exactly the ambiguous cases these methods remove from training loses its only exposure to them); in that situation, informed undersampling can leave the model brittle on precisely the hardest, most consequential cases, and a validation set drawn from the ORIGINAL (unedited) distribution is essential to catch this before shipping.

Direct answer

Compute each user's own mean and standard deviation over their trailing history using window functions, flag a transaction when it falls far enough outside that user-specific range, and handle new or thin-history users explicitly rather than letting a near-empty window silently produce a meaningless flag.

Structured elaboration

WITH user_stats AS (
    SELECT
        transaction_id,
        user_id,
        amount,
        occurred_at,
        AVG(amount) OVER (
            PARTITION BY user_id
            ORDER BY occurred_at
            ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING
        ) AS user_avg_amount,
        STDDEV(amount) OVER (
            PARTITION BY user_id
            ORDER BY occurred_at
            ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING
        ) AS user_stddev_amount,
        COUNT(*) OVER (
            PARTITION BY user_id
            ORDER BY occurred_at
            ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING
        ) AS user_prior_count
    FROM transactions
)
SELECT
    transaction_id, user_id, amount, occurred_at,
    user_avg_amount, user_stddev_amount, user_prior_count
FROM user_stats
WHERE user_prior_count >= 5   -- require enough history before trusting the flag at all
  AND user_stddev_amount > 0  -- avoid a division-by-zero-equivalent flag on a perfectly flat history
  AND amount > user_avg_amount + 5 * user_stddev_amount

Using ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING (rather than including the current row) means each transaction is flagged against the user's history strictly BEFORE it, avoiding the circularity of a transaction contributing to its own comparison baseline; the user_prior_count >= 5 and user_stddev_amount > 0 filters explicitly handle the "reasonable behavior" requirement for thin-history users, a brand-new user's second transaction has no meaningful personal baseline yet, and the query deliberately declines to flag anything for them rather than producing an unreliable flag from a near-empty or perfectly-uniform history.

Worked example

For a user with a stable history averaging $50 with a $10 standard deviation, a new $300 transaction (over 5 standard deviations above their mean) gets flagged; a brand-new user's first 4 transactions never get evaluated at all (excluded by the user_prior_count >= 5 filter), avoiding a meaningless flag computed from almost no personal history.

Trade-offs and pitfalls

Performance on a large table: partitioned window functions like this scale reasonably well with a good index on (user_id, occurred_at), since the database can compute each partition's running statistics in a single ordered pass per user rather than a full unindexed scan; for a genuinely enormous transaction table, materializing the user-level running statistics incrementally (updated as new transactions arrive, rather than recomputed from full history on every query) is a further optimization worth considering once this naive version's performance becomes a real bottleneck.