Data Preparation and Class Imbalance for ML Questions

Direct answer

Combine two families of custom transformers: a comprehensive, leakage-safe preprocessing transformer (impute, log-transform skewed columns, standardize, one-hot encode) and a set of automated regression tests around it, both essential since the preprocessing logic and its correctness need to be treated as first-class, testable code.

Structured elaboration

from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
import numpy as np
import joblib

class TabularPreprocessor(BaseEstimator, TransformerMixin):
    """Median-impute, log-transform skewed numeric columns, standardize, one-hot encode categoricals.
    Safe for use inside a scikit-learn Pipeline / cross_val_score."""
    def __init__(self, numeric_cols, skewed_cols, categorical_cols):
        self.numeric_cols = numeric_cols
        self.skewed_cols = skewed_cols
        self.categorical_cols = categorical_cols

    def fit(self, X, y=None):
        self.num_imputer_ = SimpleImputer(strategy="median").fit(X[self.numeric_cols])
        self.scaler_ = StandardScaler().fit(
            np.hstack([
                self.num_imputer_.transform(X[self.numeric_cols])[:, [i for i, c in enumerate(self.numeric_cols) if c not in self.skewed_cols]],
                np.log1p(self.num_imputer_.transform(X[self.numeric_cols])[:, [i for i, c in enumerate(self.numeric_cols) if c in self.skewed_cols]].clip(min=0)),
            ])
        )
        self.encoder_ = OneHotEncoder(handle_unknown="ignore").fit(X[self.categorical_cols].fillna("missing"))
        return self

    def transform(self, X):
        num = self.num_imputer_.transform(X[self.numeric_cols])
        skew_idx = [i for i, c in enumerate(self.numeric_cols) if c in self.skewed_cols]
        plain_idx = [i for i in range(len(self.numeric_cols)) if i not in skew_idx]
        transformed_num = np.hstack([num[:, plain_idx], np.log1p(num[:, skew_idx].clip(min=0))])
        num_scaled = self.scaler_.transform(transformed_num)
        cat_encoded = self.encoder_.transform(X[self.categorical_cols].fillna("missing")).toarray()
        return np.hstack([num_scaled, cat_encoded])

# serialize/deserialize for production use
prep = TabularPreprocessor(numeric_cols=["age"], skewed_cols=[], categorical_cols=["region"]).fit(X_train)
joblib.dump(prep, "preprocessor.joblib")
prep_loaded = joblib.load("preprocessor.joblib")

The fit/transform separation is what guarantees no leakage: fit stores every learned statistic (num_imputer_, scaler_, encoder_) as fitted attributes, and transform reuses them without ever recomputing from whatever X it's currently given, so calling transform on validation, test, or a single production request row always applies exactly the training-time statistics.

Worked example

Fitting TabularPreprocessor on a small training set, serializing it with joblib.dump, and loading it back in a separate process with joblib.load reproduces byte-identical output for the same input row: prep.transform(X_new) and prep_loaded.transform(X_new) match exactly, confirming the saved object genuinely captures everything fit learned rather than depending on any in-memory state that wouldn't survive serialization. Around this transformer, a concrete test suite adds: schema checks confirming expected columns exist with expected types before the transformer runs; a distributional check (a Kolmogorov-Smirnov test comparing the two distributions directly, or a simpler mean/standard-deviation comparison) between the data used to fit and a new incoming batch, flagging when it has drifted enough to warrant investigation; a null-rate threshold check per column, failing loudly if a normally-clean column suddenly shows heavy missingness; and a unit test confirming transform on a known small input produces the exact expected output, including the unseen-category case.

Trade-offs and pitfalls

Serializability (via joblib or pickle) is what makes this transformer deployable, not just testable: a production scoring service loads the exact fitted object trained offline and calls .transform() on incoming requests, guaranteeing the production transform is bit-for-bit identical to what the model was trained on, rather than a separately-maintained reimplementation that can silently drift out of sync with the training-time logic over time.

Direct answer

Jittering adds small random noise to each point, scaling multiplies the whole series by a random factor, permutation shuffles segments of the series, and time-warping locally stretches or compresses the time axis; all preserve the label for classification, but forecasting tasks need to be far more careful, since the very thing being predicted (the future trajectory) can be corrupted by an augmentation that alters temporal dynamics.

Structured elaboration

• Jittering: $x'_t = x_t + \epsilon_t,\ \epsilon_t \sim \mathcal{N}(0,\sigma^2)$, simulating sensor noise; safe for both classification and forecasting as long as the noise scale is small relative to the signal.
• Scaling: $x'_t = \alpha \cdot x_t,\ \alpha$ drawn from a narrow range around 1; simulates amplitude variation (different sensor calibration, different individual baseline); safe for classification of a pattern's SHAPE, riskier for forecasting if the model needs to predict actual magnitudes rather than shape.
• Permutation: shuffling the order of fixed-length segments within the series; reasonable for classification tasks where the overall pattern across segments matters more than strict short-range order, but generally UNSAFE for forecasting, since it directly breaks the temporal ordering the forecast depends on.
• Time-warping: locally stretching or compressing sections of the time axis, simulating a pattern happening slightly faster or slower; useful for classification robustness to timing variation, but requires care for forecasting since it changes the effective time-to-event the model is learning to predict.

Preserving label semantics: for classification, the label usually describes the whole sequence's category, which most of these transforms leave intact as long as they don't distort the signal beyond recognition; for forecasting, the "label" is the actual future values, so any augmentation applied to the input HISTORY must not implicitly change what the correct future continuation should have been, which rules out permutation entirely and requires jittering/scaling/warping to be applied consistently across both the historical input and any future window used for evaluation during training.

Preserving temporal coherence across channels when augmenting sliding windows: for multivariate series (several sensors read together), applying independently-random jitter or warping per channel can break realistic cross-channel relationships (two correlated sensors that should move together no longer do after independent augmentation), so the SAME random parameters (the same warp function, the same scaling factor) should typically be applied consistently across all channels within one augmented window, not drawn independently per channel.

Worked example

For a 3-sensor multivariate window being time-warped, applying the identical warping function to all 3 channels preserves their relative timing and correlation structure; applying three INDEPENDENTLY-random warps would desynchronize sensors that are supposed to move together, potentially teaching the model a relationship between sensors that doesn't exist in real data.

Trade-offs and pitfalls

The single most important rule specific to time-series augmentation, beyond what applies to images or text, is that TEMPORAL ORDER and CROSS-CHANNEL CONSISTENCY are both first-class constraints an augmentation must respect, not incidental details, since violating either one can produce training examples that actively teach the model incorrect temporal or cross-sensor relationships rather than just adding harmless noise.

Direct answer

Focal loss adds a modulating factor to standard cross-entropy that automatically down-weights easy, already-well-classified examples, letting hard and minority-class examples dominate the gradient more than they would under plain weighted cross-entropy alone.

Structured elaboration

where $p_t$ is the model's predicted probability for the TRUE class of a given example. The $(1-p_t)^{\gamma}$ term is the modulating factor: when the model is already confident and correct ($p_t$ close to 1), $(1-p_t)^{\gamma}$ is close to 0, shrinking that example's contribution to the loss almost to nothing; when the model is wrong or uncertain ($p_t$ small), $(1-p_t)^{\gamma}$ stays close to 1, leaving that example's loss largely unshrunk.

gamma controls how AGGRESSIVELY easy examples get down-weighted: $\gamma=0$ recovers plain (optionally alpha-weighted) cross-entropy exactly; larger gamma (commonly 2) increasingly focuses training on hard examples. alpha is a more conventional class-balancing weight (like inverse-frequency weighting), applied alongside the modulating factor rather than instead of it.

Focal loss tends to outperform simple class weighting specifically when the majority class contains a large number of EASY examples that a plain weighted loss would still spend a lot of gradient budget on (correctly classifying an obvious majority example over and over contributes little useful signal but still adds up in the total loss); focal loss's modulating factor suppresses exactly that wasted signal, concentrating training on the genuinely informative hard and minority-class cases.

Worked example

For an easy, correctly-classified example with $p_t = 0.95$ and $\gamma=2$: the modulating factor is $(1-0.95)^2 = 0.0025$, so its contribution to the loss is scaled down to a quarter of one percent of what plain cross-entropy would assign it. For a hard example with $p_t = 0.4$: the modulating factor is $(1-0.4)^2 = 0.36$, 144 times larger relative weight than the easy example's factor, concentrating the effective gradient budget heavily on the harder case.

Trade-offs and pitfalls

Focal loss adds two hyperparameters (alpha and gamma) that need tuning, unlike a single class-weight ratio, and an overly large gamma can destabilize training early on, when most examples are still poorly classified and the modulating factor barely shrinks anything, making the effective learning signal noisier than plain weighted cross-entropy until the model starts to separate the classes.

Direct answer

Mixup blends two training examples and their labels linearly, producing a synthetic example that's a weighted average of both; CutMix instead pastes a rectangular patch from one image onto another and mixes the labels in proportion to the patch area. Both push the model toward smoother, better-calibrated decision boundaries between classes rather than sharp memorized ones.

Structured elaboration

Mixup: $\tilde{x} = \lambda x_i + (1-\lambda) x_j,\quad \tilde{y} = \lambda y_i + (1-\lambda) y_j,\quad \lambda \sim \text{Beta}(\alpha, \alpha)$. The new input is a literal pixel-wise (or feature-wise) blend of two images, and the label is blended in the same proportion, so the model is trained to output partial confidence for both classes on an input that genuinely looks like an interpolation of both.

CutMix instead cuts a rectangular region from image $j$ and pastes it into image $i$, keeping the rest of image $i$ intact; the label mix ratio is set by the AREA of the pasted patch (a patch covering 30% of the image gives labels $\tilde{y} = 0.7\, y_i + 0.3\, y_j$), so unlike mixup the result is a spatially coherent, locally realistic image rather than a ghostly blend, which tends to preserve more natural-looking local structure while still forcing the model to attend to multiple regions.

Both improve generalization by discouraging the model from being maximally overconfident on any single training example (a smoothed target of $0.7/0.3$ rather than a hard $1.0/0.0$ directly softens the loss landscape), and both tend to improve calibration for the same reason: a model trained on hard 0/1 labels alone has no incentive to ever output an intermediate confidence, while mixed labels explicitly reward doing so when appropriate.

Preference by task: CutMix tends to be preferred for localization-sensitive tasks (object detection, segmentation) since it preserves spatially coherent regions the model can still meaningfully attend to; mixup is a reasonable default for plain classification and works cleanly for multi-label settings too, since blending soft multi-label vectors is straightforward, whereas CutMix's area-based label proportion is a slightly less natural fit when an image can carry more than one true label at once.

Worked example

Two images with one-hot labels "cat" $=[1,0]$ and "dog" $=[0,1]$, mixed with $\lambda = 0.7$ under mixup: the resulting soft label is $[0.7, 0.3]$, and the model is trained to predict a 70/30 confidence split on an input that is a genuine 70/30 pixel blend of the two original images.

Trade-offs and pitfalls

Sampling $\lambda$ from a $\text{Beta}(\alpha,\alpha)$ distribution rather than a fixed value matters because it controls how aggressive the mixing typically is: small $\alpha$ (like 0.2) concentrates $\lambda$ near 0 or 1 (mild mixing most of the time, occasionally strong), while $\alpha$ near 1 gives a roughly uniform mixing strength, and tuning it is often necessary since too-aggressive mixing on a small dataset can hurt more than help.

Direct answer

Flag a row as an outlier when its value falls below $Q1 - k\cdot IQR$ or above $Q3 + k\cdot IQR$, computing Q1, Q3, and IQR from the column while gracefully skipping any missing values rather than letting them break the calculation.

Structured elaboration

import numpy as np
import pandas as pd

def detect_outliers_iqr(df, column, k=1.5):
    """Return the index labels of rows considered outliers via the IQR method."""
    values = pd.to_numeric(df[column], errors="coerce")   # tolerate numeric-looking strings, NaN otherwise
    valid = values.dropna()
    q1, q3 = valid.quantile(0.25), valid.quantile(0.75)
    iqr = q3 - q1
    lower, upper = q1 - k * iqr, q3 + k * iqr
    is_outlier = (values < lower) | (values > upper)   # NaN comparisons are False, so NaNs are never flagged
    return df.index[is_outlier.fillna(False)]

df = pd.DataFrame({"amount": [10, 12, 11, 13, 9, 500, None, 14]})
outlier_idx = detect_outliers_iqr(df, "amount", k=1.5)
print(outlier_idx.tolist())

Handling NaNs gracefully: computing the quantiles on valid = values.dropna() means missing values never distort Q1/Q3/IQR, and the final boolean mask's comparisons against NaN naturally evaluate to False (never flagged as an outlier), with an explicit .fillna(False) as a defensive safeguard in case a comparison chain produces something other than a clean boolean.

Worked example

Running the code above on [10, 12, 11, 13, 9, 500, None, 14], Q1 and Q3 are computed from the 7 valid values, giving a fence that easily flags 500 while leaving None untouched (not flagged, correctly excluded rather than erroring); the function returns the index position of the 500 row only.

Trade-offs and pitfalls

The IQR method's known limitation on a skewed distribution: because Q1 and Q3 are computed from the RAW (unlogged, untransformed) values, a heavily right-skewed feature can have its upper fence pulled disproportionately high by the skew itself, causing the method to under-flag genuinely unusual values on the long tail; for such features, applying a skew-reducing transform (log, Box-Cox) before running the IQR check, or using a method less sensitive to skew, often catches outliers this straightforward version misses.