Data Preparation and Class Imbalance for ML Questions

Direct answer

Combine two families of custom transformers: a comprehensive, leakage-safe preprocessing transformer (impute, log-transform skewed columns, standardize, one-hot encode) and a set of automated regression tests around it, both essential since the preprocessing logic and its correctness need to be treated as first-class, testable code.

Structured elaboration

from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
import numpy as np
import joblib

class TabularPreprocessor(BaseEstimator, TransformerMixin):
    """Median-impute, log-transform skewed numeric columns, standardize, one-hot encode categoricals.
    Safe for use inside a scikit-learn Pipeline / cross_val_score."""
    def __init__(self, numeric_cols, skewed_cols, categorical_cols):
        self.numeric_cols = numeric_cols
        self.skewed_cols = skewed_cols
        self.categorical_cols = categorical_cols

    def fit(self, X, y=None):
        self.num_imputer_ = SimpleImputer(strategy="median").fit(X[self.numeric_cols])
        self.scaler_ = StandardScaler().fit(
            np.hstack([
                self.num_imputer_.transform(X[self.numeric_cols])[:, [i for i, c in enumerate(self.numeric_cols) if c not in self.skewed_cols]],
                np.log1p(self.num_imputer_.transform(X[self.numeric_cols])[:, [i for i, c in enumerate(self.numeric_cols) if c in self.skewed_cols]].clip(min=0)),
            ])
        )
        self.encoder_ = OneHotEncoder(handle_unknown="ignore").fit(X[self.categorical_cols].fillna("missing"))
        return self

    def transform(self, X):
        num = self.num_imputer_.transform(X[self.numeric_cols])
        skew_idx = [i for i, c in enumerate(self.numeric_cols) if c in self.skewed_cols]
        plain_idx = [i for i in range(len(self.numeric_cols)) if i not in skew_idx]
        transformed_num = np.hstack([num[:, plain_idx], np.log1p(num[:, skew_idx].clip(min=0))])
        num_scaled = self.scaler_.transform(transformed_num)
        cat_encoded = self.encoder_.transform(X[self.categorical_cols].fillna("missing")).toarray()
        return np.hstack([num_scaled, cat_encoded])

# serialize/deserialize for production use
prep = TabularPreprocessor(numeric_cols=["age"], skewed_cols=[], categorical_cols=["region"]).fit(X_train)
joblib.dump(prep, "preprocessor.joblib")
prep_loaded = joblib.load("preprocessor.joblib")

The fit/transform separation is what guarantees no leakage: fit stores every learned statistic (num_imputer_, scaler_, encoder_) as fitted attributes, and transform reuses them without ever recomputing from whatever X it's currently given, so calling transform on validation, test, or a single production request row always applies exactly the training-time statistics.

Worked example

Fitting TabularPreprocessor on a small training set, serializing it with joblib.dump, and loading it back in a separate process with joblib.load reproduces byte-identical output for the same input row: prep.transform(X_new) and prep_loaded.transform(X_new) match exactly, confirming the saved object genuinely captures everything fit learned rather than depending on any in-memory state that wouldn't survive serialization. Around this transformer, a concrete test suite adds: schema checks confirming expected columns exist with expected types before the transformer runs; a distributional check (a Kolmogorov-Smirnov test comparing the two distributions directly, or a simpler mean/standard-deviation comparison) between the data used to fit and a new incoming batch, flagging when it has drifted enough to warrant investigation; a null-rate threshold check per column, failing loudly if a normally-clean column suddenly shows heavy missingness; and a unit test confirming transform on a known small input produces the exact expected output, including the unseen-category case.

Trade-offs and pitfalls

Serializability (via joblib or pickle) is what makes this transformer deployable, not just testable: a production scoring service loads the exact fitted object trained offline and calls .transform() on incoming requests, guaranteeing the production transform is bit-for-bit identical to what the model was trained on, rather than a separately-maintained reimplementation that can silently drift out of sync with the training-time logic over time.

Direct answer

IQR-based capping, z-score methods, and Winsorization are simple univariate tools for a single numeric feature; Isolation Forest, Local Outlier Factor, and DBSCAN-based approaches catch multivariate outliers, points that look normal on every individual feature but are jointly unusual. Treatment differs by model: tree-based models tolerate outliers reasonably well, while linear and distance-based models need them addressed directly.

Structured elaboration

Univariate methods (IQR, z-score) only see one column at a time and will miss a point that's individually unremarkable on every feature but jointly anomalous (an unusually high transaction amount for THIS particular customer, even if it's unremarkable across the whole population); multivariate methods are needed to catch that.

Treatment by model type: tree-based models split on thresholds and are naturally robust to extreme values (an outlier just ends up in a leaf by itself, largely uncontaminating other splits), so light or no outlier treatment is often fine; linear models and distance-based models can have their entire fit distorted by even a few extreme points, so capping, transforming, or removing outliers matters much more there.

Worked example

For a feature with $Q1=20$, $Q3=60$ (so $IQR=40$), the IQR fence is $[20 - 1.5\times 40,\ 60 + 1.5\times 40] = [-40, 160]$; any value outside that range is flagged. For skewed data (say, response times heavily right-skewed), this same rule tends to over-flag legitimate large values on the long tail, which is exactly the "advantages and assumptions" caveat: IQR assumes a roughly symmetric distribution, and applying it naively to a skewed one produces more false flags than intended.

Trade-offs and pitfalls

Small-sample caution: with only a handful of data points, quantile-based estimates like Q1/Q3 are themselves noisy, so an IQR fence computed on 20 rows is much less trustworthy than one computed on 20,000.

Direct answer

99.9% accuracy with zero recall on a 0.05% positive class means the model learned the cheapest possible solution: always predict negative. It isn't broken, it's optimizing exactly what you asked it to (minimize overall error), and overall error is dominated by the 99.95% of examples that are negative.

Structured elaboration

Diagnose first, then fix. Confirm the model is actually predicting all-negative (check the distribution of predicted labels, not just accuracy) rather than something more subtle like a broken feature or a leaked label that happens to correlate with the majority class. Once confirmed, three concrete changes address the root cause from different angles:

1. Change what the model is optimized for. Random Forest's default split criterion and vote counting have no notion that a false negative is expensive; passing class_weight='balanced' reweights each sample so minority mistakes cost proportionally more during training.
2. Change the data the model sees. Oversample the minority class (or undersample the majority) so the model doesn't have the option of ignoring the rare class to win on aggregate error.
3. Change how you turn probabilities into decisions. Even a model that DOES separate the classes reasonably well in its predicted probabilities can still show zero recall at the default 0.5 threshold, because 0.5 was never a meaningful cutoff for a 0.05% base rate. Lowering the threshold, chosen via the validation set against a recall or cost target, is often the single highest-leverage fix and requires no retraining.

Worked example

Suppose the model's predicted probabilities for the 5 true positives in a 10,000-row test set are $[0.02, 0.08, 0.15, 0.30, 0.55]$. At the default threshold of 0.5, only the last one crosses it, so recall is $1/5 = 0.2$, still poor but not literally zero, illustrating that the "zero recall" you're diagnosing is often a threshold artifact layered on top of a genuinely under-trained minority signal, not necessarily a model that learned nothing at all.

Trade-offs and pitfalls

Controlling false positives while improving recall is the real balancing act: class weighting and resampling both trade some precision for recall, and the right amount depends on the actual cost of a false positive in your application. The common mistake is stopping at "I improved recall" without checking what happened to precision and whether the new operating point is one the business can actually act on.

Direct answer

Measure label quality with a periodic, stratified audit sample scored for inter-annotator agreement (Cohen's kappa or similar), set explicit relabeling rules for what triggers a re-check, estimate the annotation cost of scaling audits up or down, and use model-confidence drift as an early, cheap signal for when to trigger an incremental relabeling pass between full audits.

Structured elaboration

Stratified sampling for audits: rather than auditing a purely random sample, stratify by segments where label quality is most likely to be weak or most consequential (a newer labeling guideline still being learned, a particularly ambiguous category, a high-stakes minority class), so limited audit capacity is spent where it catches the most real problems.

Inter-annotator agreement metrics: Cohen's kappa (for two annotators) or Fleiss' kappa (for more than two) on the audit sample gives a quantified reliability measure, and tracking it OVER TIME (not just as a one-off snapshot) surfaces whether quality is improving, stable, or degrading as guidelines evolve or new annotators onboard.

Rules for relabeling: define concrete triggers, a kappa dropping below a chosen threshold on a specific category, or a specific labeling function/annotator's disagreement rate with the audit ground truth exceeding some bound, that automatically flag a batch for relabeling rather than relying on someone noticing informally that quality "feels off."

Cost estimates for annotation: track cost per label and per audit round explicitly, so scaling the audit process up (auditing more examples, more frequently) is a deliberate, budgeted trade-off against how much labeling-quality risk the project can tolerate, not an afterthought.

Triggers based on model confidence drift: between full audit rounds, monitor whether the model's predicted-confidence distribution on newly-labeled data is drifting (systematically becoming less confident, or newly disagreeing more with the given labels than historically), which is a cheap, always-on signal that something may have changed in either the data or the labeling process, worth investigating even before the next scheduled full audit.

Worked example

A quarterly audit of 500 stratified examples might find kappa dropping from 0.85 to 0.71 specifically on a recently-added subcategory, triggering a targeted relabeling of that subcategory's recent batch and a guideline clarification session with annotators, rather than either ignoring the drop or unnecessarily re-auditing the ENTIRE dataset when the problem was actually isolated to one identifiable segment.

Trade-offs and pitfalls

The real design tension is audit frequency and coverage against cost: auditing too rarely or too narrowly risks a real quality problem going undetected for a long stretch (and potentially poisoning a lot of downstream model training in the meantime), while auditing too broadly or too frequently burns a labeling budget that could otherwise go toward genuinely new labels; the model-confidence-drift signal above is specifically valuable because it's a cheap, continuous proxy that lets you catch problems between the more expensive, scheduled full audits, rather than choosing between "audit constantly" and "audit rarely" as the only two options.

Direct answer

Pick the threshold by sweeping candidate cutoffs against validation-set predicted probabilities, choosing the smallest threshold that still keeps recall at or above your target (or symmetrically, the largest that keeps precision at or above a floor), rather than trusting the model's default 0.5 cutoff.

Structured elaboration

A reproducible procedure:

1. Get out-of-fold or held-out validation predicted probabilities (never threshold on training-set predictions, they're optimistic).
2. Sort unique predicted probabilities as threshold candidates, compute recall and precision at each.
3. Filter to thresholds where recall meets your floor (or precision meets your floor, in the symmetric version of the problem), then pick the one that best satisfies the OTHER metric among the survivors, typically the one minimizing false positives when recall is the hard constraint.
4. Re-validate the chosen threshold on a separate fold or later time window before shipping, since a threshold chosen and evaluated on the exact same validation set can be slightly optimistic.

To estimate the expected false-positive volume at a chosen threshold given a daily event count: multiply the false-positive RATE at that threshold (from the validation confusion matrix) by the expected daily negative volume.

Label lag matters because your "recall" measured today only reflects the outcomes you've already confirmed; positives that haven't been labeled yet (fraud not yet confirmed, churn not yet observed) are invisible in the count, so recall computed too soon after a threshold change is biased optimistic and should be revisited once enough time has passed for labels to mature.

Worked example

With 1,000,000 daily events and a threshold whose validation false-positive rate is 2%, expected daily false positives are $1{,}000{,}000 \times 0.02 = 20{,}000$. If review capacity is only 5,000 alerts a day, that threshold is operationally infeasible even if it hits the recall target, and the conversation needs to go back to the stakeholder about a lower recall floor or added review capacity.

Trade-offs and pitfalls

The symmetric version (maximize recall subject to a minimum precision) uses the exact same sweep, just filtering on precision first. The common mistake is choosing a threshold purely from the validation confusion matrix without translating it into an operational volume (alerts per day) that someone downstream actually has to act on; a technically correct threshold that produces an unworkable alert volume isn't actually a solution.