Data Preparation and Class Imbalance for ML Questions

Direct answer

Train AUC of 0.98 with validation AUC of 0.75 on a pipeline that SMOTEs before splitting is the textbook leakage signature: synthetic points generated from BOTH what becomes the training set and what becomes the validation set let information about the validation examples leak into training through their nearest neighbors, so the model looks far better than it will ever perform on genuinely unseen data.

Structured elaboration

Debugging checklist, roughly in order of how cheap each check is:

1. Confirm the order of operations. Check the code path directly: does the SMOTE call happen before or after train_test_split? If before, that alone is very likely the whole story.
2. Reproduce with the fix and compare. Move SMOTE inside the training fold only (fit and oversample after splitting, apply nothing to validation/test) and rerun. If validation AUC jumps toward the training number, leakage is confirmed as the dominant cause.
3. Check SMOTE's neighbor computation directly. If it's genuinely computed only within the training set post-fix, verify no validation indices appear among the neighbor indices used to generate synthetic points.
4. Rule out compounding causes. Even after fixing the order, check whether other preprocessing (scaling, encoding) is ALSO fit on the combined data, since leakage bugs frequently travel in pairs within the same pipeline.
5. Look at declining minority recall in production separately. This is a distinct, second-order symptom: even after fixing the leakage, a validation set that itself doesn't represent the true production distribution (for example, still class-balanced from an earlier bad step) will overstate real-world performance, so re-check that validation and production both reflect the true class ratio.

Worked example

If SMOTE runs before the split, a validation example $x_v$ can be selected as a nearest neighbor when generating a synthetic point $x_s$ that ends up in the training set; the model then trains on $x_s$, which is a near-copy of $x_v$'s own feature values, so when the model is evaluated on $x_v$ it is effectively grading itself on a variant of a question it already saw the answer to.

Trade-offs and pitfalls

Short-term remediation is the pipeline reorder above, verified by rerunning and confirming the AUC gap narrows to something explainable by genuine model variance rather than leakage. Long-term remediation is structural: wrap the entire imputation-plus-resampling-plus-model sequence in a single scikit-learn Pipeline (with imblearn.pipeline.Pipeline specifically, since resampling steps must live inside cross-validation folds, unlike a plain sklearn Pipeline) so the ordering bug becomes structurally impossible to reintroduce, and add a CI check that fails if any resampling call appears before a split call in the training code.

Direct answer

99.9% accuracy with zero recall on a 0.05% positive class means the model learned the cheapest possible solution: always predict negative. It isn't broken, it's optimizing exactly what you asked it to (minimize overall error), and overall error is dominated by the 99.95% of examples that are negative.

Structured elaboration

Diagnose first, then fix. Confirm the model is actually predicting all-negative (check the distribution of predicted labels, not just accuracy) rather than something more subtle like a broken feature or a leaked label that happens to correlate with the majority class. Once confirmed, three concrete changes address the root cause from different angles:

1. Change what the model is optimized for. Random Forest's default split criterion and vote counting have no notion that a false negative is expensive; passing class_weight='balanced' reweights each sample so minority mistakes cost proportionally more during training.
2. Change the data the model sees. Oversample the minority class (or undersample the majority) so the model doesn't have the option of ignoring the rare class to win on aggregate error.
3. Change how you turn probabilities into decisions. Even a model that DOES separate the classes reasonably well in its predicted probabilities can still show zero recall at the default 0.5 threshold, because 0.5 was never a meaningful cutoff for a 0.05% base rate. Lowering the threshold, chosen via the validation set against a recall or cost target, is often the single highest-leverage fix and requires no retraining.

Worked example

Suppose the model's predicted probabilities for the 5 true positives in a 10,000-row test set are $[0.02, 0.08, 0.15, 0.30, 0.55]$. At the default threshold of 0.5, only the last one crosses it, so recall is $1/5 = 0.2$, still poor but not literally zero, illustrating that the "zero recall" you're diagnosing is often a threshold artifact layered on top of a genuinely under-trained minority signal, not necessarily a model that learned nothing at all.

Trade-offs and pitfalls

Controlling false positives while improving recall is the real balancing act: class weighting and resampling both trade some precision for recall, and the right amount depends on the actual cost of a false positive in your application. The common mistake is stopping at "I improved recall" without checking what happened to precision and whether the new operating point is one the business can actually act on.

Direct answer

Make reproducibility a checklist with teeth: version the code and dependencies that produce a transform, fix every random seed, serialize the fitted transformers themselves (not just their parameters), enforce data contracts with schema checks, and wire all of it into CI so a broken or non-reproducible pipeline fails loudly rather than shipping quietly.

Structured elaboration

• Code and dependency versioning: pin the preprocessing code to a commit hash and the library versions (scikit-learn==1.3.0, not a floating >=) alongside every trained model artifact, since a library version bump can silently change a transform's numerical behavior.
• Deterministic transforms: fix random seeds everywhere randomness enters (train/test split, any resampling, any model with stochastic initialization) and verify determinism by rerunning the same pipeline twice and diffing outputs.
• Serialization of fitted transformers: save the actual fitted scaler/encoder/imputer objects (via joblib or similar), not just a text description of what they do, so production scoring loads the exact object trained offline rather than a separately-maintained reimplementation that can drift.
• Data contracts and schema checks: assert expected columns, types, and value ranges at the pipeline's entry point, failing fast on an unexpected schema rather than silently producing garbage downstream.
• Unit and integration tests: test each transformer in isolation (known input, known expected output) and the full pipeline end-to-end.
• CI/CD enforcement: run all of the above automatically on every change, blocking a merge that breaks reproducibility or an existing contract.

Concrete tooling for versioning raw data, transformed datasets, and code together: DVC or Delta Lake for data versioning (tracking exactly which data snapshot a given model was trained on), MLflow for experiment and artifact tracking (which code version, which parameters, which resulting metrics), enabling both rollback (revert to a prior data or code version) and lineage queries (trace a production prediction back to the exact data and code that produced the model).

For a small team producing weekly model-backed dashboards, a MINIMAL viable stack matters more than a maximal one: version-controlled transformation code (even just git) plus a lightweight artifact/metadata log (a simple database table or MLflow's free tier) plus fixed seeds is often sufficient, reserving a heavier data-versioning tool like DVC or Delta Lake for when data volume or team size genuinely outgrows the lightweight approach.

For a compliance-sensitive context specifically, add an explicit AUDIT layer on top: log every transformation applied (with its exact parameters) to each data run, index logs by run or commit ID, and ensure you can reconstruct the exact training input for any historical model version on demand, which is a stronger requirement than ordinary reproducibility (being ABLE to reproduce it) since it also requires being able to PROVE what was done, after the fact, to a third party.

Worked example

A weekly dashboard-driving model retrain pipeline might log, for every run: the git commit hash of the pipeline code, the DVC hash of the input data snapshot, the exact library versions from a locked requirements file, the random seed used, and the resulting model's MLflow run ID; six months later, reproducing that week's exact numbers means checking out that commit, that data snapshot, and that seed, and confirming the output matches bit-for-bit or within documented numerical tolerance.

Trade-offs and pitfalls

The most common way reproducibility silently breaks in practice isn't a deliberate change, it's an UNPINNED dependency quietly updating between runs (a library patch release that changes a default parameter, or a floating-point computation order), which is exactly why pinning exact versions, not just "compatible" version ranges, matters more than it might seem for a system that otherwise looks fully reproducible.

Direct answer

Assign each data index to a rank based on a deterministic (seeded) shuffle, then partition those shuffled indices round-robin across replicas so each replica's local shard maintains roughly the same class proportion as the global dataset, guaranteeing a minimum number of positive examples per batch by drawing them from a separately-tracked positive-index pool.

Structured elaboration

Algorithm outline:

1. Separately shuffle (with a seed derived from the current epoch, so the shuffle differs epoch-to-epoch but is reproducible given the seed) the positive-example indices and the negative-example indices.
2. Partition the shuffled positive indices round-robin across the $R$ replicas (positive_indices[rank::R] for each rank), guaranteeing each replica gets an (approximately) equal share of positives every epoch, which is what makes the "at least N positives per batch" guarantee possible in the first place.
3. Do the same for negative indices, at whatever ratio produces the desired per-batch class balance.
4. Each replica's DataLoader then draws from its own assigned local shard (its share of positives interleaved with its share of negatives) using a DistributedSampler-style mechanism, so no cross-replica communication is needed during the actual data-loading step, only at shard-assignment time.

Reproducible shuffles across epochs: deriving each epoch's shuffle seed from a base seed plus the epoch number (seed = base_seed + epoch) means every replica, given the same epoch number, produces the exact same global shuffle before partitioning, so all replicas agree on which indices belong to which shard without needing to communicate the assignment explicitly, they can each independently recompute it.

Worked example

With 4 replicas, 1,000 positive and 100,000 negative examples, and a target of 10 positives per batch per replica: round-robin partitioning gives each replica 250 positive indices per epoch (1,000 / 4), and each replica's local DataLoader draws from its 250 positives interleaved with its share of negatives, guaranteeing every batch on every replica sees close to the target 10 positives, rather than some replicas getting lucky with more and others starved entirely, which is the real risk a naive global-random-shuffle-then-split-across-replicas approach runs into with a rare class.

Trade-offs and pitfalls

A remaining subtlety worth naming: guaranteeing a MINIMUM positive count per batch, rather than just an expected one, generally requires the sampler to actively interleave positives into each batch deterministically (not purely randomly), since pure random sampling from even a correctly-balanced shard can still occasionally produce a batch with fewer positives than the target by chance, so a genuinely hard guarantee needs the batch construction itself, not just the shard assignment, to be structured around it.

Direct answer

Estimate the timeline and cost from concrete unit economics (cost per label, throughput per annotator, required quality-control overhead), then propose specific levers, active learning, synthetic data, data partnerships, to reduce the naive estimate meaningfully rather than just accepting the linear scale-up.

Structured elaboration

Assumptions to state explicitly: cost per label (say, $0.05-$0.15 for a simple classification label via a crowd platform, higher for something requiring specialized expertise or careful bounding-box-style annotation); throughput per annotator per hour (varies hugely by task complexity, from hundreds of simple labels per hour to single digits for complex expert review); a quality-control overhead factor (double-labeling a subset, rejecting and relabeling a fraction, typically adding 20-40% to the naive cost/time estimate).

Naive baseline calculation: at $0.10 per label and 1,000,000 labels needed, raw labeling cost is roughly $100,000 before quality-control overhead; at a throughput of, say, 200 labels/hour/annotator, that's 5,000 annotator-hours, which with a team of 20 annotators working 8-hour days is roughly 31 working days, before accounting for ramp-up, quality review cycles, or any data-collection lag before annotation can even start.

Levers to accelerate and reduce cost:

1. Active learning: rather than labeling 1,000,000 examples randomly, prioritize the most informative ones, potentially reaching comparable model quality with a fraction of the raw label count, directly reducing both cost and timeline.
2. Synthetic data: generate additional training examples for at least some portion of the target distribution, reducing how many REAL labels are strictly necessary, though with the fidelity and privacy caveats covered elsewhere.
3. Data partnerships: licensing or partnering for an already-labeled dataset covering part of the target distribution can shortcut a meaningful fraction of the raw labeling effort, if a suitable partner or existing dataset exists.

Worked example

A revised plan combining active learning (targeting 300,000 carefully-selected labels instead of 1,000,000 for comparable model quality) and a data partnership covering an additional 200,000 examples might bring the REMAINING gap down to 500,000 needed labels, roughly halving both the naive cost ($50,000 plus QC overhead) and timeline (roughly 15-16 working days at the same throughput and team size assumptions) relative to the naive baseline.

Trade-offs and pitfalls

The estimate's honesty depends entirely on how defensible the underlying assumptions are: presenting a single point estimate without stating cost-per-label, throughput, and QC-overhead assumptions explicitly invites (deserved) skepticism, while showing the assumptions AND a sensitivity range (what the estimate looks like at a pessimistic versus optimistic cost-per-label) demonstrates the kind of estimation rigor that's actually being tested by this question, distinct from just landing on a plausible-sounding final number.