Data Preparation and Class Imbalance for ML Questions

Direct answer

Behavioral answer, using the STAR structure: describe a specific situation where a preprocessing or data-quality decision (an imputation choice, a resampling strategy, a fix to a broken date-parsing routine) turned out to materially change a model's behavior or a business outcome, and walk through the reasoning, the validation, and the lesson learned.

Structured elaboration

Situation: set up the concrete context, what model or report was affected and why it mattered (a churn model whose predictions were being used to prioritize a retention team's outreach).

Task: what you were responsible for, specifically (diagnosing why the model's predictions seemed off, or improving a pipeline you inherited).

Action: the actual preprocessing decision and the reasoning behind it, described concretely (discovering that a "days since last purchase" feature was being computed with a bug that included future purchases for churned customers, a leakage bug; or deciding that median imputation on an income field was distorting predictions for a specific segment and switching to a grouped, segment-aware imputation instead).

Result: the measurable outcome, tied to real numbers where possible (validation AUC improved from X to Y, or the retention team's outreach precision improved by Z%, or a specific downstream business metric moved), plus what you validated to confirm the improvement was real (an A/B test, a time-based holdout, not just a training-set number).

Worked example

"I inherited a churn model whose predictions correlated suspiciously well with a 'days_to_cancel' feature. Tracing the feature's derivation, I found it was computed from the FULL customer history including records after cancellation, a leakage bug. I removed the feature, retrained, and validation AUC dropped from an unrealistic 0.94 to a more modest but honest 0.79. I then worked with the team to build genuinely predictive features from PRE-cancellation behavior only, recovering AUC to 0.83 over the following two weeks, a number the team could actually trust for prioritizing outreach, versus the original number which would have looked great in a demo and failed in production."

Trade-offs and pitfalls

The strongest version of this answer names a decision that could have gone WRONG, and shows the reasoning and validation that confirmed it was right, rather than a story where the "right" choice was obvious in hindsight; interviewers are listening for genuine judgment under uncertainty, not just a happy ending.

Direct answer

A pre-modeling checklist walks from raw ingestion through readiness for training: verify the schema and types, quantify data quality (missingness, outliers, duplicates, label noise, class imbalance, and distribution differences across important slices), then move into missing-value handling, encoding, scaling, splitting, all while keeping the whole sequence reproducible and leakage-free.

Structured elaboration

1. Ingestion and schema checks: confirm column types match expectations, no unexpected nulls in required fields, row counts match the source system, and timestamps parse correctly.
2. Data-quality assessment, with a concrete check per dimension:
   • Missingness: per-column missing rate and whether it's concentrated in specific segments.
   • Outliers: quick IQR or z-score scan per numeric column.
   • Duplicates: exact and fuzzy duplicate row detection.
   • Label noise: spot-check a sample of labels against source truth if possible, or check inter-annotator agreement if labels came from multiple sources.
   • Class imbalance: the positive rate for the target, and whether it matches what you'd expect from domain knowledge.
   • Distribution differences across slices: does the feature distribution look meaningfully different across time periods, regions, or other important segments (a sign of drift or a data-collection artifact worth investigating before training).
3. Missing-value handling, encoding, scaling: apply consistently, using only training-fold statistics.
4. Splitting: stratified and/or time-aware as appropriate.
5. Reproducibility and leakage discipline throughout: every statistic-computing step lives inside a Pipeline scoped to the training fold, every transform is logged with its parameters, and random seeds are fixed.

Automation at scale means these checks run as code (a schema-validation library, a profiling tool, or custom assertions), not manual spreadsheet inspection, so they can run on every new data drop without someone re-eyeballing it each time.

Worked example

For a fresh 10-million-row, 200-column tabular dataset: a quick automated pass might flag that 3 columns exceed 40% missingness (candidates for dropping or a missingness indicator), 1 column has an unexpected string type where a numeric was expected (a schema bug worth fixing at the source before proceeding), and the target's positive rate is 0.3%, well below the naive assumption of "roughly balanced" someone might have started with, immediately reframing the rest of the modeling plan around imbalance-handling.

Trade-offs and pitfalls

The single highest-value habit in this whole checklist is running it BEFORE any modeling decision is made, since discovering an unexpectedly extreme positive rate, a data-type bug, or heavy missingness after you've already built a model wastes far more time than catching it up front; treat this pass as a hard prerequisite gate, not an optional nice-to-have.

Direct answer

99.9% accuracy with zero recall on a 0.05% positive class means the model learned the cheapest possible solution: always predict negative. It isn't broken, it's optimizing exactly what you asked it to (minimize overall error), and overall error is dominated by the 99.95% of examples that are negative.

Structured elaboration

Diagnose first, then fix. Confirm the model is actually predicting all-negative (check the distribution of predicted labels, not just accuracy) rather than something more subtle like a broken feature or a leaked label that happens to correlate with the majority class. Once confirmed, three concrete changes address the root cause from different angles:

1. Change what the model is optimized for. Random Forest's default split criterion and vote counting have no notion that a false negative is expensive; passing class_weight='balanced' reweights each sample so minority mistakes cost proportionally more during training.
2. Change the data the model sees. Oversample the minority class (or undersample the majority) so the model doesn't have the option of ignoring the rare class to win on aggregate error.
3. Change how you turn probabilities into decisions. Even a model that DOES separate the classes reasonably well in its predicted probabilities can still show zero recall at the default 0.5 threshold, because 0.5 was never a meaningful cutoff for a 0.05% base rate. Lowering the threshold, chosen via the validation set against a recall or cost target, is often the single highest-leverage fix and requires no retraining.

Worked example

Suppose the model's predicted probabilities for the 5 true positives in a 10,000-row test set are $[0.02, 0.08, 0.15, 0.30, 0.55]$. At the default threshold of 0.5, only the last one crosses it, so recall is $1/5 = 0.2$, still poor but not literally zero, illustrating that the "zero recall" you're diagnosing is often a threshold artifact layered on top of a genuinely under-trained minority signal, not necessarily a model that learned nothing at all.

Trade-offs and pitfalls

Controlling false positives while improving recall is the real balancing act: class weighting and resampling both trade some precision for recall, and the right amount depends on the actual cost of a false positive in your application. The common mistake is stopping at "I improved recall" without checking what happened to precision and whether the new operating point is one the business can actually act on.

Direct answer

Median imputation disproportionately affecting predictions for a protected group is a real fairness signal worth a structured audit: quantify how much the imputation choice specifically (not the model overall) drives the disparity, test alternative preprocessing strategies designed to be more fair, and present the accuracy-versus-fairness trade-off explicitly to stakeholders rather than resolving it unilaterally.

Structured elaboration

Step 1, quantify the harm: isolate the imputation step's specific contribution by comparing model predictions (or a fairness metric like demographic parity or equalized odds) for the protected group under the current global-median imputation versus under a baseline that doesn't touch this feature at all (drop it, or use only complete cases), confirming the disparity is genuinely attributable to the imputation choice and not some other factor entirely.

Step 2, test alternative strategies:

• Group-wise imputation: compute the median SEPARATELY within each protected-group subpopulation, so no group's missing values get filled with a value drawn from a different, systematically-different population's typical value.
• Missingness indicator: add an explicit "was missing" flag alongside the imputed value, letting the model learn any differential effect of missingness itself rather than having it silently absorbed into a single, potentially misleading imputed number.
• Model-based imputation: predict the missing value from OTHER features (which may or may not include group membership, itself a further ethical judgment call worth surfacing explicitly to stakeholders rather than deciding silently), potentially producing more individually-accurate imputations than a single group-level statistic.

Step 3, present trade-offs to stakeholders: quantify, for each alternative, both the fairness improvement (how much the disparity shrinks) AND any accuracy cost (does overall model performance change), since a stakeholder needs both numbers to make an informed decision about which trade-off point the organization is comfortable with, this isn't a purely technical decision to resolve alone.

Worked example

If global median imputation for "income" uses a value drawn predominantly from the majority group (because the majority group makes up most of the dataset), and the protected group's TRUE income distribution differs meaningfully, switching to group-wise median imputation might close a substantial fraction of a measured fairness gap (say, reducing a demographic-parity gap from 0.15 to 0.06) while costing only a small amount of overall accuracy (say, 0.3 points of AUC), a trade-off most stakeholders would consider clearly worth making once they see both numbers side by side.

Trade-offs and pitfalls

Group-wise imputation itself carries a subtler fairness question worth surfacing, not hiding: computing statistics SEPARATELY by protected-group membership means the model's treatment of missing values now explicitly depends on group membership, which is exactly the kind of design decision that should be documented and discussed with stakeholders (and possibly legal/compliance, depending on the domain) rather than treated as an obviously-correct technical fix, since "using group membership to compute a fairer statistic" and "using group membership as a direct model input" sit in genuinely different, and sometimes contested, ethical and regulatory territory depending on context.

Direct answer

Compute each user's own mean and standard deviation over their trailing history using window functions, flag a transaction when it falls far enough outside that user-specific range, and handle new or thin-history users explicitly rather than letting a near-empty window silently produce a meaningless flag.

Structured elaboration

WITH user_stats AS (
    SELECT
        transaction_id,
        user_id,
        amount,
        occurred_at,
        AVG(amount) OVER (
            PARTITION BY user_id
            ORDER BY occurred_at
            ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING
        ) AS user_avg_amount,
        STDDEV(amount) OVER (
            PARTITION BY user_id
            ORDER BY occurred_at
            ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING
        ) AS user_stddev_amount,
        COUNT(*) OVER (
            PARTITION BY user_id
            ORDER BY occurred_at
            ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING
        ) AS user_prior_count
    FROM transactions
)
SELECT
    transaction_id, user_id, amount, occurred_at,
    user_avg_amount, user_stddev_amount, user_prior_count
FROM user_stats
WHERE user_prior_count >= 5   -- require enough history before trusting the flag at all
  AND user_stddev_amount > 0  -- avoid a division-by-zero-equivalent flag on a perfectly flat history
  AND amount > user_avg_amount + 5 * user_stddev_amount

Using ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING (rather than including the current row) means each transaction is flagged against the user's history strictly BEFORE it, avoiding the circularity of a transaction contributing to its own comparison baseline; the user_prior_count >= 5 and user_stddev_amount > 0 filters explicitly handle the "reasonable behavior" requirement for thin-history users, a brand-new user's second transaction has no meaningful personal baseline yet, and the query deliberately declines to flag anything for them rather than producing an unreliable flag from a near-empty or perfectly-uniform history.

Worked example

For a user with a stable history averaging $50 with a $10 standard deviation, a new $300 transaction (over 5 standard deviations above their mean) gets flagged; a brand-new user's first 4 transactions never get evaluated at all (excluded by the user_prior_count >= 5 filter), avoiding a meaningless flag computed from almost no personal history.

Trade-offs and pitfalls

Performance on a large table: partitioned window functions like this scale reasonably well with a good index on (user_id, occurred_at), since the database can compute each partition's running statistics in a single ordered pass per user rather than a full unindexed scan; for a genuinely enormous transaction table, materializing the user-level running statistics incrementally (updated as new transactions arrive, rather than recomputed from full history on every query) is a further optimization worth considering once this naive version's performance becomes a real bottleneck.