Handling Class Imbalance Questions

Addressing scenarios where one class significantly outnumbers others (common in fraud, churn, disease detection). Problems: accuracy becomes misleading (95% accuracy can be trivial if 95% are negative class), model biased toward majority class. Solutions: Resampling (undersampling majority, oversampling minority, or SMOTE), adjusting class weights in loss function, choosing appropriate metrics (F1, precision-recall instead of accuracy), ensemble methods. For junior level, recognize imbalance problems, understand why accuracy fails, and know multiple approaches to handling it.

EasyTechnical

51 practiced

Contrast unsupervised anomaly detection with supervised imbalanced classification. For a fraud detection product with very limited labeled fraud samples, discuss when an unsupervised anomaly detector is preferable, and how you would evaluate and combine both approaches in production.

Sample Answer

Unsupervised anomaly detection vs supervised imbalanced classification — key contrast:- Unsupervised anomaly detection: trains on (mostly) normal data to identify outliers by density, reconstruction error, or distance (e.g., isolation forest, autoencoders). No fraud labels required; flags novelty.- Supervised imbalanced classification: learns discriminative patterns from labeled fraud/non-fraud examples, often requires class-weighting, resampling, or specialized losses (focal loss) to handle skew. Strong when labels are sufficient and representative.

When to prefer unsupervised for fraud with very few labeled frauds:- Labels are scarce, stale, biased, or unrepresentative of emerging fraud types.- You need fast detection of novel attacks where labeled supervision can’t anticipate patterns.- You have abundant clean transactional data to model normal behavior.

How to evaluate and combine in production:- Evaluation: - Use precision@k, recall@k, and PR-AUC (precision-recall is better under heavy class imbalance) plus business KPIs (false positives cost, investigator throughput). - Create realistic held-out test sets: curated labeled frauds, synthetic/seeded frauds, and time-split backtests to measure drift. - Use human-in-the-loop labeling to expand ground truth over time.- Combining approaches: - Ensemble / score fusion: compute anomaly score and supervised fraud probability; combine via weighted score, logistic stacking model, or rule-based prioritization (e.g., anomaly score triggers investigation when supervised prob < threshold). - Use unsupervised detector as a recall-oriented layer (catch novel cases) and supervised model for precision (confirm known patterns). - Implement a feedback loop: human-reviewed alerts are fed back to retrain supervised model and to tune anomaly thresholds. - Deploy with monitoring: track score distributions, label arrival rates, concept drift, and run periodic A/B tests to measure net fraud reduction and operational cost.Trade-offs: unsupervised yields better novelty detection but more false positives; supervised yields higher precision for known frauds but misses new modes. Hybrid systems balance recall and precision while enabling continuous learning.

MediumTechnical

48 practiced

Describe a robust algorithm to choose a decision threshold that maximizes recall subject to a minimum precision constraint (e.g., precision >= 90%). Provide step-by-step procedure using validation predicted probabilities and explain how to make it robust under class imbalance and over cross-validation folds.

Sample Answer

Situation: You have a classifier that outputs predicted probabilities on a validation set and you need a single decision threshold that maximizes recall while guaranteeing precision >= 90%, robust to class imbalance and variability across folds.

Step-by-step procedure

1. Prepare validation data- Use stratified K-folds so each fold preserves class ratio.- If probabilities are uncalibrated, calibrate them (Platt scaling or isotonic) using a held-out calibration fold or nested CV.

2. Per-fold threshold selectionFor each validation fold:- Compute precision-recall pairs by sweeping thresholds t from 0→1 (or use unique predicted probabilities sorted desc).- Build the precision-recall curve: for each t compute precision(t), recall(t).- Determine feasible thresholds T_feasible = {t | precision(t) >= 0.90}.- If T_feasible non-empty, pick t* that maximizes recall(t) among T_feasible. Record recall(t*), precision(t*), t*.- If none meets precision constraint, record t* = min threshold (conservative fail-safe) or mark fold as infeasible.

3. Aggregate across folds (robustness)- You now have K thresholds {t*_i}. Avoid pooling raw predictions across folds (data leakage) — aggregate the selected t*_i.- Choose final threshold t_final as a conservative summary: e.g., median(t*_i) or lower quantile (25th percentile) to ensure precision constraint under variation. Alternatively choose the threshold with best worst-case performance: pick t that maximizes average recall subject to min_precision >= 0.90 across folds.- Also compute cross-fold statistics: mean recall, std, and min precision. If min precision < 0.90 for chosen t_final, move to a more conservative quantile.

4. Uncertainty & robustness- Use bootstrap over validation samples within each fold to compute confidence intervals for precision/recall at candidate thresholds.- If class imbalance is extreme, report metrics per-class and use class-weighted sampling during calibration/validation to ensure small-class behavior is well estimated.- Consider using interpolated precision-recall curve (monotonic) to avoid zig-zag due to discrete thresholds.

5. Final validation & deployment- Validate t_final on a holdout test set (never used during threshold tuning). Check precision >=90% with CI; if violated, back off threshold.- In production, monitor precision and recall drift; implement automatic alerting and periodic re-tuning (e.g., monthly or when data distribution shifts).

Why this works- Precision-recall is the right tool under class imbalance.- Per-fold selection plus a conservative aggregation guards against overfitting to a single split.- Calibration and bootstrapping quantify uncertainty so you don't choose thresholds that look good due to sampling noise.

Practical tips- Use sorted unique probs to limit thresholds; vectorized computation is fast with numpy/pandas.- If you need deterministic behavior, store t_final and the procedure used to derive it in model metadata.- If multiple operating points are acceptable, expose a small set (conservative, balanced, aggressive) and choose at runtime based on business risk.

EasyTechnical

55 practiced

Explain the tradeoff between precision and recall. Provide three concrete real-world scenarios where precision should be prioritized and three where recall should be prioritized. For a disease screening system describe which to prioritize and the consequences of the choice.

Sample Answer

Precision measures the proportion of positive predictions that are correct (TP / (TP+FP)); recall (sensitivity) measures the proportion of true positives that are found (TP / (TP+FN)). The tradeoff arises because moving the classifier threshold higher usually increases precision (fewer false positives) but reduces recall (more false negatives), and lowering the threshold does the opposite. The right balance depends on the costs of FP vs FN.

When to prioritize precision (avoid false positives):- Email spam filter for a corporate inbox: FP = important emails marked spam → missed business. Prefer precision to avoid blocking valid mail.- Fraud alert system that triggers account suspension: FP causes customer friction and support costs, so high precision avoids unnecessary locks.- Legal document e-discovery: marking non-relevant documents as relevant wastes lawyer time; prioritize precision.

When to prioritize recall (avoid false negatives):- National security/airport screening for weapons: missing a threat (FN) is catastrophic, so maximize recall even with more FPs to inspect.- Search/recommendation for medical literature retrieval: missing a relevant paper harms research; better to surface more (high recall) and let users filter.- Defect detection in safety-critical manufacturing (e.g., airplane parts): missing a defect is dangerous; tolerate false alarms to catch all defects.

Disease screening system: prioritize recall (sensitivity) especially for serious, treatable diseases (e.g., early-stage cancer). Missing a sick patient (FN) delays treatment and increases morbidity/mortality. Consequences of prioritizing recall: more false positives → additional diagnostic tests, patient anxiety, cost. Consequences of prioritizing precision instead: fewer unnecessary follow-ups but higher risk of undetected disease, potentially severe health outcomes. Practical approach: choose high-recall screening followed by a high-precision confirmatory test; set thresholds based on prevalence, downstream test costs, and stakeholder risk tolerance.

HardTechnical

50 practiced

Design a statistical significance testing approach for A/B experiments that measure imbalanced metrics like recall where event rates are low. Explain how to compute required sample size for desired power, how bootstrapping or permutation tests can be used when assumptions are weak, and how to control for sequential testing and multiple comparisons.

Sample Answer

Requirements & goals:- Metric: recall (true positives / actual positives) with low event rate (rare positives).- Desired: detect absolute lift Δ (or relative), at significance α (e.g., 0.05) with power 1−β (e.g., 0.8).- Constraints: non-normality, imbalance, multiple arms, potential sequential looks.

Sample size computation:1. Work on the denominator (actual positives). Let p0 = baseline recall, Npos per arm = number of actual positive events observed. For proportions, approximate with normal on logit or using Wilson; but for rare events use exact/binomial or Poisson approximation.2. For two-arm test comparing recalls p0 vs p1 = p0+Δ, required positive events per arm: Npos = [ (z_{1−α/2}√(p̄(1−p̄)) + z_{1−β}√(p0(1−p0)+p1(1−p1))/2 )^2 ] / (p1−p0)^2 where p̄=(p0+p1)/2. Convert Npos to total users per arm by dividing by expected event rate r (positives per user): Nusers = Npos / r.3. If r is very small, sample size explodes—consider enrichment (targeted sampling), longer test, or composite metrics.

When assumptions are weak — bootstrapping & permutation:- Bootstrapping: resample users (not events) with replacement to preserve user-level correlation, compute recall per arm repeatedly to get empirical distribution of Δ; derive CI and p-value by percentile or bias-corrected methods. Use stratified bootstrap if covariates matter.- Permutation test: pool all user labels, shuffle treatment assignments many times, recompute Δ to build null distribution. This is exact under exchangeability and robust to distributional shape.- For rare events, use Poisson bootstrap or block/cluster bootstrap for dependent data (sessions/users).

Controlling sequential testing & multiple comparisons:- Sequential looks: use alpha-spending (O’Brien–Fleming, Pocock) or group sequential methods; compute adjusted boundaries so interim looks preserve family-wise α. Alternatively, use always-valid p-values / martingale-based tests (e.g., mixture sequential probability ratio tests) to allow continuous monitoring.- Multiple comparisons: if multiple variants or metrics, control false discovery rate (Benjamini–Hochberg) for many metrics, or family-wise error (Bonferroni or Holm) for strict control. For hierarchical testing, pre-specify primary metric (recall) and gate secondary tests on primary significance.Practical notes:- Simulate end-to-end with realistic user-level data generating process (event rate, variance, clustering) to validate power and Type I error.- Report user-level effect, CI, and absolute counts of positives — with rare events, transparency on counts is essential.

EasyTechnical

41 practiced

Define class prior shift and covariate shift. Explain how a change in class priors between training and production can affect predicted probabilities and thresholds, and outline concrete methods to detect and correct prior shift in a deployed model.

Sample Answer

Class prior shift vs covariate shift- Class prior shift: P(y) changes between training and production while the class-conditional features P(x|y) remain (approximately) the same. Example: fewer frauds in production than in training.- Covariate shift: P(x) changes but the conditional label distribution P(y|x) stays the same (or can be corrected by importance weighting).

How changed class priors affect probabilities and thresholds- A discriminative model typically learns P(y|x) that depends on training priors. If production P(y) differs, predicted probabilities will be biased: odds scale by ratio of new/prior class priors (via Bayes’ rule).- Concretely, for binary labels: P_prod(y=1|x) ∝ P_train(y=1|x) * P_prod(y=1)/P_train(y=1) after renormalization. If you don’t adjust, decision thresholds tuned on training data will produce wrong precision/recall in production (e.g., too many false positives if positive class becomes rarer).

Detecting prior shift in deployment- Monitor class prevalence for labeled production data and compare to training priors.- When labels are scarce: use mixture-proportion / mixture-density or classifier-based estimation (train a binary classifier to distinguish training vs production X; calibration of its outputs can estimate prior change).- Statistical tests on score distributions (KS test, PSI — population stability index) and monitoring downstream business metrics (conversion, false positive rates).- Estimate P(x|y) stability by checking feature distributions conditioned on labels when labels available.

Correcting for prior shift- Direct prior adjustment (Bayes correction): adjust model scores via P_prod(y|x) = P_train(y|x) * (P_prod(y)/P_train(y)) / Z(x) where Z(x) normalizes to [0,1]. Requires estimate of P_prod(y).- Re-calibrate using recent labeled production data (Platt scaling / isotonic) after retraining if needed.- Importance reweighting: reweight training examples by ratio of new/prior class weights when retraining.- EM / iterative methods (Saerens et al.): estimate new class priors from unlabeled data and update posterior probabilities iteratively.- Operational strategies: periodically retrain with up-to-date labeled data, update thresholds based on target operating point (precision/recall) using recent labels, and add guardrails (adaptive thresholds, human-in-loop for low-confidence cases).

Best practices- Instrument pipelines to log scores, predicted probabilities, and eventual labels; compute running priors, PSI, KS, and business KPIs.- Automate alerts when priors or score distributions drift beyond thresholds and trigger recalibration or retraining.- Combine quick prior-correction (Bayes scaling) with periodic retraining and calibration for robust production performance.

Unlock Full Question Bank

Get access to hundreds of Handling Class Imbalance interview questions and detailed answers.

Join thousands of developers preparing for their dream job.