Handling Class Imbalance & Special Modeling Scenarios Questions

Techniques for building and evaluating machine learning models when confronted with imbalanced datasets and other specialized modeling scenarios. Includes data-level methods (oversampling, undersampling, SMOTE and variants), algorithmic approaches (class weights, focal loss, cost-sensitive learning), evaluation strategies and metrics suited for imbalanced problems (precision-recall AUC, F1, balanced accuracy), threshold tuning, calibration, and robust validation (stratified cross-validation). Also covers anomaly/rare-event detection, multi-class and multi-label considerations, and practical production considerations such as model monitoring, fairness implications, and deployment trade-offs in skewed data settings.

EasyTechnical

32 practiced

Describe two common probability calibration methods for binary classifiers (Platt scaling and isotonic regression). When is calibration particularly important in imbalanced problems?

HardSystem Design

27 practiced

Design an end-to-end system to train and serve a multi-label classifier with 1000 labels where label frequency follows a heavy-tailed distribution. Include data storage, training strategies to handle rare labels, efficient prediction serving, and how to support per-label thresholds.

Sample Answer

Requirements & constraints:- 1,000 possible labels, heavy-tailed frequencies, need low-latency predictions and per-label binary decisions (thresholds), retrainable and monitorable.

High-level architecture:- Ingest: Kafka streams / batch ETL into raw event lake (S3 / GCS) + metadata DB (Postgres).- Feature Store: Redis/Feast for online features; Parquet in S3 for offline features and training sets.- Training infra: Kubeflow/Terraform pipelines, GPUs for deep models, CPU pool for classical baselines.- Serving: Online predictor (TF-Serving or TorchServe + Redis cache) behind API gateway and autoscaler.- Monitoring: Prometheus/ELK for latency/throughput + model metrics DB (BigQuery) for label-wise performance and thresholds.

Model & training strategies (handle rare labels):- Base architecture: shared encoder (Transformer/Deep FM/FFNN) with a 1,000-dim sigmoid output layer (multi-label).- Make label-aware representations: label embeddings + attention to model label correlations.- Loss & sampling: - Use class-weighted binary cross-entropy with per-label weights inverse to log(freq) to avoid extreme weights. - Complement with focal loss to focus on hard/rare positives. - Oversample rare-label positive examples in mini-batches (controlled upsampling factor) and use importance-sampling correction to keep calibration.- Two-stage approach for efficiency & improved rare-label recall: - Stage 1: lightweight candidate generator (e.g., linear model or shallow NN) that outputs top-K label candidates per instance very fast. - Stage 2: powerful reranker (deep model) scoring only candidates; reduces computation and improves rare-label focus by boosting candidates from label-similarity or metadata.- Transfer & meta-learning: - Pretrain encoder on large self-supervised task (masked input, contrastive) to help rare-label generalization. - Use label hierarchy or text descriptions to initialize label embeddings (zero-shot assistance).

Training pipeline details:- Offline feature extraction with consistent transformation code (feature store) and dataset slices stratified by label frequency.- Validation: per-label PR-AUC and calibrated probability distributions; use stratified cross-validation ensuring rare-label presence.- Checkpointing, reproducibility, and lineage via ML metadata store.

Serving & latency optimizations:- Cache hot predictions for repeated user queries.- Use sparse output representation: only return positive labels above lightweight thresholds; internally compute dense logits but only materialize top-K using partial topk or approximate nearest neighbor (if label embeddings).- Two-stage serving ensures <100ms: candidate generator in memory/Redis, reranker in GPU pool for batch requests.- Model distillation: compress deep model into smaller student for low-latency edge serving, keep teacher for offline re-ranking.

Per-label thresholds & calibration:- Calibration: - Use temperature scaling or isotonic regression per-label on held-out calibration set to produce well-calibrated probabilities (store calibration params per label).- Threshold selection: - Compute per-label thresholds by optimizing business metric (F1, precision@k, or expected utility) using validation data; store thresholds in a Threshold DB (keyed by label + model version). - Support dynamic thresholds: online A/B experiments or recreation based on drift detection.- Serving flow: - After reranker outputs calibrated probabilities, apply per-label threshold lookup; optionally apply business rules (min support, margin over baseline).- Retraining thresholds: automated weekly/rolling recalibration pipeline; track threshold stability and push alerts when large changes occur.

Monitoring, feedback & continual learning:- Track label-level metrics: precision, recall, PR-AUC, count of positives. Use alerting for performance drops especially on rare labels.- Log false negatives/positives for human-in-the-loop review; use active learning to request labels for ambiguous/rare cases and feed back to training store.- Maintain model versions and rollback capability; shadow deployment to validate behavior before full rollout.

Trade-offs & considerations:- Two-stage reduces compute but adds complexity and potential latency from orchestration.- Oversampling + weighting improves rare recall but can harm calibration—compensate with importance correction and calibration steps.- Per-label calibration scales (1,000 calibrators manageable), but for many more labels consider grouping by frequency or label clusters.

Why this works:- Shared encoder + label embeddings leverages commonalities; pretraining and transfer help rare labels.- Two-stage serving balances latency and accuracy.- Per-label calibration and thresholds provide business-aligned binary decisions while monitoring and active learning sustain performance over time.

MediumTechnical

34 practiced

Design a small offline experiment to test whether combining SMOTE with Tomek link cleaning improves generalization on a held-out imbalanced test set. Include hypothesis, experimental steps, metrics, and decision criteria.

Sample Answer

Hypothesis:Applying SMOTE followed by Tomek link cleaning on the training data will improve model generalization (better discrimination and precision on minority class) on a held-out imbalanced test set versus (a) no resampling and (b) SMOTE alone.

Experiment steps:1. Data split: Hold out a stratified test set (e.g., 20%). Use the remaining data for repeated stratified k-fold CV (e.g., 5x5) for model selection and estimate variance. Fix random seeds.2. Pipelines (apply resampling only inside training folds): - Baseline: standard preprocessing → model - SMOTE: preprocessing → SMOTE → model - SMOTE+Tomek: preprocessing → SMOTE → TomekLinks → model Use identical preprocessing and hyperparameter tuning for all pipelines (tune within CV).3. Model choice: 2 representative models (e.g., XGBoost and logistic regression) to check robustness.4. Metrics: - Primary: PR-AUC (focuses on minority class), and average precision. - Secondary: ROC AUC, F1 at business threshold, recall at fixed precision, calibration (Brier score). - Resource: confusion-matrix-derived costs if business cost available.5. Statistical evaluation: - Within CV: collect metric distributions across folds. - On holdout: train final models on full training set (with chosen hyperparams) and evaluate on test. - Test significance: bootstrap the difference in primary metric between treatments (e.g., 10k bootstrap samples) and report 95% CI and p-value; compute effect size (Cohen’s d or median difference).6. Robustness checks: - Vary SMOTE ratio and Tomek aggressiveness. - Repeat with different seeds, and inspect minority-class synthetic sample characteristics.7. Sanity checks: ensure no leakage (resampling only on training), monitor class overlap after Tomek, verify model calibration.

Decision criteria:- Accept SMOTE+Tomek improvement if on the held-out test: - Primary metric (PR-AUC) increases by a practically meaningful margin (predefine, e.g., Δ ≥ 0.02) AND - Bootstrap 95% CI for the difference excludes 0 (p < 0.05) OR effect size is medium/large.- If primary metric ties, prefer method with better recall at target precision or lower Brier score.- If improvements vary by model or are not robust across seeds/hyperparams, conclude no reliable benefit and recommend further targeted experiments (e.g., different oversampling strategies or cost-sensitive learning).

Notes / pitfalls:- Never resample before splitting; tune hyperparams inside CV; watch for overfitting to synthetic samples; interpret gains in business context (false-positive cost).

EasyTechnical

34 practiced

Given a pandas DataFrame with columns 'features' and 'label', show how to perform a stratified train-test split in Python ensuring the test set contains at least N positive examples. Include code and brief explanation.

Sample Answer

Approach: if you need the test set to contain at least N positive examples, do a controlled split by (1) checking there are >= N positives, (2) sampling N positives into the test set, (3) sampling negatives (or additional positives if you want a larger test set) to achieve desired test size or class balance, and (4) putting the rest into train. This gives deterministic control while preserving stratification for the required positive count.

python

import pandas as pd
from math import ceil

def stratified_split_min_positive(df, label_col='label', test_frac=0.2, min_positives=50, random_state=42):
    """
    Returns train_df, test_df such that test_df contains at least min_positives
    examples with label==1. If not enough positives in dataset, raises ValueError.
    test_frac is target fraction of total rows for test set (approx).
    """
    rng = df.sample(frac=1, random_state=random_state)  # shuffle
    positives = rng[rng[label_col] == 1]
    negatives = rng[rng[label_col] == 0]

    if len(positives) < min_positives:
        raise ValueError(f"Not enough positive examples: found {len(positives)}, need {min_positives}")

    total = len(df)
    target_test_size = max(ceil(test_frac * total), min_positives)  # ensure at least min_positives overall
    # take min_positives positives first
    test_pos = positives.sample(n=min_positives, random_state=random_state)
    remaining_pos = positives.drop(test_pos.index)

    # fill remaining slots in test with negatives (or more positives if negatives are scarce)
    remaining_slots = target_test_size - len(test_pos)
    if remaining_slots <= 0:
        test_df = test_pos
    else:
        take_neg = min(len(negatives), remaining_slots)
        test_neg = negatives.sample(n=take_neg, random_state=random_state)
        remaining_slots -= take_neg

        # if still need slots, take more positives
        extra_pos = remaining_pos.sample(n=remaining_slots, random_state=random_state) if remaining_slots > 0 else pd.DataFrame(columns=df.columns)
        test_df = pd.concat([test_pos, test_neg, extra_pos])

    train_df = df.drop(test_df.index)
    return train_df.reset_index(drop=True), test_df.reset_index(drop=True)

# Example usage:
# train, test = stratified_split_min_positive(df, label_col='label', test_frac=0.2, min_positives=100)

Key points:- This guarantees at least min_positives in test without repeated random splits.- If you care about exact class ratio in test (stratification), you can adapt the number of negatives/positives sampled to match desired proportion; here we prioritize the min positives constraint.- Complexity: O(n) time and O(n) space (shuffling and sampling).Edge cases:- Less than min_positives available -> raise error.- Very small dataset may force test fraction to exceed intended; we set target to at least min_positives.- If labels are not binary, filter/select appropriate class values.

EasyTechnical

32 practiced

How would you explain the concept of balanced accuracy and where it is useful versus regular accuracy? Provide an equation for balanced accuracy and an example where it reveals hidden performance issues.

Unlock Full Question Bank

Get access to hundreds of Handling Class Imbalance & Special Modeling Scenarios interview questions and detailed answers.

Join thousands of developers preparing for their dream job.