Model Diagnosis & Improvement Questions

Techniques for diagnosing underperforming machine learning models, including error analysis, feature importance assessment, data quality checks, model monitoring, drift detection, hyperparameter tuning, retraining strategies, and deployment considerations.

MediumTechnical

54 practiced

For a high-firehose streaming feature with thousands of updates per second, propose an efficient approach to detect covariate distribution change. Discuss use of PSI, KS test, Earth Mover's Distance, and streaming/approximate techniques such as quantile sketches and histograms. Explain choices for sampling, window size, thresholds, and how to make the approach compute- and memory-efficient.

Sample Answer

Approach (summary): use a two-tier streaming pipeline: lightweight continuous monitoring with approximate summaries + on-demand exact tests when an alarm fires. This balances cost and sensitivity.

1) Streaming summaries (continuous, low-cost)- Maintain quantile sketches (e.g., t-digest or KLL) per feature for reference and sliding-window current distributions. These are mergeable, memory-efficient (KBs per feature), and support approximate CDF queries.- Also keep compact histograms (adaptive fixed-bin or streaming histograms) for quick EMD approximations.

2) Alarm criteria (cheap checks)- Compute fast distance proxies from sketches each evaluation window: L1 between histogram approximations, approximate KS via max CDF difference from t-digest, and a cheap EMD estimate by cumulative bin differences. If any exceeds coarse thresholds, trigger deeper analysis.

3) Deep checks (on alarm)- Sample recent raw points (stratified by key if needed) and run: - PSI for binned comparisons (good for discrete or binned numerical features; interpretable contribution per bin). - Two-sample KS test (nonparametric CDF test) for continuous features. - Exact EMD (Wasserstein-1) if shift in support/ordering matters.

Design choices:- Window size: use dual windows — a long-term reference (weeks) and short sliding window (minutes/hours) depending on traffic and model sensitivity. Choose window length so sample size gives statistical power (e.g., short window >= 1k–10k samples).- Sampling: reservoir or Poisson sampling at ingress to bound downstream cost; stratify by critical subpopulations to catch localized drift.- Thresholds: set coarse thresholds for sketches conservatively (avoid noise) using historical null distribution of distances; for statistical tests use alpha corrected for multiple features (Benjamini-Hochberg) and effect-size thresholds (e.g., PSI > 0.1 moderate, >0.25 large).

Compute & memory efficiency:- Keep sketches per feature keyed by model or segment; merge at Kafka/stream processors (Flink/Beam) to distribute work.- Use approximate tests as gating to avoid running O(N) exact tests every window.- Parallelize deep checks off the hot path and persist only summaries; use bloom filters for categorical high-cardinality features to detect new categories cheaply.- Monitor false alarm rate and adjust sampling/thresholds; log contributory bins from PSI to aid debugging.

Why these choices:- t-digest/KLL provide accurate tail/quantile estimates for KS/percentile checks with tiny memory.- PSI gives interpretable bin-wise contributions; KS is distribution-agnostic; EMD captures transport-like shifts affecting order.- Combining approximate streaming checks with occasional exact tests yields scalable, sensitive, and actionable drift detection suitable for high-throughput production.

EasyTechnical

87 practiced

Explain how learning curves can help you decide whether to gather more training data or to increase model capacity. Describe the typical learning curve patterns for underfitting and overfitting, how to interpret the gap between training and validation errors, and recommended actions in each scenario.

MediumTechnical

48 practiced

A model's average inference latency tripled after a new release. Describe a step-by-step debugging plan you would follow to isolate whether the regression is due to the model, serving infrastructure, preprocessing, or input distribution changes. Include telemetry to capture, profiling tools or metrics to run, isolation experiments, and quick mitigations to reduce latency while you investigate.

Sample Answer

1) Triage & collect baseline telemetry (immediately)- Capture per-request latency histograms (P50/P90/P99), CPU/GPU utilization, memory, I/O, network, queue lengths, concurrency, batch sizes, and request rate. Use Prometheus/Grafana, Cloud Monitoring, or Datadog.- Log model version, serving container id, hardware, input payload size, and trace IDs. Enable distributed tracing (OpenTelemetry/Jaeger) to get per-stage durations.

2) Narrow scope: model vs infra vs input vs preprocessing- Compare latency distributions by model version (new vs previous) using logs/traces.- Break down total latency into stages: client -> ingress -> preprocessing -> model inference -> postprocessing -> egress. Traces should show which stage increased.

3) Profiling per-stage- Preprocessing: add micro-benchmarks for parsers, feature transforms; run CPU profiler (py-spy, perf) to find hotspots.- Serving infra: collect container/OS metrics, run pprof/ebpf/tcpdump for syscalls/network stalls; check GC pauses (JVM/Python) and cold-start times.- Model inference: run the new model in isolation on identical hardware with synthetic inputs; use framework profilers (torch.autograd.profiler / torch.profiler, TensorFlow profiler, NVIDIA Nsight / nvprof) and generate flamegraphs to find ops causing spikes.- I/O: measure storage latency if model loads artifacts or embeddings.

4) Input-distribution check- Compare recent inputs vs historical: payload sizes, feature distributions, token lengths. Compute simple KS/Chi-square tests for major features; watch for outliers or long sequences causing longer runtimes.

5) Isolation experiments- Canary rollback: route a fraction of traffic to previous release and compare latency to confirm regression.- A/B test: run new model behind same preprocess and serving infra but with identical inputs to previous model (replay logs).- Local replay: replay production requests against (a) old model on new infra, (b) new model on old infra, (c) new model in isolated machine — this isolates model vs infra.- Disable nonessential preprocessing/postprocessing to see impact.

6) Quick mitigations (while investigating)- Route traffic to previous stable release (rollback or higher weight to old canary).- Reduce concurrency or batch sizes to avoid queuing; or temporarily increase replicas/CPU/GPU to reduce queuing.- Enable CPU/GPU pinning and increase resource requests to prevent throttling.- Apply input validation to drop/reject extreme inputs or limit sequence length.- Use async responses/timeouts or degrade gracefully (serve cached responses or fallback model).- Deploy a lighter quantized/ distilled model if acceptable.

7) Root-cause confirmation & fix- Once a culprit is found (e.g., a new op in model causing long GPU kernel), produce a reproducible test, patch (optimize model graph, fuse ops, change operator implementations, fix memory copies), and run stress tests.- Add new telemetry: op-level latency, queue time, input-size histograms, and alerting on P90 regressions.

8) Postmortem & prevention- Document cause, fix, and test harness.- Add pre-deploy performance tests (replay load, perf budgets), automatic canary latency comparisons, and run model profilers as part of CI.

Key tools: Prometheus/Grafana, OpenTelemetry/Jaeger, torch/tf profilers, py-spy/perf, nvprof/Nsight, pprof/ebpf, load-testing and replay tools.

MediumTechnical

50 practiced

Implement in Python a function 'walk_forward_cv_split(dates, n_splits, min_train_period, test_period)' that yields train and test index pairs for time-series walk-forward validation. Ensure no leakage by disallowing future data in training windows and handle edge cases where data is sparse or dates are irregular.

Sample Answer

Approach: convert input dates to pandas Timestamps for robust arithmetic, sort while keeping original indices, then slide a test window of length test_period forward. For each split, use all observations strictly earlier than the test window as training (so no future leakage) and observations inside the test window as test. Enforce min_train_period (time span) and minimum sample counts. Skip splits where train or test would be empty.

python

import pandas as pd
import numpy as np

def walk_forward_cv_split(dates, n_splits, min_train_period, test_period, min_train_size=1, min_test_size=1):
    """
    dates: array-like of datelike objects (strings, datetime, etc.)
    n_splits: number of folds to yield
    min_train_period: pandas offset alias or Timedelta-like (e.g., '90D')
    test_period: pandas offset alias or Timedelta-like (e.g., '30D')
    min_train_size: minimum number of training rows
    min_test_size: minimum number of test rows
    Yields: (train_idx, test_idx) where indices are positions in the original input
    """
    dates = pd.to_datetime(dates)
    order = np.argsort(dates.values)
    sorted_dates = dates.iloc[order].reset_index(drop=False)  # columns: original index, date
    sorted_dates.columns = ['orig_idx', 'date']
    sorted_dates = sorted_dates.reset_index(drop=True)
    
    min_train_td = pd.Timedelta(min_train_period)
    test_td = pd.Timedelta(test_period)
    
    # earliest possible test start: earliest date + min_train_period
    test_start = sorted_dates['date'].min() + min_train_td
    splits_yielded = 0
    # advance test_start so that at least min_train_size exists before it
    while splits_yielded < n_splits:
        # define test window: [test_start, test_start + test_td)
        test_end = test_start + test_td
        # boolean masks on sorted order
        train_mask = sorted_dates['date'] < test_start
        test_mask = (sorted_dates['date'] >= test_start) & (sorted_dates['date'] < test_end)
        train_idx = sorted_dates.loc[train_mask, 'orig_idx'].to_numpy()
        test_idx = sorted_dates.loc[test_mask, 'orig_idx'].to_numpy()
        
        # ensure time span of train satisfies min_train_period and sample size constraints
        if train_idx.size >= min_train_size and test_idx.size >= min_test_size:
            train_span = None
            if train_idx.size >= 1:
                train_span = sorted_dates.loc[train_mask, 'date'].max() - sorted_dates.loc[train_mask, 'date'].min()
            if train_span is None or train_span >= min_train_td or train_idx.size >= min_train_size:
                yield (np.sort(train_idx), np.sort(test_idx))
                splits_yielded += 1
        
        # advance test_start by test_period; if no data in future, break
        next_start = test_start + test_td
        if next_start >= sorted_dates['date'].max():
            break
        test_start = next_start

Key points:- Uses strict "< test_start" for training to avoid leakage.- Handles irregular/sparse dates because selection is by timestamp, not fixed indices.- Skips splits without enough samples or without required training time span.Time complexity: O(n log n) for initial sort, then O(n * n_splits) in worst-case for masks (can be optimized with pointers). Space: O(n).Alternatives: incremental pointer-based implementation to make splits O(n) across all folds; use expanding window with fixed-size test by index rather than time if business needs require.

HardTechnical

84 practiced

You lead ML engineering for a service that trains large models nightly and must reduce carbon footprint while maintaining SLAs. Propose optimizations across model architecture, training practices, infrastructure scheduling and ops such as lower-precision training, reducing unnecessary hyperparameter trials, spot instances, regional scheduling during low-carbon hours, dataset curation, and model selection trade-offs. Include measurement and targets.

Sample Answer

Framework: define constraints (SLA: nightly completion, accuracy delta ≤0.5% absolute, retrain window), measure baseline energy/CO2 per full training run, then optimize across model, training, infra, ops with measurable targets (e.g., 40–60% CO2 reduction, ≤0.5% accuracy loss).

1) Measurement & targets- Baseline: kWh per run, CO2e/run (use cloud provider energy or Watt usage * carbon intensity API), PUE, and cost.- Targets: reduce CO2e/run by 50% in 6 months, keep SLA (nightly completed) and accuracy within 0.5%.- Instrumentation: export per-job metrics to ML observability (experiment tracker) + energy dashboard; tag metadata (region, instance type, precision).

2) Model & dataset optimizations- Model selection trade-offs: prefer smaller or efficient architectures (MobileNet/Vit-lite, transformer distillation) when performance loss is minimal. Use knowledge distillation to keep inference quality while training smaller students.- Neural architecture search constrained by energy budget: include FLOPs/latency penalty in objective.- Dataset curation: deduplicate, remove label noise, stratified subsampling for diminishing returns (use learning curves to find required dataset size); use active learning to prioritize high-value samples—reduces training epochs/data processed.

3) Training practices- Lower-precision: adopt mixed precision (FP16 + dynamic loss scaling) and bfloat16 where supported — validated for stability.- Gradient accumulation to use larger effective batch sizes on smaller/GPU types.- Efficient optimizers and regularization: LAMB/AdamW with fewer steps; layer-wise learning rates to converge faster.- Early stopping & adaptive training: use validation-driven stopping and successive halving (ASHA/Hyperband) to cull trials early.- Reduce unnecessary HPO: switch from exhaustive grid/random to Bayesian HPO with pruning, multi-fidelity HPO (FAIR/BOHB).- Checkpoint warm-starting and transfer learning to avoid training from scratch nightly.

4) Infrastructure & scheduling- Spot/preemptible instances with robust checkpointing and elastic trainer; replicate critical jobs on mixed fleet to meet SLA.- Region-aware scheduling: query electricity grid carbon intensity API and prefer low-carbon regions/times; shift non-urgent retrains to periods with low carbon intensity.- Right-sizing: choose GPU types that minimize energy per training step (measure FLOPS/W). Use multi-node only when scaling reduces time sufficiently to offset added power.- Containerized reproducible jobs, autoscaling worker pools, and placement constraints in orchestration layer.

5) Ops, monitoring & safeguards- CI gating: require energy/CO2 estimate and accuracy delta before promoting models.- Experiment tracking: log energy, runtime, CO2, hyperparams, checkpoints.- Rollback/SLOs: maintain SLA alerting if nightly window at risk; failover to previous model if newer model misses objective.- Automation: automated scheduler which applies policies (spot fallback, region choice, precision) based on current carbon intensity and remaining window.

Example implementation snippets:- Use PyTorch AMP for mixed precision, integrate Ray Tune with ASHA for HPO, and Terraform modules to spin spot fleets. Add carbon-intensity checks via API to Airflow operator.

Reasoning: combine algorithmic efficiency (less compute per model) + smarter searching (fewer wasted trials) + infra-level energy-aware scheduling yields multiplicative reductions. Measurement ensures transparent trade-offs; targets focus engineering effort and enable iterative improvements.

Unlock Full Question Bank

Get access to hundreds of Model Diagnosis & Improvement interview questions and detailed answers.

Join thousands of developers preparing for their dream job.