Problem Solving and Debugging Persistence Questions

Stories about challenging technical problems you have debugged or solved: your systematic approach, the multiple strategies you tried before finding one that worked, when and how you sought help appropriately, and what you learned from failure along the way. Strong answers walk through a specific problem end to end: how you formed and tested hypotheses, what evidence (logs, metrics, reproduction steps) you gathered, how you decided when to escalate versus keep digging, and what you changed afterward so the same class of problem was less likely to recur.

MediumTechnical

52 practiced

How would you debug a GPU Out-Of-Memory (OOM) error during a large model training job that only occurs intermittently on certain nodes? Cover steps including environment checks, profiling memory allocation, data loading patterns, mixed precision options, and identifying memory leaks in custom operations.

Sample Answer

Situation: Intermittent GPU OOMs across some nodes during large-model training.

Approach: methodically rule out environment, data, and code causes; profile allocations; apply mitigations (mixed precision, checkpointing); find leaks in custom ops.

Steps and checks

1) Environment sanity- Confirm CUDA, driver, cuDNN, NCCL, OS, and container images are identical across nodes (nvidia-smi, nvcc --version). Mismatched drivers or CUDA ABI can change allocator behavior.- Check GPU memory available and other processes (nvidia-smi -q -d MEMORY). Look for daemon/monitor processes stealing memory.- Ensure same GPU model and ECC/BIOS settings.

2) Reproduce deterministically- Run the job with a fixed seed and a reduced config (smaller batch/model) on a failing node to reproduce.- Run on a healthy node with same config to confirm node-specific behavior.

3) Profile memory allocations- Use framework tools: - PyTorch: torch.cuda.memory_summary(), torch.cuda.max_memory_allocated(), torch.autograd.profiler.profile(record_shapes=True, use_cuda=True). - NVIDIA: nsight-systems, nsight-compute, or nvprof to see CUDA API allocations. - Set TORCH_CPP_LOG_LEVEL=info or use ENV CUDA_LAUNCH_BLOCKING=1 to catch where error occurs.- Capture memory snapshots before/after key phases (model init, forward, backward, optimizer.step()).

Example (PyTorch):

python

with torch.autograd.profiler.profile(use_cuda=True, record_shapes=True) as prof:
    out = model(batch)
    loss = criterion(out, labels)
    loss.backward()
print(torch.cuda.memory_summary())
print(prof.key_averages().table(sort_by="cuda_time_total"))

4) Data loading and input patterns- Check batch size variability from last batch or uneven sharding in distributed training — last micro-batches can be larger.- Inspect DataLoader workers for leaks: ensure workers exit cleanly, close file handles, and don't retain references to tensors. Try num_workers=0 to test.- Use pin_memory=True appropriately; excessive pinned host memory can have side effects.

5) Mixed precision and optimizer options- Use AMP (torch.cuda.amp) with dynamic loss scaling to reduce memory footprint. Example: GradScaler for PyTorch.- Monitor whether automatic mixed precision increases peak memory in some layers (some kernels use more workspace). Test with and without AMP and with different autocast settings.- Try gradient accumulation to reduce per-step batch size, or gradient checkpointing (torch.utils.checkpoint) to trade compute for memory.

6) Distributed training specifics- Ensure consistent partitioning: check sampler/sharder so nodes get equal-sized sub-batches.- Validate rendezvous and reduce-scatter operations aren’t allocating temporary buffers differently on some nodes (NCCL versions matter).- Check that DDP is initialized correctly and that only model parameters are on device (no accidental device mismatch).

7) Finding memory leaks in custom operations- Inspect custom CUDA ops and kernels: verify every cudaMalloc has corresponding cudaFree, and that returned buffers are freed when no longer needed.- In Python extensions, ensure no lingering references to tensors from C++/Python hold GPU memory. Use weakrefs in callbacks or clear caches.- Use gc.collect() and check refcounts (sys.getrefcount) for tensors after iteration. Example:

python

import gc
del batch, outputs
gc.collect()
torch.cuda.empty_cache()
print(torch.cuda.memory_allocated())

- Check hooks (forward/backward hooks) — they can capture tensors prolonging lifetime; replace with functional code or detach() copies.

8) Edge debugging tips- Run with CUDA_LAUNCH_BLOCKING=1 to get accurate backtraces.- Log torch.cuda.memory_reserved(), memory_allocated() at phases.- If intermittent, run long-lived tests to reproduce leak growth; plot allocated memory per iteration.- If a node shows lower total GPU memory, test kernel workspace sizes (cuDNN heuristics) and disable benchmarking: torch.backends.cudnn.benchmark = False.

Result and reasoning- Follow these steps to narrow cause: environment mismatch (drivers/NCCL) or data/partitioning issues often explain node-specific OOMs; true leaks typically show monotonically increasing allocated memory per iteration and survive gc.empty_cache(). Mixed precision and checkpointing are practical mitigations while you fix leaks.

MediumTechnical

45 practiced

You suspect label leakage is inflating cross-validation scores in a supervised model. Outline concrete tests and experiments to detect leakage, quantify its impact, and fix the issue (for example, time-based splits, forward-chaining validation, feature ablation, and permutation tests).

Sample Answer

Situation: I suspect label leakage because cross-validation scores are unrealistically high and model performance drops in production/holdout.

Concrete tests and experiments to detect leakage, quantify impact, and fix it:

1. Quick detective checks- Inspect feature generation: any feature computed using future data (e.g., rolling aggregates that include the target row), IDs that map directly to labels, or join keys leaking future labels.- Correlation/chi-square between each feature and the label; extremely high single-feature correlation is suspicious.

2. Validation redesign (detect + quantify)- Time-based split / forward-chaining: train on t0..tN, validate on tN+1..tN+k. If CV score drops sharply vs. random CV, likely time leakage.- Out-of-time holdout: keep a completely later period as production-like test. Measure delta in metric: the larger the drop, the larger the leakage effect.- Group-k fold: if data has groups (user, account), use GroupKFold to prevent leakage across same entity.

3. Feature ablation & permutation- Iteratively remove top features (by importance) or ablate suspicious feature sets; measure performance drop. If removing one feature collapses score, it’s likely leaking.- Permutation test: permute a feature across rows and measure metric change; leaking features will drop performance dramatically.- Train model with only suspected features to quantify how much of the score they explain.

4. Data pipeline tests- Ensure preprocessing is fit only on train folds (scalers, encoders). Add unit tests that assert no future timestamps are used in feature creation.- Re-run CV with "train-only transforms" enforced; compare performance.

5. Hardening and fixes- Remove or re-compute leaking features using only past information (lagged features computed up to t).- Use causal/time-aware features; add feature engineering guards (max lookahead windows).- Use conservative validation (forward-chaining, group folds) for model selection and hyperparameter tuning.- Implement monitoring in production: track prediction distribution drift, calibration, and performance on new data.

Expected outcomes / metrics- Report initial CV vs. time-split vs. OOT holdout metrics and per-feature ablation impacts (ΔAUC, ΔRMSE).- A true model should have similar performance on forward-chaining CV and OOT holdout; if not, iterate on removing/recomputing leaking features.

This process both detects leakage (large CV → OOT gaps, single-feature dominance) and quantifies its impact (metric deltas from ablation/permutation), then fixes it via time-aware validation, feature guards, and pipeline unit tests.

HardBehavioral

42 practiced

Describe the most technically challenging debugging problem you've solved involving ML systems. Explain context, hypotheses you tested, tools and experiments used, why it was difficult, how you persisted over time, and what the ultimate outcome and organizational learning were. Highlight technical depth and leadership where relevant.

Sample Answer

Situation: At my previous company we deployed a PyTorch-based fraud-detection model that performed well offline (AUC 0.92) but production metrics dropped ~8% within two weeks and false positives spiked — causing customer complaints and a 12% increase in manual reviews.

Task: As the ML engineer on-call, I needed to find the root cause quickly, stabilize production, and prevent recurrence.

Action:- Immediate triage: I froze model rollouts and enabled a "shadow" mode to replay live traffic to both the old and new models for controlled comparison.- Formed a small cross-functional task force (data engineering, infra, product) and set a 48-hour debugging cadence.- Hypotheses I enumerated and tested: 1) Data drift — compared feature distributions (histograms, KS tests) between training, validation, and live data. 2) Preprocessing mismatch — suspected different featurization in the online pipeline (e.g., categorical hashing vs. training-time lookup). 3) Label leakage/regression — checked whether a new ETL changed timestamps or joined tables differently. 4) Model serving nondeterminism — GPU vs CPU numerical differences (batchnorm, mixed precision). 5) Downstream thresholds/score normalization changed.- Experiments & tools: - Used MLflow and DVC to retrieve exact training artifacts and inputs; replayed serialized preprocessing in a local container. - Instrumented feature logging in production and exported samples to S3 for side-by-side comparison. - Ran statistical tests (SciPy KS, Wasserstein) and feature importance drift metrics. - Created unit tests for preprocessing using pytest and Great Expectations to assert schema and value ranges. - Replayed traffic through a locally-hosted model server (Docker, Kubernetes port-forward) to compare outputs deterministically between CPU/GPU and float32/float16. - Added Prometheus/Grafana dashboards for per-feature anomaly alerts and distribution drift.

Why it was difficult:- The failure manifested only in production traffic patterns (rare edge combinations), making it non-reproducible from offline datasets.- Multiple moving parts: ETL changes shipped by a different team, an infra upgrade that switched serving instances to mixed-precision, and a hidden fallback in the online encoder for unseen categories.- The bug was subtle: a recent change in the online categorical encoder replaced unknown tokens with an index that collided with an existing category used during training, causing systematic misrepresentation of several high-weight features.

Persistence & debugging over time:- I maintained a prioritized hypothesis board, ran quick A/B experiments (10% traffic shadowing) to validate or rule out causes, and iterated daily.- When initial tests didn’t surface the issue, I instrumented additional logging (feature-level hashes, counts) while minimizing latency impact.- I coordinated with data engineering to roll back an unrelated ETL change to test impact, then progressively isolated the encoder behavior by replaying identical records through both pipelines.

Result:- We identified two root causes: (1) the categorical encoder fallback collision, and (2) mixed-precision inference on GPU causing small numerical shifts that amplified due to model calibration. Fixes: updated the encoder to use a safe OOV token mapping and added deterministic hashing; forced float32 inference for this model and retrained with calibration-aware techniques.- Within 48 hours of fixes, production AUC returned to baseline and false positives dropped to prior levels. We resumed rollouts after a controlled canary.

Organizational learning and leadership:- I led a postmortem, documenting the end-to-end failure chain and action items: implement data contracts, add schema and value-range checks (Great Expectations) at ingestion, version feature encoders, and require compatibility tests between infra changes and model numerics.- I introduced CI checks that replay a small sample of historical traffic on any serving change and added automated per-feature drift alerts to Grafana.- I mentored two engineers on writing reproducible preprocessing unit tests and championed a "shadow replay" pattern for future deployments.

This incident reinforced that production ML reliability requires engineering rigor: deterministic preprocessing, artifact/version management, reproducible inference environments, and cross-team communication. My persistence, structured hypotheses, and coordinated experiments were key to resolving a multifaceted, high-impact failure.

EasyTechnical

57 practiced

Describe your approach to documenting debugging steps, fixes, and postmortem findings so future engineers can reproduce investigations and avoid repeating mistakes. Include formats, minimum information to capture, and how you keep documentation discoverable.

Sample Answer

Situation brief: When an incident or hard-to-reproduce bug occurs in an ML pipeline, my documentation goal is twofold — make the exact investigation reproducible, and capture lessons so the team avoids the same mistake.

Format & storage:- Short postmortem (one page) in the team knowledge base (Confluence / Notion) with standard headings.- Repro notebook or script (Jupyter / Python) committed to the repo and tagged with the incident ID.- Issue ticket (Jira/GitHub) linking postmortem, code commits, and alerts.- Artefacts stored in model registry / S3 with clear path and lifecycle tags.

Minimum information to capture (always):- Incident ID, date/time, and reporter- Symptom and business impact (quantified: users, error-rate, latency)- Exact environment: commit SHA, docker image tag, Python package versions (requirements.txt / conda.yml), GPU/CPU, OS- Data snapshot: dataset version, sample record(s), seed(s) used to reproduce- Reproduction steps: exact commands, scripts, notebook cell order, CLI args- Logs & metrics: relevant log excerpts, monitoring dashboards, pre/post metrics- Root cause analysis: hypothesis, evidence, and why the fix addresses it- Fix implemented: code diff, config changes, rollout plan, rollback steps- Postmortem action items: owner, priority, ETA- Preventive measures: tests, alerts, automation added

How I make it reproducible:- Include a single-run script or notebook that, given a data snapshot and the specified environment, reproduces the failure and the fix path.- Pin seeds and nondeterministic settings (torch.manual_seed, deterministic flags) and document randomness sources.- Attach minimal reproducer data (or synthetic equivalent) to avoid privacy issues.

Keeping documentation discoverable:- Enforce naming conventions and tags (e.g., ML-INCIDENT, model-name, root-cause) and index them in the KB.- Link incident IDs across alerts, Jira, repo branches, and model registry entries.- Add the incident to a searchable changelog and to a quarterly “lessons learned” digest.- Make runbooks and playbooks discoverable via README in repo root and a short Slackbot command that returns links.- Automate: CI creates incident skeletons from alerts; successful fixes create annotated releases that reference the postmortem.

Example quick checklist (for everyday use):- Commit SHA + Docker tag- Command(s) to reproduce- Data version + sample- Deterministic seeds- Log snippet + metric snapshot- Fix diff + rollback steps- Action items with owners

This approach balances speed (short postmortem + runnable reproducer) with durability (indexed KB, linked artefacts) so future engineers can rerun investigations and avoid repeat mistakes.

MediumTechnical

84 practiced

How would you discover and resolve a mismatch between preprocessing used during training and the one used at inference time that is causing skewed predictions? Describe tooling, automated tests, versioning, and operational processes you'd deploy to catch and prevent such mismatches.

Sample Answer

Situation: When predictions differ between training and production, a common root cause is a preprocessing mismatch (feature scaling, categorical encoding, imputation, feature ordering).

Approach — discover:- Compare feature distributions and schemas between training feature store and live inference inputs (mean, std, cardinality, null rates, histograms).- Compute a replay/shadow inference on a sample of production inputs using the training pipeline and compare outputs to production inference.

Tooling:- Data validation: Great Expectations or TFDV to assert schema, types, ranges, nulls, and cardinality.- Feature-store: Use Feast or similar to serve identical feature values and transformations for train/infer.- Pipeline orchestration: TFX/Prefect/Airflow to package preprocessors as reusable components.- Serving: Containerized inference with same preprocessing code (e.g., packaged Python wheel) in the model image.

Automated tests:- Unit tests for each transform (e.g., test that StandardScaler.fit_transform then transform on held-out yields same results).- Integration tests in CI: run a small dataset through training pipeline and the packaged inference pipeline and assert feature-wise equality/tolerance and end-to-end prediction parity.- Regression tests: record canonical inputs and expected outputs (golden files) for preprocessors and model.

Versioning:- Version preprocessors as artifacts (semantic versioning), store in artifact registry alongside model versions.- Tag models with the preprocessor version used for training; include hashes of preprocessing code/config in model metadata.- Use feature-store versioning and dataset snapshot IDs for reproducibility.

Operational processes:- Shadow/canary deploy: run new model and preprocessing in parallel on production traffic and compare predictions; block rollout if mismatch exceeds threshold.- Monitoring & alerts: monitor prediction distribution drift, PSI, feature drift, and model confidence shifts; alert on sudden changes.- On-call playbook: steps to rollback, switch to previous preprocessor/model, and run local reconciliation tests.

Concrete example test snippet (unit test concept):

python

def test_scaler_roundtrip():
    scaler = load_artifact("scaler:v1")
    train_batch = np.load("train_sample.npy")
    # ensure applying scaler used at train time equals stored transformed values
    transformed = scaler.transform(train_batch)
    golden = np.load("golden_transformed_v1.npy")
    assert np.allclose(transformed, golden, atol=1e-6)

Why this works: enforcing the same code/artifacts, validating inputs continuously, and automating tests and monitoring creates a traceable pipeline that prevents silent preprocessing drift and speeds root-cause resolution.

Unlock Full Question Bank

Get access to hundreds of Problem Solving and Debugging Persistence interview questions and detailed answers.

Join thousands of developers preparing for their dream job.