Debugging, Testing, and Optimization Questions

Core engineering skills for identifying, diagnosing, testing, and improving code correctness and performance. Covers approaches to finding and fixing bugs including reproducible test case construction, logging, interactive debugging, step through debugging, and root cause analysis. Includes testing strategies such as unit testing, integration testing, regression testing, test driven development, and designing tests for edge cases, boundary conditions, and negative scenarios. Describes performance optimization techniques including algorithmic improvements, data structure selection, reducing time and space complexity, memoization, avoiding unnecessary work, and parallelism considerations. Also covers measurement and verification methods such as benchmarking, profiling, complexity analysis, and trade off evaluation to ensure optimizations preserve correctness and maintainability.

HardTechnical

54 practiced

Given a fixed total GPU-hours budget and dataset size, design a hyperparameter optimization strategy (choose between Bayesian optimization, Hyperband, Population-Based Training, or hybrid approaches). Explain budget allocation, early stopping rules, reproducibility requirements, and tests to validate that the chosen hyperparameters generalize.

Sample Answer

I would choose a hybrid: Bayesian Optimization + Hyperband (BOHB) as the primary strategy, with a small Population‑Based Training (PBT) component for dynamic schedules if online adaptation matters. BOHB combines multi‑fidelity early stopping (Hyperband’s brackets) with model‑based sampling (Bayesian) to use GPU-hours efficiently while focusing on promising regions.

Budget allocation- Let B be total GPU‑hours and N the dataset size. Decide a minimum resource r_min (e.g., 1 epoch or 0.5 GPU‑hour) and maximum r_max (full training cost per config). Use Hyperband brackets to split B into successive halving rounds. Compute expected number of full runs: k ≈ B / r_max (upper bound). Allocate ~70–80% of B to BOHB main search (many low‑cost evaluations with aggressive promotion), 10–15% to confirmatory full trainings of top candidates, and 5–10% reserved for PBT warmups or sanity checks.

Early stopping rules- Use successive halving: evaluate many configs at r_min, keep top 1/η, increase resources multiplicatively. η=3–4 is common. Supplement with Bayesian predictive stopping: if surrogate predicts < threshold improvement with high probability (p>0.95), stop. Also stop if validation loss plateaued for s steps or metric variance high across seeds.

Reproducibility- Fix and version-control random seeds, environment (container + CUDA/CUDNN versions), dataset preprocessing code, and hyperparameter search seed. Log deterministic metadata (hardware, batch sizes, mixed‑precision flags). Save checkpoints and reproduce top-k trials with identical seeds and full r_max to confirm results.

Validation and generalization tests- Hold out a stratified test set not used in search. After selection, re-run top candidates 5–10 times with different seeds, report mean±CI of metrics. Perform k‑fold CV if dataset size allows (or stratified repeated CV). Run ablation on hyperparameters to ensure stability (vary top hyperparams ±10–20%). Stress tests: OOD slices, corrupted inputs, and scale training data down/up to check robustness. Use statistical tests (paired t-test or bootstrap) vs baseline to confirm significant gains.

Trade-offs and rationale- BOHB maximizes useful evaluations per GPU‑hour via multi‑fidelity, while Bayesian sampling avoids wasting full runs. PBT is added only if schedules (learning rates, augmentations) should evolve during training. This hybrid balances exploration, exploitation, and resource constraints for reliable, reproducible hyperparameter selection.

HardSystem Design

53 practiced

Architect an end-to-end testing, debugging, and optimization framework for training a large transformer model across multi-node, multi-region clusters using spot instances. Include reproducibility, sharded checkpoints, profiling hooks, automatic failover and requeueing, cost control, and verification tests that assert model quality after resume. Describe components and interactions.

Sample Answer

Requirements:- Multi-node, multi-region large-transformer training on spot instances with minimal interruption- Reproducibility, sharded+incremental checkpoints, fast resume, profiling hooks, automatic failover/requeue, cost controls, and post-resume verification asserting model quality

High-level architecture:- Orchestrator (control plane): job scheduler, failover/requeue logic, global state store- Fleet Manager: manages spot instance pools across regions, bidding policies, autoscaling, preemption detectors- Trainer Workers: distributed training processes (e.g., PyTorch/XLA or DeepSpeed, FSDP) with sharded checkpointing and deterministic RNG seeds- Checkpoint Store: object storage (multi-region replication) + manifest service that records shard metadata, optimizer state, RNG seeds, versioned config- Profiler & Telemetry: lightweight in-process hooks to capture perf counters, memory, NVTX traces, network I/O, aggregated to central telemetry DB- Verifier Service: runs deterministic validation/replay pipelines on resumed checkpoints to assert model metrics (loss, eval tokens, calibration) within tolerance- Cost Controller: policy engine that enforces budgets, uses spot risk scores, dynamic region switching, and graceful degradation (smaller batch / gradient accumulation)

Component interactions and flows:1. Job submit: user submits job + seed/config + cost policy to Orchestrator. Orchestrator writes immutable run manifest to global state.2. Launch: Fleet Manager provisions spot pools across preferred regions; Trainer Workers launch with the run manifest. RNG seeds and hostlist in manifest ensure reproducibility.3. Training loop: workers perform distributed optimizer with sharded states (FSDP). Periodic coordinated checkpoint: - Each worker uploads its shard; manifest updated atomically with shard checksums and epoch/step. - Optional incremental checkpoint stores only changed tensors (delta) to save cost.4. Profiling: profiler hooks sampled (low-overhead continuous) and high-fidelity traces on-demand. Telemetry aggregates to detect regressions.5. Preemption handling: Fleet Manager detects spot loss via cloud events. Workers persist final local shard to ephemeral fast storage and Orchestrator triggers requeue: - Orchestrator picks new spot pool (based on cost/risk) and assigns resume point from latest consistent manifest. - Resume preserves RNG seeds + optimizer states to ensure bitwise or statistically equivalent continuation.6. Automatic requeue and fast restart: use container images with persisted environment; use node-local caches and parallel shard streaming to reduce restart time.7. Verification after resume: Verifier Service runs a deterministic micro-evaluation (subset of validation tokens, forward-only) and statistical tests (paired t-test on loss/metric over last N steps). If deviation exceeds thresholds, flag and optionally rollback to earlier checkpoint.8. Cost control and preemption mitigation: - Dynamic bidding strategy, multi-region diversification, and hybrid on-demand fallback for critical phases (final checkpoints, evaluation). - Adaptive batch-size/accumulation to reduce memory and allow more granular checkpoints.9. Observability and alerts: - Dashboards for throughput, step time, GPU utilization, interconnect bandwidth, preemption events. - Alerts for drift in profiler metrics or verification failures that automatically pause job and trigger debug snapshot.

Reproducibility details:- Immutable run manifest: code git hash, pip/conda env, container digest, hardware topology, seeds for all RNGs, deterministic dataloader shuffle plan (pre-generated shuffle indices in object store).- Deterministic ops where possible (cuDNN deterministic modes), fallback tests detect non-determinism and record variance.- Checkpoint manifest guarantees atomic consistency across shards using two-phase commit semantics.

Sharded checkpoints & storage optimization:- Use FSDP/ZeRO-3 style sharding; upload shards in parallel; manifest stores mapping worker->shard.- Content-addressed storage + deduplication for large embeddings; delta checkpoints for infrequent large tensors.- Multi-tier storage: hot (regional SSD), warm (regional S3), cold (cross-region archival) with lifecycle policies to control cost.

Profiling hooks:- Lightweight counters in training loop (step time, comp/comm overlap)- NVTX/Trace export on sampled steps- GPU kernel timeline + memory snapshots on anomaly triggers (OOM risk, sudden perf drop)- Automatic profiler-driven recommendations (increase micro-batch, change grad accumulation, fuse ops)

Testing and verification:- Deterministic unit tests for checkpoint load/save, shard reconstruction- Integration tests: simulated preemption (inject spot termination) and resume validation- Post-resume quality tests: run fixed-seed eval; assert metrics within configured epsilon or run compensating calibration (learning rate warmup)- Canary runs: small-scale reproduction across zones before scaling

Trade-offs and failure modes:- Full bitwise determinism is expensive and sometimes impossible across hardware — allow statistical equivalence thresholds.- Cost vs reliability: hybrid fallback to on-demand increases cost but reduces restart latency.- Checkpoint frequency vs throughput: frequent checkpoints reduce recompute on preemption but raise I/O cost; use adaptive checkpoint scheduler based on spot risk and remaining budget.

This framework balances reproducible, testable training with cost-aware spot usage, fast failover, and automated verification to ensure resumed models meet quality standards.

HardTechnical

71 practiced

Scenario: Production A/B shows a small model accuracy increase (0.5%) but a noticeable drop in conversions. Outline your investigation plan to find the root cause: calibration checks, thresholding effects, cohort analysis, feature distribution changes, logs/telemetry inspection, and counterfactual offline analysis. Propose steps to mitigate or roll back safely.

Sample Answer

High-level goal: find why a small accuracy gain (+0.5%) coincides with meaningful conversion loss. I’d run a prioritized, evidence-first investigation covering calibration, decision thresholds, cohorts, features, logs/telemetry, and offline counterfactuals — then propose safe mitigations/rollback.

Investigation plan (ordered, with methods & observables)1. Validate A/B metrics & experiment integrity- Recompute primary/secondary metrics, check randomization, sample sizes, and statistical significance (confidence intervals, p-values, sequential tests). Verify no instrumentation bias (missing events, platform differences).2. Calibration & confidence checks- Compare predicted probabilities (calibration curves, reliability diagrams, ECE) between control and treatment. Check Brier score and distribution of confidences. If calibration shifted, model may be over/under-confident causing different downstream decisions.3. Thresholding / decision logic effects- Reproduce production decision rule (thresholds, business rules). Simulate varying thresholds on holdout data to see conversion response curve. Inspect whether small probability shifts move many examples across a hard cutoff.4. Cohort analysis- Break down impact by user segments: device, geography, new vs returning users, traffic source, time-of-day, top product buckets. Look for concentrated negative lift in a segment (size, conversion delta, p-value).5. Feature-distribution & data drift- Compare feature distributions (KS test, PSI) pre/post rollout and between variants. Check for covariate shift, missing/NaN rates, anomalous categories. Validate label distribution in recent training/serving windows.6. Explainability / counterfactual offline analysis- Use SHAP/feature attribution to see what features changed decision-making most vs control. Run counterfactual scoring: for users who converted under control but not under treatment, score with both models to see predicted class/confidence delta and top feature changes.7. Logs & telemetry inspection- Correlate model decisions with downstream service logs (errors, latency, fallback rules). Check for increased retries, timeouts, or downstream code paths that treat model outputs differently. Inspect request payloads for truncation or schema changes.8. Sanity checks on data pipeline & deployment- Confirm model code, featurization, normalization, and preprocessing are identical to offline assumptions. Verify model version and weights deployed match tested artifact; check for A/B tagging bugs.

Root-cause hypothesis framing- Likely causes: (A) calibration shift causing many borderline users to be classified differently due to threshold; (B) covariate/feature distribution drift causing different decision boundary behavior for a key cohort; (C) instrumentation/serving bug changing inputs or downstream handling; (D) small accuracy gain came from easy negatives while hurting high-value positive cohort.

Mitigation & rollback plan (safe, staged)1. Immediate safety actions (within minutes)- Hit the kill-switch: route traffic back to control or enable traffic-weight rollback to 100% control if high-risk. If kill-switch not available, shift to routing model to "shadow" mode.- Enable strict monitoring and alerts (conversion delta, revenue, error rates).2. Short-term mitigations (hours)- Canary/gradual rollback: reduce treatment fraction in steps (e.g., 50% → 10% → 0%), verifying metric recovery.- If calibration/threshold issue suspected, temporarily adjust thresholds to match control decision rates (match precision/recall targets) or apply calibrated mapping (Platt/isotonic) computed on recent production-like data.- Apply cohort-based routing: exclude affected segments from treatment while keeping others.3. Medium-term fixes (days)- Reproduce issue offline using counterfactual simulation and fix model or preprocess accordingly (retrain with corrected features, fix featurization bug, regularize to reduce harmful behavior).- If instrumentation bug, deploy corrected featurization and validate in staging shadow before rolling.4. Validation & safe re-release- Run offline uplift-by-cohort validation and online canary with small fraction, plus ramp plan with automated stop criteria (statistical thresholds for conversion/regression).- Add regression tests: per-cohort guardrails, calibration checks, PSI thresholds, and model-card limits.5. Postmortem & preventative controls- Document root cause, actions taken, and lessons. Add automated monitoring: ECE, PSI, cohort-level conversion deltas, and model explainability drift alerts. Add automated rollback triggers.

Key tools & tests- Statistical: bootstrap CIs, sequential testing, permutation tests.- Drift: KS test, PSI, population stability index, MLflow data drift.- Calibration: reliability diagram, isotonic/Platt calibration routines.- Explainability: SHAP, LIME, counterfactual scoring.- Observability: ELK/Datadog/Grafana for logs, trace correlation from request → model → downstream.

What I’d report quickly- Repro steps, top hypotheses with evidence (plots: calibration curves, threshold crossing histogram, cohort lift table), immediate action taken, and next steps with ETA for root-cause fix and safe re-release.

MediumSystem Design

62 practiced

Design a debugging and testing strategy for distributed data-parallel training on 64 GPUs where one process occasionally lags (a straggler). Describe instrumentation to collect per-process timings, what tests to run to reproduce, how to isolate node vs network vs data pipeline issues, and mitigation strategies like timeouts, rebalancing, or skipping slow nodes.

Sample Answer

Requirements & goals:- Detect and root-cause intermittent straggler in 64-GPU distributed data-parallel (DDP) training.- Fast reproduction, isolate node vs network vs data pipeline, and propose mitigations that preserve correctness and throughput.

Instrumentation (per-process / per-GPU, sampled at each step/batch):- Wall-clock timings: forward, backward, optimizer step, all-gather/collectives, CUDA kernel times, CPU preprocessing time, data fetch + decode, queue wait time.- System metrics: GPU utilization, memory, SM/mem bandwidth, PCIe/NVLink throughput, CPU load, disk I/O, network interface counters and packet drops, interrupts.- Fine-grained traces: torch.autograd.profiler / NVIDIA Nsight metrics, NCCL tracing (nccl-tests logging, NCCL_TRACE environment), timestamps for RPC/collective entry/exit.- Attach lightweight heartbeat with lamport-like sequence numbers and per-step latency histograms to central aggregator (Prometheus + Grafana + traces).

Reproduction tests:- 1) Deterministic small-batch run with synthetic data to eliminate data variability.- 2) Run the exact model on subsets: single node (8 GPUs), full 8-node run, and scaled runs to see where straggler appears.- 3) Stress tests: heavy network load on suspected node, I/O-heavy synthetic workloads, CPU throttling.- 4) Injected delays: artificially sleep in data loader or CPU thread to reproduce timing patterns.

Isolation strategy:- Node vs network: run identical workload collocated on same node (intra-node) vs across nodes; compare all-gather times and NIC counters. High/all-gather wait with NIC errors => network.- Data pipeline: run synthetic in-memory dataset; if straggler disappears, root cause is data loading/IO.- GPU/driver: run microbenchmarks (cudaMemcpy, kernel latency, sm efficiency) and check differences vs neighbors; if GPU busy/low utilization, check clock/driver, ECC errors.- OS contention: inspect dmesg, cgroup limits, cron jobs, thermal throttling.

Mitigations:- Short-term: per-step timeout with replacement of gradient from previous step or skip contribution (careful: impacts correctness). Prefer: restart worker process if stuck, requeue job on node.- Medium-term: dynamic bucketing/rebalancing — move data shards away from slow node; adaptive batch-size per worker (gradient accumulation to equalize throughput).- Algorithmic: asynchronous or stale-synchronous training, elastic DDP (torch.distributed.elastic) to remove/readd nodes, gradient compression to reduce network waits.- Operational: isolate problematic node (cordon), firmware/drivers updates, NIC reprovision, improve data locality and prefetching, add retries for transient NCCL errors.

Trade-offs:- Timeouts and skipping speed up wall-clock but may bias training; prefer elastic rebalancing + checkpointing for correctness.- Instrumentation overhead: sample rate trade-off; collect detailed traces only on failures.

Key deliverable:- Dashboards (per-rank latency, collective wait, GPU util), automated test suite to reproduce, and runbook with prioritized remediation (isolate node → run microbenchmarks → reimage or remove from pool).

EasyBehavioral

58 practiced

Behavioral: Tell me about a time you diagnosed a tricky model bug (not a syntax error) — for example caused by data drift, label leakage, or a subtle preprocessing mismatch. Use the STAR format: Situation, Task, Action, Result. Emphasize tools you used, how you prioritized hypotheses, and what you changed to prevent recurrence.

Unlock Full Question Bank

Get access to hundreds of Debugging, Testing, and Optimization interview questions and detailed answers.

Join thousands of developers preparing for their dream job.