Debugging and Code Optimization Questions

Practical debugging skills and techniques for improving code performance and complexity. Topics include tracing and reproducing bugs, stepping through execution, reasoning about time and space complexity, refactoring for performance, and applying algorithmic optimizations. Candidates should be able to demonstrate logical debugging approaches and make safe, measurable performance improvements to working code.

HardTechnical

89 practiced

You see consistent differences in floating-point outputs between CPU and GPU inference for the same model, resulting in slightly different top-1 predictions. Explain possible causes (precision, different kernels, nondeterministic reductions) and how to debug and mitigate these differences when reproducibility is required.

Sample Answer

Situation: When the same trained model produces slightly different floating‑point outputs on CPU vs GPU and that changes top‑1 labels, it's usually due to numerical differences rather than a bug in high level model code.

Causes (concise):- Precision differences: GPU (CUDA) often uses mixed precision or FP16/FP32 accumulation; CPU commonly uses FP32 with x86 FMA behavior. Rounding/associativity differences matter.- Different kernels: BLAS/cuDNN/cuBLAS implementations use different algorithms (tiling, vectorization, Winograd conv) producing different rounding patterns.- Nondeterministic reductions/atomic ops: Some GPU kernels use parallel reduction ordering or atomics that are nondeterministic across runs.- Library/version differences: Different PyTorch/CUDA/cuDNN/oneDNN versions change kernels/optimizations.- Threading and parallelism: CPU multithreading (OpenMP) can reorder ops.

Debug & repro steps:1. Fix software stack: record exact PyTorch, CUDA, cuDNN/oneDNN, MKL versions and drivers.2. Make seeds deterministic: torch.manual_seed, torch.cuda.manual_seed_all and set env vars (CUBLAS_WORKSPACE_CONFIG, CUDA_LAUNCH_BLOCKING=1).3. Force deterministic algorithms: torch.use_deterministic_algorithms(True) and set TORCH_USE_CUDA_DSA or relevant flags; for cudnn: torch.backends.cudnn.deterministic=True; torch.backends.cudnn.benchmark=False.4. Compare intermediate tensors: run forward on a small batch and compare layer outputs (e.g., after each conv/BN/FC) with tolerances (absolute and relative) to locate divergence point.5. Check precision: ensure both run in same dtype—cast CPU tensors to float32/float64 or run GPU in FP32. If using mixed precision, disable AMP for comparison.6. Re-run single-threaded CPU: set OMP_NUM_THREADS=1 and MKL_NUM_THREADS=1 to remove reorder differences.7. Use double precision for debugging: run model.double() on both devices to reduce rounding noise and find logical mismatches.8. Compare model parameters: ensure state_dict identical after load (no device-dependent conversions).

Mitigations when reproducibility required:- Standardize dtype: run inference in FP32 (or FP64) on both devices.- Use deterministic kernels and disable nondeterministic optimizations (at cost of performance).- Use fixed reduction algorithms or libraries with deterministic modes (set CUBLAS_WORKSPACE_CONFIG and cudnn deterministic flags).- Quantize/round outputs: if small numerical differences don't matter, apply robust decision rules (top‑k majority, thresholding, or softmax temperature).- Acceptance testing: add numeric unit tests comparing CPU vs GPU tolerances and log drifts in production.

Example quick PyTorch checklist:- torch.backends.cudnn.deterministic = True- torch.backends.cudnn.benchmark = False- torch.use_deterministic_algorithms(True)- export CUBLAS_WORKSPACE_CONFIG=:4096:8- set CUDA_LAUNCH_BLOCKING=1 for debugging- force dtype consistency: model.float(); input = input.float()

Tradeoffs: deterministic + higher precision increases latency and memory. Choose strictness based on production needs: exact bitwise reproducibility vs statistically stable decisions.

HardSystem Design

103 practiced

Design an inference-serving approach that batches incoming single-record requests into micro-batches to improve throughput without violating tight latency SLOs (e.g., 50ms p95). Describe batching heuristics, maximum batch size, timeout-based batching, and the trade-offs between latency and throughput.

Sample Answer

Requirements & constraints:- Functional: accept single-record requests, serve with micro-batching to improve throughput.- SLO: 50 ms p95 end-to-end.- Non-functional: maximize GPU/accelerator utilization, avoid SLO violations.

High-level design:- Components: request ingress → per-model Batch Coalescer → Batch Scheduler → Inference Worker(s) (GPU) → response.- Each model has its own coalescer and scheduler to avoid cross-model interference.

Batching heuristics:- Profile model to get per-batch inference latency curve L(b) (e.g., L(1), L(8), L(16)). Use these to compute throughput gain and marginal latency per added item.- Admission: accumulate requests into current batch until either batch_size == max_batch_size OR timeout hits OR SLO headroom reached.- Prioritize based on client/request priority and remaining SLO headroom (computed as SLO - elapsed_time).

Maximum batch size:- Determine offline by benchmarking: pick the largest b such that L(b) gives acceptable per-item latency and good hardware utilization. Example: if L(1)=10ms, L(8)=22ms, L(16)=35ms, choose max_batch_size=16 but only use full 16 at high QPS.- Enforce hard upper limit to prevent OOM and nonlinear latency blowups.

Timeout-based batching:- Adaptive timeout T dynamically computed from SLO budget and current queuing latency target. Split SLO into queuing + inference + network (e.g., for 50ms p95: reserve 30ms inference+network → 20ms queuing/headroom).- Compute T = min(max_wait, SLO_headroom - expected_inference_latency). Use EWMA of arrival rate λ to adjust T: at low λ increase T to fill some batch (but capped), at high λ decrease T to meet latency target.- If batch reaches max_batch_size early, dispatch immediately.

Adaptive strategies & trade-offs:- Dynamic batching: when QPS high, batches hit max_size and throughput is excellent; when QPS low, small batches reduce queuing latency.- Headroom-aware early dispatch: if a request's remaining SLO headroom is small, send a smaller/individual batch (fast-path) to avoid tail violations — trades throughput for latency on a few requests.- Probabilistic batching: occasionally sample and send singletons to bound tail latency.

Backpressure and fairness:- Limit the queue length per-model, apply backpressure (429 / rate-limit) under overload.- Use priority queues so latency-sensitive requests get smaller batches / immediate dispatch.

Metrics & monitoring:- Track p50/p95/p99 latency, batch size distribution, GPU utilization, queue time, inference time.- Alert when p95 queue_time > configured quota.

Example parameterization for 50ms p95:- Offline: L(1)=8ms, L(8)=18ms, L(16)=32ms → choose max_batch_size=16.- Runtime SLO split: 50ms total → 10ms network + 32ms inference (for 16) → allocate 8ms queuing headroom. Set T such that expected wait ≤8ms at target QPS; use EWMA λ to reduce T as λ falls so small batches meet p95.

Edge cases & mitigations:- Burst arrivals: spin up more workers or use model sharding/replicas; use admission control.- Model stalls: circuit-breaker to avoid queue buildup.- Nonlinear latency growth: enforce lower batch sizes for models with poor scaling.

This approach balances throughput and tight latency SLOs by combining offline profiling, adaptive timeouts, SLO-aware early dispatch, and runtime monitoring to trade a small amount of throughput for guaranteed latency tail behavior.

EasyTechnical

95 practiced

Your data loader is reading a 200GB CSV file and training stalls due to I/O. Describe practical optimizations to reduce I/O bottlenecks for training: formats, sharding, prefetching, caching, compression trade-offs, and local vs network storage strategies.

Sample Answer

Approach: treat this as a data pipeline problem — minimize bytes read, maximize sequential throughput, and overlap I/O with compute.

Practical optimizations- File format: convert CSV to a binary, splittable format (Parquet, Avro, TFRecord). Parquet is columnar (good if you read subsets of columns) and highly compressible; TFRecord is simple and fast for TF/PyTorch when using fixed-schema tensors.- Compression trade-offs: use fast block compression (snappy, LZ4) for good throughput + space savings. Gzip gives smaller files but higher CPU decompression cost and higher latency; avoid for hot training loops.- Sharding: write the dataset into N large shards (e.g., 100–1000 files sized ~100MB–1GB) to allow parallel reads and avoid millions of tiny files. Ensure shards are balanced and deterministically shufflable.- Read parallelism & prefetching: use framework features (tf.data.map(..., num_parallel_calls), tf.data.prefetch, PyTorch DataLoader with num_workers + prefetch_factor, DataPipes). Set prefetch to overlap I/O and batch preparation with GPU.- Caching: if dataset fits or a shard fits in RAM, use in-memory caching for repeated epochs. Otherwise use local SSD/NVMe as a staging cache to avoid repeated network reads. OS pagecache will help but explicit caching is more reliable.- Local vs network storage: stage training data onto local ephemeral SSD/NVMe for best throughput and lowest latency. If using object storage (S3/GCS), use multipart download, cached FUSE layers, or a distributed caching layer (e.g., s3fs with local cache, Cloud Storage Transfer). For multi-node training, distribute shards so each worker reads mostly-local data.- Other optimizations: avoid parsing CSV at training time — parse/encode features offline into binary tensors. Use efficient serializers (Protobuf, msgpack) or NumPy memmap for very large arrays. Tune shuffle buffer size to balance randomness and memory.

Implementation checklist- One-time conversion: CSV -> Parquet/TFRecord with shard creation and snappy compression.- Producer writes deterministic shard list and manifests.- Training pipeline: parallel reader -> map (decode) -> batch -> prefetch_to_device.- Monitor: I/O bandwidth, CPU decompression, read latency, GPU utilization to ensure compute is the bottleneck.

Expected impact- Reduced I/O bytes and CPU parsing, higher sequential throughput, lower stall time. Metrics to track: bytes/sec read, IOPS, GPU utilization, epoch time.

EasyTechnical

75 practiced

You have this PyTorch training loop in Python:

for batch in loader: outputs = model(batch['x']) loss = loss_fn(outputs, batch['y']) loss.backward() optimizer.step()

Training loss rapidly becomes NaN and validation accuracy collapses. List the most likely causes, explain how to reproduce and trace the exact cause, and describe concrete fixes you would apply to make the loop robust in production.

Sample Answer

Most likely causes- Missing optimizer.zero_grad() so gradients accumulate and explode.- Too-large learning rate or bad optimizer hyperparams (e.g., momentum/eps).- Numerical issues in loss (log(0), division by zero, sqrt of negative), bad labels (NaN/Inf/one-hot mistakes).- Exploding gradients from model architecture or poor weight init.- Mixed-precision (AMP) without GradScaler or improper scaling.- BatchNorm/Dropout mode mistakes (eval vs train) or dataset with extreme outliers.

How to reproduce and trace the exact cause- Reproduce deterministically: set torch.manual_seed, deterministic cudnn, small dataset, single-batch runs.- Run a single batch and print/inspect: - loss.item(), torch.isfinite(loss) - parameter and gradient stats: min/max/mean, torch.isnan checks- Use torch.autograd.set_detect_anomaly(True) to get stack traces on NaNs/Infs during backward.- Insert hooks or small wrappers to catch when values become non-finite: - register_full_backward_hook or check after backward for any non-finite gradients.- Isolate components: replace loss with MSE of outputs and zeros to confirm model vs loss function issue.- Try FP32 only if using AMP to see if mixed precision causes it.

Concrete fixes and production-robust loop- Always zero grads: optimizer.zero_grad() (or model.zero_grad()) at start of step.- Validate loss finite before backward; skip batch and log if not.- Use gradient clipping: torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm).- Lower LR / tune optimizer; add weight decay.- Add eps safeguards in loss (e.g., log(x + 1e-8)), use stable numerics (log-sum-exp).- For AMP, use torch.cuda.amp.GradScaler() and scale/unscale properly.- Add regular monitoring and alerting for NaN/Inf, record batch index and inputs that cause failure.- Initialize weights appropriately and consider spectral norm / gradient penalties for unstable networks.

Robust training loop example

python

scaler = torch.cuda.amp.GradScaler()  # if using AMP
for batch_idx, batch in enumerate(loader):
    optimizer.zero_grad()
    x, y = batch['x'].to(device), batch['y'].to(device)
    with torch.cuda.amp.autocast():  # remove if not using AMP
        outputs = model(x)
        loss = loss_fn(outputs, y)
    if not torch.isfinite(loss):
        logger.error(f"Non-finite loss at batch {batch_idx}: {loss}")
        continue  # skip or inspect batch
    scaler.scale(loss).backward()  # use loss.backward() if no AMP
    # optional: detect anomalies (slower)
    # torch.autograd.set_detect_anomaly(True)
    # clip grads
    scaler.unscale_(optimizer)  # required before clipping with AMP
    torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
    scaler.step(optimizer)
    scaler.update()

Extra production best-practices- Unit-test loss on synthetic inputs, include numeric-stability tests.- Log gradients/statistics and failing batches to storage for post-mortem.- Automatic rollback or safe checkpoints when metric collapse detected.- Use smaller LR and warmup schedule; consider learning-rate finder during experimentation.

These steps let you identify whether the issue is optimizer usage, numerical instability in the loss, data problems, or precision-related—and apply targeted, production-ready fixes.

EasyTechnical

90 practiced

Explain how to reason about time and space complexity for common ML preprocessing operations. Give Big-O for: loading N samples from disk, applying a mapping transform that is O(k) per sample, sorting features across N samples, and building a hash index. Describe how these asymptotics influence choices in a production data pipeline.

Unlock Full Question Bank

Get access to hundreds of Debugging and Code Optimization interview questions and detailed answers.

Join thousands of developers preparing for their dream job.