Debugging and Recovery Under Pressure Questions

Covers systematic approaches to finding and fixing bugs during time pressured situations such as interviews, plus techniques for verifying correctness and recovering gracefully when an initial approach fails. Topics include reproducing the failure, isolating the minimal failing case, stepping through logic mentally or with print statements, and using binary search or divide and conquer to narrow the fault. Emphasize careful assumption checking, invariant validation, and common error classes such as off by one, null or boundary conditions, integer overflow, and index errors. Verification practices include creating and running representative test cases: normal inputs, edge cases, empty and single element inputs, duplicates, boundary values, large inputs, and randomized or stress tests when feasible. Time management and recovery strategies are covered: prioritize the smallest fix that restores correctness, preserve working state, revert to a simpler correct solution if necessary, communicate reasoning aloud, avoid blind or random edits, and demonstrate calm, structured troubleshooting rather than panic. The goal is to show rigorous debugging methodology, build trust in the final solution through targeted verification, and display resilience and recovery strategy under interview pressure.

HardTechnical

86 practiced

You are debugging a nondeterministic divergence that happens only under Distributed Data Parallel (DDP) across nodes. Propose a prioritized experiment plan to isolate whether the cause is data inconsistency, RNG/seed problems, NCCL misconfiguration, optimizer state synchronization, or a hardware issue. Include quick commands and tools you would use.

Sample Answer

Priority 1 — Verify data consistency (fast, likely):- Hypothesis: different nodes see different data/shards.- Experiment: compute deterministic checksums for dataset shards and first N minibatches on each rank.- Commands: - In dataset loader add: print(rank, sha256(batch.numpy().tobytes())) or write to file. - Example: python -c "import hashlib,sys; print(hashlib.sha256(open(sys.argv[1],'rb').read()).hexdigest())"- Expected: different hashes → fix data sharding, random transforms, file read race.

Priority 2 — Confirm RNG/seed determinism:- Hypothesis: per-rank RNG divergence (augmentations, dropout, dataloader workers).- Experiment: set all seeds and make RNG deterministic; log first few parameter grads and inputs.- Code snippet: - torch.manual_seed(0); torch.cuda.manual_seed_all(0); np.random.seed(0); random.seed(0) - torch.use_deterministic_algorithms(True)- Expected: divergence disappears → root cause RNG/worker/transform. Also try single-worker dataloader to rule worker-seed bug.

Priority 3 — Rule out NCCL/network issues:- Hypothesis: all-reduce hangs/lossy/timeout causing silent divergence.- Experiment: enable NCCL debug and use GLOO as control backend, and run NCCL tests.- Commands: - export NCCL_DEBUG=INFO; export NCCL_IB_DISABLE=1 (test over TCP) - python train.py --backend=gloo - Run nccl-tests: git clone https.../nccl-tests && build && ./all_reduce_perf -b 1 -e 8M -f 2 -g <gpus> - ipc: nvidia-smi topo --matrix; ibv_devinfo; iperf between nodes- Expected: errors/timeouts or different behavior between nccl/gloo → network/NCCL.

Priority 4 — Check optimizer & state sync:- Hypothesis: optimizer state (momentum/EMA) not synchronized leading to drift.- Experiment: after init, broadcast model.state_dict() and optimizer.state_dict() and compare hashes per rank each N steps.- Commands: - torch.distributed.broadcast_object_list([...]) or save optimizer.state_dict() to tempfile and compare sha256. - Insert assert_allclose on param and optimizer buffers across ranks using torch.distributed.all_reduce.- Expected: mismatch → fix broadcast/restore logic, ensure same gradient accumulation schedule, no per-rank learning-rate schedulers.

Priority 5 — Isolate to single-node vs multi-node:- Hypothesis: problem only when crossing nodes (network, config).- Experiment: replicate run with same world_size on single node (multiple GPUs) and with same seed/data.- Commands: torchrun --nproc_per_node=4 vs torchrun --nnodes=2 --nproc_per_node=2- Expected: if single-node ok but multi-node fails → network/NCCL or node-config.

Priority 6 — Hardware checks (last, costly):- Hypothesis: faulty GPU/PCIe/driver causing silent compute errors.- Experiment/tools: - Run cuda-memcheck / cuda-gdb on small kernels. - Run stress tests: nvidia-smi dmon, stress-ng cpu/mem, memtest for host RAM. - Check ECC errors: nvidia-smi -q | grep -i ecc - Swap suspected GPU to another node, re-run.- Expected: ECC/driver/kernel errors or failing memcheck → hardware replacement or driver update.

Quick general debugging tips & commands:- Add per-step deterministic logging (param hashes, grad norms): hashlib.sha256(tensor.cpu().numpy().tobytes())- Use barriers: torch.distributed.barrier() to align logs.- Lower LR and run for few steps while logging to make divergence visible early.- Use GDB/CUDA tools only after reproducing minimal failing script.

Interpretation & next steps:- If data/RNG fixes it → enforce seed + deterministic transforms + worker init_fn.- If NCCL/network shows issues → tune NCCL env (NCCL_IB_DISABLE, NCCL_SOCKET_IFNAME), upgrade drivers, check topology.- If optimizer mismatch → enforce explicit broadcast of model+optimizer and identical LR schedule.- If hardware → replace or isolate faulty hardware.

HardTechnical

72 practiced

Case study: a model trained on synthetic data performs well in validation but fails catastrophically in real deployment. You have a 2-hour window to run forensic analysis. Outline the steps you would take to isolate the sim-to-real gap causes (e.g., sensor differences, label mismatch, preproc differences), rapid experiments to test hypotheses, and immediate mitigations to restore functionality.

Sample Answer

Framework: prioritize rapid hypothesis isolation (highest-probability causes first), reproducible tests, and safe mitigations to restore partial functionality.

0–10 min: Triage & gather artifacts- Confirm failure mode (what “catastrophic” means: blank outputs, high error, unsafe actions).- Collect: sample real inputs, corresponding model outputs, logs, pre/postproc code, synthetic data spec, training checkpoints, model config, inference env (OS, libs), sensor metadata.

10–30 min: Quick differential checks (fast, deterministic tests)- Input distribution drift: compute simple statistics (mean/std, histograms, image brightness, frequency spectra, sensor latencies) between synthetic vs real batches.- Preproc parity: run identical sample through training-preproc and deployment-preproc; diff outputs (pixel stats, tokenization, scaling).- Label/annotation mismatch: inspect a small hand-labeled set of real inputs vs expected labels used in synthetic training.- Runtime env mismatch: verify model uses same precision (FP32 vs FP16), same library versions, and identical normalization constants.

30–60 min: Targeted rapid experiments (A/B tests)- Re-run inference on real inputs with: (A) training preproc pipeline, (B) deployment pipeline, (C) normalized to synthetic statistics. Measure metric delta.- Ablation on input channels: mask or clamp channels/sensors to see sensitivity (isolates noisy sensors).- Simple domain-adaptive transforms: apply histogram matching / color jitter / noise to synthetic → see if model responses on synthetic become closer to real behavior.- Check thresholds/decision logic: ensure calibration/thresholds tuned on synthetic aren’t misapplied.

60–100 min: Deeper diagnostics- Feature-space comparison: pass synthetic and real through intermediate layers; use PCA/t-SNE or distance metrics to detect representation shifts.- Confusion analysis on small hand-annotated real set to find systematic error types.- If possible, run lightweight fine-tune (few-shot) on 50–200 labeled real examples to see if performance recovers quickly (indicates label gap vs representational gap).

100–120 min: Mitigations to restore functionalityImmediate (minutes):- Swap to conservative safe mode: reduce autonomous actions, increase human-in-the-loop, lower-confidence abstain threshold.- Apply runtime preproc alignment: live-normalize real inputs to synthetic stats (histogram matching, standard scaling).- Revert model precision/env mismatches (e.g., force FP32) or deploy fallback baseline model that was validated on real data.Short-term (hours):- Deploy small calibration model or lightweight domain-adapter (batch-norm/adapters) trained on limited labeled real samples.- Rollback to earlier stable release if regression introduced by recent changes.

Communicate: summarize findings, recommended root causes, confidence levels, and next steps (collect more labeled real data, add synthetic realism, incorporate domain randomization, long-term domain adaptation). Metrics to report: delta in core metric, representation distance, success of quick fine-tune.

MediumTechnical

85 practiced

A multithreaded Python data loader produces deterministic shuffling (same order every epoch) causing training issues. Given this snippet, identify the likely bug and propose a fix and a test to verify shuffling randomness across workers:

python

def init_worker(seed):
    random.seed(seed)

for epoch in range(epochs):
    random.seed(42)
    sampler.shuffle()
    # start workers that call init_worker(42)

Explain proper seed handling for workers.

Sample Answer

Approach — bug and root cause- Bug: every epoch you call random.seed(42) in main and init_worker(42) in each worker, so all RNG state (main + every worker) is re-seeded to the same value each epoch -> deterministic identical shuffle/order across epochs and workers.- Proper handling: pick a single master base seed per epoch, then derive unique seeds for each worker (e.g., base_seed + worker_id + epoch) or use a cryptographic/safe RNG to generate per-worker seeds. Do not repeatedly reseed with the same constant.

Fix (example):

python

import random
import os

def init_worker(worker_seed):
    # set per-worker independent seed for stdlib random and numpy if used
    random.seed(worker_seed)
    try:
        import numpy as np
        np.random.seed(worker_seed)
    except ImportError:
        pass

for epoch in range(epochs):
    base_seed = 42  # could be time-based or passed in; keep stable for reproducibility
    epoch_seed = base_seed + epoch
    random.seed(epoch_seed)   # seed main RNG used for sampler.shuffle()
    sampler.shuffle()

    # when launching workers, derive unique seeds per worker
    for worker_id in range(num_workers):
        worker_seed = epoch_seed + worker_id + 1
        start_worker(init_worker, worker_seed)

Alternative safer pattern: use a per-worker Random instance so worker code never touches global RNG:

python

from random import Random
def init_worker(worker_seed):
    worker_rng = Random(worker_seed)
    # use worker_rng.choice/shuffle in worker rather than global random

Test to verify randomness across workers/epochs- Run a short loop that collects the order produced per epoch (or per-worker chunk), and assert that orders differ across epochs and that workers do not all produce identical sequences:

python

orders = []
for epoch in range(5):
    base = 100
    epoch_seed = base + epoch
    random.seed(epoch_seed)
    sampler.shuffle()
    # simulate workers with derived seeds
    epoch_orders = []
    for wid in range(4):
        worker_seed = epoch_seed + wid + 1
        random.seed(worker_seed)
        # simulate worker producing indices chunk
        epoch_orders.append(tuple(some_worker_fragment()))
    orders.append(tuple(epoch_orders))

# Check not all epochs identical
assert any(orders[i] != orders[0] for i in range(1, len(orders)))
# Check workers differ within an epoch
assert len({o for o in orders[0]}) > 1

Key points / best practices- Use one source-of-truth seed per epoch, then derive worker seeds (base + epoch + worker_id).- Avoid repeatedly setting global RNG to same constant.- For full reproducibility also seed numpy, torch (torch.manual_seed) and set cudnn deterministic flags when needed.- Consider using per-worker RNG instances to avoid accidental global-state clashes.

HardTechnical

64 practiced

Implement a Python utility that finds the smallest batch size that causes model outputs to contain NaN values. The function signature is:

python

def find_nan_batch_size(run_fn, min_bs=1, max_bs=1024, retries=2):
    """run_fn(batch_size) -> True if run succeeded without NaN, False otherwise"""

Return the minimal batch size that causes NaN or None if none found. Consider flaky runs, timeouts, and efficiency (use exponential + binary search). Provide robust code.

Sample Answer

To find the smallest batch size that causes NaNs while accounting for flaky runs and timeouts, first expand exponentially until you find a batch size that fails (or hit max_bs). Use repeated trials (retries) to reduce flakiness: treat the majority outcome across attempts as the true outcome. Then do a binary search between the last-known-good and first-known-bad sizes, again using majority voting for each tested size. Treat exceptions (including timeouts) as failures but still count them toward retries; you can re-raise non-runtime/system-critical exceptions if desired.

python

import math
from collections import Counter

def _majority_run(run_fn, batch_size, retries):
    """Run run_fn up to `retries+1` times and return True if majority succeeded."""
    outcomes = []
    for _ in range(retries + 1):
        try:
            ok = bool(run_fn(batch_size))
        except Exception:
            ok = False  # treat exceptions/timeouts as failures
        outcomes.append(ok)
        # early stop: if majority already decided
        counts = Counter(outcomes)
        if counts[True] > (retries + 1) // 2:
            return True
        if counts[False] > (retries + 1) // 2:
            return False
    return Counter(outcomes).most_common(1)[0][0]

def find_nan_batch_size(run_fn, min_bs=1, max_bs=1024, retries=2):
    """
    Return the minimal batch size that causes NaN (i.e., run_fn returns False/exception)
    or None if no failing batch size in [min_bs, max_bs].
    Uses exponential search to find an upper bound then binary search to pinpoint boundary.
    """
    if min_bs < 1 or max_bs < min_bs:
        raise ValueError("Invalid min_bs/max_bs")

    # Check min_bs first
    if _majority_run(run_fn, min_bs, retries) is False:
        return min_bs

    # Exponential search to find first failing batch size <= max_bs
    lo = min_bs  # known good
    hi = lo
    while True:
        hi = min(max_bs, hi * 2 if hi > 0 else 1)
        if hi == lo:
            break
        if _majority_run(run_fn, hi, retries) is False:
            break
        lo = hi
        if hi >= max_bs:
            break

    # If no failing batch size found in range
    if _majority_run(run_fn, hi, retries) is True:
        return None

    # Binary search between lo (good) and hi (bad) for minimal bad
    left, right = lo, hi
    while left + 1 < right:
        mid = (left + right) // 2
        if _majority_run(run_fn, mid, retries) is False:
            right = mid
        else:
            left = mid
    return right if _majority_run(run_fn, right, retries) is False else None

Key points:- Exponential expansion reduces number of calls to find a failing region quickly.- Majority voting across retries addresses flakiness.- Exceptions/timeouts treated as failures; adjust if you want different behavior.- Worst-case calls: O(log(max_bs/min_bs)) tests (each with retries+1 runs).

EasyTechnical

68 practiced

You have an LRU cache implementation used to memoize model inference results. It does not evict items correctly when capacity is reached. Without writing full code, describe the typical mistakes that cause incorrect eviction behavior, how you'd reproduce the issue quickly, and minimal unit tests you'd add to fix and prevent regressions.

Sample Answer

Common mistakes that cause incorrect LRU eviction- Using a plain dict for order: relying on insertion order without updating when an item is accessed.- Not moving a node to “most-recent” on get() (only on put()).- Faulty doubly-linked list pointers: off-by-one when removing head/tail or reconnecting neighbors.- Incorrect capacity checks: checking after insertion and forgetting to evict when size == capacity, or evicting before insertion but failing when updating an existing key.- Race conditions in concurrent access: missing locks leads to lost updates and inconsistent size/count.- Mixing logical size vs. actual nodes (e.g., decrementing size twice on eviction).

How I’d reproduce quickly1. Create a small capacity cache (capacity=2).2. Sequence tests that expose the bug: - put(A), put(B), get(A) -> ensure A becomes most-recent - put(C) -> expected eviction: B (if A was accessed), but current bug may evict A or wrong item3. Repeat with updating existing key: put(A), put(B), put(A, new_val), put(C) to check whether update avoids eviction.4. If concurrency suspected, run many threads doing random gets/puts and assert invariants.

Minimal unit tests to add (pytest-style descriptions)- Test access moves to MRU: - setup capacity=2; put("a"), put("b"); get("a"); put("c"); assert cache contains "a" and "c" only.- Test update doesn't increase size: - capacity=2; put("a"), put("b"); put("a", new_val); assert size==2 and values correct.- Test eviction order LRU: - capacity=3; put(1), put(2), put(3); get(2); get(1); put(4); assert contains 1,2,4 (3 evicted).- Test edge cases: - put same key repeatedly; get non-existent key returns None/KeyError as API defines.- Concurrency smoke test (if concurrent code): spawn N threads doing randomized ops for short time, then verify size <= capacity and internal linkage valid (walk list from head to tail counts nodes and matches dictionary).

Why these help- The sequences force movement and eviction logic to reveal pointer/order bugs and size-accounting mistakes. Concurrency test surfaces race conditions. Together they provide quick, deterministic checks and prevent regressions.

Unlock Full Question Bank

Get access to hundreds of Debugging and Recovery Under Pressure interview questions and detailed answers.

Join thousands of developers preparing for their dream job.