Code Quality and Defensive Programming Questions

Covers writing clean, maintainable, and readable code together with proactive techniques to prevent failures and handle unexpected inputs. Topics include naming and structure, modular design, consistent style, comments and documentation, and making code testable and observable. Defensive practices include explicit input validation, boundary checks, null and error handling, assertions, graceful degradation, resource management, and clear error reporting. Candidates should demonstrate thinking through edge cases such as empty inputs, single element cases, duplicates, very large inputs, integer overflow and underflow, null pointers, timeouts, race conditions, buffer overflows in system or embedded contexts, and other hardware specific failures. Also evaluate use of static analysis, linters, unit tests, fuzzing, property based tests, code reviews, logging and monitoring to detect and prevent defects, and tradeoffs between robustness and performance.

HardSystem Design

37 practiced

Architect a distributed training system for large-scale DNNs (multi-node GPUs) that tolerates node failures. Ensure checkpoint consistency, minimal wasted work, and correctness of optimizer state. Describe coordination mechanisms (parameter-server vs all-reduce), checkpoint frequency and strategy, reconfiguration on node loss, and how you'd test fault injection at scale.

Sample Answer

Requirements & constraints:- Large DNN across multi-node GPU cluster; tolerate node failures without corrupting model/optimizer; minimize wasted work; keep correctness of optimizer state (momentum, Adam moments, sharded states).- Low training throughput loss; support elastic scale-up/down.

High-level architecture:- Worker-only synchronous training using all-reduce (NCCL/UCX) + optimizer state sharding (ZeRO stage 2/3) for bandwidth and memory efficiency. Use parameter-server only as fallback for extremely large models or when asynchronous updates are needed.- Cluster control plane: rendezvous & membership service (etcd/consul) + orchestration (Kubernetes) + a checkpoint coordinator (lightweight service).

Coordination & consistency:- Synchronous global-step barrier: every iteration ends with an all-reduce to produce consistent gradients; only after successful all-reduce workers advance the global step.- Checkpoint protocol: coordinated consistent checkpoints at global-step boundaries using a two-phase commit: 1. Prepare: coordinator asks all workers to quiesce at next global step boundary. 2. Commit: workers flush optimizer sharded state + model shards to durable storage (object store like S3) using atomic versioned keys (model-v{step}). Workers report success to coordinator; coordinator marks checkpoint as successful only when all shards present.- Optimizer correctness: store optimizer sharded states (e.g., Adam's m,v or ZeRO shards) together with metadata (global step, RNG state, mapping of shard -> node). Ensure deterministic ordering of parameter partitions for reproducible restore.

Checkpoint frequency & strategy:- Hybrid: frequent lightweight incremental checkpoints + periodic full checkpoints. - Lightweight: save optimizer deltas / only changed parameter shards every K steps (e.g., every 500–1k steps) to reduce I/O. - Full: every M minutes or N steps (e.g., every 10k steps or 30 minutes) produce full consistent snapshot.- Adaptive frequency: if node health degrades or spot instances likely to be preempted, increase checkpoint frequency.- Streaming upload with multipart + checksum; use multi-part finalize to ensure atomic visibility.

Reconfiguration on node loss (elastic recovery):- Failure detection via heartbeat from coordinator + Kubernetes node events.- On single-node failure: - If using ZeRO shard: consult shard-to-node mapping, restore missing shards from latest successful checkpoint. - Coordinator pauses training, signals remaining workers to load checkpoint (or the failed worker is removed and its shard is reconstructed from checkpoint) then re-initiate rendezvous with reduced world size or with replacement nodes.- Elastic resizing: - If replacement nodes available: spawn replacement pods, load required shards and join via rendezvous; rebuild NCCL rings automatically (Horovod/torch.distributed rendezvous supports ring rebuild). - If not: shrink world size deterministically: recompute data-parallel mapping (re-shard optimizer and parameters) using same deterministic partitioning algorithm; use checkpoint to reconstruct full model then redistribute shards across remaining workers; resume from that checkpoint to ensure correctness (avoids partial-apply of updates).- Minimize wasted work: keep incremental checkpoint that allows rolling back only a few steps; upon failure, resume from latest successful checkpoint rather than redoing entire epoch.

Parameter-server vs All-reduce tradeoffs:- All-reduce (synchronous): - Pros: deterministic, simpler correctness for sync SGD, efficient with NCCL for GPU rings, good for throughput and avoiding stale gradients. - Cons: requires ring reconstruction on node loss; entire run stalls until reconfiguration.- Parameter-server (async or sync): - Pros: more flexible elastic behavior; can tolerate worker drop without global barrier. - Cons: harder to keep optimizer correctness (staleness), more complex consistency; network bottleneck at PS; additional state durability needed.- Preferred: all-reduce + optimizer-shard (ZeRO) for throughput and correct optimizer state; parameter-server reserved for specific use-cases (e.g., very sparse embeddings).

Testing & fault injection at scale:- Chaos engineering pipeline: - Node kill: randomly kill GPU pods mid-step and observe recovery/resume; vary timing to hit different code paths (during all-reduce, during checkpoint). - Network partition/packet loss: use tc/netem to introduce latency/packet drop between subsets of nodes, simulate partial rings and ensure reconfiguration handles NCCL timeouts gracefully. - Disk/object-store failures: simulate slow or failing S3 uploads, corrupted chunks (checksum mismatch), and test coordinator abort/rollback logic. - Preemption: simulate spot instance termination signals; ensure preemption hook triggers expedited checkpoint.- Automated regression: deterministic replay harness with synthetic workloads and seeded RNGs to reproduce failures; unit/integration tests for two-phase checkpoint commit, shard reconstruction, and optimizer state merge.- Scale tests: run large-scale test with fault schedules (Netflix Simian Army style) on staging cluster that mimics prod scale; measure wasted steps, time-to-recover, and model divergence vs golden baseline.- Metrics & alarms: instrument checkpoint latency, time-to-commit, number of retried iterations, model divergence (e.g., L2 diff from baseline), and throughput.

Implementation notes & best practices:- Use NCCL/UCX for high-performance all-reduce; use Horovod or torch.distributed elastic + rendezvous for reconfiguration.- Use versioned, immutable checkpoints and metadata catalog (e.g., DynamoDB/etcd) to avoid partial read.- Ensure deterministic sharding and RNG seeds; persist RNG state to guarantee bitwise restart where needed.- Timeout tuning: set NCCL/all-reduce timeouts conservatively; use exponential backoff for retries before aborting to checkpoint.- Security: encrypt checkpoints at rest and in transit.

This design balances correctness (coordinated checkpoints, optimizer state durability), minimal wasted work (incremental checkpoints, quick failover), and practical orchestration for large-scale GPU training.

HardTechnical

25 practiced

Given the following simplified PyTorch training loop snippet, identify defensive programming issues (seed handling, resource leaks, missing exception handling, non-atomic checkpointing, possible OOM from gradient accumulation). Then rewrite the loop to address these issues using best practices (deterministic seeding, safe checkpointing with atomic rename, try/finally or context managers, gradient clipping, and optional mixed-precision loss scaling).

for epoch in range(epochs):
    for batch in dataloader:
        optimizer.zero_grad()
        outputs = model(batch['x'])
        loss = loss_fn(outputs, batch['y'])
        loss.backward()
        optimizer.step()
    torch.save(model.state_dict(), 'checkpoint.pt')

Provide the corrected version and explain each change.

Sample Answer

Approach: make training deterministic where practical, avoid resource leaks and unhandled exceptions, prevent non-atomic checkpoint overwrite, protect against OOM during accumulation, and optionally use AMP + loss scaling and gradient clipping.

python

import os
import torch
from torch.cuda.amp import GradScaler, autocast

def set_seed(seed):
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    import random, numpy as np
    random.seed(seed)
    np.random.seed(seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False

def save_checkpoint_atomic(state, path):
    tmp = f"{path}.tmp"
    torch.save(state, tmp)
    os.replace(tmp, path)  # atomic on POSIX and Windows since Python 3.3

def train(model, dataloader, optimizer, loss_fn, epochs,
          device='cuda', seed=42, clip_norm=None,
          accumulate_steps=1, use_amp=True, ckpt_path='checkpoint.pt'):
    set_seed(seed)
    model.to(device)
    scaler = GradScaler(enabled=use_amp and device=='cuda')
    try:
        for epoch in range(epochs):
            model.train()
            optimizer.zero_grad()
            for step, batch in enumerate(dataloader, 1):
                inputs = batch['x'].to(device, non_blocking=True)
                targets = batch['y'].to(device, non_blocking=True)

                with autocast(enabled=scaler.is_enabled()):
                    outputs = model(inputs)
                    loss = loss_fn(outputs, targets) / accumulate_steps

                scaler.scale(loss).backward()

                if step % accumulate_steps == 0:
                    if clip_norm:
                        # unscale before clipping
                        scaler.unscale_(optimizer)
                        torch.nn.utils.clip_grad_norm_(model.parameters(), clip_norm)
                    scaler.step(optimizer)
                    scaler.update()
                    optimizer.zero_grad()

            # safe atomic checkpointing and persist optimizer/scaler state
            checkpoint = {
                'epoch': epoch,
                'model_state': model.state_dict(),
                'optim_state': optimizer.state_dict(),
                'scaler_state': scaler.state_dict() if scaler is not None else None,
                'seed': seed
            }
            save_checkpoint_atomic(checkpoint, ckpt_path)

    except Exception:
        # optionally log exception, ensure checkpointing of progress before re-raising
        try:
            save_checkpoint_atomic({
                'model_state': model.state_dict(),
                'optim_state': optimizer.state_dict(),
                'scaler_state': scaler.state_dict() if scaler is not None else None,
                'note': 'exception_checkpoint'
            }, ckpt_path + ".failed")
        except Exception:
            pass
        raise
    finally:
        # best-effort cleanup of GPU memory
        try:
            del inputs, targets, outputs, loss
        except NameError:
            pass
        torch.cuda.empty_cache()

Key changes and why:- Deterministic seeding: set_seed sets PyTorch, CUDA, random, numpy seeds and disables cuDNN nondeterminism to improve reproducibility.- non_blocking=True and .to(device): improves throughput and avoids accidental host/GPU copies blocking.- Mixed precision (autocast + GradScaler): reduces memory and speeds up FP16-capable GPUs; scaler handles loss scaling to avoid underflow.- Gradient accumulation: divide loss by accumulate_steps and only step optimizer after accumulated steps to control effective batch size and OOM risk.- Unscale before gradient clipping: scaler.unscale_ then clip_grad_norm_ to correctly clip true gradients.- Atomic checkpointing: save to a tmp file then os.replace to avoid corrupt/partial checkpoints if crash occurs during save.- Save optimizer and scaler states: necessary to resume correctly.- try/except/finally: captures exceptions, attempts to save a final checkpoint, re-raises for caller to handle, and performs cleanup to free GPU memory.- torch.cuda.empty_cache() and explicit del: best-effort resource cleanup to reduce memory leaks.- Minimal logging omitted—add structured logging in real code.

Edge cases: full determinism on CUDA is not always guaranteed across drivers/hardware; atomic replace semantics rely on filesystem semantics (network filesystems may differ).

MediumTechnical

25 practiced

You're observing occasional numerical overflow when computing softmax in inference and training, producing NaNs and unstable training. Describe defensive coding and architectural fixes you would implement across code, model definitions, and the training loop to prevent this (include numerically-stable softmax, gradient clipping, loss scaling, input normalization, and monitoring).

Sample Answer

Start with a clear separation of layers: enforce numerical stability at the math layer, add model-level guards, and add training-loop/runtime protections and monitoring.

Code-level fixes- Use numerically stable softmax: subtract max before exponentiating.

python

import numpy as np

def stable_softmax(x, axis=-1):
    # x: numpy array or tensor-like
    x_max = np.max(x, axis=axis, keepdims=True)
    e = np.exp(x - x_max)            # avoid huge exponents
    return e / np.sum(e, axis=axis, keepdims=True)

- For frameworks (PyTorch/TF) prefer builtin stable ops (F.log_softmax + exp) and use log-softmax + nll when possible to avoid separate softmax then log.

Model-definition fixes- Output scaling: add a learnable temperature parameter or constrain logits (e.g., clip logits to a reasonable range) to avoid extremely large magnitudes.- Use activation choices that limit extremes (e.g., scaled tanh where appropriate).- Mixed precision: if using FP16, annotate layers sensitive to precision and keep logits / loss in FP32.

Training-loop/runtime fixes- Gradient clipping: clip by norm or value to prevent exploding gradients.

python

# PyTorch-style
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)

- Loss scaling for mixed precision: use dynamic loss scaling (Apex or native AMP) to avoid underflow/overflow.- Input normalization: standardize/scale inputs, embeddings, and any features to expected ranges.- Regularization: weight decay and gradient noise can reduce runaway weights.

Defensive programming & monitoring- Add assertions and checks: detect NaNs/Infs after forward, before backward, and after optimizer.step; raise informative errors.- Automatic recovery: on detecting NaN/Inf save model state, reduce lr, rollback optimizer, or restart from checkpoint with reduced LR or stronger clipping.- Logging and metrics: track max/min logits, gradient norms, loss scale, and FP16/FP32 status. Alert thresholds when logits exceed e.g. |logit|>1e3.- Unit tests: synthetic inputs that exercise extremes to validate softmax and loss computations remain finite.

Trade-offs & reasoning- Subtracting max is cheap and removes primary overflow cause in softmax. Gradient clipping and loss scaling address downstream instability. Monitoring + automatic rollback prevents silent model corruption in long runs.- Constraining logits or temperature slightly biases outputs but is acceptable versus failing training.

These combined layers—stable numerics in functions, guarded model definitions, conservative training-loop operations, and proactive monitoring—prevent overflow/NaN propagation and make training robust in production.

MediumTechnical

31 practiced

Describe a time (or design a plan) where you led a push to improve code quality for ML engineers: introducing linters, pre-commit hooks, test coverage goals, and CI gating. Explain how you gained buy-in, measured impact (KPIs), iterated on rollout, and minimized developer friction.

EasyTechnical

24 practiced

Explain why small, single-responsibility functions and modular design are particularly important in AI codebases (training loops, data transforms, model layers). Give two concrete examples where modularity reduced bugs or sped up experimentation and describe criteria you use to split or merge modules.

Unlock Full Question Bank

Get access to hundreds of Code Quality and Defensive Programming interview questions and detailed answers.

Join thousands of developers preparing for their dream job.