Debugging and Troubleshooting AI Systems Questions

Covers systematic approaches to find and fix failures in machine learning and artificial intelligence systems. Topics include common failure modes such as poor data quality, incorrect preprocessing, label errors, data leakage, training instability, vanishing or exploding gradients, numerical precision issues, overfitting and underfitting, optimizer and hyperparameter problems, model capacity mismatch, implementation bugs, hardware and memory failures, and production environment issues. Skills and techniques include data validation and exploratory data analysis, unit tests and reproducible experiments, sanity checks and simplified models, gradient checks and plotting training dynamics, visualizing predictions and errors, ablation studies and feature importance analysis, logging and instrumentation, profiling for latency and memory, isolating components with canary or shadow deployments, rollback and mitigation strategies, monitoring for concept drift, and applying root cause analysis until the underlying cause is found. Interviewers assess the candidate on their debugging process, ability to isolate issues, use of tools and metrics for diagnosis, trade offs in fixes, and how they prevent similar failures in future iterations.

HardSystem Design

40 practiced

A complex Apache Beam/TF Transform pipeline produces features offline that yield good validation metrics but in production the features differ and model metrics degrade. Propose a systematic approach to isolate whether windowing/sharding/serialization/seed differences are causing the issue and how to validate fixes.

Sample Answer

Requirements and goal clarification:- Goal: determine why offline TF Transform/Beam features differ from production features and cause model degradation.- Constraints: large-scale streaming/batch production, deterministic transforms desired, minimal downtime.

High-level approach (steps):1. Reproduceable data path parity - Capture identical raw inputs used offline and in prod for a representative time window. - Snapshot raw records (or hashes) and store with timestamps and keys.

2. Instrumented feature logging - Add a lightweight feature-logging layer in production that records: raw payload hash, pipeline stage hashes (post-window, post-shard, post-transform), metadata (window, shard id, seed, SDK/runtime versions, serialization format/codec). - Ensure logs are sampled to avoid cost explosion and respect PII.

3. Determinism checks - Windowing: compare window assignments between offline and prod for same event timestamps; mismatch indicates windowing config or watermark differences. - Sharding: verify that grouping/combining keys are identical; non-deterministic combiner order can alter aggregated results. - Serialization: compare serialized bytes or schema evolution; mismatched proto/avro/beam coders change values. - Randomness/seed: ensure RNG/Hash seed parity for any sampling, bucketing, or augmentation.

4. Pairwise replay tests - Replay captured raw records through the offline Beam/TF-Transform code and through the production runtime (in a staging environment) using the same runner and config. Compare per-record features and hashes to pinpoint first divergence stage.

5. Binary diffing & assertions - Compute strong fingerprints of intermediate feature dicts (sorted keys, canonical serialization) to diff quickly. - Add inline assertions in staging that fail when differences exceed tolerance.

6. Automated tests & CI - Unit tests for transform functions with fixed seeds. - Integration tests that run a small sample through full Beam pipeline using portable runner (Direct/Flink) with seeded RNG and assert equality with TF Transform outputs. - Contract tests for schema and coder compatibility.

7. Metrics and monitoring - Production monitoring of feature distributions (histograms, percentiles) vs baseline expected distributions; alert on drift. - Sampling-based end-to-end shadow inference comparing model outputs from prod features and replayed offline features.

Validation of fixes:- After a fix, run replay tests on historical snapshots and live shadow runs for N days; require no significant feature deltas and model metric recovery.- Rollout with canary: enable fixed pipeline for a small percentage of traffic, compare model decisions and business KPIs before full rollout.

Trade-offs and practical notes:- Sampling reduces cost but may miss rare divergences.- Ensure privacy/compliance when logging raw inputs.- Prefer deterministic Beam coders and explicit windowing/watermarks; avoid implicit runner-dependent semantics.- Document seeds and environment versions as part of artifact metadata.

This systematic pipeline—capture → instrument → replay/diff → assert → monitor—isolates windowing/sharding/serialization/seed causes and provides verifiable validation before rollout.

HardTechnical

31 practiced

Implement a numerical gradient-checking utility for a custom PyTorch autograd Function. Include finite-difference approximations, handling of vector-valued outputs, and configurable tolerances. Provide code skeleton and mention numerical pitfalls (eps selection, relative vs absolute error).

Sample Answer

Approach: compute analytical gradients via autograd, compute numerical Jacobian via central finite differences for each scalar input element, handle vector outputs by flattening outputs and comparing Jacobian rows, and compare using configurable absolute and relative tolerances.

python

import torch

def numerical_gradcheck(fn, inputs, eps=1e-6, atol=1e-5, rtol=1e-3, device=None):
    """
    fn: function mapping tuple(inputs) -> tensor (possibly multi-dim). fn should be in eval mode.
    inputs: tuple of torch.Tensor with requires_grad=True
    Returns: (ok:bool, details:str)
    """
    # Ensure inputs are tensors requiring grad
    inputs = tuple(inp.clone().detach().requires_grad_(True) for inp in inputs)
    # Evaluate output and analytic gradients
    out = fn(*inputs)
    out_flat = out.detach().reshape(-1)
    # Compute analytic Jacobian: for each output scalar, backprop a unit vector
    analytic_jacobians = []
    for i in range(out_flat.numel()):
        grads = torch.autograd.grad(out_flat[i], inputs, retain_graph=True, create_graph=False)
        analytic_jacobians.append(torch.cat([g.reshape(-1) for g in grads]).detach())
    analytic = torch.stack(analytic_jacobians)  # shape [M, N] where M outputs, N total input elems

    # Numerical Jacobian via central differences
    num_elems = sum(inp.numel() for inp in inputs)
    numerical = torch.zeros_like(analytic)
    # Precompute flattened views
    flat_inputs = [inp.reshape(-1) for inp in inputs]
    for idx in range(num_elems):
        # map idx to (tensor_i, offset)
        cum = 0
        for t_idx, flat in enumerate(flat_inputs):
            if idx < cum + flat.numel():
                offset = idx - cum
                break
            cum += flat.numel()
        # perturb
        orig = flat_inputs[t_idx][offset].item()
        # +eps
        flat_inputs[t_idx][offset] = orig + eps
        out_p = fn(*[x.reshape(inp.shape) for x, inp in zip(flat_inputs, inputs)]).detach().reshape(-1)
        # -eps
        flat_inputs[t_idx][offset] = orig - eps
        out_m = fn(*[x.reshape(inp.shape) for x, inp in zip(flat_inputs, inputs)]).detach().reshape(-1)
        # restore
        flat_inputs[t_idx][offset] = orig
        # central difference
        numerical[:, idx] = (out_p - out_m) / (2 * eps)

    # Compare
    abs_err = (analytic - numerical).abs()
    rel_err = abs_err / (numerical.abs().clamp(min=1e-12) + analytic.abs().clamp(min=1e-12))
    failed = (abs_err > atol) & (rel_err > rtol)
    if failed.any():
        first = torch.nonzero(failed)[0]
        i_out, j_in = int(first[0]), int(first[1])
        return False, f"Mismatch at output {i_out}, input idx {j_in}: analytic={analytic[i_out,j_in]}, numerical={numerical[i_out,j_in]}, abs_err={abs_err[i_out,j_in]}, rel_err={rel_err[i_out,j_in]}"
    return True, "OK"

Key concepts:- Use central differences for O(eps^2) accuracy.- Flatten inputs and outputs to build full Jacobian for vector outputs.- Computationally expensive: O(M*N) forward evaluations (M outputs, N input elements).

Numerical pitfalls:- eps too large → truncation error; too small → catastrophic cancellation due to floating-point; typical eps ~1e-6 for float32, 1e-8-1e-6 for float64.- Use central difference rather than forward/backward.- Compare using both absolute and relative tolerances because small true gradients need absolute checks while large values need relative checks.- Ensure deterministic fn (no randomness) and avoid in-place ops that break autograd.

MediumTechnical

57 practiced

Training dynamics show loss oscillations and unstable accuracy curves. You suspect optimizer or learning-rate schedule problems. Describe a set of experiments (e.g., LR range test, disabling momentum, different optimizers) to localize the issue and principled changes you'd try to stabilize training.

Sample Answer

Situation: Training shows large loss oscillations and unstable accuracy — likely optimizer or LR-schedule related.

Experiment plan (isolated, one change at a time; fix random seed; log gradients, norms, LR, loss per step):

1) LR range test (Smith):- Sweep LR exponentially from 1e-6 to 10 over a few epochs; plot loss vs LR.- Diagnosis: steep loss decrease then explosion indicates max LR too high; flat/no change suggests LR too low or optimizer issues.

2) Disable or reduce momentum:- For SGD, set momentum=0 then 0.5; for Adam, set beta1=0 (or 0.5).- If oscillations stop when momentum=0, momentum or its interaction with LR/warmup is culprit.

3) Try different optimizers:- Run small runs with SGD+momentum, AdamW, RMSProp, LAMB (if large-batch).- Compare stability/early loss curves. AdamW often more stable initially; SGD generalizes better but needs tuned LR.

4) Test LR schedule and warmup:- Add linear warmup (e.g., 500–2000 steps) then cosine/linear decay.- Try removing aggressive cyclic schedules that spike LR.

5) Gradient diagnostics and mitigations:- Log gradient norms and per-layer spikes.- Apply gradient clipping (global norm 1–5) and see if oscillations reduce.

6) Regularization and batch effects:- Check batch size: smaller batches increase noise — try larger batch or adjust LR proportionally.- Verify weight decay and normalization layers (BatchNorm in train mode).

Principled changes to apply if experiments indicate issues:- Reduce base LR to below LR range-test optimum by 5–10x if instability persists.- Use short linear warmup before schedule.- If momentum causes overshoot, lower momentum and pair with slightly lower LR.- Enable gradient clipping and consider switching to AdamW for robustness.- Repeat with batch-size/LR scaling rules; monitor gradient norms and per-layer learning rates.

Decision rules:- If loss explodes at specific LR → lower LR or stronger warmup.- If disabling momentum stabilizes → lower momentum or use Nesterov off.- If optimizer switch stabilizes → adopt robust optimizer (AdamW) for prototyping, then search for SGD match for final run.

Expected outputs: LR-loss plot, gradient-norm timeline, per-experiment stable/unstable tag — use these to pick final config and run full training.

MediumTechnical

43 practiced

You observe gradients near zero in early layers and large gradients in later layers (vanishing/exploding gradient pattern). Provide a systematic debugging and mitigation plan: initialization schemes, normalization layers, residual connections, activation choices, and learning-rate strategies. Which experiments would you run to validate fixes?

Sample Answer

Start with a diagnostic checklist, then apply targeted fixes and experiments.

Diagnosis (quick tests)- Log per-layer gradient norms and parameter updates during a few training steps to confirm pattern (near-zero in early layers, huge in later).- Check forward activations distribution (saturation/clipping) and weight magnitudes.- Try a tiny learning rate and single-batch backward to see if pattern persists (isolates optimizer effects).

Mitigation plan1. Initialization- Use variance-preserving initializers: He (for ReLU/LeakyReLU) or Glorot/Xavier (for tanh/sigmoid). For deep nets, consider scaled variants (e.g., He normal with fan_in).- For RNNs or very deep MLPs, orthogonal initialization can help.

2. Normalization- Insert BatchNorm or LayerNorm after linear/conv and before activation to stabilize signal and gradients.- For small batch sizes, prefer GroupNorm or LayerNorm.

3. Residual / Skip connections- Add residual connections (identity skips) every few layers to create direct gradient paths and avoid attenuation.- Use pre-activation residual blocks (BN->ReLU->Conv) which empirically improves gradient flow.

4. Activation choices- Replace saturating activations (sigmoid/tanh) with ReLU, LeakyReLU, or SiLU/Swish for smoother gradients.- If dying ReLUs appear, use LeakyReLU or ParametricReLU.

5. Learning-rate and optimizer strategies- Reduce base learning rate; use LR warmup (linear warmup over first N steps).- Use adaptive optimizers (Adam/AdamW) initially, then tune weight decay; try gradient clipping to cap explosions.- Consider layer-wise adaptive LR (LAMB, LARS) for very deep models.

Experiments to validate fixes- Baseline: record per-layer gradient norms, activation means/std, training loss curve.- Ablation experiments (one change at a time): init only; init+norm; init+norm+residual; activation swap; LR warmup. Compare gradient norm heatmaps and training stability.- Convergence/speed: measure epochs-to-target loss and final validation accuracy.- Robustness: run with different seeds and batch sizes.- Visualize gradient flow plots (layers vs steps) to confirm early-layer gradients increase and late-layer extremes reduced.

Interpret results: prefer minimal changes that restore balanced gradient norms and improve convergence without overfitting.

MediumTechnical

40 practiced

You use saliency maps and confusion matrices to debug a multiclass image classifier that systematically fails on a certain subpopulation (e.g., images taken at night). Describe the set of visualization analyses and follow-up experiments you'd perform to confirm root cause and propose remediation.

Sample Answer

First clarify the failure pattern: quantify how much worse performance is on “night” images vs overall (per-class precision/recall, F1, calibration error) and confirm it’s statistically significant (bootstrap or permutation test).

Visualization analyses- Confusion matrix broken out by condition (day vs night): reveal which classes are confused more at night.- Per-class ROC/PR curves and calibration plots for day vs night.- Dataset-level t-SNE/UMAP of deep feature vectors colored by time-of-day and label: check if night images form a separate cluster or overlap with other classes.- Saliency maps (Grad-CAM/Integrated Gradients) for representative correct vs incorrect night examples: examine whether the model focuses on relevant object regions or on spurious cues (e.g., headlights, reflections).- Input distribution diagnostics: histograms of brightness, contrast, color channels; PCA of pixel statistics to show covariate shift.- Failure-case gallery: montage of typical failure examples to spot patterns (motion blur, low contrast, sensor noise, occlusions).- Activation/feature map comparison: visualize intermediate filters for day vs night to see degraded representations.

Follow-up experiments (to confirm root cause)- Controlled ablation: evaluate model on synthetically darkened day images (reduce brightness/contrast) to see if performance drops similarly — isolates lighting vs other covariates.- Train/eval split test: fine-tune a copy using only night images and evaluate on held-out night set to assess whether capacity/data is limiting.- Domain adaptation probe: train classifier on day images plus night-style-transferred images (CycleGAN/Style Transfer) and measure improvement.- Metadata conditioning: add time-of-day or brightness as an input feature to test whether the model benefits from explicit context.- Noise/blur augmentation test: add motion blur, sensor noise to day images to reproduce errors and check sensitivity.

Remediation strategies- Data: collect more labeled night images or augment with photometric transforms (brightness, contrast, gamma) and synthetic style-transferred night images. Use stratified sampling or upweight night examples in loss.- Model: use architectures or normalization robust to illumination (e.g., instance normalization, contrastive pretraining), or add an illumination-invariant front-end (learned histogram equalization).- Training: curriculum or mixup focusing on hard/night examples; domain-adaptive pretraining (self-supervised on unlabeled night images) then fine-tune.- Inference: test-time augmentation (multi-exposure syntheses), ensembling, or confidence-aware abstention for low-illumination inputs.- Calibration & monitoring: recalibrate probabilities separately for night/day and add monitoring alerts when input distribution drifts (brightness histogram triggers).

Evaluation plan- Re-run per-condition confusion matrices, per-class F1, and calibration after each remediation.- Use held-out, real night dataset for final validation and report uplift and remaining gap with confidence intervals.

This sequence confirms whether failure is due to covariate shift (lighting), label scarcity, spurious cues, or model sensitivity — and maps each confirmed cause to targeted fixes (data, model, training, or inference).

Unlock Full Question Bank

Get access to hundreds of Debugging and Troubleshooting AI Systems interview questions and detailed answers.

Join thousands of developers preparing for their dream job.