Systematic Debugging and Root Cause Analysis Questions

Root cause analysis for an AI system layers standard software RCA discipline with a few concerns unique to models: the "bug" can be in data, labels, features, training code, or the trained artifact itself, not only in application logic.

What distinguishes AI RCA from general software debugging

1. The failure surface includes data and statistical drift, not just code paths: a metric can degrade with zero code changes if the input distribution shifted.
2. Behavior is probabilistic, so a single wrong prediction is not automatically a bug; RCA needs an aggregate, statistically-grounded view (a cohort, a distribution) rather than one anecdote.
3. Training and serving are separate systems that must match (features, preprocessing, versions); a mismatch between them is a distinct, common AI-specific failure class with no equivalent in ordinary software.

Typical steps when a key metric degrades

Check for a training/serving skew or preprocessing mismatch first (fast, common, cheap to rule in/out); check whether the input data distribution shifted versus training time; check whether the evaluation/metric computation itself changed (a silent bug in the eval script is common); only then suspect the model weights/training process itself. Concretely: suppose precision drops from 0.91 to 0.74 in production. First confirm the eval script's feature computation still exactly matches the serving pipeline's (a training/serving skew would explain a drop this size on its own); if that checks out, compare the day's input feature distributions against training-time statistics for drift before ever concluding the model itself needs retraining.

Four common mistakes teams make

1. Assuming a metric drop means "the model got worse" without first ruling out a measurement bug in the eval pipeline.
2. Treating one bad prediction as proof of a systemic issue instead of checking whether it's within expected error-rate variance.
3. Skipping the check for training/serving skew, which is one of the most common and most fixable causes.
4. Retraining reflexively as the default fix before confirming the root cause, which is slow, expensive, and doesn't help if the cause was a data or measurement bug that retraining won't touch.

Trade-offs and pitfalls

Because AI failures are probabilistic, RCA needs a large-enough sample before drawing conclusions; acting on a handful of anecdotal bad predictions without checking statistical significance is the single most common way an AI RCA goes down the wrong path.

MECE (mutually exclusive, collectively exhaustive) means the set of hypotheses you generate should not overlap with each other, and together should cover every plausible explanation, so the investigation doesn't waste effort on redundant branches or miss an entire category.

Example: a 15% drop in purchase conversions over 48 hours

A MECE hypothesis set might be: (1) a technical/instrumentation issue (tracking broke, so the drop is an artifact, not real); (2) a product/UX change (a recent release made checkout harder); (3) a traffic-mix change (a shift toward a lower-converting channel or user segment, with underlying per-segment conversion unchanged); (4) an external factor (a competitor promotion, a payment processor outage). Each branch is a genuinely different underlying story, and together they cover "the number changed" without needing a fifth vague catch-all.

Teaching a junior to build one

Give them a real past incident's headline symptom only (not the answer), have them independently produce 4-6 hypothesis branches, then grade against two criteria: exclusivity (do any two branches actually describe the same underlying mechanism, just worded differently) and exhaustiveness (is there a plausible cause that fits none of the branches, which usually means a whole category, like "instrumentation," was missed entirely). A common junior mistake is generating several branches that are really the same hypothesis restated, which feels thorough but isn't actually MECE.

Trade-offs and pitfalls

A perfectly MECE set takes real effort to construct and can slow down an initial response under time pressure; the practical compromise is starting with a fast, rougher hypothesis list to drive the first triage actions, then tightening it into a properly MECE set as the investigation continues past the first hour.

The training loop never calls optimizer.zero_grad(), so gradients (a gradient is the direction and size of the nudge the optimizer applies to each of the model's parameters to reduce the loss, computed fresh from each batch by loss.backward()) from every batch accumulate on top of previous batches' gradients instead of being reset each step; the optimizer then applies an update based on this ever-growing accumulated gradient, not the current batch's actual gradient.

for epoch in range(epochs):
    for x, y in train_loader:
        preds = model(x)
        loss = loss_fn(preds, y)
        loss.backward()          # gradients ADD to whatever was already there
        optimizer.step()
        # missing optimizer.zero_grad()
    val_preds = model(val_x)
    val_acc = accuracy(val_preds, val_y)

A tiny concrete trace of the accumulation

Suppose batch 1 alone would produce a gradient of 0.4 for some parameter, and with zero_grad() correctly resetting first, optimizer.step() applies an update proportional to exactly that 0.4. Without the reset, batch 2's independently-computed gradient of -0.3 for that same parameter does not replace the 0.4, it adds to it: loss.backward() accumulates onto the existing .grad tensor rather than overwriting it, so the optimizer sees 0.4 + (-0.3) = 0.1, a stale, watered-down mixture of two different batches' signals rather than either batch's actual gradient. By batch 50, the value the optimizer applies is the running sum of 50 different batches' gradients, most of them already stale relative to the model's current state, which is why the update direction becomes increasingly disconnected from what any single current batch is actually telling the model to do.

Why training loss decreases but validation accuracy doesn't improve

Early in training, the accumulated gradient still points in a broadly useful direction often enough that training loss can appear to decrease (the model is still nudged toward fitting the training batches it just saw); but as gradients keep accumulating across many batches without ever resetting, the effective step size on later batches becomes dominated by a growing sum from many stale, unrelated batches, producing increasingly erratic and progressively less meaningful parameter updates. That destabilizes generalization even while the reported training loss can look deceptively reasonable, which is exactly why validation accuracy is the signal that exposes the bug while training loss alone might not.

Minimal patch

for epoch in range(epochs):
    for x, y in train_loader:
        optimizer.zero_grad()     # reset before each step
        preds = model(x)
        loss = loss_fn(preds, y)
        loss.backward()
        optimizer.step()

Complexity

No change; zero_grad() is O(number of parameters), negligible next to the forward/backward pass cost.

Edge cases and regression test

Add an assertion-based test: run two steps on identical, fixed synthetic data with and without zero_grad(), and assert the gradient tensor's norm after the second step matches the expected single-batch gradient (not roughly double it) when zero_grad() is present; this directly catches a regression if the reset is accidentally removed again, rather than relying on noticing a slow validation-accuracy degradation much later.

Inference results differing for the same user on the same model version usually trace to a source of nondeterminism the system didn't intend: sampling, non-deterministic GPU ops, model warm-up state, or input preprocessing that isn't as stable as assumed.

Investigation plan

1. Rule out sampling first: if the model uses stochastic decoding (temperature/top-k sampling) rather than greedy/deterministic decoding, differing outputs are expected, not a bug; confirm the decoding config actually used at inference time.
2. Check for non-deterministic ops: certain GPU kernels (some cuDNN (NVIDIA's GPU library that implements the low-level convolution math deep-learning frameworks call into) convolution algorithms, atomic-based reductions) are not bit-exact deterministic by default; verify whether determinism flags were set (e.g. forcing deterministic algorithms) and whether that setting differs across the serving fleet.
3. Model warm-up state: confirm whether the model or its runtime has any stateful warm-up (cached graph optimizations, JIT compilation choices) that could differ between a cold and warm instance serving the same request.
4. Input preprocessing: verify the exact preprocessing pipeline is byte-for-byte identical for the same logical input across requests (floating-point order-of-operation differences, locale-dependent formatting, or non-deterministic tokenization can all silently vary the input tensor).

Verified: seeding gaps produce exactly this class of bug

The same root cause class shows up, and can be directly reproduced, in the RNG layer around a model rather than the model itself. Two concrete, executed examples:

1) Cross-library seeding gap (seeding random but forgetting numpy, or vice versa, so only part of the pipeline is actually reproducible):

import random
import numpy as np

def naive_seed_everything(seed):
    random.seed(seed)          # BUG: forgets numpy (or any other RNG source)

def full_seed_everything(seed):
    random.seed(seed)
    np.random.seed(seed)       # FIX: seed every RNG source the pipeline touches

# naive: random.random() is reproducible run-to-run, np.random.rand() is NOT,
# because something else in the process advances numpy's global RNG state
# between runs. Executed result:
#   naive : random() reproducible=True   numpy reproducible=False
#   fixed : random() reproducible=True   numpy reproducible=True

2) Worker-seeding bug in a multi-worker data loader (seeding every worker with the same fixed value at init makes every worker's "shuffle" produce the identical, not independently random, sequence):

import random

def buggy_worker_shuffle(worker_id, base_seed, dataset):
    rng = random.Random(base_seed)                       # BUG: ignores worker_id
    order = dataset[:]
    rng.shuffle(order)
    return order

def fixed_worker_shuffle(worker_id, base_seed, dataset, epoch=0):
    rng = random.Random(base_seed + worker_id * 1000 + epoch)  # FIX: per-worker, per-epoch
    order = dataset[:]
    rng.shuffle(order)
    return order

# Executed with 4 workers, dataset of 20 items, base_seed=7:
#   buggy_worker_shuffle -> all 4 workers produce the IDENTICAL order
#   fixed_worker_shuffle -> all 4 workers produce DISTINCT orders, and the
#                           same base_seed across two separate runs reproduces
#                           the identical set of 4 orders (still deterministic,
#                           just no longer duplicated across workers)

Both snippets were executed directly (Python 3, stdlib random + numpy); the "buggy" variants reproduce exactly the failure mode described (unintended duplication / partial irreproducibility), and the "fixed" variants confirm the corrected behavior (independent-but-reproducible per-worker sequences; fully reproducible cross-library output).

Trade-offs and pitfalls

Forcing full determinism (deterministic GPU algorithms, single-threaded data loading) is a useful diagnostic mode but often costs real throughput; it's typically enabled to investigate, then a narrower fix (proper independent-but-reproducible seeding, confirming decoding config) is shipped rather than leaving full determinism mode on in production permanently.

A nightly metrics pipeline drifting lower while the live production model stays stable is strong evidence the pipeline itself changed, not the model: the investigation should first rule out the pipeline before doubting the model.

RCA plan

1. Checksum the inputs: compare the checksum/hash of the dataset the nightly job consumed against the expected/previous dataset; a silent upstream data change (a new day's data with a schema drift, a partial load, a different sample) is the most common cause of "the eval script says worse, but production says fine."
2. Dataset versioning: pin and log the exact dataset version/snapshot used for each nightly run so a regression can be attributed to a specific version bump rather than an ambiguous "sometime between runs."
3. Golden artifacts: maintain a small, fixed, hand-verified reference dataset with known expected metric values; rerun the nightly evaluation code against that golden set specifically, and if it now produces different numbers than its own historical golden-set baseline, the bug is in the evaluation code, not the data; if the golden-set numbers are unchanged, the regression is genuinely in the day's input data. For example, the golden set's accuracy might hold steady at 0.842 across both last night's run and a rerun today, while the day's actual input checksum changed from abc123 to def456; that combination, golden set stable, real input checksum different, is the concrete evidence that the input data changed and the evaluation code did not.
4. Compare the nightly pipeline's own historical metric on a fixed replay dataset run-over-run, isolating whether the pipeline's code changed behavior independent of data.

Trade-offs and pitfalls

Skipping straight to "the model regressed" without checking the pipeline first risks an expensive, unnecessary retraining or rollback cycle for a bug that was actually in the measurement, not the thing being measured; the golden-artifact check is cheap and should run before any model-level conclusion is drawn.