Systematic Debugging and Root Cause Analysis Questions

Inference results differing for the same user on the same model version usually trace to a source of nondeterminism the system didn't intend: sampling, non-deterministic GPU ops, model warm-up state, or input preprocessing that isn't as stable as assumed.

Investigation plan

1. Rule out sampling first: if the model uses stochastic decoding (temperature/top-k sampling) rather than greedy/deterministic decoding, differing outputs are expected, not a bug; confirm the decoding config actually used at inference time.
2. Check for non-deterministic ops: certain GPU kernels (some cuDNN (NVIDIA's GPU library that implements the low-level convolution math deep-learning frameworks call into) convolution algorithms, atomic-based reductions) are not bit-exact deterministic by default; verify whether determinism flags were set (e.g. forcing deterministic algorithms) and whether that setting differs across the serving fleet.
3. Model warm-up state: confirm whether the model or its runtime has any stateful warm-up (cached graph optimizations, JIT compilation choices) that could differ between a cold and warm instance serving the same request.
4. Input preprocessing: verify the exact preprocessing pipeline is byte-for-byte identical for the same logical input across requests (floating-point order-of-operation differences, locale-dependent formatting, or non-deterministic tokenization can all silently vary the input tensor).

Verified: seeding gaps produce exactly this class of bug

The same root cause class shows up, and can be directly reproduced, in the RNG layer around a model rather than the model itself. Two concrete, executed examples:

1) Cross-library seeding gap (seeding random but forgetting numpy, or vice versa, so only part of the pipeline is actually reproducible):

import random
import numpy as np

def naive_seed_everything(seed):
    random.seed(seed)          # BUG: forgets numpy (or any other RNG source)

def full_seed_everything(seed):
    random.seed(seed)
    np.random.seed(seed)       # FIX: seed every RNG source the pipeline touches

# naive: random.random() is reproducible run-to-run, np.random.rand() is NOT,
# because something else in the process advances numpy's global RNG state
# between runs. Executed result:
#   naive : random() reproducible=True   numpy reproducible=False
#   fixed : random() reproducible=True   numpy reproducible=True

2) Worker-seeding bug in a multi-worker data loader (seeding every worker with the same fixed value at init makes every worker's "shuffle" produce the identical, not independently random, sequence):

import random

def buggy_worker_shuffle(worker_id, base_seed, dataset):
    rng = random.Random(base_seed)                       # BUG: ignores worker_id
    order = dataset[:]
    rng.shuffle(order)
    return order

def fixed_worker_shuffle(worker_id, base_seed, dataset, epoch=0):
    rng = random.Random(base_seed + worker_id * 1000 + epoch)  # FIX: per-worker, per-epoch
    order = dataset[:]
    rng.shuffle(order)
    return order

# Executed with 4 workers, dataset of 20 items, base_seed=7:
#   buggy_worker_shuffle -> all 4 workers produce the IDENTICAL order
#   fixed_worker_shuffle -> all 4 workers produce DISTINCT orders, and the
#                           same base_seed across two separate runs reproduces
#                           the identical set of 4 orders (still deterministic,
#                           just no longer duplicated across workers)

Both snippets were executed directly (Python 3, stdlib random + numpy); the "buggy" variants reproduce exactly the failure mode described (unintended duplication / partial irreproducibility), and the "fixed" variants confirm the corrected behavior (independent-but-reproducible per-worker sequences; fully reproducible cross-library output).

Trade-offs and pitfalls

Forcing full determinism (deterministic GPU algorithms, single-threaded data loading) is a useful diagnostic mode but often costs real throughput; it's typically enabled to investigate, then a narrower fix (proper independent-but-reproducible seeding, confirming decoding config) is shipped rather than leaving full determinism mode on in production permanently.

A nightly metrics pipeline drifting lower while the live production model stays stable is strong evidence the pipeline itself changed, not the model: the investigation should first rule out the pipeline before doubting the model.

RCA plan

1. Checksum the inputs: compare the checksum/hash of the dataset the nightly job consumed against the expected/previous dataset; a silent upstream data change (a new day's data with a schema drift, a partial load, a different sample) is the most common cause of "the eval script says worse, but production says fine."
2. Dataset versioning: pin and log the exact dataset version/snapshot used for each nightly run so a regression can be attributed to a specific version bump rather than an ambiguous "sometime between runs."
3. Golden artifacts: maintain a small, fixed, hand-verified reference dataset with known expected metric values; rerun the nightly evaluation code against that golden set specifically, and if it now produces different numbers than its own historical golden-set baseline, the bug is in the evaluation code, not the data; if the golden-set numbers are unchanged, the regression is genuinely in the day's input data. For example, the golden set's accuracy might hold steady at 0.842 across both last night's run and a rerun today, while the day's actual input checksum changed from abc123 to def456; that combination, golden set stable, real input checksum different, is the concrete evidence that the input data changed and the evaluation code did not.
4. Compare the nightly pipeline's own historical metric on a fixed replay dataset run-over-run, isolating whether the pipeline's code changed behavior independent of data.

Trade-offs and pitfalls

Skipping straight to "the model regressed" without checking the pipeline first risks an expensive, unnecessary retraining or rollback cycle for a bug that was actually in the measurement, not the thing being measured; the golden-artifact check is cheap and should run before any model-level conclusion is drawn.

The same binary-search reasoning that isolates a bad commit also isolates a bad element inside a large input: split the input in half, test each half (or its complement) against the failure, and recurse into whichever piece still reproduces the failure.

Applying it manually and automating it

Given a dataset or a large DAG of transforms that fails somewhere inside it: run the process on the first half; if it still fails, recurse into that half, otherwise recurse into the second half. Concretely, on a 16-row dataset where row 11 is the one that crashes the model: round 1 tests rows 1-8 (passes) and 9-16 (fails), so the search keeps 9-16; round 2 tests 9-12 (fails, since it still contains row 11) and 13-16 (passes), keeping 9-12; round 3 tests 9-10 (passes) and 11-12 (fails), keeping 11-12; round 4 tests row 11 alone (fails) and row 12 alone (passes), isolating row 11 as the minimal failing input in exactly 4 rounds instead of checking all 16 rows one at a time. Automating this needs a test function that returns pass/fail for an arbitrary subset, and the loop halves the search space each round, so a million-row dataset localizes to the failing row(s) in about 20 rounds instead of one at a time.

Nondeterministic failures

If the failure only reproduces some fraction of the time, a single test on a candidate subset is not trustworthy: a subset that doesn't fail on this run might still contain the true trigger. The standard adjustment is to run each candidate subset multiple times and only accept "does not reproduce" after several clean runs, and to keep the last few known-failing subsets so a wrongly-discarded half can be revisited if the search dead-ends.

Trade-offs and pitfalls

Binary search over data assumes the failure is caused by the presence of specific elements, not by the size of the input (e.g. a memory limit hit only at full scale); if halving the input consistently stops reproducing regardless of which half is kept, that is itself a signal the bug is scale-dependent rather than content-dependent, and the investigation should pivot to resource limits instead of continuing to bisect.

Root cause analysis for an AI system layers standard software RCA discipline with a few concerns unique to models: the "bug" can be in data, labels, features, training code, or the trained artifact itself, not only in application logic.

What distinguishes AI RCA from general software debugging

1. The failure surface includes data and statistical drift, not just code paths: a metric can degrade with zero code changes if the input distribution shifted.
2. Behavior is probabilistic, so a single wrong prediction is not automatically a bug; RCA needs an aggregate, statistically-grounded view (a cohort, a distribution) rather than one anecdote.
3. Training and serving are separate systems that must match (features, preprocessing, versions); a mismatch between them is a distinct, common AI-specific failure class with no equivalent in ordinary software.

Typical steps when a key metric degrades

Check for a training/serving skew or preprocessing mismatch first (fast, common, cheap to rule in/out); check whether the input data distribution shifted versus training time; check whether the evaluation/metric computation itself changed (a silent bug in the eval script is common); only then suspect the model weights/training process itself. Concretely: suppose precision drops from 0.91 to 0.74 in production. First confirm the eval script's feature computation still exactly matches the serving pipeline's (a training/serving skew would explain a drop this size on its own); if that checks out, compare the day's input feature distributions against training-time statistics for drift before ever concluding the model itself needs retraining.

Four common mistakes teams make

1. Assuming a metric drop means "the model got worse" without first ruling out a measurement bug in the eval pipeline.
2. Treating one bad prediction as proof of a systemic issue instead of checking whether it's within expected error-rate variance.
3. Skipping the check for training/serving skew, which is one of the most common and most fixable causes.
4. Retraining reflexively as the default fix before confirming the root cause, which is slow, expensive, and doesn't help if the cause was a data or measurement bug that retraining won't touch.

Trade-offs and pitfalls

Because AI failures are probabilistic, RCA needs a large-enough sample before drawing conclusions; acting on a handful of anecdotal bad predictions without checking statistical significance is the single most common way an AI RCA goes down the wrong path.

Correlation is "these two things moved together"; causation is "one of them made the other happen." An incident-diagnostics investigation that stops at correlation risks fixing (or blaming) the wrong thing.

Worked example

Error rate rises right after a deploy. That's a correlation. The actual cause might be the deploy itself, or it might be a third variable: a traffic reroute that happened to land at the same time, sending a different, more error-prone traffic mix to this service, with the deploy being coincidental.

Three ways to test true causation

1. Precise timing alignment: does the error rate change at the exact deploy timestamp (to the second/minute), or does it lag or lead it? A reroute event with its own distinct timestamp that better matches the error onset is evidence against the deploy being the cause.
2. Controlled comparison: roll the deploy back on a subset (a canary) while leaving the traffic conditions unchanged, and see if errors clear on just that subset; if errors persist despite the rollback, the deploy wasn't the cause.
3. Mechanism check: can you point to a specific code/config change in the deploy that plausibly produces this exact error, versus the reroute plausibly sending edge-case traffic (different geography, different client version) that the service already mishandled before the deploy.

Generalizing to model diagnostics

The identical discipline applies to a model metric correlating with a deployment event: before concluding the new model caused a metric drop, check whether a confound (a simultaneous data-pipeline change, a seasonal effect, a shift in the underlying user population) explains it just as well, and use the same three tests (timing precision, a controlled comparison holding the model fixed while varying the other candidate, and a plausible mechanism) before attributing cause.

Trade-offs and pitfalls

Under time pressure, mitigating based on correlation alone (rolling back the most recent deploy because it's the most recent) is often the right immediate action even before causation is proven, but the RCA writeup should not present that correlation as a proven root cause without the additional checks above.