Debugging and Testing ML Systems Questions

Direct answer. When offline metrics improve but the online experiment shows no lift or a regression, the offline metric and the business metric are measuring genuinely different things, and the investigation has to check each place they can diverge rather than assuming the online result is simply "noisier."

Checklist to reconcile the discrepancy.

1. Distribution shift between offline eval traffic and live production traffic. The offline test set is a snapshot; live traffic during the experiment window may have a different mix of users, devices, or time-of-day patterns that the offline set doesn't represent.
2. Calibration differences. A model can have a better AUC (better ranking) while being worse-calibrated (its absolute predicted probabilities are less trustworthy), and if any downstream logic uses the raw probability, not just the ranking, a ranking-only metric like AUC will miss this entirely. To measure it rather than hypothesize it: bucket the holdout predictions into deciles of predicted probability and compare, per bucket, the mean predicted probability against the observed outcome rate. That comparison is a reliability diagram, and its summary number is expected calibration error, the size-weighted average absolute gap between the two. Compute it for both model versions on the same holdout. A new model whose AUC rises while its expected calibration error goes from about 0.01 to about 0.04 is exactly this case, and any downstream rule with a fixed probability threshold (a bid, a notification cutoff, an eligibility gate) will fire at a different rate than it used to even though the ranking got better.
3. Feature availability mismatch. Confirm every feature the offline evaluation used is actually available, fresh, and computed the same way at serving time; a feature that's stale or a no-op in production (silently falling back to a default) can make the online model perform worse than its offline twin despite being "the same model."
4. Position or exposure bias. If the offline metric is computed on logged data that itself came from a previous model's exposure pattern, it inherits that model's blind spots; a new model that ranks differently may simply not be evaluated fairly by metrics computed against the old exposure distribution.
5. Per-segment heterogeneity. An aggregate online metric can hide a model that improved for most users but regressed sharply for a smaller, high-value segment; slice the online result the same way before concluding "no lift" is the whole story.
6. Integrity of the pipeline feeding the ONLINE metric, which is a different pipeline from the one feeding the offline metric and fails in its own ways. Check the sample ratio first: if the traffic split lands at, say, 52/48 when it was configured 50/50, the randomization or the logging is broken and the whole comparison is void before any model question is worth asking. Then confirm the business metric's definition and attribution window did not change during the experiment window, that bot and internal traffic are filtered identically in both arms, and that the randomization unit matches the analysis unit (randomizing by user and analyzing by session inflates significance and can manufacture a regression that is not there).

Two concrete shapes this takes in practice, with the numbers, to ground what "worse" looks like:

shape 1   offline AUC        0.812 -> 0.834   (+2.2 points, +2.7% relative)
          online revenue per session  -1.4%   (95% interval -2.3% to -0.5%)

shape 2   offline logloss    0.412 -> 0.396   (-3.9% relative, an improvement)
          online watch-time per session -3.1% (95% interval -4.4% to -1.8%)

(Logloss is the average negative log of the probability the model assigned to the outcome that actually occurred; lower is better, and unlike AUC it is sensitive to the ABSOLUTE probabilities rather than only their ordering, so it partly captures calibration and partly explains why a model can move the two in opposite directions.)

The intervals are the part that decides whether you have an investigation at all. In both shapes above the 95% interval excludes zero, so the regression is real and worth the checklist. A result like -0.4% with an interval of -1.6% to +0.8% is a different situation entirely: the interval spans zero, so you do not have a regression, you have an underpowered experiment, and the correct next step is more exposure or a longer window rather than a root-cause hunt. Deciding which of those two you are in comes before item 1.

Diagnostic ordering, all six items placed. Order by cost first, and let a free question re-rank the rest. Before (0), the interval check above: no interval excluding zero, no investigation. First (6), experiment and metric-pipeline integrity, because a sample-ratio mismatch or a changed metric definition invalidates every comparison downstream of it, and reading the split and the metric definition costs a query. Second (3), feature-availability mismatch, the least expensive of the model-side checks and the one that turns out to be the culprit most often. Third (5), per-segment heterogeneity, because it is a re-slice of data you already have from the running experiment and it can change what the rest of the investigation is even for: if the aggregate is hiding a sharp regression in one segment, you are now explaining a segment, not a model. Fourth (2), calibration, and note that it is CHEAPER than the two checks below it, not more expensive: it needs only the model's own predicted probabilities scored against outcomes on a fixed holdout, with no live traffic and no exposure comparison. What decides whether it runs here at all is one free question you can answer before running anything, does any downstream system consume the raw probability rather than the rank (a bid, a notification cutoff, an eligibility gate). If yes, calibration moves ahead of everything except (6) and (3), because it is both cheap and the leading hypothesis for exactly this AUC-up-business-metric-down shape. If nothing downstream reads the probability, calibration cannot be the mechanism and you can skip it entirely. Last (1) and (4) together, which genuinely require comparing live traffic and logged-exposure distributions and so cost the most to set up.

Communicating with a skeptical stakeholder. State clearly, and separately, what you have confirmed (for example, "feature parity between offline and online is confirmed, that's ruled out") versus what remains a hypothesis ("we believe this is a calibration issue based on X, but haven't yet confirmed it"), and give a concrete next step and timeline rather than a single verdict. A stakeholder who's skeptical the new model is worse is usually reacting to an unexplained gap, not to the existence of a gap, so showing the specific checks you've run (and their results) does more to build confidence than asserting a conclusion before the investigation is done. Bring the interval with every number you quote: "revenue per session is down 1.4%, interval -2.3% to -0.5%" ends an argument that "revenue looks down" only starts.

Direct answer

A chaos-engineering plan for an ML system deliberately injects a specific, named failure, feature corruption, missing upstream data, a sudden spike in one label's frequency, or delayed inputs, into a small, capped slice of real or shadow traffic, then measures how the system actually degrades against a hypothesis stated before the experiment runs. The two disciplines that separate this from just breaking things in production are a hard blast-radius cap with an automatic abort if the observed error rate crosses a threshold, and a mandatory follow-up step: every experiment that reveals a real weakness produces a tracked fix, not just a finding written down and forgotten.

Structured elaboration

The four failure injections and what each one tests.

• Feature corruption: replace a feature's value with an out-of-distribution or garbage value (large-magnitude noise, a wrong-typed value) for a fraction of requests, simulating an upstream bug that silently mangles a field rather than failing loudly. This tests whether the model (or the pipeline wrapping it) has any input-sanity check at all, or whether it will confidently score garbage.
• Missing upstream data: force a feature to fall back to its default (null, zero, or a mean-imputation value), simulating an upstream service outage or a schema field that stopped being populated. This specifically tests the fallback path, which in a healthy system runs rarely and is therefore the least-exercised code path in the whole pipeline.
• A spike in one label's frequency: flood the system with requests that skew heavily toward one predicted class or one input segment, simulating a real-world event that shifts the traffic mix suddenly (a fraud ring probing one product category, a viral post driving one query type). This tests capacity and downstream-system assumptions that were tuned for an average traffic mix, a fraud-review queue sized for a steady 2% flag rate will behave very differently if flags briefly spike to 40%.
• Delayed inputs: hold back a feature's arrival past its normal latency budget, simulating a slow upstream dependency. This tests the timeout and fallback logic specifically, does the system wait indefinitely (a latency failure), serve a stale cached value (a correctness failure), or fail the request outright (an availability failure), and which of those three is actually the intended, chosen behavior versus an accident of how the code happens to be written.

Capturing and interpreting the system's response. Before running anything, write down the falsifiable hypothesis the experiment tests ("if feature X is corrupted for 10% of traffic, end-to-end accuracy on that slice stays within Y of baseline, because the input-validation layer should catch and reject it"), then measure the actual metric on the actual injected slice against a same-window, non-injected control group, not against a historical average that could be confounded by unrelated factors. Interpretation is comparing the observed result to the hypothesis, not just reporting a number: a result that matches the hypothesis exactly is informative in a different way than one that badly misses it, both are useful, but they call for different follow-ups.

Safety controls so the experiment cannot cause real user harm. Three layers, from least to most severe:

1. A hard blast-radius cap: the experiment is only allowed to touch a small, pre-approved percentage of traffic (single digits for a first run of any new experiment type), enforced in code before the experiment starts, not as a policy someone is trusted to follow.
2. An automatic abort trigger: monitor the injected slice's error rate (or a business metric like conversion) in real time during the experiment, and kill it automatically the moment it crosses a pre-set threshold, rather than waiting for a human to notice.
3. A kill switch and rollback path tested in advance: the mechanism that stops the experiment must itself be tested before the experiment runs, an abort trigger that has never been exercised is not a safety control, it is an unverified assumption.

After a test reveals a real weakness. Treat the finding the same way a production incident would be treated: file it as a tracked issue with an owner and a severity, not just a note in a doc; prioritize the fix by the same criteria as any other production risk (how likely is this failure to occur outside of an experiment, and how bad is it if it does); and, once fixed, re-run the exact same experiment to confirm the fix actually closes the gap, rather than assuming the fix worked because it looks correct in code review.

Worked example

Training a small classifier and measuring the actual impact of the first two injection types on a genuinely held-out test set. The setup is shipped in full rather than assumed, because the accuracy drops below only mean something if you can see how strongly the label depends on the feature being corrupted (here feature 0 carries most of the signal, which is why injecting into it moves the numbers at all):

import torch
import torch.nn as nn

def make_dataset(n, seed):
    gen = torch.Generator().manual_seed(seed)
    X = torch.randn(n, 4, generator=gen)
    logits = 2.5 * X[:, 0] + 0.6 * X[:, 1] - 0.4 * X[:, 2]
    y = (logits > 0).long()
    return X, y

def accuracy(model, X, y):
    with torch.no_grad():
        return (model(X).argmax(dim=1) == y).float().mean().item()

torch.manual_seed(0)
X_train, y_train = make_dataset(2000, seed=1)
model = nn.Sequential(nn.Linear(4, 16), nn.ReLU(), nn.Linear(16, 2))
opt = torch.optim.Adam(model.parameters(), lr=0.02)
for _ in range(400):
    opt.zero_grad()
    nn.functional.cross_entropy(model(X_train), y_train).backward()
    opt.step()
model.eval()

g = torch.Generator().manual_seed(3)   # separate seeded generator for injection sampling

X_test, y_test = make_dataset(300, seed=2)
baseline_acc = accuracy(model, X_test, y_test)
print(f"baseline accuracy (no injection): {baseline_acc:.3f}")

def corrupt(frac):                      # Experiment A: garbage value
    X = X_test.clone()
    idx = torch.randperm(len(X_test), generator=g)[:int(frac * len(X_test))]
    X[idx, 0] = torch.randn(len(idx), generator=g) * 10
    return X

def drop_to_default(frac):              # Experiment B: fallback to the default 0.0
    X = X_test.clone()
    idx = torch.randperm(len(X_test), generator=g)[:int(frac * len(X_test))]
    X[idx, 0] = 0.0
    return X

for title, inject in [("Experiment A: feature corruption (feature 0 replaced with noise)", corrupt),
                      ("Experiment B: missing upstream data (feature 0 falls back to 0.0)", drop_to_default)]:
    print()
    print(title)
    for frac in [0.1, 0.5, 1.0]:
        acc = accuracy(model, inject(frac), y_test)
        print(f"  injected fraction={frac:.0%}: accuracy={acc:.3f} (drop={baseline_acc - acc:.3f})")

Actual output:

baseline accuracy (no injection): 0.993

Experiment A: feature corruption (feature 0 replaced with noise)
  injected fraction=10%: accuracy=0.943 (drop=0.050)
  injected fraction=50%: accuracy=0.753 (drop=0.240)
  injected fraction=100%: accuracy=0.547 (drop=0.447)

Experiment B: missing upstream data (feature 0 falls back to 0.0)
  injected fraction=10%: accuracy=0.953 (drop=0.040)

At every injected fraction, garbage-value corruption hurts accuracy more than a sane zero-fallback (0.050 against 0.040 at 10%, 0.240 against 0.207 at 50%, and 0.447 against 0.423 at 100%), a concrete, falsifiable version of the intuition that a defined default is safer than an undefined one, and exactly the kind of result a chaos experiment is supposed to produce: a number to argue from instead of an assumption. Note how small the gap is at 100% injection, 0.447 against 0.423. Both injections destroy the feature's signal; the defined default is better, but only slightly, which is a more useful finding than the intuition alone, since it says the real mitigation is not a better default value but detecting that the feature is broken at all.

The blast-radius safety control, exercised on all three of its outcomes: a request that violates the cap, one that respects it and completes, and one that respects the cap but trips the abort threshold mid-experiment. The third case matters most and is the one usually left untested, because an abort trigger that has never fired is an assumption rather than a control:

def run_capped_experiment(inject_fn, frac, max_blast_radius=0.05, abort_error_rate=0.5):
    if frac > max_blast_radius:
        return {"status": "REJECTED_BEFORE_START",
                "reason": f"requested {frac:.0%} exceeds blast-radius cap {max_blast_radius:.0%}"}
    error_rate = 1 - accuracy(model, inject_fn(frac), y_test)
    if error_rate > abort_error_rate:
        return {"status": "ABORTED_MID_EXPERIMENT", "error_rate": round(error_rate, 3)}
    return {"status": "COMPLETED", "error_rate": round(error_rate, 3)}

print("requesting a 50% blast radius (should be rejected before it ever runs):")
print(run_capped_experiment(corrupt, 0.50))
print("requesting a 3% blast radius (within cap, should run):")
print(run_capped_experiment(corrupt, 0.03))
print("a 4% experiment whose injected slice errors past the abort threshold:")
print(run_capped_experiment(corrupt, 0.04, abort_error_rate=0.01))

Output:

  injected fraction=50%: accuracy=0.787 (drop=0.207)
  injected fraction=100%: accuracy=0.570 (drop=0.423)
requesting a 50% blast radius (should be rejected before it ever runs):
{'status': 'REJECTED_BEFORE_START', 'reason': 'requested 50% exceeds blast-radius cap 5%'}
requesting a 3% blast radius (within cap, should run):
{'status': 'COMPLETED', 'error_rate': 0.023}
a 4% experiment whose injected slice errors past the abort threshold:
{'status': 'ABORTED_MID_EXPERIMENT', 'error_rate': 0.02}

Trade-offs and pitfalls

• A capped blast radius trades statistical power for safety, and that is the correct trade. A 3% slice gives a noisier estimate of the true impact than 50% would, but the entire point of the cap is that you are not yet confident enough in the outcome to accept the larger slice's risk; run small first, and only widen the blast radius once the small run's result matches the hypothesis.
• Injecting on shadow or synthetic traffic first, and only later on a capped slice of real traffic, catches a different class of bug than either alone. Shadow traffic is safe but might not reproduce real timing and concurrency conditions; a real capped slice is realistic but risks a real (small) user impact, so a mature program runs both, in that order, rather than picking one.
• The 'spike in one label' injection is the easiest of the four to get wrong by testing the model instead of the system. The model itself usually handles a skewed prediction mix without complaint; the actual fragile point is almost always a downstream system (a review queue, a rate limiter, a notification budget) that was capacity-planned around the normal mix, so point this experiment at the whole pipeline, not just at the model's own accuracy.
• A finding that reveals no weakness is not a wasted experiment. It is tempting to only report and act on experiments that find a problem; a clean result that matches the stated hypothesis is evidence the system's fallback behavior works as designed, which is exactly what justifies the next experiment's larger blast radius.

Direct answer

Unit tests are fastest, least flaky, and cheapest to maintain, but for ML systems specifically they catch a smaller share of the bugs that actually matter than the classic test-pyramid advice would predict, because the highest-cost ML failures, data leakage, distribution shift, a silently wrong loss computation, live at the boundary between components and real data, not inside a single pure function. Given a limited testing budget for an ML system, I would invest more heavily in integration-level tests running against realistic (not toy) data than the standard web-service pyramid recommends, because that is where this system's actual failure modes concentrate, even though those tests are slower and somewhat more expensive to maintain than a typical web service's integration layer.

Structured elaboration

The three levels, compared on ML systems specifically.

Where the highest-value ML bugs actually live. A unit test of impute_median in isolation, tests a pure function on a five-row fixture. That is real and useful, but it cannot catch data leakage, because leakage is a property of how two pipeline stages interact with real data (a feature computed with future information, a preprocessing step accidentally fit on the validation split), not a property of any single function tested alone. The same is true for training-serving skew (the training pipeline and the serving pipeline computing a feature slightly differently) and for a mis-specified loss that only manifests once real label distributions and real batch compositions are involved. These are exactly the class of bug that tends to dominate real ML incident reviews, and unit tests structurally cannot see them, because by construction a unit test isolates one function from the very interactions where the bug lives.

Given a limited budget, where to invest for the highest return. Spend the first, cheapest slice of budget on unit tests for anything with a genuinely deterministic, checkable output (metric formulas, tensor-shape-transforming utility functions), since these are nearly free and catch real bugs. Spend the next slice, and the largest single slice if the budget is truly limited, on integration tests that run the pipeline's actual stages against realistic synthetic or sampled data, specifically targeting the seams: does the train-time feature computation match the serve-time feature computation on the same input, does a held-out split stay genuinely held out through every preprocessing step. Reserve E2E tests for the smallest slice, a handful of full-pipeline smoke tests that confirm the whole system still produces a valid model artifact and a valid prediction, since E2E tests are the most expensive to write, run, and maintain, and mostly duplicate coverage the integration layer already provides for this kind of system.

Why this differs from the usual advice for a typical web-service codebase. The standard test pyramid (many unit tests, fewer integration tests, very few E2E tests) is calibrated for systems where most of the business logic lives inside individually-testable functions and classes, and where the main risk of testing "too high" in the pyramid is flakiness and slow CI (continuous integration, the automated pipeline that runs checks on every proposed change) for not much extra bug-catching power. That calibration assumes the bugs that matter are reachable by testing components in isolation. In an ML pipeline, the components can each be individually correct, the imputation function imputes correctly, the scaler scales correctly, the loss function computes the right formula, and the system can still be badly wrong, because the failure is in how data flows between them, not in any one function's logic. That is the concrete reason the investment should shift toward the integration layer here specifically: not because integration tests are generically more valuable, but because this system's dominant bug class only exists at that layer.

Worked example

A concrete demonstration of the exact gap the direct answer points at. A three-stage pipeline: impute_median fills missing values with the column median, fit_scaler computes a min/max range from a dataset, apply_scaler rescales a dataset using a previously-fit range:

import numpy as np

def impute_median(v):
    v = np.asarray(v, dtype=float)
    return np.where(np.isnan(v), np.nanmedian(v), v)

def fit_scaler(v):
    return (float(np.min(v)), float(np.max(v)))

def apply_scaler(v, scaler_range):
    lo, hi = scaler_range
    return (np.asarray(v, dtype=float) - lo) / (hi - lo)

Three unit tests, each on a small isolated fixture:

def test_impute_median():
    v = np.array([1.0, np.nan, 3.0, np.nan, 5.0])
    out = impute_median(v)
    assert not np.isnan(out).any() and out[1] == 3.0

def test_fit_scaler():
    v = np.array([2.0, 4.0, 6.0, 8.0, 10.0])
    assert fit_scaler(v) == (2.0, 10.0)

def test_apply_scaler():
    v = np.array([2.0, 6.0, 10.0])
    assert np.allclose(apply_scaler(v, (2.0, 10.0)), [0.0, 0.5, 1.0])

for t in (test_impute_median, test_fit_scaler, test_apply_scaler):
    t()
    print(f"unit test: {t.__name__[5:]:14s} -> PASS")

Output:

unit test: impute_median  -> PASS
unit test: fit_scaler     -> PASS
unit test: apply_scaler   -> PASS

All three individually correct. Now the actual bug: the pipeline's wiring code calls fit_scaler on the FULL dataset (train and validation rows together) before splitting, then applies that shared range to each partition separately:

def BUGGY_pipeline(full_data, val_start_idx):
    scaler_range = fit_scaler(full_data)  # BUG: fit on train+val together
    train, val = full_data[:val_start_idx], full_data[val_start_idx:]
    return apply_scaler(train, scaler_range), apply_scaler(val, scaler_range), scaler_range

def FIXED_pipeline(full_data, val_start_idx):
    train, val = full_data[:val_start_idx], full_data[val_start_idx:]
    scaler_range = fit_scaler(train)      # fit on train ONLY
    return apply_scaler(train, scaler_range), apply_scaler(val, scaler_range), scaler_range

An integration test on a synthetic fixture built specifically to make the leak observable (validation values drawn from 100-110, training values from 0-10, so a leaked range is unmistakable), asserting the fitted range must equal the train partition's own range. The fixture is seeded, because an unseeded fixture would make the printed range values change on every run and there would be no way to tell a real change in behavior from a new draw:

rng = np.random.default_rng(0)
train_part = rng.uniform(0, 10, size=50)
val_part = rng.uniform(100, 110, size=50)
full_data = np.concatenate([train_part, val_part])
VAL_START = 50

def integration_test(pipeline, label):
    _, _, fitted = pipeline(full_data, VAL_START)
    expected = fit_scaler(full_data[:VAL_START])
    ok = np.allclose(fitted, expected)
    print(f"  {label}: fitted range=({fitted[0]:.3f}, {fitted[1]:.3f}), "
          f"train-only range=({expected[0]:.3f}, {expected[1]:.3f}) -> {'PASS' if ok else 'FAIL'}")

print("integration test: scaler must be fit on the TRAIN partition only")
integration_test(BUGGY_pipeline, "buggy wiring (fit on train+val) ")
integration_test(FIXED_pipeline, "fixed wiring (fit on train only)")

Actual output:

integration test: scaler must be fit on the TRAIN partition only
  buggy wiring (fit on train+val) : fitted range=(0.027, 109.951), train-only range=(0.027, 9.972) -> FAIL
  fixed wiring (fit on train only): fitted range=(0.027, 9.972), train-only range=(0.027, 9.972) -> PASS

The three unit tests above pass unmodified against either pipeline version, because none of them exercises the wiring; only the integration test, which runs the real end-to-end call sequence, distinguishes the buggy version (fitted upper bound balloons from 9.972 to 109.951 because it silently absorbed the validation partition) from the fixed one. This is the concrete version of "the components can each be individually correct and the system can still be wrong": three green unit tests and one integration test that correctly fails on the buggy wiring and correctly passes on the fixed wiring, on the exact same three functions.

Trade-offs and pitfalls

• Overweighting integration tests without any unit tests loses fast, cheap, precise failure localization. When an integration test fails, it tells you the pipeline is broken somewhere in a multi-stage sequence; a unit test failing tells you exactly which function and which line. Keep a baseline of unit tests on the pipeline's individual transformation functions specifically so failures are diagnosable quickly, not just detectable.
• Integration tests against "realistic" data need a genuinely disciplined definition of realistic, or they degrade back into toy tests with extra steps. A synthetic integration fixture that does not reproduce the real data's key structural properties (correlated features, realistic missingness patterns, realistic class imbalance) will pass cleanly while the same leakage or skew bug ships to production, since the fixture never had the property that would trigger it.
• E2E tests earn their cost specifically around infrastructure and environment risk, not logic risk. A dependency-version mismatch, a missing environment variable, a broken artifact-serialization format between training and serving, these are exactly the bugs an E2E test catches and an integration test (which typically runs each stage's logic without necessarily matching production's exact runtime environment) might not; keep a small number of E2E tests specifically for this class of risk rather than eliminating the tier entirely.
• "Catches a real bug" is not the same question as "catches a real bug quickly and cheaply." A slow, flaky E2E suite that occasionally catches a leakage bug is a worse investment than an integration test purpose-built to catch that exact class of bug directly; when a bug type is catchable at a lower tier with a targeted test, build that test rather than relying on a higher, more expensive tier to happen to catch it as a side effect.

Direct answer

Turn the vague request into a concrete, falsifiable investigation plan before doing anything else: pin down what "better" actually means, enumerate the specific, testable ways the model could be underperforming rather than guessing at one, and prioritize those hypotheses by how much diagnostic value each one yields per unit of the scarcest resource, mostly labeling budget, rather than by which one merely seems most likely. Under real time and labeling constraints, the checks that need zero new labels come first, since they are effectively free relative to the ones that require fresh ground truth.

Structured elaboration

Step 1: clarify the objective and constraints. Before investigating anything, ask what metric "better" refers to (precision, recall, a ranking metric, calibration), what the relative business cost of a false positive versus a false negative actually is, what time horizon the churn label is defined over, and what the current baseline value is. This step alone often reframes the whole investigation: a stakeholder who actually cares about catching the highest-value accounts, not overall accuracy, needs a different investigation than one who cares about total volume of correctly flagged churners.

Step 2: enumerate candidate failure modes as testable hypotheses, not a single guess. A standard checklist for "why is a model underperforming" covers several genuinely distinct causes, and a structured investigation names them explicitly rather than jumping at the first plausible one: label definition drift (the business definition of churn changed without the training label being updated to match), covariate shift in the model's most important input features, feature leakage or a broken feature that quietly stopped carrying real signal, a calibration-only problem (the model still ranks users correctly by risk, but the decision threshold is stale relative to a shifted score distribution), and segment-specific underperformance (the aggregate metric looks fine or mildly degraded while one segment is badly wrong and gets diluted into the average).

Step 3: prioritize by diagnostic value per unit of scarce resource, not by probability alone. Score each hypothesis as its estimated probability of being a real, fixable contributor divided by its cost to investigate, in labeling-budget units. This matters because a hypothesis that is only moderately likely but essentially free to check (an audit of the label definition and pipeline configuration needs no new labels at all) can be worth investigating before a hypothesis that is more likely but expensive, since the free check either resolves a real issue immediately or is ruled out at negligible cost, freeing the full budget for the remaining hypotheses. This priors-and-cost ranking should update as each step returns evidence, since finding nothing wrong with the cheapest hypothesis is itself informative and should reweight what is investigated next, not just work down a fixed list computed once at the start.

Step 4: work around a limited labeling budget. Order investigation by what each check actually costs in labels, cheapest first. At the top is the check that needs no new labels at all: a label-definition and pipeline-configuration audit inspects the label logic and the pipeline's own configuration rather than model outputs on fresh data, so it costs nothing in ground truth. Next come the checks that run mostly against historical, already-labeled data and need only a small confirmation sample of new labels, a feature-leakage check and a calibration check being the usual examples. Reserve the bulk of the budget for hypotheses that genuinely cannot be tested without substantial fresh ground truth, most commonly confirming or ruling out segment-specific underperformance on recent traffic. Where fresh labels are needed, spend them where they carry the most diagnostic value: targeted sampling of the cases the current model is least confident about or that fall in the suspected weak segment, rather than a uniformly random sample, extracts more information per label spent, while keeping a small separate random holdout specifically to measure overall metrics honestly, since a purely targeted sample would bias any metric computed directly on it.

Worked example

Illustrative priors an investigating engineer might assign to each hypothesis (informed judgment going in, not a measured result) and their estimated investigation cost in labels needed, scored as probability divided by cost, with a cost of 0 meaning the check needs no new labels at all:

With a 300-label total budget, this ordering (free checks first, then descending score among the ones that cost labels) investigates the label-definition audit for free, then the feature-leakage check for 50 labels (cumulative 50), then the miscalibration check for 200 labels (cumulative 250), leaving 50 labels unspent rather than enough to also fully investigate covariate shift or segment-specific underperformance this round. Note that miscalibration has the highest individual prior (0.45) but is not investigated first, because the free label-definition check and the far cheaper leakage check both deliver more diagnostic value per label spent. To see what that ordering choice actually buys, work both plans all the way through the same 300-label budget, skipping any check whose cost exceeds what is left at that point:

Two things fall out of that trace, and the first one is not the intuitive answer. Both orderings end up testing the same three hypotheses for the same 250 labels, so the case for value-per-label ordering is not that it buys more checks; at this budget it does not. What differs is WHEN the information arrives. The value-per-label plan has one hypothesis answered before it has spent a single label, two by label 50, and all three by label 250. The probability-first plan commits 200 labels, two thirds of the whole budget, to the 0.0023-per-label check before it has learned anything at all, and only then reaches the free audit and the 0.0030-per-label leakage check.

That timing is the entire argument, for two reasons. Priors are supposed to be updated as evidence arrives, and an ordering that defers every cheap answer until after the expensive one is already paid for has given up the chance to re-rank on what it learned. And investigations are cut short by time, by an incident, or by a stakeholder wanting an answer this afternoon far more often than they run exactly to the end of a labeling budget: stopped after two hours, the value-per-label plan has two answers in hand and the probability-first plan has one.

The trace also settles one thing worth being precise about, because it is easy to overclaim in the other direction: a zero-cost check is reached under EVERY ordering, since nothing can crowd out something that costs nothing. Scheduling free checks first is not about protecting them from the budget, it is about getting their answers back before the budget has been committed elsewhere.

Trade-offs and pitfalls

• Sorting purely by probability and ignoring cost is the most common way this kind of prioritization goes wrong. It defers cheap, high-value checks behind expensive ones that merely seem more likely, so most of the budget is committed before any of the cheap evidence is in and there is nothing left to re-rank with, exactly what the worked example's side-by-side spend trace shows.
• The priors themselves are judgment calls, not measurements, and should be treated as a starting point to update, not a fixed ranking. Finding nothing in the free label-definition audit is itself evidence that should lower that hypothesis's weight and free attention for the next one, not a wasted step.
• Targeted, uncertainty-focused labeling extracts more diagnostic value per label but introduces its own sampling bias if the same targeted sample is later used to report an overall metric. Keep a small, genuinely random holdout specifically for honest overall-metric reporting even while spending the bulk of the scarce budget on targeted diagnostic sampling.
• Skipping step 1 and diving straight into re-running dashboards is the other common wrong turn. "Make it better" is under-specified enough that investigating the wrong metric, one nobody actually cares about, is a real risk, and it is a risk step 1 removes at essentially no cost.

Direct answer. An intermittent multi-GPU failure (one node's CUDA OOM, or results diverging across otherwise-identical nodes) needs evidence gathered from every rank, not just the one that visibly failed, because the actual fault often originates on a DIFFERENT rank than the one that shows the symptom.

Debugging plan.

1. Collect NCCL, CUDA, and system logs from every rank, not just the failing one, and turn them on BEFORE the run, since none of this can be recovered retroactively. A rank that silently produces a NaN, for instance, often only shows up as an OOM or a hang on a DIFFERENT rank once the collective operation (all-reduce) tries to synchronize with it. The three layers each need their own switch.
   • NCCL / collectives: NCCL_DEBUG=INFO (plus NCCL_DEBUG_SUBSYS=ALL when you need the ring topology) shows which ranks joined which communicator and where a collective stalled. Pair it with TORCH_NCCL_ASYNC_ERROR_HANDLING=1 so a rank that dies tears the job down with an error instead of leaving every other rank blocked forever in the all-reduce, and TORCH_DISTRIBUTED_DEBUG=DETAIL to get shape and dtype mismatches across ranks reported as errors rather than as silent corruption.
   • CUDA: on a REPRO run (not the production one, because it serializes kernel launches and changes timing), set CUDA_LAUNCH_BLOCKING=1 so the traceback points at the kernel that actually failed rather than at whatever later call happened to synchronize. For the OOM specifically, wrap the step in a handler that dumps torch.cuda.memory_summary() at the moment of failure, and for a repeat offender turn on the allocator trace with torch.cuda.memory._record_memory_history() and dump it with torch.cuda.memory._dump_snapshot(), which gives you the call sites holding every live block instead of a single total.
   • System: check the kernel log on the suspect node (dmesg -T) for Xid errors and for the host OOM killer, and nvidia-smi -q -d ECC,TEMPERATURE,POWER for uncorrectable ECC counts, retired pages and thermal or power throttling. This layer is the one people skip and it is the one that answers the question actually being asked, which is why THIS node. A repeating Xid or a rising uncorrectable-ECC count on exactly one node turns a week of software debugging into a node replacement, and a host-side OOM kill explains a node dying with no GPU memory problem at all.
2. Separate the three memory failure shapes: a genuine leak, fragmentation, and a legitimate peak-allocation spike. They look identical at the moment of the OOM and completely different across many steps, so log TWO numbers per step, not one: memory ALLOCATED (the bytes currently held by live tensors) and memory RESERVED (the bytes the caching allocator is holding from the driver, including free-but-cached blocks). In PyTorch those are torch.cuda.memory_allocated() and torch.cuda.memory_reserved(). On a 16 GiB device the three shapes read like this:

step              1      50     100    150    200
LEAK        alloc 8.1    8.4    8.7    9.0    9.3   (GiB, monotonic climb)
            resv  8.6    8.9    9.2    9.5    9.8
SPIKE       alloc 6.2    9.8    6.2    9.8    6.2   (sawtooth, returns to baseline)
            resv  9.9    9.9    9.9    9.9    9.9
FRAGMENT    alloc 6.0    6.0    6.1    6.0    6.0   (flat, far below capacity)
            resv  9.5   10.2   10.9   11.4   11.8   (climbs away from alloc)

• Leak: allocated climbs monotonically across steps and never returns to baseline. Something is retaining references (a loss tensor accumulated into a Python list without .detach() or .item(), a growing cache, a hook holding activations). Fix the retention. Raising the memory ceiling only buys steps.
• Spike: allocated is a sawtooth that returns to the same baseline every step, with the peak driven by a particularly large batch or an activation-checkpointing boundary. Reserved sits flat at the high-water mark. This is normal behavior against an unlucky ceiling, and reducing batch size or enabling activation checkpointing genuinely fixes it.
• Fragmentation: allocated stays flat and well below device capacity while reserved climbs away from it, and the OOM message itself gives it away, reading something like "tried to allocate 2.00 GiB, 3.50 GiB free": there IS enough free memory in total, just not in one contiguous block. The other tell is that failure depends on allocation ORDER rather than step count, so it can fire at step 12 on one run and step 400 on the next, and it is strongly associated with varying tensor shapes (variable sequence lengths, ragged batches) that make each allocation a slightly different size. The fixes are different in kind from the other two: set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (or tune max_split_size_mb) so the allocator stops carving unusable slivers, call torch.cuda.empty_cache() at a safe step boundary to hand cached blocks back to the driver, bucket or pad variable-length inputs so allocation sizes repeat instead of drifting, and pre-allocate the large buffers once at start-up. Note what is NOT on that list: reducing batch size, which is the standard reflex and does not reliably help here, because the problem was never total capacity.

Confusing any two of the three leads to the wrong fix, and the fragmentation case is the one where the wrong fix is most tempting, since the allocator is reporting plenty of free memory while the allocation still fails.
3. Validate that batch size and sharding are actually consistent across ranks. A configuration bug (one node launched with a stale config, or an uneven data-sharding split) can silently give one rank a larger effective batch than the others; log the actual batch size and shard boundaries each rank believes it has, and diff them across ranks rather than assuming the launch config was applied uniformly everywhere.
4. Isolate WHICH rank produces an anomaly (for example a NaN) in a multi-GPU job: add a lightweight per-rank check right after the forward pass (before the collective all-reduce) that logs a boolean "this rank saw a NaN" flag, tagged with the rank ID, and aggregate these flags centrally. This turns "the job produced a NaN somewhere" into "rank 3 produced the NaN, ranks 0/1/2 were clean," which narrows the investigation from the whole cluster to one node's data shard, environment, or hardware.

Short-term mitigation to keep the job running while investigating. Reduce the per-rank batch size (or enable gradient accumulation to compensate) to lower peak memory pressure and buy headroom while you investigate the OOM's root cause, and enable periodic checkpointing (if not already in place) so an eventual failure doesn't lose the whole run's progress. This is explicitly a stopgap, not a fix: if the true cause is a genuine leak or a sharding bug, reducing batch size only delays the eventual failure, and if the cause is fragmentation it may not delay it at all, since a smaller batch changes allocation sizes without making the free memory any more contiguous. In that case the cheapest stopgap is instead the allocator setting plus a periodic empty_cache() at a step boundary.