Debugging and Testing ML Systems Questions

Direct answer

A bug report for an intermittent, GPU-type-specific training failure has to do the work a live debugging conversation would otherwise do informally, because the person who eventually picks it up will not have access to the machine, the moment, or your memory of what you tried. That means an exact environment matrix (down to driver, CUDA, the NVIDIA GPU computing toolkit, and cuDNN, NVIDIA's deep-learning kernel library, versions, since this exact bug class is version-sensitive by nature), a minimal reproduction that isolates the GPU type as the variable, every seed pinned so any attempted repro is comparing apples to apples, the specific logs to attach and why each one matters, an honest impact assessment, and a concrete next experiment, not just a description of the symptom.

Structured elaboration

Why this bug class specifically demands more than a symptom description. "Training sometimes crashes" is not actionable; a bug report is only as useful as the next person's ability to reproduce, or rule out, the same failure without your involvement. For a bug that is intermittent (does not happen every run) and hardware-specific (only certain GPU types), the report has to actively narrow the search space: which axis is the variable, hardware, driver version, a specific kernel, and provide enough detail that the report itself becomes evidence, not just a complaint.

Timestamps, and why they belong in the report as their own field, not folded into "environment." Record the exact wall-clock time of every failing run, alongside the exact wall-clock time of every successful run in the same investigation, in a consistent timezone, UTC (Coordinated Universal Time, the timezone-independent standard most infrastructure logs already use). This is what lets a later reader correlate a failure against something that changed at a specific moment but is not itself part of the environment matrix: a driver auto-update that rolled out to part of the fleet, a concurrent job that briefly contended for the same GPU, or a thermal-throttling event visible in a separate infrastructure dashboard. A bug report that only states "this happens sometimes" gives a later investigator nothing to line up against those external timelines; a report with exact timestamps lets them check in minutes whether the failures cluster around a specific external event instead of being uniformly distributed, which is itself a strong clue about the root cause.

What the environment matrix needs to capture, and why each field matters. GPU driver version and CUDA (Compute Unified Device Architecture, NVIDIA's parallel-computing platform) toolkit version together determine which low-level kernel implementations get selected for a given operation; a driver/CUDA combination that works on one GPU architecture can expose a genuine race condition or an unsupported code path on another. cuDNN (CUDA Deep Neural Network library, NVIDIA's library of GPU-accelerated primitives for deep learning) version matters separately because it ships its own kernel selection heuristics, independent of the CUDA toolkit version. Framework version (the training library itself) matters because kernel dispatch logic lives partly in the framework, not only in CUDA/cuDNN. Recording all four together, rather than just one, is what lets someone later correlate this exact failure against a known upstream bug report in any of the four projects.

Minimal reproduction steps. State the smallest model, batch size, and number of training steps that still reproduces the failure, and the specific GPU type(s) it reproduces on versus the GPU type(s) it was also tried on and did not reproduce on, since the negative result (GPU type X: 0 failures in N attempts) narrows the search space just as much as the positive one does.

Seeds. Pin and report every seed used in the reproduction attempts, since without them a colleague's "I couldn't reproduce it" is uninformative, they might simply have drawn a different random path through the same non-deterministic kernel. Also report whether the failure reproduces on the SAME seed across multiple attempts (which would suggest the non-determinism lives somewhere other than the obvious PyTorch/NumPy random state, likely the CUDA kernel scheduling itself) or only appears on some seeds and not others.

Which logs to attach. The full stdout/stderr from a failing run (the actual CUDA error string and code, not a paraphrase of it), the nvidia-smi output captured at or near the time of failure (GPU memory usage and utilization can distinguish an out-of-memory-adjacent race from a pure kernel-launch bug), and, if available, output from CUDA_LAUNCH_BLOCKING=1 (an environment variable that forces synchronous kernel launches, which converts an asynchronous, hard-to-localize CUDA error into one that points at the actual failing kernel call, at the cost of much slower execution, which is exactly why it is a debugging-only flag rather than something left on by default).

Impact assessment. State concretely which training runs and which GPU types are affected, and the failure rate observed so far in this specific investigation (framed honestly as "N failures in M attempts on this GPU type, this seed set," not extrapolated into a general production failure rate unless that rate has actually been measured at production scale).

Recommended next experiment or workaround. Propose the next diagnostic step that would most efficiently narrow the remaining hypothesis space (for example, running with CUDA_LAUNCH_BLOCKING=1 to localize the failing kernel, or bisecting across driver versions on the affected GPU type specifically), and a concrete interim workaround if one exists (routing training for the affected GPU type to a known-good driver version, or avoiding the specific operation suspected of triggering it), so the team is not blocked on training while the root cause is still under investigation.

Worked example

A filled bug report for exactly the scenario in the question:

Title: Intermittent CUDA kernel launch failure during training, reproduces only on GPU type A100, not on V100 or H100 (A100, V100, and H100 are successive generations of NVIDIA's data-center training GPUs)

Environment matrix

Field	Affected runs	Unaffected runs
GPU type	NVIDIA A100 (80GB)	NVIDIA V100, NVIDIA H100
GPU driver	535.104.05	535.104.05 (same driver, different GPU type)
CUDA toolkit	12.2	12.2
cuDNN	8.9.4	8.9.4
Framework	PyTorch 2.3.0	PyTorch 2.3.0

Observed: 6 failures in 40 training attempts on A100 (15% failure rate in this investigation's sample; not yet measured at full production scale). 0 failures in 25 attempts on V100 and 0 failures in 15 attempts on H100, same code, same driver/CUDA/cuDNN/framework versions.

Timestamps of each A100 failure (UTC): 2026-07-14 03:12, 2026-07-14 09:47, 2026-07-15 03:20, 2026-07-16 03:15, 2026-07-18 03:09, 2026-07-19 22:58.

Tally of that list (a clustering claim that has not actually been counted is not evidence, so the count belongs in the report next to the raw timestamps):

classification	timestamps	count
inside the 03:00-03:20 UTC window	07-14 03:12, 07-15 03:20, 07-16 03:15, 07-18 03:09	4 of 6, on 4 distinct days
outside it	07-14 09:47, 07-19 22:58	2 of 6

Four of six landing inside a 20-minute window is the detail worth flagging first, ahead of the raw failure count. Twenty minutes is 20/1440 = 1.39% of a day, so if failures were spread uniformly through the clock you would expect 6 x 0.0139 = 0.08 of them to land in that window, not 4; the probability of seeing 4 or more of 6 there by chance is about 5.5e-07, roughly 1 in 1.8 million. That is far too strong to write off as coincidence, so it suggests correlation with a recurring scheduled job on the shared cluster (a nightly maintenance window is the working hypothesis, not yet confirmed) rather than a purely code-triggered race condition. The two failures outside the window are exactly why it stays a working hypothesis: a maintenance-window cause accounts for 4 of the 6 observations and something else, a second trigger or plain chance, still has to account for 07-14 09:47 and 07-19 22:58.

Minimal reproduction: a 2-layer transformer block, batch size 32, sequence length 128, fails intermittently within the first 50 training steps on A100. Does not require the full model or dataset; the minimal repro script isolates a single forward/backward pass repeated in a loop.

Seeds: reproduced with seeds 0, 17, and 42 on A100 (not seed-specific: same seed 0 both failed and succeeded across different attempts, which points toward non-determinism in kernel scheduling itself rather than in the framework-level random state).

Logs attached: (1) full stdout/stderr from a failing run, including the raw CUDA error string and code, not a summary; (2) nvidia-smi output captured within 5 seconds of the failure; (3) a run with CUDA_LAUNCH_BLOCKING=1 set, which converts the normally-asynchronous CUDA error into one that names the specific failing kernel call, at the cost of roughly 3 to 5x slower execution for that run, which is why it is not run by default.

Impact: blocks reliable overnight training runs specifically on the team's A100 nodes; V100 and H100 nodes are unaffected and available as an immediate workaround.

Recommended next experiment: (1) immediate workaround: route affected training jobs to V100 or H100 nodes while this is investigated; (2) next diagnostic step: check the cluster's job scheduler and infrastructure logs for anything recurring in the 03:00-03:20 UTC window to confirm or rule out the maintenance-window hypothesis the 4-of-6 clustering points at, and separately check whether the two out-of-window failures (07-14 09:47, 07-19 22:58) share any property with each other that the four in-window ones do not, since a hypothesis that only ever explains 4 of 6 is not yet a complete account; (3) in parallel, re-run the minimal repro with CUDA_LAUNCH_BLOCKING=1 on A100 across 20 attempts to identify the specific failing kernel by name, then check that kernel against NVIDIA's published A100 driver 535.x errata before assuming this is a novel bug rather than a known one.

Trade-offs and pitfalls

• Reporting failure rate without stating the sample size invites the reader to misjudge severity in either direction. "15% failure rate" sounds alarming or mild depending on whether it is 6 of 40 or 150 of 1000; always report both numbers together, and be explicit about whether the rate has been measured at production scale or only within this specific investigation's sample, per the reproducibility discipline of never presenting an estimate as a broader measurement than it actually is.
• CUDA_LAUNCH_BLOCKING=1 is a debugging tool, not a permanent fix. It serializes kernel launches, which can slow training by several times; attaching a run captured with it on is valuable evidence, but the recommendation should never be to leave it on in production training as a workaround for the underlying bug.
• A same-seed-both-outcomes observation is a stronger clue than it looks, and is easy to bury in a wall of text. It specifically points toward non-determinism at the CUDA kernel-scheduling level rather than the framework's own random-number state, which changes where the next investigator should look first; surface it explicitly near the top of the report, not just in a logs appendix.
• A stated pattern that has not been counted is the same mistake as a failure rate quoted without its sample size. "The failures cluster overnight" is an assertion; "4 of 6, on 4 distinct days, inside a 20-minute window where 0.08 would be expected by chance" is evidence the next investigator can check and either build on or refute. Put the tally in the report next to the raw list, and let the prose summarise the tally rather than the other way round.
• A minimal reproduction that is not actually minimal wastes the next investigator's time. If the smallest failing case still requires the full model and dataset, keep reducing it (smaller batch size, fewer layers, synthetic data instead of the real dataset) before filing, since every extra unnecessary component in the repro is one more variable a colleague has to also rule out.

Direct answer

A model that passed every pre-deployment test and still regressed in production tells you something specific: the tests were measuring the wrong thing, or measuring the right thing on the wrong data. My first move is forensic, not corrective: freeze and gather evidence before anything about the environment changes further, compare the world the tests ran against to the world production actually sees, and only then propose a fix. The closing, and in my view most important, part of the response is diagnosing exactly which gap in the test suite let this through, because fixing this one regression without closing that gap just means the next one repeats it.

Structured elaboration

Evidence to gather, before it rotates out of retention. Pull the exact model artifact and its version identifier that was deployed (not "the version that was supposed to be deployed"; deployment mismatches are themselves a real cause), the serving configuration active at the time, and a sample of real inference logs from the incident window (inputs as hashes or fingerprints if raw payloads cannot be retained, plus outputs, confidence scores, and latencies). Separately capture a feature-store snapshot for the incident window, since features can be point-in-time and the values a request saw when it happened may not match what you would compute by re-querying now. Note the exact deployment timestamp and cross-reference it against any other change that landed around the same time (a feature-store schema change, an upstream data-source migration, a dependency bump), since "new model version" and "new bug" being simultaneous does not prove the model version caused it.

Compare pre- and post-deployment input distributions and schemas. For every feature the model consumes, compare the distribution in the incident window's production traffic against both the training distribution and the pre-deployment test distribution, using a two-sample test such as Kolmogorov-Smirnov (KS) for continuous features or a chi-square test for categorical ones. Separately, and more cheaply, diff the schema itself: field names, types, nullability, and cardinality, since a schema drift (a field silently becoming nullable, a categorical feature gaining new values) is a harder, more binary signal than a distributional shift and is usually faster to confirm or rule out.

Check for a data-pipeline or feature-transformation change the pre-deployment tests did not cover. This is the step that specifically explains "passed all tests and still failed": walk the feature-transformation code path end to end and ask, for each step, whether the pre-deployment test suite exercised it against data shaped like what actually reached production. Common gaps: the test suite used a static, curated golden dataset (a fixed, checked-in set of example inputs with their expected outputs, captured once and used as the pre-deploy reference ever since) that no longer represents the live feature distribution; the test suite validated the model in isolation but not the full serving path, so a change in an upstream feature-extraction service between test and deploy went unchecked; or the test suite covered the model's own code but a shared preprocessing library was bumped independently and neither the model tests nor a compatibility test caught the interaction. Confirm by re-running the pre-deployment test suite's own inputs through the CURRENT (post-deployment) feature pipeline and diffing the outputs against what the test suite recorded at deploy time; a difference here directly proves the pipeline moved out from under a test that never re-checked itself against fresh production-shaped data.

What to change about the test suite so this class of regression is caught next time. Once the specific gap is identified, the fix is almost always one of three shapes: replace a static golden dataset with a rolling, periodically-refreshed sample of real (or synthetic but representative) recent production data, so the test data cannot silently drift out of sync with what production actually sees; extend the pre-deployment suite from model-only testing to full serving-path integration testing, so a change anywhere in the request's actual path is exercised, not just the model's own forward pass; and add a schema and distribution CONTRACT check as a hard pre-deploy gate (not just a test that can be skipped under time pressure), so any feature schema or distribution change beyond a defined tolerance blocks the deploy automatically rather than depending on a human noticing. Define that tolerance in numbers or it will not survive its first deadline: block the deploy if any feature's Kolmogorov-Smirnov statistic against the last 7 days of production exceeds 0.1, if its null rate moves by more than 2 percentage points, or if its median moves by more than 20 percent. A 0.1 gate is deliberately loose relative to sampling noise: two clean samples of the sizes below land around 0.01 to 0.02, so 0.1 blocks real movement without firing on ordinary variation.

Worked example

Concretely: precision at the deployed threshold fell from 71 percent to 63 percent over the first two days after the release, an 8-point drop. The post-deployment schema diff shows nothing (all fields present, all types unchanged), but the KS test on one input feature, "days since last activity," shows a clear shift between the pre-deployment test distribution and the incident-window production distribution. The numbers: the golden set holds 5,000 rows with a median of 14 days since last activity; the incident window holds 20,000 production rows with a median of 31 days; the two-sample KS statistic between them is 0.294 with a p-value below $10^{-300}$. For scale, two clean samples drawn from the same distribution at those sizes give a KS statistic of about 0.013, with 0.020 at the 95th percentile, so 0.294 is not a borderline call, it is roughly fifteen times the noise floor. Re-running the pre-deployment test suite's own golden inputs through the current feature pipeline reproduces the ORIGINAL test-time values exactly (the pipeline code did not change), which rules out a pipeline bug. The remaining explanation, confirmed by checking the golden dataset's last-refreshed timestamp, is that the golden dataset itself was captured eight months earlier and the live user population's activity pattern had genuinely moved since then; the pre-deployment tests kept passing because they were comparing the new model against stale data, not because the new model actually generalized to current production inputs. Why this version and not the earlier ones that tested against the same stale golden set: the new version was tuned considerably harder against that distribution (more capacity and a longer hyperparameter search, selected on golden-set performance), so it fit the eight-month-old activity pattern more tightly and had correspondingly more to lose when production stopped matching it. The earlier, less tightly-fit versions were leaving performance on the table against the golden set, which is exactly what made them less sensitive to the golden set being wrong. The test-suite fix that follows directly from this evidence is the rolling-refresh golden dataset described above, specifically for this case, not a full serving-path integration test, since the pipeline code itself was proven unchanged.

Trade-offs and pitfalls

The most common wrong turn is rolling back and closing the incident once metrics recover, without completing the forensic comparison. A rollback restores service; it does not tell you whether the next model version will hit the exact same gap, and if the golden dataset is stale, every future release trained and tested the same way is at risk until that is fixed.

A second pitfall is over-correcting the test suite into something too expensive to run on every deploy. A full production-scale, fully-refreshed integration suite on every commit can become slow enough that teams start skipping it under deadline pressure, recreating the exact failure mode you were trying to close. Split the strategy by cost: fast, deterministic schema and distribution CONTRACT checks as a hard, cheap gate on every deploy, and a periodic (not necessarily per-deploy) refresh of the golden dataset and full-path integration run, so the properties that change slowly are not re-verified at a cost that does not match how often they actually move.

A third pitfall is confirming only ONE candidate cause and stopping. In this example, the schema diff, pipeline replay, and distribution test were run as a set specifically because any one check alone (say, the KS test showing drift) would not have distinguished "the golden dataset is stale" from "the pipeline itself changed and happens to now match old test values by coincidence." Running the full battery, and confirming the pipeline replay reproduced the ORIGINAL recorded values, is what turned a plausible hypothesis into a confirmed root cause.

Direct answer. When leakage only shows up on a full end-to-end run, the fix has to live in the platform, not in any single step's code: the feature store's API has to make it structurally hard to accidentally read a future value, validation has to run against the assembled pipeline output rather than each step in isolation, and the SHAPE of every feature has to be a contract the platform enforces rather than an assumption each step makes privately.

Time-travel / point-in-time API semantics. The feature store's read API should require an explicit as_of timestamp for every historical feature lookup, and should physically be unable to return a value with an effective timestamp later than that cutoff, this is a structural guarantee (the API can't return the wrong thing) rather than a convention that a careless caller can bypass. Joins for training-set construction should go through this API exclusively, never a raw table join that has no concept of "as of when."

One feature, traced. Customer C-8814, feature orders_last_30d, prediction made at 2026-01-10 09:00, label resolved 30 days later. The feature table holds one row per recomputation:

effective_ts          value
2026-01-05 02:00        3
2026-01-09 02:00        4     <- newest row at or before the prediction time
2026-01-12 02:00        9     <- computed AFTER the prediction, and after part of the outcome window

get("orders_last_30d", key="C-8814", as_of="2026-01-10 09:00")  ->  4
SELECT value FROM f WHERE key='C-8814' ORDER BY effective_ts DESC LIMIT 1  ->  9

Both queries run without error and both return a number that looks like a plausible order count. The raw join returns 9, a value that partly reflects behavior from inside the label window, which is the leak. This is why the guarantee has to be in the API's type signature rather than in a review checklist: the wrong answer is not detectably wrong by inspection.

Versioning. Every feature has a versioned definition (the transformation logic that produces it), and a training run records exactly which version of each feature it used. If a feature's definition changes, that's a new version, not a silent overwrite, so a leakage bug introduced by a definition change is traceable to a specific version rather than invisible.

Schema guarantees. Versioning covers what the transformation DOES; schema guarantees cover what the value LOOKS LIKE, and they catch a different class of multi-step bug. Every feature is registered with a declared type, a nullability rule, an entity key it is defined on, and where applicable a domain or range (a categorical's allowed value set, a rate's [0, 1] bound). Three enforcement points make it a guarantee rather than documentation:

• At write time, a batch that violates the registered schema is REJECTED, not coerced. Silent coercion is the mechanism: a float64 quietly downcast to float32, a null quietly filled with 0, a categorical gaining an unseen level that maps to an implicit default. Each of those is invisible in the step that causes it and changes what a later step computes.
• On change, a compatibility rule decides whether a schema edit is allowed in place or must mint a new feature version. Additive and widening changes (adding a genuinely new nullable field, widening int32 to int64) are backward compatible and can land in place. Retyping a column, narrowing a range, making a nullable column non-nullable, or changing the unit or semantics of an existing column are BREAKING, and the platform's job is to refuse the in-place edit and force a new version, so that no already-trained model silently starts receiving a differently shaped input.
• At serve time, every trained model carries a schema fingerprint of each feature it consumed (name, type, nullability, domain). Before a model is served, the platform compares that fingerprint against the live registry and fails the model load on divergence, rather than serving predictions on inputs the model never saw the shape of.

This is exactly the multi-step interaction case: a column silently widened between two steps, or a null rate that jumps because an upstream join started missing rows, changes the model's effective input without one line of the model's own code changing, and no single step's unit test can see it because each step's contract with its neighbor was never written down.

Retrospective validation. Periodically (and specifically after any pipeline or feature-definition change), re-run a set of historical predictions using the point-in-time API and compare them against what the pipeline actually served at the time, if they diverge, either the point-in-time guarantee has a bug, or the original serving path had a leak the offline reconstruction doesn't have (which itself is a strong hint about where to look). This check specifically catches issues that only appear end-to-end, since it exercises the exact same join and windowing logic the full pipeline uses, not a simplified reconstruction.

Why this needs to be a platform decision, not a per-question fix. Any individual data scientist checking "is my feature safe" against a single step in isolation can get a clean answer while the multi-step join still leaks, because the leak lives in the INTERACTION between steps. Two concrete interactions: a join key resolving differently depending on execution order, and a windowing boundary computed against wall-clock time in one step and event-time in another. Event time is the timestamp carried by the event itself, when the order was actually placed. Wall-clock time is when the job happened to run. In the trace above, the feature is orders_last_30d, so a step that builds its 30-day window ending at event time covers 2025-12-11 to 2026-01-10 and correctly stops at the prediction time. A step that builds the same 30-day window ending at wall-clock time, on a job that runs on 2026-02-01, covers 2026-01-02 to 2026-02-01 instead, so it silently sweeps in the 22 days from 2026-01-10 to 2026-02-01, about three weeks of post-prediction activity, and drops most of the pre-prediction window it was supposed to measure. Each step is individually defensible; only the composition leaks. Only a platform-level, enforced guarantee removes the possibility rather than relying on every future feature author to individually get a multi-step interaction right by hand.

Direct answer

Structure the test strategy as a funnel of increasingly expensive, increasingly realistic checks: cheap, fast, deterministic tests run on every pull request and block the merge; slower, statistical tests run on a schedule against a larger held-out sample; and full production validation (shadow traffic, then a small live canary) runs only on release candidates, since that stage is the only one that measures real user-facing impact but also the only one with real cost and real risk. The cadence of both testing and retraining should be set from the same question: how fast does the world the model describes actually change, not from a fixed calendar habit like "we retrain every Monday."

Structured elaboration

The speed/quality/cost funnel, stage by stage.

Why the probabilistic output specifically changes the fast-tier design. For a model producing probabilities rather than hard labels, "does the output look reasonable" cannot be a single fixed expected value the way it can for a deterministic function; the fast tier needs a property check (is this a valid probability distribution, does the highest-probability class match a small set of known-unambiguous golden examples) rather than an exact-match assertion, since two legitimate model versions can differ slightly in their exact probability values while both being correct.

Cost trade-offs of retraining and test-execution cadence. These are two different cadences with two different cost drivers, and conflating them is a common mistake:

• Test-execution cadence trades CI (continuous integration, the automated pipeline that runs checks on every proposed change) wall-clock time and compute cost against how quickly a regression is caught. Running the full statistical-oracle suite on every commit instead of nightly catches a regression a day earlier, at the cost of running an expensive statistical comparison dozens of times a day instead of once; most teams land on "fast tier on every commit, expensive tier nightly or per-release-candidate" because the fast tier alone already catches the majority of defects (data leakage, a broken shape, a NaN, short for not-a-number, an invalid floating-point value) cheaply, and the expensive tier's marginal catch rate does not justify its cost at a higher frequency.
• Retraining cadence trades the cost of training compute against how stale the model's view of the world is allowed to get, worked through with real numbers below.

Where to put the retraining decision in the funnel. Retraining should never happen automatically off the calendar alone; a new candidate model, however it was triggered (schedule, drift alert, or a manual retrain), still has to pass through the exact same funnel above, pre-merge checks, scheduled statistical checks, shadow, then canary, before it replaces the production model. Treating "retrained on schedule" as equivalent to "validated and safe to serve" is exactly the assumption a rigorous test strategy exists to remove.

Worked example

A concrete illustration of the retraining-cadence trade-off's shape (hypothetical numbers, chosen to be easy to verify by hand, not a measured result from any real system): suppose a single retraining run costs 10 GPU-hours.

$\text{monthly cost}_{\text{weekly}} = 10 \times 4 = 40 \text{ GPU-hours}$
$\text{monthly cost}_{\text{daily}} = 10 \times 30 = 300 \text{ GPU-hours}$
$\text{ratio} = \frac{300}{40} = 7.5$

(Those two lines round the month differently, 4 weeks on one and 30 days on the other. On a consistent 30-day month the weekly cadence is 30/7 = 4.3 retrains, or 42.9 GPU-hours, and the ratio is 7.0 rather than 7.5. Nothing below turns on which rounding you pick, but it is worth being able to say which one you used.)

The benefit side needs its own arithmetic rather than an eyeballed number, because it is the half of this comparison that is easiest to overstate. A prediction request arrives at a uniformly random moment between two retrains, so the model's average staleness is half the retraining interval:

$\text{average staleness} = \frac{\text{interval}}{2} \;\Rightarrow\; \text{weekly} = \frac{7}{2} = 3.5 \text{ days}, \quad \text{daily} = \frac{1}{2} = 0.5 \text{ days}$
$\text{average freshness gain} = 3.5 - 0.5 = 3 \text{ days}$

Six days is a real number about this system, but it is a different one: it is the worst-case gap, comparing the moment just before each retrain, when the weekly model is 7 days stale and the daily model is 1. Quoting that as the average silently doubles the benefit side of the trade, which is the kind of error that flips a cost decision without anyone noticing. Dividing the two sides gives the unit the decision actually needs:

$\frac{300 - 40 \text{ GPU-hours}}{3 \text{ days}} \approx 87 \text{ GPU-hours per day of average freshness gained}$

So the move from weekly to daily buys three days of average freshness at roughly 87 GPU-hours each. Whether that 7.5x is worth paying depends entirely on how fast the underlying data distribution actually drifts: a fraud model facing an adaptive adversary might need daily retraining to stay ahead of attackers actively probing for the current decision boundary, while a model predicting a slow-moving demographic trend would see no measurable quality gain from anything faster than monthly, making the extra compute pure waste for that specific model. The right cadence is the one where the marginal cost of one more retrain equals the marginal value of the freshness it buys, not a cadence chosen because it is a round number on a calendar.

Trade-offs and pitfalls

• A fast tier that is too permissive defeats the purpose of having a funnel at all. If the pre-merge checks let obviously broken candidates through because they only check for a valid probability distribution and nothing else, every regression gets pushed to the expensive, slow, less frequent tier, which is the opposite of the intended cost structure; the fast tier needs at least a few golden-input exact-behavior checks alongside the property checks, not property checks alone.
• Shadow testing catches a different class of bug than a canary does, and skipping straight to canary to save time is a common, costly shortcut. Shadow traffic is safe (nothing is served to real users) and lets a comparison run at full traffic volume, which surfaces the rare-input bugs a limited canary slice might miss for weeks; a canary is the only stage that measures real business-metric impact, which shadow cannot. Neither stage substitutes for the other.
• Retraining cadence set purely by calendar habit, rather than by measured drift, wastes compute in the common case and under-reacts in the rare case that actually matters. A drift-triggered retraining policy (retrain when a monitored input or prediction distribution crosses a threshold, not on a fixed schedule) generally beats a fixed calendar cadence on both cost and responsiveness, at the cost of needing a reliable drift-detection system in the first place, which is itself something that has to be tested and monitored.
• The cost side of this trade-off is not only compute. A daily retraining cadence also means daily exposure to the full validation funnel's risk surface (a bad candidate reaching canary more often), and daily operational load on whoever is on call for release approvals; a slower, better-justified cadence can be the right call even when compute cost alone would favor going faster.

Direct answer. Scaling up GPU count or enabling mixed precision changes several things simultaneously (effective batch size, gradient synchronization behavior, numeric precision), so a training run that was stable before and diverges after needs a plan that isolates ONE variable at a time rather than reverting the whole change and guessing.

Three terms this depends on. A replica is one full copy of the model living on one GPU. DDP (DistributedDataParallel) is the synchronous data-parallel wrapper that runs one process per GPU and keeps the replicas in step. All-reduce is the collective operation that combines a tensor across every replica and hands every replica back the same combined result, which is how the per-replica gradients become one shared gradient each step.

Plausible causes, and what to test first. The single highest-prior cause when scaling GPU count is learning-rate scaling: if the effective batch size grows with GPU count but the learning rate doesn't, or is scaled incorrectly (linear scaling isn't always the right rule for every optimizer), training can become unstable purely from that mismatch, and it's the cheapest hypothesis to test. Worked with real numbers so "adjusted for the new batch size" is computable: a stable baseline of 8 GPUs at 32 examples per device is an effective batch of 256 at LR 0.1. Scaling to 64 GPUs at the same 32 per device gives an effective batch of 2048, an 8x increase. The linear rule says LR = 0.1 x 2048/256 = 0.8. The square-root rule says LR = 0.1 x sqrt(8) = 0.283. If the job was launched at 0.8 and diverges, the diagnostic rerun is at 0.283, and the sanity rerun (which trades throughput for certainty) is at the original 0.1.

Next, check batch-norm statistics: if BatchNorm computes per-replica statistics independently rather than synchronizing across replicas, a network with BatchNorm can behave differently, and less stably, at high replica counts purely because each replica now sees a smaller effective batch for normalization purposes. Then check all-reduce precision (is gradient synchronization happening in a lower precision than intended) and synchronization/seed-order differences (are all replicas actually initialized identically before training starts).

For mixed precision (AMP) specifically. Gradient-accumulation interactions with AMP are a common culprit. The mechanics, since the ordering is the whole bug: loss scaling multiplies the loss by a large factor S before the backward pass so that small fp16 gradients do not underflow to zero, and the gradients must later be divided by S ("unscaled") before the optimizer step. With gradient accumulation over K micro-batches, the correct order is accumulate-then-unscale: keep every micro-batch's gradients scaled, sum all K of them, and unscale ONCE immediately before the optimizer step. The wrong order, unscale-then-accumulate, divides each micro-batch's gradients by S as it goes and then accumulates already-unscaled values, while the scaler still believes it owes the division, so the effective gradient magnitude ends up off by a factor tied to K and the scaler's overflow check runs against tensors that are no longer on the scale it expects. Confirm which order your framework's AMP integration actually uses before assuming the bug is elsewhere. Separately, some DDP jobs show gradient divergence where SOME workers produce NaN and others don't purely from AMP's dynamic loss scaling adjusting independently per worker if it isn't explicitly synchronized, worth checking whether your framework synchronizes the loss-scale factor across replicas or lets each pick its own.

What you would expect to observe if each cause were the real one. This is the half that turns the list into a diagnosis:

• Learning-rate / batch-size mismatch: divergence arrives within the first few hundred steps and arrives EARLIER the more GPUs you add, and the step at which it blows up moves monotonically as you sweep LR. Rerunning the same 64-GPU job at 0.283 (or at 0.1) with everything else untouched restores stability. Gradient norms grow smoothly over several steps before the blow-up rather than jumping in one step.
• BatchNorm computing per-replica statistics: dump each replica's BatchNorm running mean and running variance buffers and compare them across ranks. Synchronized normalization keeps them near-identical; the unsynchronized failure shows them visibly diverging, with the spread widening as per-device batch size falls. Crucially, the gradients themselves stay finite the whole time, so this looks like instability without any numeric red flag. It gets worse as you hold total batch fixed and add GPUs (shrinking per-device batch), and it disappears when you swap in synchronized BatchNorm or raise the per-device batch back up.
• All-reduce running in a lower precision than intended: checksum or norm each rank's gradient tensor immediately BEFORE the collective and immediately AFTER it. Before, the ranks legitimately differ, since each saw a different data shard. After, every rank must hold a bit-identical tensor. The tell is that post-collective gradients agree only to the first three or four significant digits rather than exactly, and that the disagreement grows with rank count, because more partial sums mean more accumulated rounding. Compare against a reference reduction performed in fp32 to size the error.
• Seed or initialization-order mismatch: hash every replica's parameters at step 0, before any data is seen. They must be identical. If rank 0's hash differs from rank 1's at step 0, the replicas never started from the same weights, and the shared averaged gradient is being applied to divergent parameter sets. The signature is that divergence appears essentially at step 1, not after a hundred healthy steps.
• AMP unscale-then-accumulate ordering: instrument the global gradient norm immediately before the optimizer step, and compare against the same total batch run with K=1 (no accumulation). The two should agree within numerical noise. The ordering bug shows up as a clean multiplicative discrepancy tied to K, a factor of K or 1/K rather than a vague drift, which is what makes it identifiable rather than merely suspicious.
• AMP loss-scale desync across workers: log the current loss-scale value per rank per step. Healthy runs show every rank holding the same value and stepping it in lockstep. The failure shows ranks holding different scales at the same step (one at 1024 while another is at 32768), SOME ranks reporting NaN while others report clean gradients, and the identity of the NaN-reporting ranks changing from run to run.

Why the effective batch size is the thread running through all of this. Scaling GPU count in synchronous data-parallel training multiplies the effective batch size by the number of new replicas, which is exactly what makes the learning-rate hypothesis the highest-prior one and why the arithmetic above matters. It also means that if the instability appears only past a specific scale rather than at every scale, that threshold is itself evidence: a genuine LR/batch mismatch only becomes destabilizing once the step size and the curvature disagree badly enough, whereas a hardware or environment fault would be present regardless of how many replicas are running.

Prioritized order to test. Order by what each check actually costs to run, then by prior probability, because two of these are nearly free and one of them is a precondition for trusting any of the others.

0. Free, so run these before spending a single GPU-hour on a rerun. Hash every replica's parameters at step 0: one hash per rank, zero training steps, and if the hashes disagree you are done, because none of the cross-rank comparisons below mean anything until the replicas provably start from the same weights. Under AMP, also log each rank's loss-scale value per step, which is one scalar per rank and settles the desync hypothesis outright. Neither of these is the most likely cause; they go first because they are cheaper than everything else and because they are the two that can invalidate the other measurements.
1. Learning-rate scaling, the highest-prior cause and the first hypothesis worth real GPU-hours: rerun the 64-GPU job at 0.283, and at the original 0.1 if that is still unstable. One short run, one variable, directly falsifiable.
2. AMP accumulate-then-unscale ordering, whenever gradient accumulation is on: compare the pre-optimizer global gradient norm against the same total batch run at K=1. One extra short run and one scalar, and the tell is a clean factor of K or 1/K rather than a drift.
3. BatchNorm synchronization: dump the running-mean and running-variance buffers on every rank and diff them. More instrumentation than anything above, still far less than the last item.
4. All-reduce precision, last: checksumming every rank's gradient tensor before and after the collective is the heaviest instrumentation in this list, and it is also the least common root cause for a symptom that appeared specifically at a scaling boundary.