Debugging and Testing ML Systems Questions

Direct answer

Predictions that swing wildly on tiny pixel changes can come from four genuinely different places: a preprocessing mismatch that makes "tiny" not actually tiny by the time the model sees it, floating-point precision differences that get amplified through the network, a model that legitimately learned a sharp decision boundary near that input (brittleness that is a property of the trained function, not a bug), or a targeted adversarial vulnerability. The four look identical from the outside (small input change, large output change) but need different experiments to tell apart and different fixes once you know which one you have, so the investigation has to run each hypothesis's specific experiment rather than assuming the flashiest explanation (adversarial attack) by default.

Structured elaboration

Hypothesis 1: preprocessing mismatch. A "tiny" change in raw pixel values is not necessarily tiny after preprocessing if the perturbed pixels interact with a nonlinear or discontinuous step in the pipeline: a color-space conversion, a resize using a different interpolation method at the boundary, or a normalization constant applied inconsistently between where the "tiny change" was introduced and where the model actually receives its input.
Targeted experiment: feed the same raw image through the two preprocessing code paths in question (for example, however the perturbed sample was generated versus the model's actual serving preprocessing) and diff the preprocessed tensors directly, not just the final predictions. If the preprocessed tensors differ by more than the raw pixel change alone would explain, the "tiny" input perturbation was amplified before the model ever saw it, and this is your cause.
Mitigation: unify preprocessing into one shared, versioned implementation used both for whatever generates or tests these perturbations and for serving, and add an explicit invariant test asserting that a small raw-pixel change produces a proportionally small preprocessed-tensor change.

Hypothesis 2: numerical precision. Running inference at reduced precision (mixed precision, or hardware-specific fused kernels) can make an already close decision boundary flip for inputs that land near it, purely from rounding differences between two runs that are not actually bit-identical even on the same nominal input.
Targeted experiment: run the exact same input, and the exact same perturbed input, through the model at full 32-bit floating-point (fp32) precision, deterministic mode enabled (fixed seeds, non-deterministic kernel selection disabled), on the same hardware. If the drastic flip disappears or shrinks substantially under fp32 and determinism, precision is a meaningful contributor.
Mitigation: for any input near a genuinely close decision boundary, precision-related noise is somewhat irreducible without changing the model, but you can reduce its practical impact by using higher precision for the final classification layer specifically (a common mixed-precision pattern: bulk of the network in reduced precision, final logits computed in fp32) and by monitoring the confidence margin, treating low-margin predictions as lower-confidence regardless of the raw predicted class.

Hypothesis 3: genuine model brittleness (a sharp decision boundary). The model may have legitimately learned a function that is highly sensitive near this specific input region, without any adversarial intent and without any bug: this happens when training did not expose the model to enough natural variation around similar inputs, so the learned boundary is steeper there than it would be with better coverage.
Targeted experiment: apply natural, non-adversarial perturbations at a comparable magnitude, small random Gaussian noise, mild Gaussian blur, or small brightness and contrast shifts, and measure how often the prediction flips across many random draws at that perturbation magnitude. If the model is frequently unstable under RANDOM small perturbations of similar size, not just the one specific perturbation direction you tested, that is evidence of general local brittleness (a genuinely sharp boundary) rather than something targeted.
Mitigation: data augmentation with exactly this class of natural perturbation during training (so the model sees and is trained to be stable under similar variation), label smoothing (training against slightly softened targets, for example 0.9 for the correct class instead of 1.0, which discourages the model from driving its outputs to extreme confidence and tends to produce a less knife-edged boundary), or test-time augmentation and ensembling (averaging predictions across several small random perturbations of the input) to reduce sensitivity to any single realization. Of the three, data augmentation with the right perturbation class is the primary response, since it addresses the training-coverage gap that produced the sharp boundary; label smoothing and test-time augmentation reduce the symptom without changing what the model saw.

Hypothesis 4: an adversarial-robustness problem. Distinct from general brittleness, this is a case where a small, specifically-DIRECTED perturbation (not just any small change) reliably flips the prediction, while random perturbations of the same or even larger magnitude mostly do not.
Targeted experiment: compare the random-perturbation flip rate from hypothesis 3's experiment against the flip rate from a small number of targeted gradient-based perturbations at the SAME perturbation budget (for example, a fast gradient-based attack that moves the input in the direction that most increases the loss, constrained to a small maximum per-pixel change). If targeted perturbations flip the prediction far more reliably than random perturbations of the same magnitude, the vulnerability is directional, which is the specific signature of an adversarial rather than a general-brittleness problem: general brittleness predicts instability in most directions near that input, while an adversarial vulnerability predicts instability concentrated in a narrow, exploitable direction.
Mitigation: adversarial training (including examples generated by the same style of targeted perturbation during training) for models where this is a real deployment risk (an attacker actually has the incentive and access to craft such inputs), or, where guarantees are needed rather than empirical robustness, certified-robustness methods such as randomized smoothing, which trade some clean accuracy for a provable robustness radius, at higher inference cost since they require many forward passes.

How the four experiments interlock as a differential. Run preprocessing and precision checks first, since both are comparatively cheap and, if either explains the anomaly, the other two hypotheses become moot for this specific case. If both come back clean (preprocessed tensors match, and the flip persists under deterministic fp32), run the random-perturbation experiment before the targeted-attack experiment, since it is cheaper and its outcome tells you what to expect from the targeted experiment: a model that is already unstable under random noise at this magnitude will likely also be vulnerable to a targeted perturbation, but a model that is stable under random noise and only flips under a specifically-directed one has a much narrower, more targeted vulnerability.

Worked example

Applying the sequence to a concrete case: a traffic-sign classifier flips from "speed limit 60" to "speed limit 80" when a handful of pixels in the sign's border are changed by an amount well within normal camera sensor noise. Running hypothesis 1's experiment first: the preprocessed tensors from the two preprocessing code paths (the pipeline that generated the perturbed sample, and the model's actual serving preprocessing) are diffed directly and match within floating-point rounding, ruling out a preprocessing mismatch. Hypothesis 2's experiment: re-running both the original and perturbed input at full fp32 precision with determinism enabled still reproduces the flip, ruling out precision as the driver. Moving to hypothesis 3, and fixing the budget in real units first, since the whole comparison is meaningless without it: the perturbation budget is plus or minus 3 pixel levels out of 255 per pixel, comparable to ordinary sensor noise. Applying 200 independent draws of random Gaussian noise inside that budget, 7 of the 200 draws flip the prediction, a 3.5 percent flip rate, so the model is NOT broadly unstable at this input under random perturbation, which argues against general brittleness as the dominant explanation. Finally, hypothesis 4's experiment: 10 targeted gradient-based perturbations, constrained to the identical plus-or-minus-3-out-of-255 budget, flip the prediction 9 times, a 90 percent flip rate. Read the pair as a ratio rather than as two separate numbers: 90 percent against 3.5 percent is a factor of about 26, which is the directional signature. The rough decision rule is that a random flip rate in the same ballpark as the targeted one (say within a factor of two or three) says the neighbourhood is broadly unstable and hypothesis 3 is the finding, while a targeted rate an order of magnitude or more above the random rate says the instability is concentrated in a narrow direction. Had the random draws flipped, for example, 120 of 200 times (60 percent) against the same 90 percent targeted rate, hypothesis 3 would have been the confirmed cause instead and the mitigation would have been augmentation rather than adversarial training. The observed pattern (stable under random perturbation, unstable under a specifically-directed one at the identical budget) is exactly the signature that isolates the cause to hypothesis 4, a directional, adversarial-style vulnerability, rather than hypothesis 3's general sensitivity, and the mitigation that follows is adversarial training or, if a provable guarantee is needed for a safety-relevant classifier like this one, a certified-robustness method, rather than the broader data-augmentation response that would have been the right call had hypothesis 3 been confirmed instead.

Trade-offs and pitfalls

The most common wrong turn is assuming "prediction flipped on a tiny change" automatically means adversarial vulnerability and reaching for adversarial training as the first response. Adversarial training is expensive (it roughly multiplies training cost, since each step needs an inner attack computation) and specifically targets directional vulnerability; applied to a case that was actually a preprocessing bug or a precision artifact, it does not fix the real cause and burns significant compute chasing the wrong hypothesis.

A second pitfall is treating hypothesis 3 (general brittleness) and hypothesis 4 (adversarial vulnerability) as the same finding once you rule out preprocessing and precision. They call for genuinely different responses: broader data augmentation is a reasonable, relatively cheap fix for general brittleness, but it does not reliably close a narrow, directional adversarial vulnerability, since an attacker who can compute gradients will find the specific direction augmentation did not happen to cover.

A third pitfall is evaluating the random-perturbation and targeted-perturbation experiments at different perturbation budgets and then comparing their flip rates, which invalidates the comparison. The whole point of the differential is that random and targeted perturbations of the SAME magnitude behave differently for an adversarial vulnerability and similarly for general brittleness; mismatched budgets make that comparison meaningless and can point you at the wrong hypothesis entirely.

Direct answer

A numerical-stability test suite for mixed-precision or quantized training needs four independent checks, because each one catches a failure mode the others cannot: a scan for NaN (not-a-number) or Inf (infinity) values in weights and gradients, an underflow/overflow check on gradient magnitudes specifically at 16-bit floating point (FP16) resolution, a correctness check on the dynamic loss-scaling policy that is supposed to prevent that underflow, and a final accuracy-delta check comparing the mixed-precision model's converged accuracy against a full-precision baseline. Each check has its own automated threshold and its own remediation, from "skip this step" up to "fail the pipeline and fall back to full precision."

Structured elaboration

Check 1: NaN/Inf scan. After every N training steps (or every step, if cheap enough), scan every parameter and gradient tensor for NaN and Inf values. This is the cheapest and most unambiguous check: there is no legitimate reason a weight or gradient should ever be NaN or Inf, so any occurrence is an automatic failure, not a threshold judgment call. Remediation: halt training immediately and dump the last N steps' inputs and gradient norms for offline diagnosis, since by the time a NaN appears, the step that caused it has usually already passed.

Check 2: gradient underflow and overflow at FP16 resolution. FP16 has a much narrower representable range than 32-bit floating point (FP32): a minimum positive normal magnitude around 6.10e-5 and a minimum positive subnormal (a smaller, less precise representable value just above zero) around 5.96e-8, versus FP32's minimum normal around 1.18e-38. A gradient element smaller than FP16's subnormal floor rounds to exactly zero, silently dropping that gradient's contribution to the update, with no error raised anywhere. Overflow is the opposite failure: FP16's maximum representable magnitude is 65504, so a gradient or activation that exceeds it becomes Inf. The check: periodically sample gradient magnitudes and report the fraction that would underflow or overflow if cast to FP16 at the current loss scale, automated threshold example: alert if more than 1% of gradient elements underflow to zero. Remediation on breach: raise the loss scale (or lengthen the growth interval so the scale spends more of the run high) and re-measure; if the underflow fraction is still above 1% at the largest scale that does not push the same layer into overflow, that layer's gradients do not fit inside FP16's range at all, so pin it to FP32 and re-run. This is a tuning response rather than a build failure on the first breach: it escalates to failing the run only when the same layer keeps breaching after both remedies, since at that point the mixed-precision path is silently training a different model than the baseline.

Check 3: dynamic loss-scaling correctness. Dynamic loss scaling exists specifically to fix Check 2's underflow problem: multiply the loss by a scale factor before the backward pass (which multiplies every gradient by the same factor, moving small-but-real values up out of the underflow range), then divide the gradients back down by that same factor before the optimizer step. The scale adjusts automatically: if a step's scaled gradients overflow to Inf or NaN, skip that optimizer step entirely and halve the scale; if a fixed number of consecutive steps stay clean, double the scale to keep gradients using as much of FP16's range as safely possible. The check: verify the scale only ever changes on these two triggers (never drifts for any other reason) and that a step with detected overflow is genuinely skipped, not applied with corrupted gradients. Automated threshold: zero tolerance on the policy itself, so any scale change not attributable to an overflow backoff or a growth-interval doubling fails, as does any step where an overflow was detected and the parameters moved anyway (assert the parameter tensors are bitwise unchanged across a skipped step). Note what is not a failure here, because it is the mildest remediation in the whole suite and it is the scaler working as designed: a single overflowing step is not an incident, the correct response is to skip this step and halve the scale. Remediation on a genuine policy failure: pin the scaler to a fixed, conservative scale and block the mixed-precision path from merging until the policy bug is fixed, because a misbehaving scaler is largely invisible to the rest of the suite. A scale that grows on some third trigger pushes a large fraction of steps into overflow, and every one of those steps is correctly skipped, so the run quietly trains on a fraction of its intended updates while Check 1 sees no NaN and Check 2 sees no underflow. The opposite bug, applying a step whose gradients did overflow, does eventually surface in Check 1, but only at the next periodic scan and only after the corrupted values are already in the master weights, which makes it a diagnosis rather than a defense.

Check 4: accuracy-delta check against the full-precision baseline. After both models converge on the same task with the same seed, compare final accuracy (or loss) between the mixed-precision run and a full-precision baseline run. Automated threshold example: fail if the relative delta exceeds 5%. Remediation on failure: fall back to full precision for this model/task combination and open a numerical-stability investigation, since a real accuracy gap at this stage usually means the earlier checks did not catch something (an underflow that Check 2 missed because it only sampled, or a loss-scale oscillation Check 3 missed because it only checked the trigger logic, not the resulting training dynamics).

The four checks at a glance, since the whole point of automating them is that an on-call engineer can read the failure and know the response without re-deriving it:

The escalation ladder is deliberate: Checks 2 and 3 mostly produce tuning actions, Check 1 stops the run, and only Check 4 gives up on mixed precision for that model, because it is the only one measuring the thing anyone actually cares about.

Worked example

Check 1, a NaN/Inf scan:

import torch

def nan_inf_check(tensors: dict) -> dict:
    violations = {}
    for name, t in tensors.items():
        n_nan, n_inf = torch.isnan(t).sum().item(), torch.isinf(t).sum().item()
        if n_nan or n_inf:
            violations[name] = {"nan_count": n_nan, "inf_count": n_inf}
    return violations

torch.manual_seed(0)
clean = {"layer1.weight": torch.randn(4, 4), "layer1.bias": torch.randn(4)}
poisoned = {k: v.clone() for k, v in clean.items()}
poisoned["layer1.weight"][0, 0] = float("nan")
poisoned["layer1.weight"][1, 1] = float("inf")
print("clean tensors:", nan_inf_check(clean))
print("poisoned tensors:", nan_inf_check(poisoned))

Output on a clean tensor dict versus one with a planted NaN and Inf:

clean tensors: {}
poisoned tensors: {'layer1.weight': {'nan_count': 1, 'inf_count': 1}}

Check 2, underflow at FP16 resolution, three gradient magnitudes spanning the boundary:

small_grad_fp32 = torch.tensor([1.0e-8, 1.0e-6, 8.0e-5], dtype=torch.float32)
small_grad_fp16 = small_grad_fp32.to(torch.float16)
print("fp32 values:", small_grad_fp32.tolist())
print("fp16 values:", small_grad_fp16.tolist())
print("underflowed to exactly zero in fp16:", (small_grad_fp16 == 0).tolist())

Output:

fp32 values: [9.99999993922529e-09, 9.999999974752427e-07, 7.999999797903001e-05]
fp16 values: [0.0, 1.0132789611816406e-06, 7.998943328857422e-05]
underflowed to exactly zero in fp16: [True, False, False]

The 1e-8 value flushes to exactly zero (below FP16's subnormal floor); 1e-6 and 8e-5 both survive, just at reduced precision. This confirms the mechanism Check 3's loss scaling exists to counteract.

Check 3, a re-implementation of the standard dynamic loss-scaling algorithm, run for 10 steps with overflow injected at steps 2 and 7:

class ToyLossScaler:
    def __init__(self, init_scale=2.0 ** 16, growth_interval=3, backoff_factor=0.5, growth_factor=2.0):
        self.scale, self.growth_interval = init_scale, growth_interval
        self.backoff_factor, self.growth_factor = backoff_factor, growth_factor
        self._clean_streak = 0

    def step(self, grad_has_overflow: bool):
        if grad_has_overflow:
            self.scale *= self.backoff_factor
            self._clean_streak = 0
            return "SKIP step, halve scale"
        self._clean_streak += 1
        if self._clean_streak >= self.growth_interval:
            self.scale *= self.growth_factor
            self._clean_streak = 0
            return "apply step, double scale"
        return "apply step"

scaler = ToyLossScaler()
OVERFLOW_STEPS = {2, 7}
for step in range(10):
    overflow = step in OVERFLOW_STEPS
    action = scaler.step(overflow)
    print(f"step {step}: overflow={str(overflow):5s} -> {action:24s} -> scale={int(scaler.scale)}")

Output:

step 0: overflow=False -> apply step               -> scale=65536
step 1: overflow=False -> apply step               -> scale=65536
step 2: overflow=True  -> SKIP step, halve scale   -> scale=32768
step 3: overflow=False -> apply step               -> scale=32768
step 4: overflow=False -> apply step               -> scale=32768
step 5: overflow=False -> apply step, double scale -> scale=65536
step 6: overflow=False -> apply step               -> scale=65536
step 7: overflow=True  -> SKIP step, halve scale   -> scale=32768
step 8: overflow=False -> apply step               -> scale=32768
step 9: overflow=False -> apply step               -> scale=32768

The scale only ever moves on the two defined triggers, halving exactly on the two injected overflow steps and doubling exactly after three consecutive clean steps, matching the intended policy exactly.

Check 4, training the same linear-regression task to convergence (200 steps, Adam optimizer, fixed seed) once in full precision and once with activations downcast to FP16 and back every forward pass (a simplified stand-in for autocast's forward-in-FP16, master-weights-in-FP32 pattern):

def train_linear(cast_activations_to_fp16: bool, steps=200, seed=0):
    torch.manual_seed(seed)
    X = torch.randn(256, 8)
    w_true = torch.randn(8, 1)
    y = X @ w_true + 0.01 * torch.randn(256, 1)
    model = torch.nn.Linear(8, 1)
    opt = torch.optim.Adam(model.parameters(), lr=0.05)
    for _ in range(steps):
        opt.zero_grad()
        out = model(X)
        if cast_activations_to_fp16:                 # forward in FP16, master weights in FP32
            out = out.to(torch.float16).to(torch.float32)
        loss = ((out - y) ** 2).mean()
        loss.backward()
        opt.step()
    return loss.item()

fp32_loss = train_linear(False)
fp16_loss = train_linear(True)
rel_delta = abs(fp16_loss - fp32_loss) / fp32_loss
print(f"full-precision final loss:  {fp32_loss:.10f}")
print(f"mixed-precision final loss: {fp16_loss:.10f}")
print(f"relative delta: {rel_delta * 100:.6f}%  (threshold: 5%)")
print("PASS" if rel_delta < 0.05 else "FAIL")

Output:

full-precision final loss:  0.0000822348
mixed-precision final loss: 0.0000828672
relative delta: 0.769005%  (threshold: 5%)
PASS

Trade-offs and pitfalls

• Check 4's result above is honest but weak, and that weakness is itself worth naming in review. The relative delta came out at 0.77%, well below the 5% threshold. It is not zero, so the FP16 rounding is genuinely perturbing the converged loss rather than being optimized away entirely, but it clears the gate comfortably only because this task is simple and well-conditioned enough that gradient descent lands in essentially the same minimum regardless of per-step rounding noise. A convergence-based accuracy-delta check on an easy task can pass even when a genuine precision problem exists that would only bite on a harder-conditioned or deeper model; choose the CI (continuous integration, the automated pipeline that runs these checks on every change) task deliberately to be sensitive to the failure you are trying to catch, not just any task that happens to be fast to run.
• Check 1 (NaN/Inf) alone is necessary but not sufficient. A model can silently underflow a meaningful fraction of its gradients (Check 2's failure mode) for many steps before that ever manifests as an outright NaN, so a suite that only checks for NaN/Inf will pass while quietly training worse than it should.
• The loss-scale growth interval trades responsiveness against stability. A short growth interval climbs back toward a higher, more precision-preserving scale faster after a backoff, but also re-triggers overflow (and wastes the just-skipped step) sooner if the underlying instability has not actually resolved; a longer interval is more conservative but leaves gradients at reduced precision for longer after every backoff.
• Quantized training (as opposed to FP16 mixed precision) needs its own version of Check 2 and Check 4, since quantization's failure mode is a fixed, coarse step size (not a floating-point exponent range), which fails differently: values do not vanish to zero the way FP16 underflow does, they clip or alias to the nearest quantization level, which needs a different automated check (a per-layer clipping-rate check) than the underflow-rate check above.

Direct answer. A NaN loss almost always means a single operation produced a NaN or Inf somewhere in the forward pass, and that value then poisoned every gradient downstream of it once backprop multiplies through it. The job is to find the earliest point in the computation graph where a NaN first appears, not the point where you happened to notice it (loss is usually the last place it shows up, not the first).

Step-by-step checklist.

1. Confirm it's really NaN, not just "very large." Print the loss and a couple of intermediate tensors every step for the last few steps before the failure; a loss that grows explosively for several steps before going NaN points to gradient explosion, while a loss that goes NaN in a single step points to a specific bad input or a numerically unstable operation (log of zero, division by zero, an unclipped exponential).
2. Localize with an anomaly detector. PyTorch's torch.autograd.set_detect_anomaly(True) (or the equivalent forward hook approach in other frameworks) will raise at the exact operation in the backward pass that first produced a NaN gradient, instead of letting you discover it three layers downstream.
3. Check the inputs and labels, not just the model. A single corrupted input (an inf slipped in from a bad join, a label encoded as -1 where the loss expects a valid class index) is one of the most common root causes and is invisible if you only look at the model code.
4. Review what changed recently in the data or the labels. Ask what landed since the last clean run, not just whether today's batch is valid: a new data partition or backfill, a relabeling job, an upstream schema or unit change (a rate that used to be a fraction now arriving as a percentage), a feature whose null-fill default moved from 0 to NaN, or a label vocabulary that gained a class the loss function's index range does not cover. Two things make this fast. Pin the data version to the last snapshot that trained cleanly and rerun with today's code, which separates "the data changed" from "the code changed" in one experiment. And diff the summary statistics of the current batch of data against that snapshot, per column: min, max, null rate, and cardinality for categoricals. A column whose max jumped several orders of magnitude, or whose null rate went from 0 to 3 percent, is the change that made an operation that was always slightly unstable finally produce an Inf.
5. Check for exploding gradients before you check the model architecture. If gradient norms are growing every step before the NaN, that's the signal, not a coincidence.
6. Only after 1-5 are ruled out, suspect the architecture or hyperparameters: an unclipped learning rate, a missing normalization layer, or (for mixed precision specifically) fp16 overflow, where the fix is dynamic loss scaling rather than gradient clipping.

import torch

def check_for_nans(tensor, name):
    if torch.isnan(tensor).any() or torch.isinf(tensor).any():
        n_nan = torch.isnan(tensor).sum().item()
        n_inf = torch.isinf(tensor).sum().item()
        raise ValueError(f"{name}: {n_nan} NaNs, {n_inf} Infs out of {tensor.numel()} values")

# in the training loop, right after the forward pass:
with torch.autograd.set_detect_anomaly(True):
    output = model(batch_inputs)
    check_for_nans(output, "model output")
    loss = loss_fn(output, batch_labels)
    check_for_nans(loss, "loss")
    loss.backward()

What dynamic loss scaling actually is, since it is named as the first thing to reach for under mixed precision. fp16 can only represent magnitudes down to roughly 6e-8, so small gradients underflow to exactly zero and the corresponding weights simply stop learning. Loss scaling multiplies the loss by a large constant S (65536 is a common starting value) BEFORE the backward pass, which multiplies every gradient by S too and lifts them back into fp16's representable range, then divides the gradients by S again before the optimizer step so the update size is unchanged. The DYNAMIC part is the control loop around S: the framework watches for an inf or NaN appearing in the gradients, and when one does it skips that optimizer step entirely and halves S, and after some number of clean steps in a row it doubles S back up. So a run with dynamic loss scaling working correctly will show occasional skipped steps early on and then settle, and a run where S is stuck at its floor while steps are being skipped constantly is telling you the overflow is not a scaling problem at all.

Remediation, and how to confirm each one actually worked. The first two levers are gradient clipping (torch.nn.utils.clip_grad_norm_) and reducing the learning rate. Both are cheap and reversible, but they are different hypotheses and they need different confirmations, so change ONE at a time.

Confirming gradient clipping. Log the pre-clip global gradient norm alongside the post-clip one, and count how often the clip actually fires. If the clip fires on a small fraction of steps and the pre-clip norm otherwise sits inside a stable band, clipping is doing the job it exists for, absorbing a rare outlier batch, and that is a genuine fix. If instead the clip fires on nearly every step and the pre-clip norm keeps climbing decade by decade (a trace like 3, 40, 800, 20000 while the post-clip norm sits pinned exactly at the threshold), clipping is masking a genuinely diverging run: you have suppressed the symptom, and the NaN returns the moment a pre-clip value exceeds fp32 range before the clip can touch it. A pinned post-clip norm with a rising pre-clip norm is the specific pattern that means "delayed, not fixed."

Confirming a learning-rate reduction. Rerun at the lower learning rate with clipping turned OFF, so the two levers are not covering for each other. If the run is now stable without clipping, the root cause really was the step size interacting with the loss surface, and you have a hyperparameter problem rather than a data or numerics bug. If it still goes NaN at a 10x smaller learning rate, the step size was never the cause and further tuning is wasted effort: go back to items 1 through 3, because a bad input, a bad label, or an unstable operation does not care how small your steps are.

The shared bar both have to clear. Rerun the exact same seed and data for at least three times as many steps as it previously took to fail, not one extra epoch. If the previous run died at step 4,100, surviving to step 5,000 proves nothing; surviving past step 12,300 with the gradient norm staying inside a stable band and the loss still decreasing is evidence. A run that survives longer while its gradient norm climbs steadily is the same failure on a slower clock.

Two variants worth knowing. In mixed precision (fp16), the more common cause is numeric underflow or overflow in a specific operation rather than a genuinely exploding gradient, so the first fix to try is toggling dynamic loss scaling before touching the learning rate. And a NaN doesn't have to be training-only: if a live inference path starts returning NaN for a narrow slice of production inputs while training was clean, the same tensor-level NaN/Inf checks apply, just triggered by whatever unusual input reaches that slice (an out-of-range feature value, a division by a count that happens to be zero for that user), so the debugging tools are identical even though the trigger and the blast radius are different.

Direct answer

A recurrent (RNN-family: recurrent neural network, an architecture that processes a sequence one token at a time while carrying a hidden state forward) model that trains well but degrades in production on variable-length sequences almost always has a padding or masking mismatch: training aggregated information over the TRUE tokens in each sequence, and serving is silently including the zero-padded positions in that same aggregation. The fix is rarely a modeling change; it is aligning exactly how padding is represented, masked, and pooled between the two paths, and proving it with a minimal, deterministic test case before touching production again.

Structured elaboration

Padding. Confirm the pad value and pad position (leading versus trailing) are identical in training and serving, and that the pad value is never a value the model could confuse for real data (a pad id of 0 is a common source of bugs if 0 also happens to be a valid vocabulary index or a valid numeric feature value elsewhere in the pipeline). A mismatch here alone, even with correct masking downstream, can leak signal if any operation touches the raw padded values before the mask is applied.

Masking. Confirm every operation that aggregates across the time dimension, mean pooling, attention weights, or the final hidden state selection, is mask-aware, not just the loss function. It is common to correctly mask the LOSS during training (so padded positions do not contribute gradient) while some other aggregation step, such as a mean-pool layer that summarizes the sequence into a fixed-size vector, silently averages over the full padded length regardless. Training can still converge to a reasonable solution despite this, and the reason is length bucketing (described just below). With bucketing, a training batch built from sequences of length 7 and 8 padded to 8 divides by a number close to each sequence's true length, and the closed form derived below says exactly how close: the length-7 example is diluted by (8-7)/8 = 12.5 percent and the length-8 example not at all. Real buckets are tighter than that toy pair and the spread shrinks with them: lengths 60 to 64 padded to 64 span 6.3 percent down to 0. So within a bucketed batch the rescaling is small and nearly constant across examples, and a near-constant rescaling is something the next layer's weights simply absorb during training. Serving gets no such protection: a single production request of length 2 padded to a fixed maximum of 8 is diluted by 75 percent, and the amount of dilution now swings from request to request instead of being nearly constant within a batch. That is why the identical aggregation code looks fine in aggregate training metrics and only becomes visible input-by-input in production, especially on short sequences where padding is a larger fraction of the sequence.

Batch bucketing. If training used length-based bucketing (grouping similar-length sequences into the same batch to minimize wasted padding), confirm serving either does the same or, if serving processes one request at a time or with very different batch composition, that this does not change the model's behavior. Bucketing itself does not usually change per-example output IF pooling is properly mask-aware, but it is worth confirming explicitly, since an under-tested serving path that never exercises heavily-padded batches (because serving happens to batch similarly-sized requests together in practice) can hide a masking bug that only appears when a genuinely short sequence lands in a batch with much longer ones.

Inference-time preprocessing. Confirm the tokenizer or encoder, truncation rule (head versus tail, and the maximum length itself), and padding logic are the literal same code or same shared configuration in both paths, not two independent reimplementations that are merely intended to match. Freeze and version this preprocessing exactly as you would the model artifact itself.

Constructing a minimal test case that reproduces the discrepancy. The key design choice is to build a batch containing sequences of genuinely different true lengths padded to the same max length, run it through both the training-style (mask-aware) aggregation and the serving-style aggregation, and compare per-example outputs directly, isolating the aggregation logic from the model itself so the test does not depend on trained weights at all. If the two aggregations agree exactly for a full-length sequence (no padding needed) but diverge for a short one, that is a clean, minimal, mechanical proof of the bug, independent of any statistical noise.

Worked example

The snippet below builds exactly that minimal test case: three synthetic sequences with true lengths 2, 5, and 8 (out of a padded max length of 8), and compares a mask-aware mean pool (what a correctly-implemented training pipeline does) against a naive full-length mean pool (the serving bug: dividing by the padded length instead of the true length).

"""
Minimal test case reproducing a padding/masking train-serve discrepancy:
the training pipeline mean-pools token embeddings over the TRUE sequence
length (mask-aware); the serving pipeline (bug) mean-pools over the padded
MAX length, silently diluting short sequences with zero-vectors.

Pinned: numpy Generator seed=7, embedding dim=4, 3 example sequences with
true lengths [2, 5, 8] padded to max_len=8.
Run with: python3 padding_mismatch.py
"""
import numpy as np

rng = np.random.default_rng(seed=7)
max_len, dim = 8, 4
true_lengths = [2, 5, 8]

# token embeddings for each sequence, generated only for the true tokens,
# zero-padded up to max_len (this is what both pipelines receive on the wire)
padded_batch = np.zeros((3, max_len, dim))
for i, length in enumerate(true_lengths):
    padded_batch[i, :length, :] = rng.normal(size=(length, dim))

mask = np.zeros((3, max_len))
for i, length in enumerate(true_lengths):
    mask[i, :length] = 1.0


def train_pool(batch, mask):
    """Mask-aware mean pool: divide by the TRUE length."""
    summed = (batch * mask[:, :, None]).sum(axis=1)
    true_len = mask.sum(axis=1, keepdims=True)
    return summed / true_len


def serving_pool_buggy(batch):
    """BUG: divides by max_len (padded length) regardless of true length."""
    return batch.sum(axis=1) / batch.shape[1]


train_repr = train_pool(padded_batch, mask)
serve_repr = serving_pool_buggy(padded_batch)

print("=== masked mean (training) vs naive full-length mean (serving bug) ===")
for i, length in enumerate(true_lengths):
    diff = np.abs(train_repr[i] - serve_repr[i]).mean()
    scale = np.abs(train_repr[i]).mean()
    print(
        f"seq {i} (true_len={length}/{max_len}): "
        f"mean_abs_diff={diff:.4f}  relative_to_train_scale={diff / scale:.2%}"
    )

Actual output from running this script:

=== masked mean (training) vs naive full-length mean (serving bug) ===
seq 0 (true_len=2/8): mean_abs_diff=0.1697  relative_to_train_scale=75.00%
seq 1 (true_len=5/8): mean_abs_diff=0.2171  relative_to_train_scale=37.50%
seq 2 (true_len=8/8): mean_abs_diff=0.0000  relative_to_train_scale=0.00%

The pattern is exactly what the bug predicts, and it is exact enough to derive analytically as a cross-check: since the padded positions are zero vectors, the sum is unaffected by the bug, only the divisor changes, from the true length to the max length. So the serving representation is simply the training representation scaled by $\text{true\_len} / \text{max\_len}$, and the relative error is exactly $1 - \text{true\_len}/\text{max\_len} = (\text{max\_len} - \text{true\_len}) / \text{max\_len}$. For sequence 0, $(8-2)/8 = 0.75$, matching the printed 75.00 percent exactly; for sequence 1, $(8-5)/8 = 0.375$, matching 37.50 percent exactly; and for sequence 2, which has no padding at all, the two aggregations are identical by construction, giving exactly 0. This closed form is also the practical severity signal for production: the shortest, most padding-heavy sequences are hit hardest, which is consistent with a model that "performs well during training but underperforms in production" being especially bad on the shorter, more common end of a real sequence-length distribution while looking fine on the longest sequences.

Trade-offs and pitfalls

The most common wrong turn is trying to fix this by retraining with more padding-heavy examples or adding regularization, treating it as a generalization problem rather than a mechanical aggregation bug. No amount of additional training data fixes a serving path that divides by the wrong number; the fix is entirely in the serving code, and retraining would only be needed afterward if you also want the model to be robust to any remaining, smaller padding-related noise.

A second pitfall is testing only with sequences at or near the maximum length during pre-deploy validation, since that is often what is convenient to hand-pick as a "representative" test case, and it is exactly the case where this bug is invisible (as sequence 2 in the worked example shows, a full-length sequence has zero discrepancy). The minimal test case has to deliberately include short sequences relative to the batch's max length to have any power to catch this bug at all.

A third pitfall once the fix is deployed: verify the fix using the SAME kind of length-diverse batch, not just a smoke test on a single request, since a masking or bucketing bug can be batch-composition-dependent (it may only appear when a short sequence shares a batch with a much longer one) and a single-request smoke test would pass regardless of whether the underlying bug is actually fixed.

Direct answer. Overfitting and leakage can produce the exact same symptom (a gap between training and validation performance), but they need genuinely different fixes, so the investigation has to isolate WHICH one is happening before touching regularization or feature engineering. The discriminator is not the direction the gap moves, it is how much the ABSOLUTE validation number moves when you change how the split is built, and you have to read the training score and the validation score separately to see it.

Experiment and validation plan.

1. Rebuild the evaluation as a strict time-based split, with the held-out slice's features recomputed as of prediction time, retrain, and record training score and validation score SEPARATELY, before and after. Overfitting is a property of model capacity against data volume, so it is essentially insensitive to how the split was drawn: both numbers land where they landed before. Leakage is a property of how the evaluation slice was constructed relative to the label, so it is extremely sensitive: the training score barely moves (the leak is still in the training rows) while the validation score falls off a cliff, which means the gap GROWS rather than shrinks. Note the trap this rules out: under the original random split a leaky model often shows a suspiciously SMALL gap, because the leak inflates validation just as much as it inflates training.
2. Remove suspect features one at a time and retrain, inside the corrected evaluation from step 1. If validation performance actually IMPROVES when a specific feature is removed (not just training performance getting worse, which is expected when you remove any useful feature), that's a strong leakage signal. The mechanism, which is worth being able to state rather than memorize: in the corrected setup the leaky feature still carries the label in the training rows but carries nothing in the held-out slice, because a point-in-time read of it returns a value computed before the outcome existed. The model therefore spends real capacity learning to lean on a signal that is simply not present at evaluation time, and its predictions are actively worse than those of a model that never saw the feature at all. Dropping it does not just stop helping, it stops hurting.
3. Check feature availability timestamps for every feature the model with high suspicion currently uses, documenting exactly when in the real pipeline each one becomes known, relative to the label's outcome time.

How steps 1 and 2 fit together rather than contradicting each other. They assume different evaluation setups, and that is the point of doing them in order. Step 1 operates on the ORIGINAL random split, where the leak is present in both slices, which is exactly why validation looks so good there and why the gap is small. Step 2 operates INSIDE the corrected setup step 1 produces, where the leak is present in training but absent from the held-out slice. Same feature, two different evaluations, two different tells. Run step 1 first to build the honest evaluation, then run step 2 inside it.

What each result implies, with numbers you can reproduce. The setup: 4,000 rows, 10 standard-normal features with the label drawn from a logistic model of them, plus one extra feature built from an aggregate that is only complete once the outcome has settled (so for settled rows it carries the label plus noise, and a point-in-time read of an unsettled row returns noise alone). Model is a gradient-boosted classifier, 80/20 split, seed pinned.

import numpy as np
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import roc_auc_score

rng = np.random.default_rng(0)                    # pinned, so these numbers are reproducible
n, n_tr = 4000, 3200
X = rng.normal(size=(n, 10))                      # 10 honest standard-normal features
beta = rng.normal(size=10)
y = (rng.random(n) < 1 / (1 + np.exp(-(X @ beta)))).astype(int)   # label from a logistic model

# The leaky feature: an aggregate that is only complete once the outcome has settled.
# Settled rows carry the label plus noise; a point-in-time read of an unsettled row is noise alone.
settled = 2.0 * (2 * y - 1) + rng.normal(0, 0.5, n)
as_of   = rng.normal(0, 0.5, n)
X_leak    = np.column_stack([X, settled])         # leak as the training pipeline sees it
X_pointin = np.column_stack([X, as_of])           # leak as production would actually read it

def aucs(Xtr, ytr, Xte, yte, high_capacity=False):
    kw = dict(random_state=0)
    if high_capacity:
        kw.update(max_iter=500, max_leaf_nodes=63, min_samples_leaf=1,
                  l2_regularization=0.0, early_stopping=False, learning_rate=0.3)
    m = HistGradientBoostingClassifier(**kw).fit(Xtr, ytr)
    return (roc_auc_score(ytr, m.predict_proba(Xtr)[:, 1]),
            roc_auc_score(yte, m.predict_proba(Xte)[:, 1]))

perm = rng.permutation(n)
r_tr, r_te = perm[:n_tr], perm[n_tr:]             # random 80/20
t_tr, t_te = np.arange(n_tr), np.arange(n_tr, n)  # chronological 80/20

def show(label, a, b):
    print("  %-48s train %.3f   val %.3f   gap %.3f" % (label, a, b, a - b))

print("LEAKAGE")
show("random split, leak present in both slices",
     *aucs(X_leak[r_tr], y[r_tr], X_leak[r_te], y[r_te]))
show("time split, held-out slice rebuilt point-in-time",
     *aucs(X_leak[t_tr], y[t_tr], X_pointin[t_te], y[t_te]))
show("same time split, leaky feature DROPPED",
     *aucs(X[t_tr], y[t_tr], X[t_te], y[t_te]))

print("\nOVERFITTING (no leaky feature, high-capacity model)")
show("random split", *aucs(X[r_tr], y[r_tr], X[r_te], y[r_te], high_capacity=True))
show("chronological split", *aucs(X[t_tr], y[t_tr], X[t_te], y[t_te], high_capacity=True))

LEAKAGE
  random split, leak present in both slices        train 1.000   val 1.000   gap 0.000
  time split, held-out slice rebuilt point-in-time train 1.000   val 0.569   gap 0.431
  same time split, leaky feature DROPPED           train 0.999   val 0.872   gap 0.127

OVERFITTING (no leaky feature, high-capacity model)
  random split                                     train 1.000   val 0.884   gap 0.116
  chronological split                              train 1.000   val 0.865   gap 0.135

Read the leakage block left to right: validation moved 0.431 (1.000 down to 0.569) purely from changing how the split was built, while training moved 0.000. Note also the first row, where the gap is 0.000: under the original random split this leaky model looks flawless and shows no gap at all, which is the trap. Read the overfitting block: validation moved 0.019 across the same change (0.884 down to 0.865), which is inside the sampling noise of an 800-row holdout, and the gap barely budged (0.116 to 0.135). That contrast, four tenths of an AUC point versus two hundredths, is the whole diagnosis. Then read the third leakage row: dropping the leaky feature RAISED validation from 0.569 to 0.872, which is step 2's tell in its clearest form and something no honest feature ever does.

So: if the validation score collapses when the split is rebuilt correctly, and a specific feature's removal then raises it, that's leakage, and the fix is removing or correctly re-deriving that feature, not more regularization. If both scores are roughly where they were under either split methodology, and removing individual features only ever makes validation worse (never better), that's genuine overfitting, and the fix is standard: more regularization, more data, or a smaller model.

If training loss is near zero AND validation accuracy is ALSO unstable across runs (not just low). This is a distinct signal from a persistently low-but-stable validation number, instability run-to-run suggests either high variance in a small validation set (the validation set itself is too small to give a stable estimate, an easy thing to check by just re-splitting validation data multiple times and looking at the spread of the resulting accuracy) or genuine sensitivity to initialization/data order interacting with a model that has memorized rather than generalized (near-zero training loss with essentially no learned structure to fall back on when the exact memorized examples aren't being tested). Both point toward the same practical next step, get a larger or more stable validation set before drawing any further conclusion, since an unstable metric can't reliably distinguish between competing hypotheses in the first place. It also invalidates the step-1 comparison above until it is fixed: a 0.02 move in validation means nothing if re-splitting alone moves it by 0.05.