Debugging and Testing ML Systems Questions

Direct answer. When offline metrics improve but the online experiment shows no lift or a regression, the offline metric and the business metric are measuring genuinely different things, and the investigation has to check each place they can diverge rather than assuming the online result is simply "noisier."

Checklist to reconcile the discrepancy.

1. Distribution shift between offline eval traffic and live production traffic. The offline test set is a snapshot; live traffic during the experiment window may have a different mix of users, devices, or time-of-day patterns that the offline set doesn't represent.
2. Calibration differences. A model can have a better AUC (better ranking) while being worse-calibrated (its absolute predicted probabilities are less trustworthy), and if any downstream logic uses the raw probability, not just the ranking, a ranking-only metric like AUC will miss this entirely. To measure it rather than hypothesize it: bucket the holdout predictions into deciles of predicted probability and compare, per bucket, the mean predicted probability against the observed outcome rate. That comparison is a reliability diagram, and its summary number is expected calibration error, the size-weighted average absolute gap between the two. Compute it for both model versions on the same holdout. A new model whose AUC rises while its expected calibration error goes from about 0.01 to about 0.04 is exactly this case, and any downstream rule with a fixed probability threshold (a bid, a notification cutoff, an eligibility gate) will fire at a different rate than it used to even though the ranking got better.
3. Feature availability mismatch. Confirm every feature the offline evaluation used is actually available, fresh, and computed the same way at serving time; a feature that's stale or a no-op in production (silently falling back to a default) can make the online model perform worse than its offline twin despite being "the same model."
4. Position or exposure bias. If the offline metric is computed on logged data that itself came from a previous model's exposure pattern, it inherits that model's blind spots; a new model that ranks differently may simply not be evaluated fairly by metrics computed against the old exposure distribution.
5. Per-segment heterogeneity. An aggregate online metric can hide a model that improved for most users but regressed sharply for a smaller, high-value segment; slice the online result the same way before concluding "no lift" is the whole story.
6. Integrity of the pipeline feeding the ONLINE metric, which is a different pipeline from the one feeding the offline metric and fails in its own ways. Check the sample ratio first: if the traffic split lands at, say, 52/48 when it was configured 50/50, the randomization or the logging is broken and the whole comparison is void before any model question is worth asking. Then confirm the business metric's definition and attribution window did not change during the experiment window, that bot and internal traffic are filtered identically in both arms, and that the randomization unit matches the analysis unit (randomizing by user and analyzing by session inflates significance and can manufacture a regression that is not there).

Two concrete shapes this takes in practice, with the numbers, to ground what "worse" looks like:

shape 1   offline AUC        0.812 -> 0.834   (+2.2 points, +2.7% relative)
          online revenue per session  -1.4%   (95% interval -2.3% to -0.5%)

shape 2   offline logloss    0.412 -> 0.396   (-3.9% relative, an improvement)
          online watch-time per session -3.1% (95% interval -4.4% to -1.8%)

(Logloss is the average negative log of the probability the model assigned to the outcome that actually occurred; lower is better, and unlike AUC it is sensitive to the ABSOLUTE probabilities rather than only their ordering, so it partly captures calibration and partly explains why a model can move the two in opposite directions.)

The intervals are the part that decides whether you have an investigation at all. In both shapes above the 95% interval excludes zero, so the regression is real and worth the checklist. A result like -0.4% with an interval of -1.6% to +0.8% is a different situation entirely: the interval spans zero, so you do not have a regression, you have an underpowered experiment, and the correct next step is more exposure or a longer window rather than a root-cause hunt. Deciding which of those two you are in comes before item 1.

Diagnostic ordering, all six items placed. Order by cost first, and let a free question re-rank the rest. Before (0), the interval check above: no interval excluding zero, no investigation. First (6), experiment and metric-pipeline integrity, because a sample-ratio mismatch or a changed metric definition invalidates every comparison downstream of it, and reading the split and the metric definition costs a query. Second (3), feature-availability mismatch, the least expensive of the model-side checks and the one that turns out to be the culprit most often. Third (5), per-segment heterogeneity, because it is a re-slice of data you already have from the running experiment and it can change what the rest of the investigation is even for: if the aggregate is hiding a sharp regression in one segment, you are now explaining a segment, not a model. Fourth (2), calibration, and note that it is CHEAPER than the two checks below it, not more expensive: it needs only the model's own predicted probabilities scored against outcomes on a fixed holdout, with no live traffic and no exposure comparison. What decides whether it runs here at all is one free question you can answer before running anything, does any downstream system consume the raw probability rather than the rank (a bid, a notification cutoff, an eligibility gate). If yes, calibration moves ahead of everything except (6) and (3), because it is both cheap and the leading hypothesis for exactly this AUC-up-business-metric-down shape. If nothing downstream reads the probability, calibration cannot be the mechanism and you can skip it entirely. Last (1) and (4) together, which genuinely require comparing live traffic and logged-exposure distributions and so cost the most to set up.

Communicating with a skeptical stakeholder. State clearly, and separately, what you have confirmed (for example, "feature parity between offline and online is confirmed, that's ruled out") versus what remains a hypothesis ("we believe this is a calibration issue based on X, but haven't yet confirmed it"), and give a concrete next step and timeline rather than a single verdict. A stakeholder who's skeptical the new model is worse is usually reacting to an unexplained gap, not to the existence of a gap, so showing the specific checks you've run (and their results) does more to build confidence than asserting a conclusion before the investigation is done. Bring the interval with every number you quote: "revenue per session is down 1.4%, interval -2.3% to -0.5%" ends an argument that "revenue looks down" only starts.

Direct answer

A generic service postmortem's artifact list (code commit, config, deploy timestamp, request logs) captures everything needed to reproduce a stateless service, because its behavior is fully determined by code and config. A machine learning (ML) system's behavior additionally depends on a large, opaque model artifact and the data that produced it, neither of which is recoverable from source code alone, so an ML-specific postmortem needs five items a generic template does not ask for: the exact model artifact version, the exact training dataset version, the training-run's experiment identifier, a feature-store snapshot of what the model actually saw at serving time, and reproduction steps that tie all of those together into one runnable recipe. Without these five, the incident can be narrated but not actually reconstructed.

Structured elaboration

Model version or artifact identifier. Record the exact model artifact identifier (a registry version, a content hash of the weights file), not a human description like "the model deployed on the 12th." This matters specifically for ML because a model's decision logic lives in a large binary weights artifact that is not derivable from source code the way a compiled service is; two "deploys of the same code" at different times can carry genuinely different model weights if a retrain happened in between, so the artifact identifier, not the deploy timestamp, is the only thing that actually pins down what was making decisions during the incident.

Training dataset version or snapshot identifier. Record the exact identifier of the dataset snapshot the model was trained on, including whatever preprocessing or feature computation was applied as of that snapshot. This matters specifically for ML because datasets are not static the way source code is: they get backfilled, corrected, or silently regenerated over time, so "the training data" as it exists today can already differ from what actually trained the incident-era model. Without a pinned snapshot identifier, a later attempt to inspect "the training data" may be looking at data that no longer matches what produced the deployed model.

Experiment or training-run identifier. Record the identifier linking to the full experiment-tracking record for the training run that produced the deployed model: hyperparameters, random seed, training code commit, and training environment. This matters specifically for ML because reproducing an ML incident sometimes requires reproducing the training process itself, for example to determine whether the incident is a one-off artifact of that specific training run or something that would recur from any training run using the same code and data. A generic service never needs this distinction, since its code is inherently, deterministically reproducible from source; a model trained again from the identical code and data can still land on meaningfully different weights depending on the random seed and other run-specific factors, so the experiment record is what tells you whether the run itself was unusual.

Feature-store snapshot at serving time. Record the actual feature values the model received for a representative sample of the affected requests, not just the raw request payload. This matters specifically for ML because a model's real input is computed by a separate feature pipeline, often running asynchronously and sometimes diverging from what the training pipeline computed for the same logical input, a failure mode commonly called training-serving skew. A generic postmortem's request log tells you what was asked; an ML postmortem additionally needs to know what the model actually saw, since those two things are not the same system and can silently disagree.

Reproduction steps. A runnable recipe that ties the four items above together: fetch this exact model artifact, feed it this exact set of feature vectors (either replayed from the feature-store snapshot or recomputed from the pinned dataset version with the pinned feature-computation code), using this exact serving code version, and confirm the same failing output reproduces. This matters specifically for ML because a generic incident's reproduction usually reduces to "revert to commit X and replay the request," a single axis to pin. An ML incident's reproduction has to pin the model artifact, the feature computation, and the serving code as three separate axes, any one of which can independently explain a discrepancy if left unpinned, so the checklist has to specify exactly which artifacts to fetch and in what order, not just point at a commit hash.

Worked example

A filled-in artifact block for a hypothetical incident, showing the shape these five items take in practice, distinct from the narrative sections (timeline, impact, corrective actions) a generic postmortem template already provides:

Each row exists because a generic postmortem's usual artifact (a code commit and a deploy timestamp) genuinely does not capture it: none of the middle three rows have any equivalent in a stateless service's incident record.

Trade-offs and pitfalls

• Capturing the model version but not the dataset version is a common half-measure. It tells you what weights were live but not what produced them, which blocks any attempt to understand whether a training-data problem caused the incident.
• A feature-store snapshot captured too late, after the incident window, is close to useless if the online feature pipeline is itself mutable or time-decaying. Snapshot capture has to happen close to the actual serving time of the affected requests, not whenever someone gets around to writing the postmortem.
• Recording artifact identifiers without a working reproduction recipe still leaves the incident unreproducible in practice. The identifiers are necessary but not sufficient; someone still has to know, and document, the exact sequence of steps to fetch and combine them.
• This checklist is deliberately narrow. It covers only the artifacts an ML system needs beyond a generic postmortem, not the postmortem's document structure, timeline narrative, or corrective-action process, which a generic incident-postmortem template already handles perfectly well for an ML incident just as it would for any other.

Direct answer. For an ML system, "smoke test," "regression test," and "integration test" mean something more specific than the generic software-testing definitions, because the thing under test includes a trained model artifact and a data pipeline, not just deterministic code.

Smoke test. A fast, shallow check that the deployed model and serving path are alive and not obviously broken, it is not checking correctness of predictions, only that the system runs. Example: send one fixed, known-valid input through the live inference endpoint after every deploy and assert that a response comes back within a latency budget, with the expected shape and no error. This should run automatically immediately after every deploy, before any real traffic is routed to the new version, because it is the cheapest possible check and should never be skipped.

Regression test. Checks that model QUALITY hasn't gotten worse compared to a previous, trusted baseline, using a frozen golden dataset with known expected outputs or acceptable metric thresholds. Example: run the newly trained model against a held-out golden set of 500 labeled examples and assert that accuracy has not dropped by more than a small tolerance (say 1 percentage point) versus the currently-deployed model's score on the same set. This runs whenever a new model candidate is produced, before it's promoted to production, since it's the direct gate against shipping a worse model.

Integration test. Checks that the full pipeline, from raw data through feature computation, model inference, and post-processing, works correctly end to end, catching bugs that live at the seams between components rather than inside any single one. Example: run a short synthetic dataset through the entire pipeline (ingestion, feature transforms, model, and any business-rule post-processing) and assert the final output schema and a couple of known invariants hold (for example, a probability output stays in [0, 1] after post-processing). This should run in CI on every merge to the main branch and again nightly against the latest real data snapshot, since integration bugs are often caused by a change on one side of a seam that the other side wasn't updated for.

Why the distinction matters in practice. A model can pass its smoke test (the endpoint responds) while badly failing its regression test (the new model is meaningfully worse than the old one), and it can pass both while failing an integration test (the pipeline upstream of the model changed a feature's units and the model is now scoring garbage input that looks, from the model's own perspective, like valid input). Treating all three as one undifferentiated "does it work" check is how a regression or an integration-seam bug reaches production despite a green CI pipeline.

Direct answer

A numerical-stability test suite for mixed-precision or quantized training needs four independent checks, because each one catches a failure mode the others cannot: a scan for NaN (not-a-number) or Inf (infinity) values in weights and gradients, an underflow/overflow check on gradient magnitudes specifically at 16-bit floating point (FP16) resolution, a correctness check on the dynamic loss-scaling policy that is supposed to prevent that underflow, and a final accuracy-delta check comparing the mixed-precision model's converged accuracy against a full-precision baseline. Each check has its own automated threshold and its own remediation, from "skip this step" up to "fail the pipeline and fall back to full precision."

Structured elaboration

Check 1: NaN/Inf scan. After every N training steps (or every step, if cheap enough), scan every parameter and gradient tensor for NaN and Inf values. This is the cheapest and most unambiguous check: there is no legitimate reason a weight or gradient should ever be NaN or Inf, so any occurrence is an automatic failure, not a threshold judgment call. Remediation: halt training immediately and dump the last N steps' inputs and gradient norms for offline diagnosis, since by the time a NaN appears, the step that caused it has usually already passed.

Check 2: gradient underflow and overflow at FP16 resolution. FP16 has a much narrower representable range than 32-bit floating point (FP32): a minimum positive normal magnitude around 6.10e-5 and a minimum positive subnormal (a smaller, less precise representable value just above zero) around 5.96e-8, versus FP32's minimum normal around 1.18e-38. A gradient element smaller than FP16's subnormal floor rounds to exactly zero, silently dropping that gradient's contribution to the update, with no error raised anywhere. Overflow is the opposite failure: FP16's maximum representable magnitude is 65504, so a gradient or activation that exceeds it becomes Inf. The check: periodically sample gradient magnitudes and report the fraction that would underflow or overflow if cast to FP16 at the current loss scale, automated threshold example: alert if more than 1% of gradient elements underflow to zero. Remediation on breach: raise the loss scale (or lengthen the growth interval so the scale spends more of the run high) and re-measure; if the underflow fraction is still above 1% at the largest scale that does not push the same layer into overflow, that layer's gradients do not fit inside FP16's range at all, so pin it to FP32 and re-run. This is a tuning response rather than a build failure on the first breach: it escalates to failing the run only when the same layer keeps breaching after both remedies, since at that point the mixed-precision path is silently training a different model than the baseline.

Check 3: dynamic loss-scaling correctness. Dynamic loss scaling exists specifically to fix Check 2's underflow problem: multiply the loss by a scale factor before the backward pass (which multiplies every gradient by the same factor, moving small-but-real values up out of the underflow range), then divide the gradients back down by that same factor before the optimizer step. The scale adjusts automatically: if a step's scaled gradients overflow to Inf or NaN, skip that optimizer step entirely and halve the scale; if a fixed number of consecutive steps stay clean, double the scale to keep gradients using as much of FP16's range as safely possible. The check: verify the scale only ever changes on these two triggers (never drifts for any other reason) and that a step with detected overflow is genuinely skipped, not applied with corrupted gradients. Automated threshold: zero tolerance on the policy itself, so any scale change not attributable to an overflow backoff or a growth-interval doubling fails, as does any step where an overflow was detected and the parameters moved anyway (assert the parameter tensors are bitwise unchanged across a skipped step). Note what is not a failure here, because it is the mildest remediation in the whole suite and it is the scaler working as designed: a single overflowing step is not an incident, the correct response is to skip this step and halve the scale. Remediation on a genuine policy failure: pin the scaler to a fixed, conservative scale and block the mixed-precision path from merging until the policy bug is fixed, because a misbehaving scaler is largely invisible to the rest of the suite. A scale that grows on some third trigger pushes a large fraction of steps into overflow, and every one of those steps is correctly skipped, so the run quietly trains on a fraction of its intended updates while Check 1 sees no NaN and Check 2 sees no underflow. The opposite bug, applying a step whose gradients did overflow, does eventually surface in Check 1, but only at the next periodic scan and only after the corrupted values are already in the master weights, which makes it a diagnosis rather than a defense.

Check 4: accuracy-delta check against the full-precision baseline. After both models converge on the same task with the same seed, compare final accuracy (or loss) between the mixed-precision run and a full-precision baseline run. Automated threshold example: fail if the relative delta exceeds 5%. Remediation on failure: fall back to full precision for this model/task combination and open a numerical-stability investigation, since a real accuracy gap at this stage usually means the earlier checks did not catch something (an underflow that Check 2 missed because it only sampled, or a loss-scale oscillation Check 3 missed because it only checked the trigger logic, not the resulting training dynamics).

The four checks at a glance, since the whole point of automating them is that an on-call engineer can read the failure and know the response without re-deriving it:

The escalation ladder is deliberate: Checks 2 and 3 mostly produce tuning actions, Check 1 stops the run, and only Check 4 gives up on mixed precision for that model, because it is the only one measuring the thing anyone actually cares about.

Worked example

Check 1, a NaN/Inf scan:

import torch

def nan_inf_check(tensors: dict) -> dict:
    violations = {}
    for name, t in tensors.items():
        n_nan, n_inf = torch.isnan(t).sum().item(), torch.isinf(t).sum().item()
        if n_nan or n_inf:
            violations[name] = {"nan_count": n_nan, "inf_count": n_inf}
    return violations

torch.manual_seed(0)
clean = {"layer1.weight": torch.randn(4, 4), "layer1.bias": torch.randn(4)}
poisoned = {k: v.clone() for k, v in clean.items()}
poisoned["layer1.weight"][0, 0] = float("nan")
poisoned["layer1.weight"][1, 1] = float("inf")
print("clean tensors:", nan_inf_check(clean))
print("poisoned tensors:", nan_inf_check(poisoned))

Output on a clean tensor dict versus one with a planted NaN and Inf:

clean tensors: {}
poisoned tensors: {'layer1.weight': {'nan_count': 1, 'inf_count': 1}}

Check 2, underflow at FP16 resolution, three gradient magnitudes spanning the boundary:

small_grad_fp32 = torch.tensor([1.0e-8, 1.0e-6, 8.0e-5], dtype=torch.float32)
small_grad_fp16 = small_grad_fp32.to(torch.float16)
print("fp32 values:", small_grad_fp32.tolist())
print("fp16 values:", small_grad_fp16.tolist())
print("underflowed to exactly zero in fp16:", (small_grad_fp16 == 0).tolist())

Output:

fp32 values: [9.99999993922529e-09, 9.999999974752427e-07, 7.999999797903001e-05]
fp16 values: [0.0, 1.0132789611816406e-06, 7.998943328857422e-05]
underflowed to exactly zero in fp16: [True, False, False]

The 1e-8 value flushes to exactly zero (below FP16's subnormal floor); 1e-6 and 8e-5 both survive, just at reduced precision. This confirms the mechanism Check 3's loss scaling exists to counteract.

Check 3, a re-implementation of the standard dynamic loss-scaling algorithm, run for 10 steps with overflow injected at steps 2 and 7:

class ToyLossScaler:
    def __init__(self, init_scale=2.0 ** 16, growth_interval=3, backoff_factor=0.5, growth_factor=2.0):
        self.scale, self.growth_interval = init_scale, growth_interval
        self.backoff_factor, self.growth_factor = backoff_factor, growth_factor
        self._clean_streak = 0

    def step(self, grad_has_overflow: bool):
        if grad_has_overflow:
            self.scale *= self.backoff_factor
            self._clean_streak = 0
            return "SKIP step, halve scale"
        self._clean_streak += 1
        if self._clean_streak >= self.growth_interval:
            self.scale *= self.growth_factor
            self._clean_streak = 0
            return "apply step, double scale"
        return "apply step"

scaler = ToyLossScaler()
OVERFLOW_STEPS = {2, 7}
for step in range(10):
    overflow = step in OVERFLOW_STEPS
    action = scaler.step(overflow)
    print(f"step {step}: overflow={str(overflow):5s} -> {action:24s} -> scale={int(scaler.scale)}")

Output:

step 0: overflow=False -> apply step               -> scale=65536
step 1: overflow=False -> apply step               -> scale=65536
step 2: overflow=True  -> SKIP step, halve scale   -> scale=32768
step 3: overflow=False -> apply step               -> scale=32768
step 4: overflow=False -> apply step               -> scale=32768
step 5: overflow=False -> apply step, double scale -> scale=65536
step 6: overflow=False -> apply step               -> scale=65536
step 7: overflow=True  -> SKIP step, halve scale   -> scale=32768
step 8: overflow=False -> apply step               -> scale=32768
step 9: overflow=False -> apply step               -> scale=32768

The scale only ever moves on the two defined triggers, halving exactly on the two injected overflow steps and doubling exactly after three consecutive clean steps, matching the intended policy exactly.

Check 4, training the same linear-regression task to convergence (200 steps, Adam optimizer, fixed seed) once in full precision and once with activations downcast to FP16 and back every forward pass (a simplified stand-in for autocast's forward-in-FP16, master-weights-in-FP32 pattern):

def train_linear(cast_activations_to_fp16: bool, steps=200, seed=0):
    torch.manual_seed(seed)
    X = torch.randn(256, 8)
    w_true = torch.randn(8, 1)
    y = X @ w_true + 0.01 * torch.randn(256, 1)
    model = torch.nn.Linear(8, 1)
    opt = torch.optim.Adam(model.parameters(), lr=0.05)
    for _ in range(steps):
        opt.zero_grad()
        out = model(X)
        if cast_activations_to_fp16:                 # forward in FP16, master weights in FP32
            out = out.to(torch.float16).to(torch.float32)
        loss = ((out - y) ** 2).mean()
        loss.backward()
        opt.step()
    return loss.item()

fp32_loss = train_linear(False)
fp16_loss = train_linear(True)
rel_delta = abs(fp16_loss - fp32_loss) / fp32_loss
print(f"full-precision final loss:  {fp32_loss:.10f}")
print(f"mixed-precision final loss: {fp16_loss:.10f}")
print(f"relative delta: {rel_delta * 100:.6f}%  (threshold: 5%)")
print("PASS" if rel_delta < 0.05 else "FAIL")

Output:

full-precision final loss:  0.0000822348
mixed-precision final loss: 0.0000828672
relative delta: 0.769005%  (threshold: 5%)
PASS

Trade-offs and pitfalls

• Check 4's result above is honest but weak, and that weakness is itself worth naming in review. The relative delta came out at 0.77%, well below the 5% threshold. It is not zero, so the FP16 rounding is genuinely perturbing the converged loss rather than being optimized away entirely, but it clears the gate comfortably only because this task is simple and well-conditioned enough that gradient descent lands in essentially the same minimum regardless of per-step rounding noise. A convergence-based accuracy-delta check on an easy task can pass even when a genuine precision problem exists that would only bite on a harder-conditioned or deeper model; choose the CI (continuous integration, the automated pipeline that runs these checks on every change) task deliberately to be sensitive to the failure you are trying to catch, not just any task that happens to be fast to run.
• Check 1 (NaN/Inf) alone is necessary but not sufficient. A model can silently underflow a meaningful fraction of its gradients (Check 2's failure mode) for many steps before that ever manifests as an outright NaN, so a suite that only checks for NaN/Inf will pass while quietly training worse than it should.
• The loss-scale growth interval trades responsiveness against stability. A short growth interval climbs back toward a higher, more precision-preserving scale faster after a backoff, but also re-triggers overflow (and wastes the just-skipped step) sooner if the underlying instability has not actually resolved; a longer interval is more conservative but leaves gradients at reduced precision for longer after every backoff.
• Quantized training (as opposed to FP16 mixed precision) needs its own version of Check 2 and Check 4, since quantization's failure mode is a fixed, coarse step size (not a floating-point exponent range), which fails differently: values do not vanish to zero the way FP16 underflow does, they clip or alias to the nearest quantization level, which needs a different automated check (a per-layer clipping-rate check) than the underflow-rate check above.

Direct answer

Roll out a feature-schema migration the same way you would roll out any breaking API change: version the schema explicitly, write an explicit transformation function between the old and new version rather than mutating the field in place, and gate the migration behind a contract test that proves the transform preserves the feature's actual meaning, not just its type. Serve both schema versions side by side during a transition window (old readers get the old shape, new readers get the new shape, both backed by the same underlying data) so training and serving never briefly disagree about which version is current.

Structured elaboration

Versioning the schema. Attach an explicit version identifier to the schema itself (a schema_version field on the record, or a versioned table/topic name), never rely on "whatever the code currently produces" as the implicit definition of the current schema. Every consumer, the training pipeline and the serving pipeline alike, should read the version explicitly and know which transformation, if any, it needs to apply, rather than assuming the shape it happens to receive matches what it expects.

Writing and testing transformation functions between versions. Write one pure, explicit function per version transition (v1_to_v2, not a general-purpose "migrate to latest" that tries to handle every version pair at once), so each transform is small enough to reason about and test in isolation. Test it two ways: with concrete example-based unit tests on a handful of known records, and with a property-based test (using a library such as Hypothesis) asserting an invariant that must hold for every valid input, not just the examples you happened to write by hand. The property-based layer is what catches the transform being correct on your fixtures but wrong in general.

Contract tests that catch a silent meaning change. The specific risk this question names, a migration that changes what a feature means without anyone noticing, needs a test built around a round-trip or an equivalence invariant, not just a type check. A type check confirms the output is a float; it says nothing about whether that float still represents the same underlying quantity. A round-trip contract test converts the migrated value back to the old representation and asserts it reconstructs the original exactly (up to the precision the two representations can express), which catches a transform that is well-typed but computes the wrong thing, for example applying the wrong unit conversion.

Worked example

A concrete migration: a price_cents integer field (v1) becomes a price_dollars float field (v2).

def transform_v1_to_v2(record_v1: dict) -> dict:
    return {
        "user_id": record_v1["user_id"],
        "price_dollars": record_v1["price_cents"] / 100.0,
    }

def contract_test_price_preserved(v1_record: dict, transform) -> bool:
    '''Contract: the migrated value, converted back to cents and rounded,
    must equal the original v1 value.'''
    v2 = transform(v1_record)
    reconstructed_cents = round(v2["price_dollars"] * 100)
    return reconstructed_cents == v1_record["price_cents"]

A plausible but wrong migration, included specifically to show what the contract test is catching, forgets the unit conversion:

def BUGGY_transform_v1_to_v2(record_v1: dict) -> dict:
    return {
        "user_id": record_v1["user_id"],
        "price_dollars": float(record_v1["price_cents"]),   # missing the /100
    }

On one concrete record:

sample = {"user_id": "u1", "price_cents": 2599}
print(contract_test_price_preserved(sample, transform_v1_to_v2))         # True
print(contract_test_price_preserved(sample, BUGGY_transform_v1_to_v2))   # False
print(transform_v1_to_v2(sample))         # {'user_id': 'u1', 'price_dollars': 25.99}
print(BUGGY_transform_v1_to_v2(sample))   # {'user_id': 'u1', 'price_dollars': 2599.0}

The buggy output is off by a factor of 100, silently, no exception, no type error, exactly the class of "meaning changed, shape didn't" bug the contract test exists to catch.

The property-based version of the same contract, checking the invariant for every valid price_cents value, not just the one example above, using Hypothesis:

from hypothesis import given, strategies as st

@given(st.integers(min_value=0, max_value=10_000_000))
def check_correct_transform_always_round_trips(price_cents):
    record = {"user_id": "u", "price_cents": price_cents}
    assert contract_test_price_preserved(record, transform_v1_to_v2), \
        f"round-trip failed for price_cents={price_cents}"

Running both the correct and buggy transforms through this property test:

correct transform: round-trips for all generated integer cent values (no counterexample found)
buggy transform: correctly REJECTED by the contract test on first counterexample: round-trip failed for price_cents=1

The correct transform survives Hypothesis's generated inputs with no counterexample; the buggy one fails immediately, on the smallest possible input Hypothesis could find (price_cents=1), which is Hypothesis's shrinking behavior working as intended, it always reports the minimal failing case rather than whatever random large value it happened to try first.

Trade-offs and pitfalls

• A round-trip invariant is not available for every migration. The price example above has a natural inverse (dollars back to cents); a migration that genuinely discards information (rounding a precise timestamp down to a date, say) has no exact round-trip to check. For those, the contract test needs a different invariant. an approximate-equality bound, or a check that the discarded information was intentionally out of scope, stated and reviewed explicitly, not silently assumed safe.
• Testing the transform function in isolation does not prove training and serving agree during the transition window. Both pipelines need to apply the identical transform, ideally the same shared code path, not two independently written implementations that happen to agree on your test fixtures; a genuinely shared library function closes this gap, two parallel reimplementations do not.
• Versioning the schema is only useful if every consumer actually checks the version rather than assuming it. A consumer that silently accepts whatever shape arrives, without verifying the version field, reintroduces the exact risk the whole scheme exists to prevent, the first time an unexpected version shows up.
• The transition window itself is a real operational cost, not a formality. Serving both schema versions simultaneously means running (and monitoring, and eventually decommissioning) two code paths at once; keep the window as short as the migration's risk tolerance allows, and have an explicit, dated plan for removing the old path, or "temporary" dual-serving quietly becomes permanent.