Debugging and Testing ML Systems Questions

Direct answer. Overfitting and leakage can produce the exact same symptom (a gap between training and validation performance), but they need genuinely different fixes, so the investigation has to isolate WHICH one is happening before touching regularization or feature engineering. The discriminator is not the direction the gap moves, it is how much the ABSOLUTE validation number moves when you change how the split is built, and you have to read the training score and the validation score separately to see it.

Experiment and validation plan.

1. Rebuild the evaluation as a strict time-based split, with the held-out slice's features recomputed as of prediction time, retrain, and record training score and validation score SEPARATELY, before and after. Overfitting is a property of model capacity against data volume, so it is essentially insensitive to how the split was drawn: both numbers land where they landed before. Leakage is a property of how the evaluation slice was constructed relative to the label, so it is extremely sensitive: the training score barely moves (the leak is still in the training rows) while the validation score falls off a cliff, which means the gap GROWS rather than shrinks. Note the trap this rules out: under the original random split a leaky model often shows a suspiciously SMALL gap, because the leak inflates validation just as much as it inflates training.
2. Remove suspect features one at a time and retrain, inside the corrected evaluation from step 1. If validation performance actually IMPROVES when a specific feature is removed (not just training performance getting worse, which is expected when you remove any useful feature), that's a strong leakage signal. The mechanism, which is worth being able to state rather than memorize: in the corrected setup the leaky feature still carries the label in the training rows but carries nothing in the held-out slice, because a point-in-time read of it returns a value computed before the outcome existed. The model therefore spends real capacity learning to lean on a signal that is simply not present at evaluation time, and its predictions are actively worse than those of a model that never saw the feature at all. Dropping it does not just stop helping, it stops hurting.
3. Check feature availability timestamps for every feature the model with high suspicion currently uses, documenting exactly when in the real pipeline each one becomes known, relative to the label's outcome time.

How steps 1 and 2 fit together rather than contradicting each other. They assume different evaluation setups, and that is the point of doing them in order. Step 1 operates on the ORIGINAL random split, where the leak is present in both slices, which is exactly why validation looks so good there and why the gap is small. Step 2 operates INSIDE the corrected setup step 1 produces, where the leak is present in training but absent from the held-out slice. Same feature, two different evaluations, two different tells. Run step 1 first to build the honest evaluation, then run step 2 inside it.

What each result implies, with numbers you can reproduce. The setup: 4,000 rows, 10 standard-normal features with the label drawn from a logistic model of them, plus one extra feature built from an aggregate that is only complete once the outcome has settled (so for settled rows it carries the label plus noise, and a point-in-time read of an unsettled row returns noise alone). Model is a gradient-boosted classifier, 80/20 split, seed pinned.

import numpy as np
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import roc_auc_score

rng = np.random.default_rng(0)                    # pinned, so these numbers are reproducible
n, n_tr = 4000, 3200
X = rng.normal(size=(n, 10))                      # 10 honest standard-normal features
beta = rng.normal(size=10)
y = (rng.random(n) < 1 / (1 + np.exp(-(X @ beta)))).astype(int)   # label from a logistic model

# The leaky feature: an aggregate that is only complete once the outcome has settled.
# Settled rows carry the label plus noise; a point-in-time read of an unsettled row is noise alone.
settled = 2.0 * (2 * y - 1) + rng.normal(0, 0.5, n)
as_of   = rng.normal(0, 0.5, n)
X_leak    = np.column_stack([X, settled])         # leak as the training pipeline sees it
X_pointin = np.column_stack([X, as_of])           # leak as production would actually read it

def aucs(Xtr, ytr, Xte, yte, high_capacity=False):
    kw = dict(random_state=0)
    if high_capacity:
        kw.update(max_iter=500, max_leaf_nodes=63, min_samples_leaf=1,
                  l2_regularization=0.0, early_stopping=False, learning_rate=0.3)
    m = HistGradientBoostingClassifier(**kw).fit(Xtr, ytr)
    return (roc_auc_score(ytr, m.predict_proba(Xtr)[:, 1]),
            roc_auc_score(yte, m.predict_proba(Xte)[:, 1]))

perm = rng.permutation(n)
r_tr, r_te = perm[:n_tr], perm[n_tr:]             # random 80/20
t_tr, t_te = np.arange(n_tr), np.arange(n_tr, n)  # chronological 80/20

def show(label, a, b):
    print("  %-48s train %.3f   val %.3f   gap %.3f" % (label, a, b, a - b))

print("LEAKAGE")
show("random split, leak present in both slices",
     *aucs(X_leak[r_tr], y[r_tr], X_leak[r_te], y[r_te]))
show("time split, held-out slice rebuilt point-in-time",
     *aucs(X_leak[t_tr], y[t_tr], X_pointin[t_te], y[t_te]))
show("same time split, leaky feature DROPPED",
     *aucs(X[t_tr], y[t_tr], X[t_te], y[t_te]))

print("\nOVERFITTING (no leaky feature, high-capacity model)")
show("random split", *aucs(X[r_tr], y[r_tr], X[r_te], y[r_te], high_capacity=True))
show("chronological split", *aucs(X[t_tr], y[t_tr], X[t_te], y[t_te], high_capacity=True))

LEAKAGE
  random split, leak present in both slices        train 1.000   val 1.000   gap 0.000
  time split, held-out slice rebuilt point-in-time train 1.000   val 0.569   gap 0.431
  same time split, leaky feature DROPPED           train 0.999   val 0.872   gap 0.127

OVERFITTING (no leaky feature, high-capacity model)
  random split                                     train 1.000   val 0.884   gap 0.116
  chronological split                              train 1.000   val 0.865   gap 0.135

Read the leakage block left to right: validation moved 0.431 (1.000 down to 0.569) purely from changing how the split was built, while training moved 0.000. Note also the first row, where the gap is 0.000: under the original random split this leaky model looks flawless and shows no gap at all, which is the trap. Read the overfitting block: validation moved 0.019 across the same change (0.884 down to 0.865), which is inside the sampling noise of an 800-row holdout, and the gap barely budged (0.116 to 0.135). That contrast, four tenths of an AUC point versus two hundredths, is the whole diagnosis. Then read the third leakage row: dropping the leaky feature RAISED validation from 0.569 to 0.872, which is step 2's tell in its clearest form and something no honest feature ever does.

So: if the validation score collapses when the split is rebuilt correctly, and a specific feature's removal then raises it, that's leakage, and the fix is removing or correctly re-deriving that feature, not more regularization. If both scores are roughly where they were under either split methodology, and removing individual features only ever makes validation worse (never better), that's genuine overfitting, and the fix is standard: more regularization, more data, or a smaller model.

If training loss is near zero AND validation accuracy is ALSO unstable across runs (not just low). This is a distinct signal from a persistently low-but-stable validation number, instability run-to-run suggests either high variance in a small validation set (the validation set itself is too small to give a stable estimate, an easy thing to check by just re-splitting validation data multiple times and looking at the spread of the resulting accuracy) or genuine sensitivity to initialization/data order interacting with a model that has memorized rather than generalized (near-zero training loss with essentially no learned structure to fall back on when the exact memorized examples aren't being tested). Both point toward the same practical next step, get a larger or more stable validation set before drawing any further conclusion, since an unstable metric can't reliably distinguish between competing hypotheses in the first place. It also invalidates the step-1 comparison above until it is fixed: a 0.02 move in validation means nothing if re-splitting alone moves it by 0.05.

Direct answer

Group (aggregate) the logs by (model_version, user_cohort), compare every non-baseline version against the baseline version's traffic within the same cohort, and use two different statistical tests for the two different kinds of signal: the Mann-Whitney U test for latency, because it is continuous and typically right-skewed, and a two-proportion z-test for error rate, because error_flag is binary and a proportion test is the direct, standard tool for comparing two proportions. Never compare across cohorts, only within one, so a cohort that is naturally slower or more error-prone for unrelated reasons (a farther region, a different device mix) can never masquerade as a model-caused regression. The tool emits one short, specific alert per detected regression rather than a wall of numbers, prioritizing that an on-call engineer can act on the output without first reverse-engineering it.

Structured elaboration

Why these two tests, and not others. Latency distributions are almost always right-skewed, a small number of very slow requests pull the mean upward without representing the typical experience, so a t-test on raw milliseconds (which assumes roughly normal, symmetric data) can be misled by a handful of outliers. The Mann-Whitney U test compares the two samples by rank rather than by mean, which makes it robust to skew and to a few extreme values, at the cost of only being able to say "one distribution tends to produce larger values than the other," not "by exactly how many milliseconds," which is an acceptable trade for a detection tool. error_flag is binary, so the quantity being compared is a proportion (the error rate), and the two-proportion z-test is the standard test for whether two independently observed proportions differ, using a normal approximation that is accurate once each group has at least a few dozen observations, comfortably satisfied by hundreds of requests per group.

Grouping and aggregation discipline. Every comparison happens strictly within one user_cohort, aggregating each (model_version, cohort) group's requests into its own latency and error-rate summary before either test runs, comparing a candidate model_version's traffic in that cohort against the baseline version's traffic in that same cohort, never against a different cohort's baseline. This is what keeps a naturally different-behaving cohort from producing a false alert purely because it differs from some other cohort's baseline, not because the model actually regressed for it.

Alert construction, and why it prioritizes explainability. Each alert names the exact model_version and user_cohort it concerns, states the direction and magnitude of whichever signal triggered it (median latency compared to baseline median, or error rate compared to baseline rate), reports the test and its p-value so the reader can judge confidence rather than trust a bare "regression detected," and reports the sample sizes on both sides so a reader can sanity-check whether the alert rests on a handful of requests or a solid sample. This is deliberately more verbose than emitting a single risk score, because the goal stated in the prompt, prioritizing correctness and explainability over raw performance, means an engineer reading the alert should not need to re-run the analysis themselves to understand what it is claiming.

Complexity. For N total log rows spanning V model versions and C cohorts, grouping is a single linear pass, O(N). Each of the up to (V-1) * C non-baseline (version, cohort) comparisons runs a Mann-Whitney U test, which is O(k log k) in the size k of the two groups being compared (dominated by the ranking step), and a two-proportion z-test, which is O(k). Total work is dominated by the sum of the per-group comparison costs, effectively O(N log k) for realistic group sizes, well within what an offline batch tool needs.

Edge cases. A (model_version, cohort) pair with zero baseline rows to compare against is skipped rather than raising an error, since a brand-new cohort just beginning to see a version has nothing to compare it to yet. A group with very few requests can still trigger a nominal p-value below the threshold by chance; the alert always reports the sample sizes precisely so an engineer can discount a low-volume alert appropriately rather than treating every alert as equally trustworthy. A tie-heavy latency distribution (many identical rounded millisecond values) is handled correctly by the Mann-Whitney implementation's tie correction rather than assuming continuous, tie-free data.

Worked example

Synthetic logs: 3,600 requests across 3 model versions (v1 baseline, v2, v3) and 3 cohorts (us_west, us_east, eu), 400 requests per (version, cohort) pair. One regression is deliberately injected at (v3, eu): latency drawn from a gamma distribution with a larger scale parameter than every other group (40 ms versus 30 ms), and error rate raised from a 2% baseline to 9%. Every other non-baseline group is drawn from the exact same distribution as the baseline, so a correct detector should flag (v3, eu) alone and nothing else. The generator is shipped with its seed pinned, because a detector demonstrated on inputs the reader cannot regenerate is a claim rather than a demonstration:

"""Generates the synthetic logs.jsonl used by the worked example. Seed pinned."""
import json
import itertools
import numpy as np

SEED, PER_GROUP = 90, 400
VERSIONS, COHORTS = ["v1", "v2", "v3"], ["us_west", "us_east", "eu"]
SHAPE, BASE_SCALE, REGRESSED_SCALE = 4, 30, 40   # gamma latency, ms
BASE_ERROR_RATE, REGRESSED_ERROR_RATE = 0.02, 0.09
REGRESSION = ("v3", "eu")                        # the one injected defect

rng = np.random.default_rng(SEED)
with open("logs.jsonl", "w") as f:
    rid = 0
    for version, cohort in itertools.product(VERSIONS, COHORTS):
        bad = (version, cohort) == REGRESSION
        latencies = rng.gamma(SHAPE, REGRESSED_SCALE if bad else BASE_SCALE, PER_GROUP)
        errors = rng.random(PER_GROUP) < (REGRESSED_ERROR_RATE if bad else BASE_ERROR_RATE)
        for lat, err in zip(latencies, errors):
            rid += 1
            f.write(json.dumps({
                "request_id": f"r{rid:05d}",
                "timestamp": f"2026-07-20T00:00:{rid % 60:02d}Z",
                "model_version": version,
                "latency_ms": round(float(lat), 1),
                "error_flag": int(err),
                "user_cohort": cohort,
            }) + "\n")
print(f"wrote {rid} rows to logs.jsonl (seed {SEED})")

"""
Offline log-analysis tool: detects statistically significant regressions in latency or error
rate caused by a new model version, from JSON-lines request logs.

Input record shape (one per line):
  {request_id, timestamp, model_version, latency_ms, error_flag, user_cohort}
"""
import json
import sys
from collections import defaultdict
from scipy import stats
import numpy as np


def load_logs(path):
    rows = []
    with open(path) as f:
        for line in f:
            line = line.strip()
            if line:
                rows.append(json.loads(line))
    return rows


def group_by_version_cohort(rows):
    groups = defaultdict(list)
    for r in rows:
        groups[(r["model_version"], r["user_cohort"])].append(r)
    return groups


def two_proportion_z_test(successes_a, n_a, successes_b, n_b):
    p_a = successes_a / n_a
    p_b = successes_b / n_b
    p_pool = (successes_a + successes_b) / (n_a + n_b)
    se = np.sqrt(p_pool * (1 - p_pool) * (1 / n_a + 1 / n_b))
    if se == 0:
        return 0.0, 1.0, p_a, p_b
    z = (p_b - p_a) / se
    p_val = 2 * (1 - stats.norm.cdf(abs(z)))
    return z, p_val, p_a, p_b


def detect_regressions(rows, baseline_version, alpha=0.01):
    groups = group_by_version_cohort(rows)
    cohorts = sorted({cohort for (_, cohort) in groups})
    versions = sorted({v for (v, _) in groups})
    alerts = []
    for cohort in cohorts:
        baseline_rows = groups.get((baseline_version, cohort))
        if not baseline_rows:
            continue
        baseline_latency = np.array([r["latency_ms"] for r in baseline_rows])
        baseline_errors = sum(r["error_flag"] for r in baseline_rows)
        for version in versions:
            if version == baseline_version:
                continue
            candidate_rows = groups.get((version, cohort))
            if not candidate_rows:
                continue
            candidate_latency = np.array([r["latency_ms"] for r in candidate_rows])
            candidate_errors = sum(r["error_flag"] for r in candidate_rows)

            u_stat, lat_p = stats.mannwhitneyu(candidate_latency, baseline_latency,
                                                alternative="greater")
            z_stat, err_p, base_rate, cand_rate = two_proportion_z_test(
                baseline_errors, len(baseline_rows), candidate_errors, len(candidate_rows))

            lat_regressed = lat_p < alpha
            err_regressed = err_p < alpha and cand_rate > base_rate
            if lat_regressed or err_regressed:
                parts = []
                if lat_regressed:
                    parts.append(
                        f"latency higher (median {np.median(candidate_latency):.0f} ms vs "
                        f"baseline {np.median(baseline_latency):.0f} ms, Mann-Whitney U p={lat_p:.2e})")
                if err_regressed:
                    parts.append(
                        f"error rate higher ({cand_rate:.1%} vs baseline {base_rate:.1%}, "
                        f"two-proportion z-test p={err_p:.2e})")
                alerts.append(
                    f"ALERT model_version={version} cohort={cohort}: " + "; ".join(parts) +
                    f". n_candidate={len(candidate_rows)}, n_baseline={len(baseline_rows)}."
                )
    return alerts


if __name__ == "__main__":
    path = sys.argv[1] if len(sys.argv) > 1 else "logs.jsonl"
    rows = load_logs(path)
    print(f"loaded {len(rows)} log rows from {path}")
    alerts = detect_regressions(rows, baseline_version="v1", alpha=0.01)
    print(f"\n{len(alerts)} regression(s) detected:\n")
    for a in alerts:
        print(a)
    if not alerts:
        print("(none)")

Actual output from generating the logs and running the tool over them (python3 gen_logs.py && python3 log_regression_detector.py logs.jsonl):

wrote 3600 rows to logs.jsonl (seed 90)
loaded 3600 log rows from logs.jsonl

1 regression(s) detected:

ALERT model_version=v3 cohort=eu: latency higher (median 142 ms vs baseline 114 ms, Mann-Whitney U p=8.94e-10); error rate higher (7.0% vs baseline 2.2%, two-proportion z-test p=1.38e-03). n_candidate=400, n_baseline=400.

Exactly one alert, exactly matching the one injected regression, (v3, eu). The other five non-baseline (version, cohort) combinations, all drawn from the same distribution as their baseline, produced no false alerts, which is the concrete evidence that the tool discriminates a real regression from ordinary sampling noise rather than flagging everything or nothing regardless of input.

Trade-offs and pitfalls

• A tool that only reports whether a regression exists, without the median or rate values and the sample sizes, forces the reader to re-derive the evidence themselves. The explainability requirement in the prompt is not decorative; an alert without those numbers is materially less useful even if the detection logic behind it is identical.
• Choosing alpha=0.01 rather than the more common 0.05 is a deliberate choice for an automated alerting tool, not a default: with many (version, cohort) combinations checked at once, a looser threshold produces more false alerts purely from running many tests, and a false alert that pages someone at 3 a.m. is more costly here than a slightly slower true detection.
• The common wrong turn is choosing one test for both signals, for example running a t-test on latency because it is simpler to reason about; that trades correctness (a t-test's normality assumption is a poor fit for skewed latency data) for convenience, exactly backwards from the stated priority of correctness and explainability over raw performance.
• Comparing across cohorts instead of within one is the second common wrong turn, and it is the one this design most directly guards against by construction: grouping every comparison to share a cohort on both sides is what prevents a cohort's ordinary baseline differences from ever being mistaken for a model regression.

Direct answer

Label noise investigation breaks cleanly into three sequential questions, and each one uses a different technique: how much noise is there overall (a measurement problem), which specific examples are most likely wrong (a ranking problem, so a limited review budget goes to the highest-value examples first), and, given that budget, whether the right response is relabeling, filtering, or changing the loss function entirely. Trying to answer all three with one tool is the usual mistake; a global noise-rate estimate and a per-example priority ranking are related but different outputs, and conflating them either wastes review budget on low-value examples or understates how bad the problem really is.

Structured elaboration

(a) Quantifying the overall rate of label noise. Train with k-fold cross-validation and record each example's prediction from the fold where it was held out (its "out-of-fold" prediction, meaning the model never saw that example during the training that produced this specific prediction, so the prediction is not contaminated by having memorized a possibly-wrong label). Compute the disagreement rate between the given label and the out-of-fold predicted label as a first estimate. This raw disagreement rate is a biased estimate on its own, since it conflates genuine label errors with ordinary model error (the model can just be wrong about a CORRECTLY labeled example), so calibrate it against a small, human-audited gold sample: have annotators re-review a random sample of, say, a few hundred examples, and use the true error rate on that gold sample to correct the full-dataset estimate. Tooling: any standard k-fold cross-validation setup, plus a lightweight annotation tool or spreadsheet for the gold-sample audit; no specialized infrastructure is required for this step.

(b) Identifying which specific examples are likely mislabeled. This is a ranking problem, not a classification problem: you want a score per example that a limited review budget can be sorted by. Three quantitative signals feed this ranking, in increasing order of cost: per-example training LOSS (an example the model fits poorly relative to similar examples is a cheap, single-model first pass, but loss alone conflates hard-but-correctly-labeled examples with mislabeled ones); per-example predicted CONFIDENCE on the given label (a low confidence on the currently-assigned class is a second cheap signal); and cross-model DISAGREEMENT from an ensemble of models with some diversity (different random seeds, different architectures, or different training folds), which is more expensive to produce but far more specific. Combine confidence and disagreement into a single score. State the rule before writing it down, because the shape of it is the actual idea: score each example as its average model confidence, multiplied by (1 minus the ensemble's internal disagreement), gated to zero whenever the ensemble's consensus class matches the label already on the example. Disagreement here is the spread of the K models' votes measured as a normalized entropy: 0 when all K models vote the same class, rising toward 1 as the votes spread evenly across classes. The (1 minus disagreement) factor is what stops the score rewarding confusion: an ensemble that is internally split is showing you an ambiguous or genuinely hard example, which is not the same finding as a mislabeled one, so a split vote is discounted even when the individual models are confident. Concretely, so the number can be traced: with K = 5 models over C = 3 classes, a 4-to-1 vote split gives vote proportions of 0.8 and 0.2, and the normalized entropy is $-(0.8\ln 0.8 + 0.2\ln 0.2)/\ln 3 = 0.500/1.099 = 0.455$, while a unanimous 5-0 vote gives exactly 0. The normalizer is the log of the number of distinct outcomes the votes could spread across, which is K when there are fewer models than classes and C otherwise, so the measure stays on a 0-to-1 scale in both regimes. So the examples most likely to be labeling errors are the ones where the ensemble confidently and consistently predicts a DIFFERENT class than the one the dataset currently assigns; confident agreement that matches the given label is not interesting (nothing to flag), and confident disagreement among the models themselves is more often a genuinely hard or ambiguous example than a labeling error specifically. At large scale (millions of examples), running a full multi-model ensemble over every single example is often too expensive to justify; a practical two-stage pattern is to use the cheap per-example loss or confidence signal as a first-pass filter to narrow the full dataset down to a much smaller candidate set, and reserve the more expensive ensemble-disagreement scoring for ranking WITHIN that narrowed set, before it ever reaches a human reviewer. Tooling: the loss and confidence signals come for free from any existing training run; the ensemble requires whatever infrastructure already exists to produce K model variants, and the scoring logic itself is lightweight, demonstrated below.

(c) Deciding, given a limited human-review budget, whether to relabel, filter, or switch to a noise-robust loss. This decision depends on the estimated noise RATE from step (a), the estimated noise PATTERN (random versus systematic, discovered by checking whether the ranked candidates from step (b) cluster in specific classes, sources, or annotators rather than spreading evenly), and the review budget itself relative to dataset size.

• Relabel: the right choice when noise is concentrated (a specific class, source, or annotator is responsible for most of it) and the review budget can plausibly cover the ranked candidate list from step (b) for that concentration. Concentrated noise with a bounded high-value candidate set is exactly what makes relabeling worth the human cost.
• Filter (remove flagged examples rather than relabeling them): appropriate when the noise rate is low enough that removing suspected-bad examples does not meaningfully shrink the useful training set, and when getting a correct replacement label is expensive, slow, or infeasible (for example, the ground truth requires a specialist annotator or a delayed real-world outcome that has not happened yet). Filtering is cheaper than relabeling in review time but destroys information, so it trades data efficiency for speed.
• Switch to a noise-robust loss function (a loss designed to be less sensitive to a fraction of incorrectly-labeled examples, such as a loss that down-weights high-loss, likely-mislabeled examples during training rather than requiring them to be identified and handled up front). The named options worth knowing are generalized cross-entropy, which interpolates between ordinary cross-entropy and mean absolute error through a single exponent parameter and so keeps cross-entropy's fast early learning while bounding how much any one badly-fit example can contribute; symmetric cross-entropy, which adds a reverse term measuring the label's divergence from the prediction, deliberately bounded so a wrong label cannot dominate the gradient; and a plain MAE-style loss, which is provably robust to symmetric label noise but trains slowly enough that it is rarely the practical choice on its own. This branch is: appropriate when the estimated noise rate is high, diffuse (spread broadly rather than concentrated, so no small ranked candidate list would cover most of it), or the review budget is simply too small relative to the dataset to make either relabeling or filtering practical at meaningful scale. This is the right default when steps (a) and (b) together tell you that manual review, however prioritized, cannot realistically move the needle.

Turning step (a)'s number into step (c)'s decision, with the metrics named. The three inputs are the calibrated noise rate from step (a), the concentration of step (b)'s ranked candidates (what share of the top-ranked candidates fall into a single class, data source, or annotator), and the review budget as a fraction of the flagged candidate set. Rough bands to argue from rather than thresholds to obey, since the right cut depends on dataset size and on how expensive a wrong label is downstream:

• Estimated noise rate under roughly 5 percent, and over half the top candidates concentrated in one class, source, or annotator: relabel that concentration. The candidate set is small and the fix is targeted.
• Estimated noise rate under roughly 5 percent but spread evenly across classes and sources, with labels expensive or slow to obtain: filter the flagged examples. You lose a few percent of the training set, which a large dataset absorbs.
• Estimated noise rate above roughly 15 to 20 percent, or a review budget covering less than about a tenth of the flagged candidates: a noise-robust loss is the primary response, with whatever review the budget allows spent on the top of the ranked list rather than spread thin.
  Between those bands, run the cheap experiment instead of arguing: train on the filtered set and on the full set with a noise-robust loss, and compare both against the current baseline on a small, human-verified clean evaluation set, which is the metric that actually decides this and is worth building before any of the three interventions.

In practice these are not mutually exclusive: a common pattern is to filter or relabel the highest-confidence, highest-value flagged examples within budget, and pair that with a moderately noise-robust loss for the remaining, unreviewed noise the budget could not reach.

Worked example

The script below implements the ranking logic from step (b): given each example's ensemble of per-model class-probability predictions and its currently-assigned label, it computes a combined disagreement-and-confidence score and ranks examples for review. One example is deliberately constructed as a "planted" confident, unanimous flip (all five models agree on a class different from the given label) to verify the score correctly surfaces it at the top of the review queue.

"""
Ranks labeled examples by a combined ensemble-disagreement + confidence
score, to prioritize a limited human-review budget toward examples most
likely to be mislabeled. In practice each row of `ensemble_probs` comes
from K independently-trained models scoring the same example; here it is
synthesized directly so the ranking logic is fully pinned and reproducible.

Pinned: numpy Generator seed=11, K=5 models, C=3 classes, 6 example rows
(one deliberately "planted" as a confident, unanimous flip vs its given
label to verify the score correctly surfaces it top-of-list).
Run with: python3 disagreement_score.py
"""
import numpy as np

rng = np.random.default_rng(seed=11)
K, C = 5, 3


def make_confident_row(true_class, agreement=1.0):
    """K models' softmax rows, `agreement` fraction voting `true_class`."""
    rows = []
    n_agree = round(K * agreement)
    for k in range(K):
        base = rng.dirichlet(alpha=[0.5] * C)
        cls = true_class if k < n_agree else int(rng.integers(0, C))
        base = np.clip(base, 0, None)
        base[cls] += 2.0  # push mass onto the chosen class
        base /= base.sum()
        rows.append(base)
    return np.stack(rows)


# 6 examples: given_label is what the dataset currently says
examples = [
    {"given_label": 0, "ensemble": make_confident_row(true_class=0, agreement=1.0)},  # models agree WITH label
    {"given_label": 1, "ensemble": make_confident_row(true_class=2, agreement=1.0)},  # PLANTED FLIP: unanimous, confident, disagrees with label
    {"given_label": 2, "ensemble": make_confident_row(true_class=2, agreement=0.6)},
    {"given_label": 0, "ensemble": make_confident_row(true_class=1, agreement=0.4)},  # split vote, moderate confidence
    {"given_label": 1, "ensemble": make_confident_row(true_class=1, agreement=1.0)},
    {"given_label": 2, "ensemble": make_confident_row(true_class=0, agreement=0.8)},
]


def score_example(given_label, ensemble_probs):
    preds = ensemble_probs.argmax(axis=1)
    confs = ensemble_probs.max(axis=1)
    vote_counts = np.bincount(preds, minlength=C) / K
    nonzero = vote_counts[vote_counts > 0]
    disagreement = max(0.0, -(nonzero * np.log(nonzero)).sum() / np.log(K if K < C else C))
    consensus_class = np.bincount(preds, minlength=C).argmax()
    agrees_with_label = consensus_class == given_label
    avg_conf = confs.mean()
    # Score rewards confident consensus that CONTRADICTS the given label.
    score = avg_conf * (1.0 - disagreement) * (0.0 if agrees_with_label else 1.0)
    return score, consensus_class, avg_conf, disagreement


results = []
for i, ex in enumerate(examples):
    score, consensus, avg_conf, disagreement = score_example(ex["given_label"], ex["ensemble"])
    results.append((i, ex["given_label"], consensus, avg_conf, disagreement, score))

results.sort(key=lambda r: r[-1], reverse=True)

print("=== label-noise review queue, ranked by score (desc) ===")
print(f"{'idx':>3} {'given':>5} {'consensus':>9} {'avg_conf':>8} {'disagree':>8} {'score':>7}")
for idx, given, consensus, avg_conf, disagreement, score in results:
    print(f"{idx:>3} {given:>5} {consensus:>9} {avg_conf:>8.3f} {disagreement:>8.3f} {score:>7.3f}")

Actual output from running this script:

=== label-noise review queue, ranked by score (desc) ===
idx given consensus avg_conf disagree   score
  1     1         2    0.842    0.000   0.842
  5     2         0    0.809    0.455   0.440
  3     0         1    0.702    0.455   0.382
  0     0         0    0.846    0.000   0.000
  2     2         2    0.707    0.000   0.000
  4     1         1    0.761    0.000   0.000

The planted flip, example index 1 (given label 1, but all five models unanimously and confidently vote class 2), correctly lands at the top of the queue with the highest score (0.842), exactly as intended. Example 5, where the models were split (80 percent agreement) but still confidently landed on class 0 against a given label of 2, ranks second. Examples 0, 2, and 4, where the ensemble's consensus class MATCHES the given label, correctly score zero regardless of the ensemble's confidence or internal disagreement, since a model bank that agrees with the current label, confidently or not, is not evidence of a labeling error at all; this is what "score rewards confident consensus that contradicts the given label" is doing mechanically, and the zero scores for those three rows confirm the score is not accidentally flagging ordinary model uncertainty as a labeling problem.

Trade-offs and pitfalls

The most common wrong turn is treating the ranking score from step (b) as itself an estimate of the OVERALL noise rate from step (a). A ranked list tells you relative priority among examples; it says nothing about what fraction of the full dataset is actually mislabeled without the separate, calibrated cross-validation-versus-gold-sample estimate, and reporting "we found 40 likely-mislabeled examples in review" as if that quantifies the dataset's noise rate, without knowing what fraction of the full dataset those 40 represent or how the review budget was allocated, overstates what was actually measured.

A second pitfall is using an under-diverse ensemble for the disagreement signal, for example five models trained with the same architecture, same random seed family, and same data split. If the ensemble members are correlated enough to make similar mistakes for similar reasons, the disagreement signal collapses even on genuinely ambiguous or mislabeled examples, understating how much review is actually warranted. Diversity in seeds, architecture, or training data subsets matters for this technique to work as intended.

A third pitfall in the decision step (c): choosing a noise-robust loss function as a substitute for ever looking at the data, rather than as a complement to a budget-limited review. A noise-robust loss reduces the HARM of remaining noise during training; it does not tell you anything about the underlying data quality, so systematic noise sources (a specific mislabeling process, a broken annotation guideline, a bad data source) can persist indefinitely and keep degrading every future model trained on the same data, even as the current model becomes more robust to their symptoms.

Direct answer. A test suite for an ML codebase needs four genuinely different kinds of tests, because each catches a different class of regression that the others miss: unit tests for the deterministic pieces of code, integration tests for whether components still work together, dataset tests for whether the data itself is still sane, and model-behavior tests for whether the model's actual outputs still make sense. Running all of them on every commit would make the pipeline too slow to be usable, so they're split across PR-time, nightly, and pre-deploy based on cost and how often each one is expected to catch something real.

The four categories, with one concrete example and a runtime budget each.

When each runs, and why. Unit and a fast subset of integration tests run at pull-request time, they're cheap enough that a slow PR check would actively discourage people from running tests at all, and they catch the majority of code-level bugs before they ever reach a shared branch. Dataset tests and the full model-behavior suite run nightly against the latest production-like data snapshot, since they depend on data that changes daily and can be too slow or too dependent on external state to gate every PR. Model-behavior tests against the golden dataset also run again specifically at pre-deploy time, as a final gate right before a new model artifact goes to production, because that's the last point where catching a regression is cheap; catching the same regression after deploy means an incident instead of a blocked release.

Two closing practical notes. First, a release-validation checklist belongs at the pre-deploy stage specifically and should include not just model-quality thresholds but also privacy checks and an explicit rollback plan, automated into the deploy pipeline as hard gates rather than a manual sign-off step that people learn to skip under time pressure. Second, keep the nightly and pre-deploy dataset/model-behavior tests deterministic and reasonably fast even though they're not PR-blocking: a nightly suite that takes six hours to fail stops being useful days before anyone notices, because the feedback loop is too slow to actually change behavior.

Direct answer. The function needs to check each record against the schema independently, collect every violation (not just fail on the first one), and return both the clean records and enough detail on each rejected record to debug it without re-running anything.

Approach. Iterate records; for each field in the schema, check presence (if required), then type, then any numeric range constraint, then timestamp format if applicable. Accumulate all violations for a record before deciding whether it's valid, since a record can fail multiple checks and you want to see all of them in one pass rather than fixing one bug at a time across repeated runs.

from datetime import datetime

def validate_events(events, schema):
    valid_events, errors = [], []
    for idx, ev in enumerate(events):
        ev_errors = []
        for field, rules in schema.items():
            present = field in ev and ev[field] is not None
            if rules.get('required') and not present:
                ev_errors.append(f"{field}: missing required field")
                continue
            if not present:
                continue
            val = ev[field]
            ftype = rules.get('type')
            if ftype == 'int':
                if not isinstance(val, int) or isinstance(val, bool):
                    ev_errors.append(f"{field}: expected int, got {type(val).__name__}")
            elif ftype == 'float':
                if not isinstance(val, (int, float)) or isinstance(val, bool):
                    ev_errors.append(f"{field}: expected float, got {type(val).__name__}")
                elif 'min' in rules and val < rules['min']:
                    ev_errors.append(f"{field}: {val} below minimum {rules['min']}")
            elif ftype == 'timestamp':
                if rules.get('format') == 'iso8601':
                    if not isinstance(val, str):
                        ev_errors.append(f"{field}: expected ISO8601 string, got {type(val).__name__}")
                    else:
                        try:
                            datetime.fromisoformat(val.replace('Z', '+00:00'))
                        except ValueError:
                            ev_errors.append(f"{field}: '{val}' is not valid ISO8601")
        if ev_errors:
            errors.append({"index": idx, "record": ev, "errors": ev_errors})
        else:
            valid_events.append(ev)
    return valid_events, errors

Run against 5 example records, one clean and one carrying each of the four defects the schema can detect:

SCHEMA = {
    'user_id': {'type': 'int',   'required': True},
    'amount':  {'type': 'float', 'min': 0},
    'ts':      {'type': 'timestamp', 'format': 'iso8601', 'required': True},
}

EVENTS = [
    {'user_id': 1,     'amount': 9.99,  'ts': '2026-07-01T10:00:00Z'},   # 0: clean
    {'user_id': 'abc', 'amount': 4.50,  'ts': '2026-07-01T10:05:00Z'},   # 1: user_id is a str, not an int
    {'user_id': 3,     'amount': 12.00},                                  # 2: required ts absent entirely
    {'user_id': 4,     'amount': -5,    'ts': '2026-07-01T10:15:00Z'},   # 3: amount below the min of 0
    {'user_id': 5,     'amount': 7.25,  'ts': 'not-a-date'},             # 4: ts present but unparseable
]

valid, errs = validate_events(EVENTS, SCHEMA)
print("valid count:", len(valid))
print("error count:", len(errs))
for e in errs:
    print(f" idx {e['index']} -> {e['errors']}")

This executes and produces:

valid count: 1
error count: 4
 idx 1 -> ['user_id: expected int, got str']
 idx 2 -> ['ts: missing required field']
 idx 3 -> ['amount: -5 below minimum 0']
 idx 4 -> ["ts: 'not-a-date' is not valid ISO8601"]

Each index is traceable back to exactly one rule in the schema. Record 2 is the one that exercises the required branch: ts is absent from the dict entirely, so present is False and rules.get('required') is True, which is the only combination that produces a missing required field message. Note that user_id is checked first (schema iteration follows the schema dict's insertion order), so record 2 passes the user_id check before failing on ts; the error names the field that is actually missing, not the first field in the schema.

One subtlety worth calling out explicitly: isinstance(val, int) alone would silently accept True/False as valid integers in Python, since bool is a subclass of int. The check above excludes booleans explicitly; skipping that guard is an easy way to let a boolean value slip through a numeric-type check undetected.

What error information to include for fast debugging. Each error entry above carries the record's index in the batch, the full original record (not just the field that failed, since a downstream engineer debugging a validation failure usually needs to see the whole record for context), and the list of every rule that record violated. Returning only "record 3 is invalid" without this detail turns every validation failure into a second investigation just to find out why; returning the field-level reasons up front means the caller (a human or an automated alert) can act immediately.