Debugging and Testing ML Systems Questions

Direct answer. Identical inputs producing different predictions between requests traces to one of three layers, the model's own computation, the serving runtime, or the surrounding infrastructure, and each has a distinct fix with a distinct performance cost, so the first job is figuring out which layer, not jumping straight to a fix.

Model-level causes. An unseeded random-number generator inside the model itself (dropout not properly disabled at inference, a sampling step in a generative model) will produce different outputs by design; the test here is simple: run the same input through the model twice in the exact same process and see if outputs differ, and read the SIZE of the difference, not just its existence. Model-level randomness (live dropout, an active sampling step) moves the output visibly, often enough to change the argmax. Runtime kernel nondeterminism moves only the last few significant digits and never the decision. So a large difference points at this layer, a last-bits difference points at the runtime layer below, and bit-identical output on a CPU path rules out both. Fix: ensure eval mode is set, and if intentional sampling is part of the design, make the seed for that sampling step an explicit, loggable input rather than left to system entropy, trading a small amount of implementation complexity for reproducibility on demand.

Runtime-level causes. Nondeterministic GPU operations (certain scatter/reduction kernels chosen for performance over determinism) can make bit-identical inputs produce slightly different floating-point results run to run. Fix: use the framework's deterministic-algorithms mode; the trade-off is a measurable performance cost, since the deterministic kernel variants are often slower than the default, non-deterministic ones, worth paying for a use case where reproducibility genuinely matters (compliance, debugging) and not worth paying for one where it doesn't (a recommendation ranking where sub-percent output jitter has no real consequence).

Infrastructure-level causes. An asynchronous feature fetch with different side effects (a race between two upstream feature sources resolving in different order under different load) can make the SAME logical request see different feature values depending on timing, which is a data problem masquerading as a model nondeterminism problem. Fix: make the feature-fetch step for a single request deterministic (all fetched features version-tagged to a single request-time snapshot), at the cost of some added latency or complexity in the feature-fetch path. A hot-patched model version, where different requests are served by different, not-yet-fully-rolled-out model artifacts during a deployment, can look exactly like model-level nondeterminism from the outside; the fix (verifying the artifact version alongside every logged prediction, so this is distinguishable from true nondeterminism after the fact) costs only a small amount of additional logging, not runtime performance.

The performance trade-off, sized by WHERE the cost lands. The useful distinction is not cheap versus expensive, and it is not one-off versus recurring either, since both the infrastructure fix and the runtime fix are paid on every request for the life of the deployment. The distinction that actually predicts the bill is whether the cost is a FIXED adder per request or a cost levied on every unit of compute, because only the second one grows with your model and your batch size:

• Model-level (eval mode, explicit sampling seed): no runtime cost at all, and slightly negative if anything, since disabling dropout at inference removes work rather than adding it. Pure implementation hygiene.
• Artifact-version logging: a few extra bytes per logged prediction and no change to the request path. Effectively free.
• Infrastructure-level (single request-time feature snapshot): a genuine latency add on every request, but a FIXED one that lands once in the fetch path, not on every unit of compute, so it does not grow when the model does. It frequently nets out near zero because collapsing several ad-hoc reads into one snapshot read removes round trips as well as adding one. It is small relative to a model whose own compute is tens of milliseconds, and it is NOT small if the snapshot forces a slower storage path than the reads it replaced, which is the case to actually measure before shipping.
• Runtime-level (deterministic kernels): the only one that charges compute on every single request for the life of the deployment. Its size depends entirely on what fraction of your model's hot path sits in operations whose default kernel is nondeterministic: if none of them do, the flag costs nothing; if a nondeterministic scatter or reduction is in the inner loop, it can be substantial. There is no percentage worth quoting here, because the honest number comes from one micro-benchmark, running the same batch with the flag on and off and taking the ratio, which takes minutes and beats any published figure for your model.

So the earlier statement stands with its qualifier attached: the model-level and logging fixes are genuinely free, the infrastructure fix is a fixed per-request latency add in the fetch path that you should measure but will usually accept, and the runtime fix is the one whose cost scales with the model's own compute, which is why it's worth reaching for deliberately for the specific requests that need it rather than turning it on globally by default.

Direct answer

Turn the vague request into a concrete, falsifiable investigation plan before doing anything else: pin down what "better" actually means, enumerate the specific, testable ways the model could be underperforming rather than guessing at one, and prioritize those hypotheses by how much diagnostic value each one yields per unit of the scarcest resource, mostly labeling budget, rather than by which one merely seems most likely. Under real time and labeling constraints, the checks that need zero new labels come first, since they are effectively free relative to the ones that require fresh ground truth.

Structured elaboration

Step 1: clarify the objective and constraints. Before investigating anything, ask what metric "better" refers to (precision, recall, a ranking metric, calibration), what the relative business cost of a false positive versus a false negative actually is, what time horizon the churn label is defined over, and what the current baseline value is. This step alone often reframes the whole investigation: a stakeholder who actually cares about catching the highest-value accounts, not overall accuracy, needs a different investigation than one who cares about total volume of correctly flagged churners.

Step 2: enumerate candidate failure modes as testable hypotheses, not a single guess. A standard checklist for "why is a model underperforming" covers several genuinely distinct causes, and a structured investigation names them explicitly rather than jumping at the first plausible one: label definition drift (the business definition of churn changed without the training label being updated to match), covariate shift in the model's most important input features, feature leakage or a broken feature that quietly stopped carrying real signal, a calibration-only problem (the model still ranks users correctly by risk, but the decision threshold is stale relative to a shifted score distribution), and segment-specific underperformance (the aggregate metric looks fine or mildly degraded while one segment is badly wrong and gets diluted into the average).

Step 3: prioritize by diagnostic value per unit of scarce resource, not by probability alone. Score each hypothesis as its estimated probability of being a real, fixable contributor divided by its cost to investigate, in labeling-budget units. This matters because a hypothesis that is only moderately likely but essentially free to check (an audit of the label definition and pipeline configuration needs no new labels at all) can be worth investigating before a hypothesis that is more likely but expensive, since the free check either resolves a real issue immediately or is ruled out at negligible cost, freeing the full budget for the remaining hypotheses. This priors-and-cost ranking should update as each step returns evidence, since finding nothing wrong with the cheapest hypothesis is itself informative and should reweight what is investigated next, not just work down a fixed list computed once at the start.

Step 4: work around a limited labeling budget. Order investigation by what each check actually costs in labels, cheapest first. At the top is the check that needs no new labels at all: a label-definition and pipeline-configuration audit inspects the label logic and the pipeline's own configuration rather than model outputs on fresh data, so it costs nothing in ground truth. Next come the checks that run mostly against historical, already-labeled data and need only a small confirmation sample of new labels, a feature-leakage check and a calibration check being the usual examples. Reserve the bulk of the budget for hypotheses that genuinely cannot be tested without substantial fresh ground truth, most commonly confirming or ruling out segment-specific underperformance on recent traffic. Where fresh labels are needed, spend them where they carry the most diagnostic value: targeted sampling of the cases the current model is least confident about or that fall in the suspected weak segment, rather than a uniformly random sample, extracts more information per label spent, while keeping a small separate random holdout specifically to measure overall metrics honestly, since a purely targeted sample would bias any metric computed directly on it.

Worked example

Illustrative priors an investigating engineer might assign to each hypothesis (informed judgment going in, not a measured result) and their estimated investigation cost in labels needed, scored as probability divided by cost, with a cost of 0 meaning the check needs no new labels at all:

With a 300-label total budget, this ordering (free checks first, then descending score among the ones that cost labels) investigates the label-definition audit for free, then the feature-leakage check for 50 labels (cumulative 50), then the miscalibration check for 200 labels (cumulative 250), leaving 50 labels unspent rather than enough to also fully investigate covariate shift or segment-specific underperformance this round. Note that miscalibration has the highest individual prior (0.45) but is not investigated first, because the free label-definition check and the far cheaper leakage check both deliver more diagnostic value per label spent. To see what that ordering choice actually buys, work both plans all the way through the same 300-label budget, skipping any check whose cost exceeds what is left at that point:

Two things fall out of that trace, and the first one is not the intuitive answer. Both orderings end up testing the same three hypotheses for the same 250 labels, so the case for value-per-label ordering is not that it buys more checks; at this budget it does not. What differs is WHEN the information arrives. The value-per-label plan has one hypothesis answered before it has spent a single label, two by label 50, and all three by label 250. The probability-first plan commits 200 labels, two thirds of the whole budget, to the 0.0023-per-label check before it has learned anything at all, and only then reaches the free audit and the 0.0030-per-label leakage check.

That timing is the entire argument, for two reasons. Priors are supposed to be updated as evidence arrives, and an ordering that defers every cheap answer until after the expensive one is already paid for has given up the chance to re-rank on what it learned. And investigations are cut short by time, by an incident, or by a stakeholder wanting an answer this afternoon far more often than they run exactly to the end of a labeling budget: stopped after two hours, the value-per-label plan has two answers in hand and the probability-first plan has one.

The trace also settles one thing worth being precise about, because it is easy to overclaim in the other direction: a zero-cost check is reached under EVERY ordering, since nothing can crowd out something that costs nothing. Scheduling free checks first is not about protecting them from the budget, it is about getting their answers back before the budget has been committed elsewhere.

Trade-offs and pitfalls

• Sorting purely by probability and ignoring cost is the most common way this kind of prioritization goes wrong. It defers cheap, high-value checks behind expensive ones that merely seem more likely, so most of the budget is committed before any of the cheap evidence is in and there is nothing left to re-rank with, exactly what the worked example's side-by-side spend trace shows.
• The priors themselves are judgment calls, not measurements, and should be treated as a starting point to update, not a fixed ranking. Finding nothing in the free label-definition audit is itself evidence that should lower that hypothesis's weight and free attention for the next one, not a wasted step.
• Targeted, uncertainty-focused labeling extracts more diagnostic value per label but introduces its own sampling bias if the same targeted sample is later used to report an overall metric. Keep a small, genuinely random holdout specifically for honest overall-metric reporting even while spending the bulk of the scarce budget on targeted diagnostic sampling.
• Skipping step 1 and diving straight into re-running dashboards is the other common wrong turn. "Make it better" is under-specified enough that investigating the wrong metric, one nobody actually cares about, is a real risk, and it is a risk step 1 removes at essentially no cost.

Direct answer. A prioritized ML CI checklist should be ordered by where a bug is cheapest to catch versus how expensive it is to let it slip through, which generally means data checks first, training-time sanity checks second, pre-deploy checks third, and post-deploy smoke tests last as a final safety net.

The checklist, with the failure each item prevents.

1. Feature/data validation (schema checks, null/range checks on incoming training data): prevents training on silently corrupted or malformed data, the cheapest bug to catch and the most expensive to discover after the fact, since a model trained on bad data has to be retrained, not just patched.
2. Training-time sanity tests (a tiny-batch overfit test, a check that loss actually decreases over the first N steps): prevents shipping a model whose training pipeline is fundamentally broken (a wiring bug, a frozen layer, a disconnected loss), catching it in minutes instead of after a full multi-hour training run completes with a mysteriously bad result.
3. Pre-deploy checks (a regression test against a frozen golden dataset, a minimum-accuracy threshold gate): prevents promoting a model that's measurably worse than what's currently in production, the last automated checkpoint before a real user is affected.
4. Post-deploy smoke tests (a simple end-to-end request against the live endpoint immediately after rollout): prevents a broken DEPLOYMENT (not a broken model, a broken serving path, wrong artifact loaded, a crashed container) from silently serving errors or garbage before anyone notices.

Why this order matters, not just this list. A team that only has post-deploy smoke tests catches problems at the most expensive possible point, after a bad model or bad deploy has already reached users. A team that only has data validation catches data problems but has no safety net if the training code itself has a bug, or if a legitimately-trained model is simply worse than the one it's replacing. The checklist is valuable specifically because each layer catches a DIFFERENT class of failure that the layers before and after it cannot, not because more checks are categorically better; a team with limited time should build layer 1 and 2 first, since they're both cheap and catch the most common real bugs before any compute is wasted on a bad training run.

Direct answer

An intermittent, load-correlated inference anomaly you cannot reproduce locally means the bug depends on something your local environment does not have: concurrency, batching, a specific hardware code path, or a state that only exists after real traffic patterns build up. The fix is to stop trying to reproduce it by guessing and instead build the reproduction pipeline itself: capture enough about each failing request to replay it exactly, capture the full environment so "works on my machine" stops being ambiguous, and then drive controlled load against a faithful replica until the failure shows up on demand.

Structured elaboration

Trace-id and payload logging, privacy-preserving. Propagate a unique trace identifier through every hop (ingress, feature lookup, model call, postprocessing) and log it alongside stage-level timing, host identifier, model version, and batch composition (batch size and position within the batch) at each stage. For the request payload itself, do not log raw content by default: log a fixed-length hash or fingerprint of the input, plus lightweight, non-identifying shape metadata (input length, feature-presence flags). Add a narrow, rate-limited full-payload capture (encrypted at rest, short retention, access-controlled) that only activates for a small sample or for requests matching a suspected failure signature, so you have enough real examples to replay without indiscriminately logging every user's raw input.

Environment and dependency capture. Snapshot everything that could plausibly change numeric or control-flow behavior under load: container image digest, language runtime and library versions (exact pins, not just major versions), driver and accelerator versions if applicable, and CPU or accelerator microarchitecture identifiers, since a fleet is rarely as homogeneous as the deployment config implies. Record this snapshot per-host, not once globally, because an intermittent failure correlated with load is a good candidate for "only reproduces on host type B under contention," which a single global snapshot would hide.

Deterministic seeding. Log every random seed involved in the request path: any sampling in preprocessing or postprocessing, and the framework's own random-number-generator state if inference involves any stochastic operation (dropout left on by mistake, temperature-based sampling). Record whether deterministic execution flags are enabled, since many frameworks silently pick faster, non-deterministic kernels by default under load and only guarantee bit-identical results when determinism is explicitly requested.

Traffic-replay strategy. Use the captured trace-ids and payload fingerprints to select candidate failing requests, then replay them in an isolated staging environment built from the exact environment snapshot for the host where the failure occurred. Replay both in isolation (single request, to rule out anything intrinsic to that one input) and at the observed concurrency and batch-composition pattern (to test whether it is a contention or batching artifact). If you cannot replay the literal captured payloads for privacy reasons, replay statistically-similar synthetic payloads generated to match the captured shape metadata, at the same volume and concurrency.

The feature-store variant, and the one extra thing it needs captured. If the hard-to-reproduce failures correlate with upstream feature-store changes rather than with load, the replay strategy above has a hole in it: features are fetched at serve time, so replaying a captured request re-queries the feature store and receives CURRENT values, quietly reproducing a different request than the one that failed. Close it by logging, per request, the feature-store dataset or artifact version and the point-in-time feature VALUES actually returned (hashed or bucketed where the values themselves are sensitive), not just the raw input payload, so a replay can be pinned to the same feature snapshot the failing request saw. With that captured, the suspected correlation becomes testable rather than suggestive: replay the same captured requests twice, holding the model artifact and environment snapshot fixed, once pinned to the pre-change feature-store version and once to the post-change version. If the failure appears only under the post-change version, the feature-store change is causally implicated; if it appears under both, the feature-store change was concurrent rather than causal and the real trigger is elsewhere.

Controlled load and chaos testing. Once single-request replay does not reproduce the failure, escalate to load generation that matches the production traffic shape (arrival rate distribution, batch size distribution, concurrency) against the replica, ramping toward the levels observed around the incident. Layer in targeted chaos: inject resource pressure (CPU contention, memory pressure, forced accelerator throttling), simulate degraded network conditions between services, and vary batch composition deliberately (mixing very different input shapes in one batch, since padding and batching code paths are a common source of load-only bugs). Vary one axis at a time where practical, so a reproduction tells you which axis actually matters.

Using the reproduced artifact to confirm a fix. Once you have a deterministic, scripted reproduction (a specific replay payload set, load profile, and environment snapshot that reliably triggers the failure), that reproduction becomes your regression test. Before shipping a candidate fix: run the reproduction against the unfixed code to confirm it still fails (rules out an already-stale repro), then against the fixed code to confirm it now passes, at the same load level and several times in a row to rule out you having only shifted the failure's probability rather than eliminated it. Add the reproduction as a permanent load or chaos test in the pre-production suite, so a regression is caught automatically next time rather than requiring another multi-day investigation.

Worked example

Suppose the anomaly is a fraction of inference responses under heavy load returning a prediction that looks like it was computed on the wrong batch position, and it never reproduces with a single request sent locally. Trace-id logs show the affected responses cluster at high concurrency and specifically at larger batch sizes (16 or more), never at batch size 1. Replaying single captured payloads in isolation, as expected, does not reproduce it: the bug is contingent on batching. Replaying the same payloads with load generation configured to force batch sizes of 16 and above, at the concurrency observed in the trace logs, does reproduce the wrong-batch-position symptom reliably. That localizes the bug to the batching or unbatching code path (for example, a race in how results are demultiplexed, meaning split back out of the single batched output and matched to the individual requests that made up the batch) rather than anything about a specific input's content, since content-only replay never triggered it and load-shaped replay did every time. The fix (correcting the demultiplexing logic to match results to requests by an explicit index rather than by array position under concurrent access) is then validated by running the same load-shaped replay against the fixed build for several repeated runs at the same batch size and concurrency, confirming zero wrong-position responses across all runs, before it is considered resolved.

Trade-offs and pitfalls

A common wrong turn is treating "cannot reproduce locally" as license to guess at fixes and ship them speculatively. Without a scripted reproduction, you cannot tell a real fix from a change that merely shifted the failure's timing or probability, and load-correlated bugs are exactly the kind that can appear to go away for weeks before recurring under a slightly different traffic pattern.

A second pitfall is over-logging in the name of reproducibility. Full-payload capture at scale is both a privacy liability and an operational cost (storage, and the logging itself can perturb the timing behavior you are trying to observe, sometimes called a probe effect). Keep default logging to hashes, fingerprints, and shape metadata, and reserve full capture for a narrow, rate-limited, access-controlled path triggered only when you have a specific reason to suspect a request.

A third pitfall is chaos-testing without varying one axis at a time. If you inject CPU contention, memory pressure, and an unusual batch composition simultaneously and the failure reproduces, you have not learned which of the three mattered, and the "reproduction" you hand to the team fixing it may not actually reflect the real production trigger, wasting their time chasing the wrong axis.

Direct answer

Route traffic with a deterministic hash of a stable per-user key so the same user always lands in the same bucket, size the canary allocation from a power calculation rather than a round number, watch a small set of signals chosen specifically to confirm or rule out the suspected bug rather than a generic dashboard, and set rollback criteria before the canary starts, combining a statistical trigger tied to the suspected effect with a hard, unconditional safety ceiling that fires regardless of significance. The tension the whole design sits on is that a smaller canary limits user harm but takes longer to reach enough evidence, and a bigger one confirms faster at the cost of exposing more users to a bug you have not yet confirmed exists.

Structured elaboration

Traffic routing. Use a deterministic hash of a stable identifier (user_id for logged-in traffic, a stable session identifier otherwise), mapped into buckets, with a fixed subset of buckets assigned to the canary. Stickiness (the same user always resolving to the same bucket for the duration of the canary window) matters for two reasons: it lets you treat canary-exposed users as a clean, non-overlapping sample instead of a mix of one-off exposures, and it avoids a user seeing the model's output flip-flop between requests, which shows up as confusing behavior rather than a controlled experiment. The canary's traffic share is a separate design decision from the bug's estimated prevalence: the fact that the bug is believed to affect roughly 0.5% of production traffic describes how often the defect triggers, not how much traffic you must route to canary to investigate it. The canary share should instead be sized from the power calculation below.

Observability signals to monitor specifically during the canary window. A generic error-rate and latency dashboard is necessary but not sufficient, because the suspected bug may not manifest as a raw error at all, for example a subtly wrong prediction that never throws an exception. Add signals targeted at the specific hypothesis: the output-distribution shift on whatever the bug is hypothesized to affect (a shift in a prediction's value range, a shift in which class gets predicted for a known input pattern), a proxy metric on the specific user segment or input pattern where the bug is suspected to trigger rather than only the aggregate population, and per-request diagnostic tags (which bucket, which model version, a hash of the input) attached to every canary request so any downstream complaint or anomaly can be joined back to a specific canary request for root-cause work, not just counted.

Automated rollback criteria. Two independent triggers, not one: a statistical trigger, a pre-registered test (chosen and its threshold fixed before the canary starts, to avoid the temptation to keep watching and stop the moment something looks bad) comparing the canary's suspected-effect proxy against the control's, and a hard safety-net trigger, an unconditional ceiling on an absolute harm metric (for example, error rate crossing a fixed cap regardless of whether it is yet statistically significant) that reverts immediately without waiting for the planned sample size to be reached. The safety-net trigger exists because a genuinely severe, fast-onset failure should never wait on statistical process; the statistical trigger exists because a subtle, slow-building effect needs enough accumulated evidence to distinguish a real signal from noise.

flowchart LR
    A[Incoming request] --> B{Deterministic hash of user_id in canary bucket?}
    B -->|No, majority| C[Route to stable production model]
    B -->|Yes, sized share| D[Route to canary model]
    D --> E[Tag request with canary_id in logs]
    C --> F[Aggregate control metrics]
    E --> G[Aggregate canary metrics]
    F --> H[Automated comparator: error rate, targeted proxy metric, distribution drift]
    G --> H
    H --> I{Rollback trigger crossed?}
    I -->|Yes| J[Auto-revert canary bucket to stable model]
    I -->|No| K[Continue canary window, accumulate diagnostic sample]

Minimizing user impact while still gathering enough diagnostic data. Three levers, used together rather than any one alone: size the canary share to the smallest allocation the power calculation says will reach a decision in an acceptable window, cap the absolute number of exposed users independent of percentage (protects against an unexpected traffic spike blowing past the intended exposure even at a fixed percentage), and extract more diagnostic value per exposed user through richer logging on the canary slice specifically, since a small canary means each request is more precious for diagnosis than it would be in a full-traffic rollout.

Worked example

Sizing the canary using a two-proportion z-test for a baseline error rate of 0.8% versus a hypothesized elevated rate of 2.0% on the affected slice, targeting 80% power at a two-sided alpha of 0.05.

Two terms carry the whole calculation, so they are worth stating plainly before the symbols arrive. Power is the chance the test actually detects the effect when the effect is really there: 80% power means that even with the bug genuinely present, a canary this size would fail to flag it about one time in five. Alpha is the chance of raising a false alarm when nothing is wrong: a two-sided alpha of 0.05 means a 1-in-20 chance of concluding the canary and control differ when they do not, counting a difference in either direction. Those two are the dials on the calculation, and both point the same way, more certainty costs a bigger sample.

Read in words before substituting anything: the numerator is how much statistical margin the chosen alpha and power demand, and the denominator is the gap you are trying to see. Because that gap is SQUARED, halving the effect size you want to be able to catch quadruples the sample you need, which is why a subtler suspected bug is disproportionately more expensive to confirm.

Substituting $z_{\alpha/2} = 1.96$ (the two-sided 5% value, and the constant to change for a different alpha), $z_{\beta} = 0.8416$ (the 80%-power value, and the constant to change for a different power, for example 1.2816 for 90%), $p_1 = 0.008$, $p_2 = 0.020$, and $\bar p = (0.008 + 0.020)/2 = 0.014$, in three traceable steps:

1. $\sqrt{2 \cdot 0.014 \cdot 0.986} = \sqrt{0.027608} = 0.16616$, so the first term is $1.96 \cdot 0.16616 = 0.3257$.
2. $\sqrt{0.008 \cdot 0.992 + 0.020 \cdot 0.980} = \sqrt{0.007936 + 0.0196} = \sqrt{0.027536} = 0.16594$, so the second term is $0.8416 \cdot 0.16594 = 0.1397$.
3. $(0.3257 + 0.1397)^2 = 0.4654^2 = 0.2166$, and $(p_2 - p_1)^2 = 0.012^2 = 0.000144$, so $n = 0.2166 / 0.000144 = 1{,}504$.

(Carrying full precision instead of the four-decimal intermediates gives 1,503.6; rounding up to a whole number of requests lands on the same 1,504 either way.)

So the canary needs 1,504 requests on its side, and 1,504 matched control requests. An algebraic sample size is a claim about power, not a demonstration of it, so the whole calculation plus a direct simulation of it is worth running rather than trusting:

import numpy as np

Z_ALPHA_2, Z_BETA = 1.96, 0.8416      # two-sided alpha 0.05, 80% power
P1, P2 = 0.008, 0.020                 # baseline vs hypothesized elevated error rate
P_BAR = (P1 + P2) / 2

term1 = Z_ALPHA_2 * np.sqrt(2 * P_BAR * (1 - P_BAR))
term2 = Z_BETA * np.sqrt(P1 * (1 - P1) + P2 * (1 - P2))
n_exact = (term1 + term2) ** 2 / (P2 - P1) ** 2
N = int(np.ceil(n_exact))
print(f"term 1 = {term1:.4f}, term 2 = {term2:.4f}")
print(f"n = {n_exact:.1f} -> {N} requests per arm")

# Does that sample size actually deliver the claimed power? Simulate it.
rng = np.random.default_rng(25)       # pinned so this number reproduces
TRIALS = 2000
control = rng.binomial(N, P1, TRIALS)
canary = rng.binomial(N, P2, TRIALS)
p_pool = (control + canary) / (2 * N)
se = np.sqrt(p_pool * (1 - p_pool) * (2 / N))
z = (canary / N - control / N) / se
rejected = (np.abs(z) > Z_ALPHA_2).mean()
print(f"simulated power over {TRIALS} trials at n={N}: {rejected:.1%}")

for share in (0.005, 0.02):
    print(f"canary share {share:.1%}: {N/share:,.0f} total production requests to reach n={N}")

Output:

term 1 = 0.3257, term 2 = 0.1397
n = 1503.6 -> 1504 requests per arm
simulated power over 2000 trials at n=1504: 81.3%
canary share 0.5%: 300,800 total production requests to reach n=1504
canary share 2.0%: 75,200 total production requests to reach n=1504

The simulation rejected the null (that is, concluded the canary and control genuinely differ rather than differing by chance) in 1,627 of 2,000 trials, 81.3%, close to the 80% target and confirming the formula's sample size actually delivers the claimed power rather than only algebraically implying it. The seed is pinned, so that figure reproduces; without one it wanders by roughly a point either side of 80% between runs, which is exactly the kind of drift that makes an unpinned simulated number worth distrusting.

How the canary-share choice trades off against how long that takes to accumulate: at a 0.5% canary share, reaching 1,504 canary-side observations needs 300,800 total production requests; at a 2% share, the same 1,504 observations need only 75,200, four times faster, at the cost of exposing four times as many users to a model still under suspicion. That is the concrete shape of the tension named in the direct answer, and the right choice depends on the service's request volume and how severe the suspected bug is believed to be: a low-traffic service or a mild suspected effect argues for the larger share to get an answer in a reasonable window, a high-traffic service or a severe suspected effect argues for the smaller share since the same statistical power arrives quickly either way.

Trade-offs and pitfalls

• The bug's estimated prevalence and the canary's traffic share are easy to conflate and should not be. Sizing the canary share directly off the bug's estimated 0.5% incidence, rather than off an actual power calculation, is a common shortcut that produces an allocation with no principled connection to how much evidence it will actually generate.
• If the suspected bug is itself confined to a narrow condition inside the canary population (say, it only triggers for a specific input pattern that itself only occurs in a small fraction of requests), the effective sample size needed is larger than this calculation shows, since the calculation above assumes the elevated rate applies broadly across canary traffic; a real investigation should sanity-check that assumption against whatever evidence suggested the bug in the first place.
• Waiting for the statistical trigger alone, with no hard safety ceiling, is the more dangerous failure mode. A slow, well-powered comparison is the right tool for confirming a subtle effect, but it is the wrong tool to rely on exclusively during an active incident, where a hard, unconditional cap that fires immediately is what actually protects users.
• A canary with generic monitoring only tells you something is different, not what. Without the request-level tagging and the hypothesis-specific proxy metric, confirming the bug and root-causing it become two separate, sequential investigations instead of one; the tagging is what lets the diagnostic and the confirmation happen from the same canary window.