Test Case Design and Edge Case Analysis Questions

Direct answer
A release testing matrix for a model change needs six stages, each answering the same four questions with different rigor: what automation runs, what makes it pass, how long it is allowed to take before it blocks the release, and who has to sign off. The stages get progressively closer to production and progressively more expensive to run, so the matrix should be read top to bottom as increasing blast radius, not as a checklist to run in parallel.

Structured elaboration

Worked example
Consider a recommendation model change. Unit tests assert that the feature-normalization function maps a known input vector to a known output vector exactly (a pinned fixture, not a live call). Integration tests hit the real feature-serving endpoint in a staging environment and assert the inference response matches the expected JSON schema, including that empty and single-item candidate lists are handled without throwing. Model-acceptance compares offline NDCG (normalized discounted cumulative gain) and a fairness guardrail metric between the new and current model on the same frozen evaluation set, and blocks promotion if NDCG regresses more than the pre-agreed threshold or if the guardrail metric regresses at all, regardless of NDCG. Shadow testing mirrors 100% of production traffic to the new model for several days without serving its output, logging prediction divergence from the live model so the team can see how often and how much the two models actually disagree on real (not held-out) traffic, which offline evaluation cannot show. Canary then serves the new model to a small live slice and watches online business metrics, with an automated rollback trigger if error rate or a KPI guardrail crosses its bound. Full rollout ramps the slice up in steps (e.g. 5% then 25% then 100%), each step gated on the same online-metric bounds holding.

Trade-offs and pitfalls
The most common mistake is treating offline model-acceptance as sufficient and skipping shadow testing to save time: offline metrics are computed on a static, already-labeled evaluation set and cannot surface issues that only show up on the live input distribution (schema drift in upstream features, timeout behavior under production load, or an input pattern absent from the offline set). The second common mistake is making the canary window too short to reach statistical power, which lets a real regression through as noise or, just as costly, blocks a genuinely neutral release on a false signal; the canary's expected runtime should be derived from the traffic volume and the minimum detectable effect the team cares about, not from a fixed calendar convenience. A senior answer also keeps the sign-off chain proportional to blast radius: gating unit tests behind a human reviewer is friction with no benefit, while letting full rollout proceed without an accountable sign-off (distinct from the automated metric checks) removes the one human checkpoint that can catch a problem the automated criteria did not anticipate.

Direct answer
Integration tests for a SHAP (SHapley Additive exPlanations) pipeline need to verify the pipeline's structural CONTRACT, not the underlying game-theoretic Shapley value math itself (that math is the library's responsibility, not the pipeline's): the attributions must sum to (approximately) the model's output difference from baseline, a zero-variance feature must contribute (approximately) zero, a missing baseline value must be rejected explicitly rather than silently propagated, and an unseen categorical level at inference must have a defined, tested fallback rather than an undefined encoding. The additivity property in particular is exact, not approximate, for the class of models where Shapley values have a closed form (linear models), which makes it possible to test the pipeline's SUMMATION logic against ground truth without depending on an external SHAP library at all.

Structured elaboration

• Zero-variance features. A feature that is constant across the calibration/background sample should contribute (approximately) zero to any individual explanation, PROVIDED the baseline used for that feature matches its actual constant value. The test that actually matters here is not "does it show near-zero," it is a NEGATIVE test: does the pipeline correctly compute zero when the baseline is right, and does it visibly show a NONZERO, misleading contribution when the baseline is WRONG (mismatched against a different reference sample), since a mismatched baseline for a genuinely constant feature is a realistic pipeline bug (a stale or wrong background-sample reference), not a hypothetical.
• Missing baseline values. If the baseline/background reference contains a missing value for some feature, computing an attribution against it produces an undefined delta; the pipeline must detect and reject this explicitly (a clear validation error) rather than letting a missing value silently propagate into a NaN explanation, which would then also silently break the additivity check (since any arithmetic involving NaN is NaN, masking rather than surfacing the problem).
• Unseen categorical levels at inference. A category never seen during the calibration/background sampling has no defined position relative to that background distribution; the pipeline needs an explicit policy (reject the explanation request, or fall back to a defined "unknown category" reference point) and a test asserting that policy fires, rather than the encoding layer silently producing a nonsensical or out-of-range attribution.
• Sum-to-output-difference. For each explained instance, $\sum_i \phi_i \approx f(x) - f(\text{baseline})$, where $\phi_i$ is feature $i$'s attribution. For subsampled backgrounds (averaging attributions across several baseline samples, a common real-world SHAP pattern instead of a single reference point), the SAME property must hold against the AVERAGE baseline prediction, not just a single point, and that needs its own explicit test since averaging is a separate code path from single-baseline attribution.

Worked example (executed): a linear-model oracle that makes additivity exact, not approximate
For a linear model $f(x) = \sum_i c_i x_i + b$, the exact Shapley value for feature $i$ is $\phi_i(x) = c_i \cdot (x_i - \text{baseline}_i)$, and by construction $\sum_i \phi_i(x) = f(x) - f(\text{baseline})$ exactly, which makes a linear model a dependency-free, mathematically-grounded oracle for testing the PIPELINE's summation and edge-case logic, without needing the actual shap package or a trained black-box model at all.

import numpy as np
rng = np.random.default_rng(7)
coefs = rng.normal(0, 2.0, size=5)
intercept = 1.5
f = lambda x: float(np.dot(coefs, x) + intercept)
shap_values = lambda x, baseline: coefs * (np.array(x) - np.array(baseline))

baseline = np.zeros(5)
x = rng.normal(0, 1, size=5)
phis = shap_values(x, baseline)
print(phis.sum(), f(x) - f(baseline), abs(phis.sum() - (f(x) - f(baseline))))   # -> 0.7396218814 0.7396218814 1.11e-16

# zero-variance feature, correct vs mismatched baseline
zv_idx = 2
x_zv = x.copy(); x_zv[zv_idx] = 3.7
correct_baseline = baseline.copy(); correct_baseline[zv_idx] = 3.7
wrong_baseline = baseline.copy(); wrong_baseline[zv_idx] = 0.0
print(shap_values(x_zv, correct_baseline)[zv_idx])   # -> 0.0
print(shap_values(x_zv, wrong_baseline)[zv_idx])      # -> -2.0286 (nonzero, misleading)

# missing baseline value
def safe_shap(x, baseline):
    if np.any(np.isnan(baseline)): raise ValueError("baseline contains NaN/missing value(s)")
    return shap_values(x, baseline)
bad_baseline = baseline.copy(); bad_baseline[1] = np.nan
try: safe_shap(x, bad_baseline)
except ValueError as e: print("raised:", e)

# unseen categorical level
known = {"US": 0, "UK": 1, "DE": 2}
def encode(cat):
    if cat not in known: raise ValueError(f"unseen category '{cat}' at inference")
    return known[cat]
try: encode("BR")
except ValueError as e: print("raised:", e)

# subsample-averaged additivity
backgrounds = [rng.normal(0, 0.3, size=5) for _ in range(20)]
avg_phis = np.mean([shap_values(x, b) for b in backgrounds], axis=0)
avg_f_baseline = np.mean([f(b) for b in backgrounds])
print(avg_phis.sum(), f(x) - avg_f_baseline)   # -> 0.7369974961 0.7369974961

Executed output: additivity holds to 1.11e-16 (floating-point noise, effectively exact); the zero-variance feature's attribution is exactly 0.0 with the correct, matched baseline, and -2.0286 (nonzero) with a deliberately mismatched baseline, confirming the mismatch bug is genuinely detectable; the missing-baseline case raises ValueError: baseline contains NaN/missing value(s); the unseen-category case raises ValueError: unseen category 'BR' at inference; the subsample-averaged additivity check matches to 0.7369974961 on both sides.

Trade-offs & pitfalls
The biggest pitfall is treating "sum equals output difference" as an APPROXIMATE property everywhere and therefore tolerating any deviation, that is correct for real SHAP estimators using sampling-based approximation (KernelSHAP, sampling-based TreeSHAP variants), but a large or systematically-biased deviation from the exact sum on a KNOWN-exact case, like the linear-model oracle here, or like TreeSHAP's exact mode on tree ensembles, indicates a genuine pipeline bug, not acceptable approximation noise, so the test tolerance must be set relative to what the SPECIFIC explainer is supposed to guarantee, not a single blanket epsilon across all explainer types. A second pitfall is testing zero-variance-feature handling only with a correctly-matched baseline, which never exercises the actual bug class (a stale or mismatched background reference), the negative test with a deliberately wrong baseline is what actually proves the pipeline is sensitive to this failure mode rather than merely looking correct on the happy path. Third, missing-value and unseen-category handling are easy to test only at the FEATURE level and forget to test again at the INTERACTION level, if the pipeline also reports pairwise or grouped interaction attributions, those need their own missing-value and unseen-category tests, since interaction computation is frequently a separate code path from single-feature attribution.

Direct answer
A test plan for an int8-quantized transformer model has to validate two different things that are easy to conflate: functional correctness (does the quantized model still run and produce sane outputs across the same input space as the float model) and numeric accuracy drift (how much did quantization degrade prediction quality, and is that degradation within an agreed threshold). Edge inputs, dynamic-shape handling, and operator availability are functional-correctness risks specific to the edge-device deployment target; drift thresholds and calibration-dataset selection are accuracy risks specific to quantization itself; both need dedicated test cases, not one combined pass/fail check.

Structured elaboration

• Functional correctness tests: run the quantized model against the same test inputs as the float32 baseline and confirm it executes without error, on the TARGET hardware or its accurate emulator, not just on the development machine, since operator support and numeric behavior can differ by device.
• Numeric accuracy drift thresholds: measure per-layer or end-to-end output deviation (not just final-task accuracy, which can mask a large deviation on an unlucky input that happens not to flip a classification decision) against a defined tolerance, agreed with the team BEFORE testing starts, not fit retroactively to whatever number the quantized model happens to produce.
• Edge inputs: sequences at or near the model's minimum and maximum supported length (too short: does the model handle it without crashing on padding/masking logic; too long: does truncation behave as documented), plus activation-distribution edge cases like all-zero or unusually large-magnitude inputs, which stress the quantization scale factor itself.
• Dynamic shape handling: if the deployment target supports variable batch size or sequence length, test the actual shape-inference and buffer-allocation paths at multiple shapes, not just the shape used during quantization calibration; a common real bug is a quantized model that only works at the exact shape it was calibrated on.
• Operator availability checks: confirm every operator in the model graph has an int8-quantized kernel available on the target hardware; an operator that silently falls back to float32 (a common runtime behavior in several inference engines) can produce a technically-working model that is far slower than expected without any test catching it unless operator dtype is explicitly asserted per-node.
• Calibration dataset selection: the calibration set (used to determine per-tensor or per-channel quantization scale factors) must be REPRESENTATIVE of production input distribution; a calibration set that is too small, or skewed away from the actual deployment traffic, produces scale factors tuned to the wrong range and silently worse accuracy than a properly calibrated model would show, even though nothing in the pipeline errors.

Worked example
Quantization drift measured directly (NumPy, seed=7, symmetric per-tensor int8 quantization simulating a post-LayerNorm activation tensor of 2,048 values):

import numpy as np
rng = np.random.default_rng(seed=7)
x = rng.normal(loc=0.0, scale=1.0, size=2048).astype(np.float32)

def quantize_int8_symmetric(values):
 scale = np.max(np.abs(values)) / 127.0
 q = np.clip(np.round(values / scale), -127, 127).astype(np.int8)
 return q, scale

q, scale = quantize_int8_symmetric(x)
x_dq = q.astype(np.float32) * scale

Actual measured results: scale = 0.025602, max absolute dequantization error = 0.012798, mean absolute error = 0.006426, MSE (mean squared error) = 0.00005515. These are the numbers a drift-threshold test would assert against a pinned tolerance (e.g. "max absolute error must stay under 0.02 on the calibration-representative distribution").

Two edge inputs were tested directly against the same quantizer, real findings, not hypothesized:

• All-zero activation (16 zeros): produces scale = 0.0. Dividing by that scale raises RuntimeWarning: invalid value encountered in divide (a 0/0), which produces transient NaN values in the intermediate float array; but casting that intermediate NaN array to int8 (as the quantizer's last step does) silently converts every NaN to 0, so the actual returned q array is [0, 0, 0, ...], not NaN, and the final dequantized output is [0.0, 0.0, 0.0, ...], confirmed with np.isnan(dequantized).any() == False. This is arguably a worse bug than the NaN a first read might expect: the function returns a plausible-looking, silently wrong all-zero result with only a runtime warning (easy to suppress or miss in a production log) rather than a loud, unmistakable failure, exactly the kind of edge input a test plan exists to catch before it reaches production, where an all-zero activation (e.g. from a masked or padded position) is a realistic occurrence, not a contrived one.
• Outlier-dominated tensor (999 values drawn from the same N(0,1) distribution as the main worked example, seed=7, plus a single value of 50.0 appended): measured directly, the typical-values-only scale (no outlier) is 0.025602, matching the main example's scale exactly since it's the same 999-sample draw; adding the single 50.0 outlier inflates the shared per-tensor scale to 0.393701 (simply 50.0 / 127), a 15.4x increase, and the mean absolute dequantization error on the 999 OTHER, non-outlier values rises to 10.4% of their standard deviation (up from a baseline near the main example's 0.006426 mean absolute error). A single extreme activation value degrades the precision of every other value sharing that tensor's quantization scale; this is a well-documented real phenomenon in large transformer models (activation outliers), which is why per-channel (rather than per-tensor) quantization or outlier-aware calibration is often required in practice, not just a generic scale computation. (Correction: an earlier draft of this answer claimed a 136x scale inflation and a 76.6%-of-stddev error, evidently from an unshown, different typical-value distribution; the numbers above are the actual measured result of running the exact quantize_int8_symmetric function shown earlier against a N(0,1)-seed=7 draw, reproducible end to end.)

Acceptance criteria for release: functional tests pass 100% on target hardware across the full edge-input and dynamic-shape matrix (no crashes, no silent float32 fallback on any operator); accuracy drift stays within the pre-agreed threshold on a held-out, production-representative evaluation set, not just the calibration set itself (testing only on calibration data overstates achieved accuracy, since the model was tuned to that exact distribution); and the all-zero and outlier edge cases specifically are either handled with a defined fallback (e.g. a scale floor to avoid division by zero) or explicitly documented as an unsupported input class with upstream validation preventing it from reaching the model.

Trade-offs and pitfalls
The most common mistake is validating quantization accuracy only with end-task metrics (like classification accuracy) on a clean test set, which can look nearly unchanged even when per-tensor drift is large, because task accuracy is often robust to small activation shifts right up until it isn't; direct numeric drift measurement catches degradation before it becomes a task-level failure. A second mistake is calibrating on a small, convenient dataset (e.g. the first 100 examples from a training set) rather than one that reflects the actual range of production inputs, including outliers, which the worked example shows can distort the shared scale factor for an entire tensor. Finally, treating "it exports and runs" as sufficient functional validation misses the operator-fallback risk: a model that silently runs slower on the target device because one operator quietly fell back to float32 still "works," but violates the actual reason for quantizing in the first place.

Direct answer
For a sequence padding function pad(sequences, pad_token), three properties fully characterize correct behavior without hand-picking specific inputs: a LENGTH INVARIANT (every output sequence has the same length, equal to the longest input sequence), TOKEN PRESERVATION (every output sequence starts with exactly the original content of its corresponding input sequence, unmodified), and IDEMPOTENCE (padding an already-padded batch produces the identical result, since every sequence is already at the target length).

Structured elaboration

• Length invariant: all(len(out_seq) == max(len(s) for s in sequences) for out_seq in pad(sequences, pad_token)). This alone would pass for a badly broken implementation that pads correctly but also silently reorders or truncates content, so it must be combined with token preservation, not used alone.
• Token preservation: all(out_seq[:len(orig_seq)] == orig_seq for orig_seq, out_seq in zip(sequences, pad(sequences, pad_token))). This is the property that actually confirms no data was lost or corrupted; it is the property most likely to catch a real bug, because a length-correct-but-content-wrong implementation (e.g. one that pads to a hardcoded maximum length instead of the batch's actual maximum, silently truncating longer sequences) will violate exactly this property while still satisfying a naively-written length check on the sequences it doesn't truncate.
• Idempotence: pad(pad(sequences, pad_token), pad_token) == pad(sequences, pad_token). Since every sequence in the once-padded output is already at the batch maximum length, padding again should add zero additional padding to any of them; this property is a cheap, powerful way to confirm the length computation is based on the CURRENT input's actual lengths, not some external or stale reference length.
• Generating variable-length input sequences: use a strategy that generates a list of lists of integers with independently varying lengths, e.g. Hypothesis's st.lists(st.lists(st.integers(), min_size=0, max_size=20), min_size=1, max_size=10), which produces batches with realistic length variance (including empty sequences and single-element sequences) rather than a fixed-shape batch, since a fixed-shape generator would never exercise the padding logic's core job at all.

Worked example (executed)

def pad(sequences, pad_token):
    if not sequences:
        return []
    max_len = max(len(s) for s in sequences)
    return [list(s) + [pad_token] * (max_len - len(s)) for s in sequences]

All 3 properties were written as Hypothesis tests (200 generated examples each, fixed seed for reproducibility) and executed against this correct implementation: 3 passed, confirming all three properties hold.

One concrete failing example Hypothesis uncovered (executed against a realistic buggy variant)
A second, buggy implementation reflecting a real-world mistake (padding to a hardcoded max_len=5 left over from a different dataset's configuration, instead of the current batch's actual longest sequence) was tested against the same TOKEN PRESERVATION property:

def pad_buggy(sequences, pad_token, max_len=5):
    out = []
    for s in sequences:
        s = list(s)[:max_len]   # silently truncates sequences longer than max_len
        s = s + [pad_token] * (max_len - len(s))
        out.append(s)
    return out

Hypothesis found a failing case and shrunk it to the minimal reproducer, quoted verbatim from the executed run:

Falsifying example: sequences=[[0, 0, 0, 0, 0, 0]]
AssertionError: assert [0, 0, 0, 0, 0] == [0, 0, 0, 0, 0, 0]

A single sequence of six zeros gets silently truncated to five elements by the hardcoded max_len=5, which the token-preservation property catches immediately: the shrunk example is the smallest possible input (one sequence, one element over the hardcoded limit) that demonstrates the bug, exactly the kind of minimal, actionable counterexample property-based testing is meant to produce, in contrast to a hand-written example-based test that would only catch this bug if a developer happened to think to test a sequence longer than 5 elements.

Trade-offs and pitfalls
The most common wrong turn is writing only the length invariant and treating that as sufficient coverage, since it is the easiest property to state; as demonstrated above, the length invariant alone does not catch silent truncation, because a truncated-then-repadded sequence still ends up at the correct target length, it just isn't the RIGHT content anymore. The other pitfall is generating only same-length or near-uniform-length batches, which under-exercises the padding amount computation; Hypothesis's default shrinking toward small, sparse examples (as seen in the single-sequence, six-element counterexample above) is specifically why an intentionally varied length-generation strategy, not a hand-picked handful of batches, is what surfaces this class of bug reliably.

Direct answer
Every one of the four named edge cases, zero-length prompts, prompts long enough to trigger truncation, special-token-saturated prompts, and streaming chunking, is really the same boundary-value-analysis question asked at a different layer of the pipeline: does the system correctly handle the boundary between "fits" and "does not fit," and does it place the beginning-of-sequence (BOS) and end-of-sequence (EOS) tokens correctly regardless of where that boundary falls.

Structured elaboration

• Maximum output length: unit test the generation loop's stop condition directly, using a mock model that never predicts EOS naturally, and assert generation halts at exactly the configured maximum token count. This is a hard boundary value analysis case: test at max_length - 1 tokens generated so far (should continue) and at max_length tokens generated (should stop).
• BOS/EOS tokens: unit test that BOS is prepended exactly once at the start of every prompt, never zero times, never duplicated if the raw input already happens to contain a string that looks like the BOS token, and that EOS is correctly appended or detected at generation end. Integration test the full tokenize-generate-detokenize path to confirm special tokens used internally never leak into the user-visible output text, a common real bug where raw special-token strings appear in a response because detokenization did not strip them.
• Truncation policies: unit test the specific documented policy, truncate from the left (oldest) end, the right (newest) end, or refuse and error, against a prompt one token longer than the maximum context length, asserting the correct end is truncated and, importantly, that BOS is still present after truncation. A naive left-truncation implementation can accidentally truncate away the prepended BOS token itself if it is not treated as a protected, pinned position.
• Zero-length prompts: unit test that an empty prompt still produces a well-formed tokenized sequence, BOS present, generation proceeds or a documented empty-input response returns, rather than an empty or malformed tensor that crashes the model call.
• Extremely long prompts that trigger truncation: integration test at the exact context-window boundary and one token past it, asserting the boundary case behaves per the documented truncation policy, and the over-boundary case is truncated rather than silently accepted past the model's real context window, which would either error deep inside the model call or silently degrade quality.
• Prompts saturated with special tokens: unit test a prompt whose text content happens to contain strings that look like special tokens, for example a user literally typing the EOS token's string representation, and assert the tokenizer treats them as literal text, not as structural signals. This is the same guardrail concern as testing for reserved tokens on the input-safety side, but framed here around whether it corrupts the model's own understanding of where the prompt ends and generation begins.
• Streaming generation where chunking occurs: integration test the token-by-token or chunk-by-chunk streaming path specifically at chunk boundaries, asserting a multi-byte or multi-token character, certain unicode characters a tokenizer may split across token boundaries, is never emitted as a broken partial byte sequence mid-stream, and that the EOS signal correctly terminates the stream exactly once, not emitted redundantly across chunks and not missed entirely; a client that never receives a clean stream-end signal hangs waiting for more.

Related case: the fixed-context-window truncation-strategy edge cases map directly onto the truncation-policy content above, a fixed context window is exactly the resource the truncation policy has to respect, so the same boundary test (context length, and context length plus one token) that validates the max-output-length boundary should also run against the max-input, context-window boundary. A system can independently get the output-length boundary right while still getting the input-context-window boundary wrong; they are two distinct enforcement points sharing the same design pattern, and a test suite that only checks one leaves the other unverified.

Worked example
A concrete boundary walkthrough (definitional arithmetic): a model with a 4096-token context window and a configured maximum output length of 512 tokens. A prompt tokenizing to exactly 3583 tokens ($3583 + 512 = 4095$, one token under the limit after the full output) is accepted with no truncation. A prompt tokenizing to 3584 tokens ($3584 + 512 = 4096$, exactly at the limit) is the true boundary case worth its own dedicated test. A prompt tokenizing to 3585 tokens ($3585 + 512 = 4097$, one token over) must trigger the documented truncation policy, either truncating the prompt down to 3584 tokens to preserve the full 512-token output budget, or truncating the eventual output short if the policy instead prioritizes preserving the full prompt. Either is a legitimate design choice, but the test needs to assert whichever one this system actually implements, and confirm it does not simply error or silently drop tokens from the wrong end.

Trade-offs and pitfalls

• Testing the max-output-length and context-window boundaries against a mock model is fast and deterministic but can miss real tokenizer quirks, a real tokenizer's token count for a given string is not simply one token per word, so the boundary token counts in the worked example above must be verified against the real production tokenizer, not assumed.
• A truncation policy that silently drops content, either prompt or output, without any signal to the caller is a common wrong turn; a senior test suite asserts not just that truncation happens correctly but that the caller receives an explicit signal, a flag or a warning field, that truncation occurred, since silent truncation is a production debugging nightmare.
• Streaming chunk-boundary tests are easy to under-specify by testing only ASCII text; the multi-byte-character-split bug named above only appears with non-ASCII input, so the test corpus must deliberately include multi-byte unicode content, or this entire edge case goes unverified despite the suite looking complete.
• BOS/EOS handling interacts with truncation in a way that is easy to get backwards: a naive "keep the last N tokens" truncation that does not protect position 0 for BOS can lose the BOS token entirely, a bug that a test asserting only "the prompt is at most N tokens" would miss completely, since a pure length check does not verify which tokens survived.