Test Case Design and Edge Case Analysis Questions

Direct answer
[This is a behavioral question; the sample answer below models the STAR structure a strong candidate would use, with a realistic composite example, since the actual content should be the candidate's own genuine experience.]

Situation
A payment-confirmation email service was sending duplicate confirmation emails to a small fraction of customers (roughly 0.3% of orders) after a message-queue consumer was scaled from one instance to three for throughput. The existing test suite covered the happy path (one consumer, one message, one email) thoroughly, but had no test exercising multiple concurrent consumers against the same queue.

Task/detection and triage
Customer support flagged a rising trend of "why did I get two receipts" tickets. Triage started by checking whether the emails were byte-identical duplicates (ruling out two DIFFERENT orders) and confirmed they were, which narrowed the cause to the delivery/consumption layer rather than the order-creation logic. Correlating ticket timestamps against a recent deploy identified the consumer-scaling change as the likely trigger.

Action/immediate mitigation
The immediate mitigation was adding an idempotency check keyed on order ID before sending an email (a fast, low-risk fix: check-then-send against a short-lived cache of recently-sent order IDs), deployed within hours to stop new duplicates while the root cause was investigated further. The root cause turned out to be a message-visibility-timeout race: two consumers occasionally picked up the same message when one consumer's processing time exceeded the queue's visibility timeout under increased load, causing the message to become re-visible and be claimed by a second consumer before the first one acknowledged it.

Result/what changed afterward
Two lasting changes followed: first, the test suite gained a NEW category of test specifically simulating multiple concurrent consumers against a shared queue with an artificially shortened visibility timeout, deliberately engineered to force the race (rather than relying on it occurring by chance under real load), which is now a permanent regression test. Second, the design-review checklist for any feature involving a message queue was updated to explicitly require the author to state the visibility-timeout-vs-processing-time relationship and how at-least-once delivery is handled downstream (idempotency, deduplication, or an explicit acceptance of the risk), since this specific edge case (concurrent consumers + a visibility-timeout race) had previously been an implicit assumption nobody wrote down.

Trade-offs & pitfalls
The honest, harder lesson from this kind of incident is that the ORIGINAL single-consumer test suite was not wrong for the system it was written against, it simply never got updated when the system's concurrency model changed; the durable fix is treating a scaling change (going from one consumer to many) as a trigger for a deliberate edge-case review, not just a performance change, since concurrency introduces an entire category of edge cases (races, duplicate processing, ordering) that a single-instance test suite structurally cannot exercise.

Direct answer
A pragmatic test plan for this model-serving system layers four test types, each targeting a different edge case: unit tests for the cheap deterministic input-validation logic, integration tests for the real dependency boundaries (features, artifacts, permissions), end-to-end (e2e) tests for the full request path measured against the stated 95th-percentile (p95) latency and memory budget, and chaos/fault-injection tests for the failure modes that only appear under a genuine live fault. Rollback is tied to concrete, pre-declared thresholds on those same latency and memory numbers, not to a vague "looks worse" judgment call.

Structured elaboration

End-to-end tests specifically run the full request path, real feature fetch, real model load, real inference, against a staging replica under synthetic load shaped like production traffic, and assert both stated service-level objectives (SLOs) directly: p95 latency under 200ms and peak memory under the 4GB budget, not just functional correctness. This is the layer where the question's two numeric constraints become first-class pass/fail criteria rather than staying implicit.

Worked example
A concrete rollback-threshold design: suppose the pre-deploy baseline feature-fetch timeout rate, measured over the prior 7 days, is 0.2% of requests. A rollback rule set at "timeout rate exceeds 3x baseline, sustained for 5 consecutive minutes" fires at 0.6% sustained for 5 minutes, tight enough to catch a real regression quickly, loose enough that ordinary noise around a 0.2% baseline (which can easily vary by a factor of 2 run to run on real traffic) does not trigger a false rollback. The specific multiplier (3x) and window (5 minutes) are themselves parameters that should come from the measured variance of the baseline metric, not a guessed round number, this is the same discipline the trade-offs section below calls out for the latency and OOM thresholds.

Trade-offs and pitfalls

• Chaos and fault-injection tests are expensive to build and maintain, they need a safe environment for genuinely destructive experiments, and are frequently the first thing cut under time pressure. That is the wrong cut: network timeouts under sustained load and GPU OOM under real contention are specifically the cases unit and integration tests with mocked dependencies cannot faithfully reproduce.
• Rollback thresholds need to be derived from measured baseline behavior, not guessed round numbers; too tight and ordinary variance triggers false rollbacks, eroding trust until teams start ignoring or disabling the automation; too loose and it fails to protect the SLO it exists to enforce.
• GPU OOM testing is often skipped because CI runners frequently lack a matching GPU, or lack one entirely. The better mitigation is running that class of test in a dedicated GPU-equipped staging environment on a slower cadence, for example pre-release rather than every commit, and stating that explicitly in the plan rather than silently having zero GPU-memory coverage.
• Checksumming model artifacts protects against corruption but not against a model that loads without error yet is simply wrong, for example the output of a bad training run. That failure mode needs the canary prediction sanity check described above; conflating "loads without error" with "is a good model" leaves a real gap if only the checksum check exists.

Direct answer
A high-stakes tabular dataset needs edge-case review across five categories before training: structural anomalies (empty columns, single-unique-value features, duplicated rows), value anomalies (unexpected negatives, NaNs, extreme outliers), categorical-specific issues (unseen or mismatched levels between train and score time), temporal issues (timezone inconsistencies), and the highest-stakes category, label leakage (a feature that encodes information from the future or from the label itself).

Structured elaboration: category, mitigation, and detection test

Worked example: the label-leakage test in more detail
For a churn-prediction model, a feature like 'number of support tickets in the 30 days after the churn decision' would trivially predict churn (customers who already churned stop generating support tickets) while being completely unusable in production, since that information doesn't exist yet at prediction time; the automated test framework described above (asserting a feature's availability timestamp against the label's determination timestamp) is exactly what catches this class of bug BEFORE a model ships with deceptively excellent offline metrics that collapse in production.

Trade-offs & pitfalls
A common shortcut is running these checks manually once during initial data exploration and never again; because a high-stakes pipeline is typically RETRAINED periodically on fresh data, every one of these checks needs to run as an automated GATE in the retraining pipeline itself, not a one-time human review, since a new data source or an upstream schema change can silently reintroduce any of these issues (most dangerously, label leakage, if a new feature is added later without the same availability-timestamp discipline being applied).

Direct answer
Differential testing runs a reference implementation and a candidate (an optimized rewrite, a ported implementation, a new model version) against the SAME inputs and treats any output divergence as a signal to investigate, not an automatic bug report. It needs three harness components (an input generator, isolated dual execution, and a comparator), an explicit rule for which divergences actually count as defects, and a triage step that groups divergences by likely root cause rather than treating each failing input as its own separate bug.

Structured elaboration

• Harness components: an input generator that produces the inputs to run both implementations against; dual execution that runs the reference and the candidate on the identical input, isolated so one implementation's side effects cannot leak into the other's result; a comparator that decides whether two outputs match (exact equality for deterministic discrete outputs, a tolerance band for floating-point outputs, or a domain-specific equivalence check where superficial reordering does not matter); and a log of every divergence paired with the exact input that produced it, so it is directly reproducible afterward.
• Oracle selection, what counts as a bug: not every divergence is a defect. An intentional, documented behavior change between the reference and the candidate (for example, the optimized version deliberately rounds differently) is a divergence but not a bug. The harness needs an explicit allowlist of known, accepted divergences with a stated reason and an owner, so a genuinely new, unexplained divergence stands out instead of disappearing into accumulated noise.
• Input generation strategy: this is where a lightweight, always-on regression-aware diff tool, comparing new_model_outputs to baseline_outputs on every candidate build, becomes a concrete, continuously-run implementation of differential testing, distinct from a one-off manual comparison before a release. Random or fuzz-style generation covers the input space broadly but has near-zero probability of landing exactly on a specific boundary value; a boundary-seeded strategy, explicit edge values mixed with random fill, is needed to reliably surface boundary-triggered divergences, demonstrated concretely below.
• Triage and prioritization: group divergences by likely root cause rather than by individual failing input, since a single off-by-one bug can generate hundreds of individually-divergent inputs that are really one underlying defect; prioritize by a combination of divergence magnitude (how far apart the two outputs actually are) and production-traffic relevance (does the real input distribution actually hit this region), since a large count of tiny, rarely-hit divergences can matter less than one large divergence on a common input shape.

Worked example (executed)
A reference clamp function and an "optimized" candidate with a deliberately injected boundary bug (mishandles exactly x == 1.0, correctly clamping everything else):

import random

def clamp_reference(x):
    if x < 0.0: return 0.0
    if x > 1.0: return 1.0
    return x

def clamp_optimized_buggy(x):
    if x < 0.0: return 0.0
    if x < 1.0: return x
    if x == 1.0:
        return round(x, 3) + 1e-3  # BUG: nudges the exact boundary instead of returning it unchanged
    return 1.0

boundary_seeds = [-2.0, -0.0001, 0.0, 0.5, 0.9999, 1.0, 1.0001, 2.0]
random.seed(7)
random_inputs = [random.uniform(-2, 2) for _ in range(200)]
inputs = boundary_seeds + random_inputs

divergences = [(x, clamp_reference(x), clamp_optimized_buggy(x)) for x in inputs
               if clamp_reference(x) != clamp_optimized_buggy(x)]
print(f"total inputs={len(inputs)}, divergences={len(divergences)}, rate={len(divergences)/len(inputs):.4f}")
print("divergences:", divergences)
print("any random-only input diverged?", any(clamp_reference(x) != clamp_optimized_buggy(x) for x in random_inputs))

Running the boundary-seeded input generator above (8 explicit edge values including 1.0, plus 200 seeded random floats in [-2, 2], total 208 inputs) through a diff harness comparing new_model_outputs (the candidate) against baseline_outputs (the reference) produces:

total inputs=208, divergences=1, rate=0.0048
divergences: [(1.0, 1.0, 1.001)]
any random-only input diverged? False

Exactly 1 divergence out of 208 inputs, isolated entirely to the seeded value x = 1.0 (baseline=1.0, new_model_output=1.001). None of the 200 random-only inputs diverged on their own; only the explicit boundary seed surfaced the bug.

Trade-offs & pitfalls
A differential harness with only random inputs gives a false sense of coverage, since many inputs were tested but the specific value that actually mattered for this class of bug was never among them, exactly as shown above: 200 random floats over the boundary's neighborhood found zero divergences, and only the single explicit seed value did. An overly strict comparator (exact floating-point equality with no tolerance) will flag benign floating-point noise constantly, training the team to ignore the tool's output entirely, so choosing a sensible tolerance is part of getting real signal rather than alert fatigue. Differential testing also only tells you the two implementations disagree, never on its own which one is correct; that determination requires a third source of truth or a fresh manual check, a distinction that is easy to skip past under release pressure when the reference implementation is assumed correct by default.

Direct answer
The riskiest boundary in padding/truncation logic sits exactly where special tokens meet the length budget: if an implementation truncates content to max_seq_len and then adds [CLS]/[SEP] on top, every sequence quietly overshoots the model's real input limit by 2 tokens. The four required test lengths (0, 127, 128, 129) look like they probe "at, just under, just over max_seq_len," but the actual content budget is max_seq_len - 2 = 126, so all of 127, 128, and 129 exceed it, and a correct implementation must truncate content identically (to 126 tokens) for all three while a buggy one will not.

Structured elaboration
Using a BERT-style scheme with a [CLS] token at the start and a [SEP] token at the end: content_budget = max_seq_len - 2. The correct sequence is built as [CLS] + content[:content_budget] + [SEP], then padded with a pad token up to max_seq_len if shorter. The attention mask must be 1 for every real token including [CLS] and [SEP] (not just the content tokens), and 0 for every padding position.

The off-by-one risk is specifically in how content_budget is derived: a naive implementation that checks if len(tokens) <= max_seq_len and only then decides whether to truncate, without ever subtracting the 2 reserved slots, will treat an input of length 127 as "fits" (since 127 <= 128) when it actually still needs to lose at least 1 token to leave room for both special tokens. A second, subtler risk is off-by-one in the subtraction itself (using max_seq_len - 1 instead of -2, accounting for only one special token), which silently drops one extra content token from every sequence.

Worked example (executed; the original draft shipped only the two function definitions with the results table narrated in prose and no driver code actually producing it, and the function bodies were flattened to a single indent level, both fixed below with a real driver loop whose printed output matches the table)

CLS_ID, SEP_ID, PAD_ID, MAX_SEQ_LEN = 101, 102, 0, 128

def pad_truncate_fixed(token_ids, max_seq_len=MAX_SEQ_LEN):
    content_budget = max_seq_len - 2
    content = token_ids[:content_budget]
    seq = [CLS_ID] + content + [SEP_ID]
    attention_mask = [1] * len(seq)
    pad_needed = max_seq_len - len(seq)
    seq = seq + [PAD_ID] * pad_needed
    attention_mask = attention_mask + [0] * pad_needed
    return seq, attention_mask

def pad_truncate_buggy(token_ids, max_seq_len=MAX_SEQ_LEN):
    # BUG: truncates to max_seq_len BEFORE adding the 2 special tokens
    content = token_ids[:max_seq_len]
    seq = [CLS_ID] + content + [SEP_ID]
    return seq, [1] * len(seq)

for length in [0, 127, 128, 129]:
    tokens = list(range(1000, 1000 + length))
    fseq, fmask = pad_truncate_fixed(tokens)
    bseq, bmask = pad_truncate_buggy(tokens)
    content_kept = min(length, MAX_SEQ_LEN - 2)
    content_dropped = length - content_kept
    print(f"len={length}: fixed_total={len(fseq)} fixed_mask_sum={sum(fmask)} "
          f"kept/dropped={content_kept}/{content_dropped} buggy_total={len(bseq)}")

Running both implementations against input lengths 0, 127, 128, and 129 (via the driver loop above) produced:

The fixed implementation always produces exactly 128 tokens with [CLS] at position 0, [SEP] immediately after the last real content token, and the attention mask matching real-token count exactly (2 for the empty input, since only [CLS]/[SEP] are real; 128 for the other three, since the content budget is fully used and no padding remains). The buggy implementation produces a 2-token sequence for empty input (itself fine) but overshoots max_seq_len by 1 token at input length 127 and by 2 tokens at 128 and 129, exactly the off-by-one/off-by-two the reserved-slots bug predicts, and exactly what the four chosen test lengths are designed to expose: the length-127 case alone already proves the bug, since a correct implementation caps at 128 total regardless.

Trade-offs & pitfalls
Teams often special-case the empty-input case inconsistently, for example skipping [CLS]/[SEP] entirely when there is no content, which then breaks any downstream code that assumes position 0 is always [CLS]; the test at input length 0 exists specifically to pin down that special tokens are still present even with zero real content, as shown above (total length 128, mask sum 2, not 0). A second pitfall is testing only the attention mask's SUM (a scalar) rather than its exact positions: a mask that has the right total count of 1s but marks the wrong positions (for example, marking padding as attended while missing [SEP]) would pass a sum-only assertion while still being wrong; the test in the worked example checks that [SEP] sits at the boundary between real tokens and padding, not just that the mask has the right count. Finally, truncation direction is a real design decision, not a given: dropping from the end of the content (as shown here) is standard for tasks where early context matters most, but a task where the END of a long document matters more (for example, a document's conclusion) would need the opposite truncation direction, and that choice deserves its own explicit test rather than being assumed.