Test Case Design and Edge Case Analysis Questions

Direct answer
Fuzz-testing log-sum-exp ($$\log \sum_i e^{x_i}$$) or softmax for numerical instability means generating inputs that deliberately push the naive formula's intermediate $$e^{x_i}$$ term past floating-point range in both directions: large positive values that overflow to infinity, large negative values that underflow to exactly zero, and mixed-magnitude inputs where both happen in the same call. The stable rewrite subtracts the maximum value before exponentiating, which keeps every intermediate term in a safe range, and both claims below are backed by an actual executed run, not a textbook statement.

Structured elaboration

Generating inputs across orders of magnitude. Cover at minimum: values near zero (the case every implementation gets right and that a test suite alone would falsely validate); large positive values (e.g. in the hundreds to thousands, common after an unnormalized logit computation); large negative values (the underflow-to-zero case, just as damaging as overflow but silent instead of an exception); and mixed-sign wide-range inputs (e.g. one very negative and one very positive value in the same call), since that combination is what a real model's raw logits actually look like before normalization, not the clean small-magnitude examples used in textbook derivations.

Detecting instability. Watch for three distinct failure signatures, not just "an exception was raised": an explicit overflow (OverflowError in Python, or inf in a language that doesn't raise on it), an explicit domain error from taking the log of a non-positive number (which happens when every term has underflowed to zero, so the sum is exactly 0.0), and silent wrong-but-finite output (a large relative error against a higher-precision reference, the hardest case to catch because nothing crashes).

The stable rewrite. Subtract $$m = \max_i x_i$$ before exponentiating:

Every exponent $$x_i - m$$ is now $$\le 0$$, so $$e^{x_i-m}$$ can never overflow, and the largest term is always exactly $$e^0 = 1$$, so the sum can never underflow to a value that makes the subsequent \log fail on a non-positive input (there is always at least one term equal to 1).

Worked example (executed)

import math

def logsumexp_naive(xs):
    return math.log(sum(math.exp(x) for x in xs))

def logsumexp_stable(xs):
    m = max(xs)
    return m + math.log(sum(math.exp(x - m) for x in xs))

cases = {
    "large_positive": [1000.0, 1001.0, 999.0],
    "large_negative": [-1000.0, -1001.0, -999.0],
}
for name, xs in cases.items():
    try:
        print(name, "naive:", logsumexp_naive(xs))
    except (OverflowError, ValueError) as e:
        print(name, "naive raised:", type(e).__name__, e)
    print(name, "stable:", logsumexp_stable(xs))

Actual run output:

large_positive naive raised: OverflowError math range error
large_positive stable: 1001.4076059644444
large_negative naive raised: ValueError expected a positive input, got 0.0
large_negative stable: -998.5923940355556

The naive version fails in TWO different, equally silent-until-it-crashes ways: on large_positive it overflows computing math.exp(1001.0) before the log is even reached, raising OverflowError; on large_negative every math.exp(-1000.0) underflows to exactly 0.0, so the sum is 0.0 and math.log(0.0) raises ValueError: expected a positive input, got 0.0. The stable version handles both without incident and produces the mathematically consistent result (1001.41 and -998.59 respectively, each within roughly 1.4 of the input values' own magnitude, matching what $$\log \sum e^{x_i}$$ should be for three inputs clustered within 2 of each other).

For softmax specifically, the same instability compounds into a worse failure: on [1000.0, 1001.0, 1002.0], softmax_naive raised OverflowError computing the exponentials before even reaching the division; on [-1000.0, -1001.0, -1002.0], every exponential underflowed to 0.0, making the normalizing sum 0.0, and the division step raised ZeroDivisionError: division by zero, a genuine 0/0 produced entirely silently up to that point. The stable softmax (subtract the max before exponentiating, exactly as in log-sum-exp) produced [0.6652409557748218, 0.24472847105479764, 0.09003057317038046] for the very-negative case, summing to 0.9999999999999999 (off from exactly 1.0 only by ordinary floating-point rounding, not by an instability bug), proving the rewrite is not just crash-free but numerically correct.

A fourth signature, an explicit NaN (not-a-number), was produced deliberately with an actual infinite input rather than merely a large finite one: softmax_naive([float('inf'), 1.0, 2.0]) returned [nan, 0.0, 0.0], confirmed with math.isnan. The genuinely interesting result is that the "stable" subtract-the-max rewrite does NOT fix this case: with m = max(xs) = inf, the shift x_i - m becomes inf - inf for the infinite element itself, which IEEE-754 defines as NaN, so softmax_stable([float('inf'), 1.0, 2.0]) also returns [nan, nan, nan], all three outputs poisoned by the one NaN propagating through the shared denominator. The subtract-the-max fix only targets large FINITE magnitudes; it does not, and cannot, make the function safe against a genuinely non-finite input reaching it, which is why a fully robust implementation needs an explicit math.isfinite check on every input rejecting NaN/inf outright, as a distinct guard from the numerical-stability rewrite, not a consequence of it.

Trade-offs and pitfalls
The most common mistake is testing only near-zero or small-magnitude inputs, which is exactly what the naive formula gets right, so a test suite built from "reasonable-looking" numbers will pass on the unstable implementation and only fail once real production logits (which are unbounded before normalization) hit it. A second pitfall is treating "no exception was raised" as proof of correctness for the stable version too; the property worth asserting is not just crash-freedom but that the stable and naive implementations AGREE on inputs small enough for the naive one to succeed (a differential test), which catches a bug in the stable rewrite itself, and that softmax outputs always sum to 1 within a small floating-point tolerance (math.isclose, not exact equality, since summing floats is not associative). A third, subtler pitfall is fixing the overflow case (clipping or catching the exception) without also fixing the underflow case, since they require the same subtract-the-max fix but are easy to test for independently and declare "done" after only one is covered, exactly why both large_positive and large_negative need to be pinned test cases, not just one representative "large magnitude" case.

Direct answer
A normalize_username test suite needs at least five cases: whitespace trimming, case folding, collapsing multiple internal spaces to a single underscore, an empty-string input, and a unicode input, each verifying one specific transformation the function claims to perform.

Structured elaboration and worked example (executed)

import re
import pytest

def normalize_username(s: str) -> str:
    s = s.strip().lower()
    s = re.sub(r' +', '_', s)
    return s

@pytest.mark.parametrize("input_s,expected", [
    (" Alice Smith ", "alice_smith"),        # leading/trailing whitespace + internal space
    ("", ""),                                 # empty string
    ("BOB", "bob"),                           # case-only
    ("multi internal spaces", "multi_internal_spaces"),  # consecutive spaces collapse to ONE underscore
    ("Café Müller", "café_müller"),           # unicode: accents preserved, only case+space normalized
])
def test_normalize_username(input_s, expected):
    assert normalize_username(input_s) == expected

Running pytest -v: 5 passed in 0.15s (all 5 parametrized cases PASSED).

Why each case matters

• Leading/trailing whitespace + internal space in one case: confirms .strip and the internal-space collapse both apply, and specifically that the OUTER whitespace does not itself become a leading/trailing underscore (a common bug: stripping after substitution instead of before would turn " Alice Smith " into "_alice_smith_").
• Empty string: the trivial identity case; confirms the function doesn't throw on an empty input, which a naive regex-only implementation with no length guard could plausibly do depending on the regex engine, though this simple implementation happens to handle it safely.
• Case-only: isolates the .lower behavior from the whitespace logic, so a failure here specifically points at case-folding, not spacing.
• Multiple internal spaces: this is the case most likely to be under-tested; a naive .replace(' ', '_') (not a regex with +) would produce "multi___internal___spaces" (one underscore per space) instead of the collapsed single-underscore form, so this test specifically distinguishes a regex-based collapse from a naive single-character replace.
• Unicode input: confirms accented characters are preserved through .lower (Python's .lower is unicode-aware by default, correctly lowercasing 'É' to 'é'), rather than being stripped, mangled, or requiring a separate ASCII-only code path.

Trade-offs & pitfalls
The unicode test above uses .lower, which is adequate for accented Latin characters but is a WEAKER transform than .casefold for languages with more complex case-folding rules (e.g. German 'ß', which .casefold maps to 'ss' but .lower leaves unchanged); if the real system needs to treat visually-distinct usernames as the same account across such languages, the test suite should include a .casefold-specific case (like 'ß' vs 'ss') to pin down which behavior is actually intended, since the two functions genuinely disagree on some real-world inputs.

Direct answer
A test oracle is the mechanism a test uses to decide pass or fail; the oracle problem is that many machine learning and simulation systems have no single correct output to assert actual == expected against, since a generative model, a probabilistic classifier, or a ranking system can be legitimately correct in more than one way. The practical fix is to replace exact-match with a layered set of weaker, checkable properties: invariants that hold regardless of the exact output, distributional checks across many outputs, statistical assertions with an explicit tolerance, and reference anchors (contract tests and golden datasets) that pin down behavior without demanding a bit-exact match.

Structured elaboration

• Invariants and metamorphic relations: a property that must hold under a controlled TRANSFORMATION of the input, regardless of what the absolute output value is (for example: "the model's output should not change under a translation of the input that shouldn't affect it" or "increasing one input while holding all others fixed should never decrease a monotonic model's score"). This sidesteps the oracle problem entirely, since the test never needs to know the "right" output value, only that the output responds correctly to a controlled input change.
• Distributional checks: assert that an aggregate statistic across MANY outputs (a class balance, a mean score, a "flagged as high-risk" rate) stays within an expected tolerance band. This catches systemic drift or regression even when no individual output has an available ground-truth label.
• Statistical assertions: compare a labeled sample's measured accuracy (or another metric) against a stated floor with an explicit tolerance and sample size, acknowledging the result carries some inherent false-positive rate rather than presenting it as a certainty.
• Contract tests: assert SHAPE and TYPE-LEVEL guarantees of the output that are checkable with total certainty even with zero ground truth (a probability distribution's values sum to 1, a bounding box stays within the image's dimensions, a classifier returns one of the known label values).
• Golden datasets: a curated, human-reviewed set of input and acceptable-output-range pairs, used as a stable regression anchor and refreshed on a deliberate schedule, distinct from a live production sample which changes underneath the test constantly.

Worked example (executed)
An invariant check and a distributional check, run against a deterministic toy credit-risk scorer that returns a 0-100 score from (income, existing_debt):

import random

def risk_score(income, existing_debt):
    base = 100 - (existing_debt / max(income, 1)) * 100
    return max(0, min(100, base))

# Invariant: holding debt fixed, more income should never DECREASE the score.
# No "correct" absolute score is asserted, only the relationship.
random.seed(42)
failures = []
for _ in range(500):
    income = random.uniform(1, 300_000)
    debt = random.uniform(0, 300_000)
    raise_amount = random.uniform(0.01, 50_000)
    s1 = risk_score(income, debt)
    s2 = risk_score(income + raise_amount, debt)
    if s2 < s1 - 1e-9:
        failures.append((income, debt, raise_amount, s1, s2))
print(f"monotonicity: {500-len(failures)}/500 held, failures={len(failures)}")

# DISTRIBUTIONAL CHECK
random.seed(7)
N = 2000
scores = [risk_score(random.uniform(20_000, 300_000), random.uniform(0, 120_000)) for _ in range(N)]
high_risk_rate = sum(1 for s in scores if s < 40) / N
print(f"high_risk_rate = {high_risk_rate:.4f}, within [0.25,0.35] band: {abs(high_risk_rate-0.30)<=0.05}")

Running the driver above (seeded, fully reproducible) produced 0 violations out of 500: 500/500 cases held. The distributional check on a 2,000-sample seeded population produced high_risk_rate = 0.2890, which falls inside the stated baseline band of 0.30 +/- 0.05 (|0.2890 - 0.30| = 0.011 <= 0.05), so the check passed.

The remaining three strategies, executed against the same toy scorer:

# CONTRACT TEST: no ground truth needed, just the shape guarantee
for income, debt in [(50_000, 10_000), (0, 0), (1, 1_000_000), (200_000, 0)]:
    v = risk_score(income, debt)
    assert 0 <= v <= 100, (income, debt, v)
print("contract test: all 4 cases within [0,100]")

# STATISTICAL ASSERTION: measured accuracy against a stated floor, using an
# independent label with 10% simulated label noise so it is not circular
random.seed(99)
n = 300
correct = 0
for _ in range(n):
    income = random.uniform(20_000, 300_000)
    debt = random.uniform(0, 120_000)
    predicted_high_risk = risk_score(income, debt) < 40
    true_high_risk = predicted_high_risk if random.random() >= 0.10 else not predicted_high_risk
    if predicted_high_risk == true_high_risk:
        correct += 1
accuracy = correct / n
print(f"accuracy = {accuracy:.4f}")

# GOLDEN DATASET: curated (input, acceptable-output-RANGE) pairs
GOLDEN_CASES = [(100_000, 0, (95, 100)), (100_000, 100_000, (0, 5))]
for income, debt, (lo, hi) in GOLDEN_CASES:
    v = risk_score(income, debt)
    assert lo <= v <= hi, (income, debt, v)
print("golden dataset: both cases within declared range")

All four extreme-input cases in the contract test stayed within the declared [0, 100] range; the statistical assertion, run against 300 seeded samples (with 10% simulated label noise against an independent ground truth so the check is not circular) and a stated floor of 0.85, measured accuracy = 0.9133, clearing the floor; both golden-dataset cases landed inside their curated acceptable range.

Trade-offs & pitfalls
Invariants only catch violations of the SPECIFIC relationship you thought to encode, so a model can pass every invariant check in the suite and still be badly wrong in an absolute sense; invariants are a necessary layer, never a sufficient one on their own. Distributional checks can mask a real regression that happens to preserve the aggregate statistic, for example if the model starts getting a different 10% of cases wrong while getting a previously-wrong 10% right, the net high-risk rate stays unchanged even though the SET of correct predictions shifted; distributional checks are for catching gross drift, not a substitute for case-level oracles wherever those are actually available. Golden datasets go stale as the true acceptable-output distribution shifts over time (for example, as a generative model's acceptable phrasing evolves with house style), so a golden dataset needs an explicit re-certification process on a schedule, not a one-time "add it and trust it forever" treatment.

Direct answer
robust_mean should use Kahan summation to control floating-point accumulation error on large or high-magnitude inputs, skip None and NaN values while counting only the valid ones, raise ValueError when no valid numbers remain, and treat infinities as a special case rather than letting them flow through Kahan's compensation arithmetic, which silently produces NaN if left unguarded.

Structured elaboration and worked example (executed)

import math
from typing import List, Optional

def robust_mean(values: List[Optional[float]]) -> float:
    total = 0.0
    compensation = 0.0
    count = 0
    saw_pos_inf = False
    saw_neg_inf = False
    for v in values:
        if v is None:
            continue
        if isinstance(v, float) and math.isnan(v):
            continue
        if v == float('inf'):
            saw_pos_inf = True; count += 1; continue
        if v == float('-inf'):
            saw_neg_inf = True; count += 1; continue
        y = v - compensation
        t = total + y
        compensation = (t - total) - y
        total = t
        count += 1
    if count == 0:
        raise ValueError("no valid numbers present")
    if saw_pos_inf and saw_neg_inf:
        return float('nan')
    if saw_pos_inf:
        return float('inf')
    if saw_neg_inf:
        return float('-inf')
    return total / count

The infinity bug this design deliberately avoids
A first implementation without the explicit infinity branches was tried and FAILED at execution: calling robust_mean([float('inf'), 1, 2]) returned nan instead of inf. The root cause is that Kahan's compensation step computes (t - total) - y; the moment an infinite value flows through it, t becomes inf, and inf - inf (inside that compensation formula) evaluates to nan, which then poisons every subsequent element because compensation stays nan for the rest of the loop. The fix shown above tracks infinities OUTSIDE the Kahan accumulator entirely and reasons about them algebraically afterward (both signs present -> nan, since +inf and -inf together have no defined mean; one sign present -> that infinity dominates the mean regardless of the finite values).

Demonstrated numerical stability gain (executed, not asserted)

N = 10_000_000
vals = [0.1] * N

naive_total = 0.0
for v in vals:
    naive_total += v
naive_mean = naive_total / N

kahan_mean = robust_mean(vals)

print("naive_running_mean =", repr(naive_mean))
print("naive abs error from 0.1 =", abs(naive_mean - 0.1))
print("kahan robust_mean =", repr(kahan_mean))
print("kahan abs error from 0.1 =", abs(kahan_mean - 0.1))

Actual output:

naive_running_mean = 0.09999999998389754
naive abs error from 0.1 = 1.6102466582346153e-11
kahan robust_mean = 0.1
kahan abs error from 0.1 = 0.0

Summing ten million copies of 0.1 with a naive running total gives naive_running_mean = 0.09999999998389754, an absolute error of about 1.6e-11 from the true value 0.1. The Kahan-based robust_mean on the identical input returned exactly 0.1, a measured error of 0.0. This is not a marginal difference: naive summation accumulates rounding error roughly proportional to the number of additions, while Kahan's compensation term cancels the dominant rounding error at each step, which is why it is the standard choice once an aggregation runs over many elements or elements of very different magnitudes.

Complexity and assumptions
O(n) time and O(1) additional space, a single pass over the input. The function assumes IEEE-754 double-precision floats (Python's native float), so float('nan') != float('nan') and NaN-detection must use math.isnan, never ==; it also assumes the caller wants NaN and None values silently excluded rather than treated as an error, which is a documented contract choice, not a universal default (a stricter variant might instead raise on the first NaN it encounters).

Trade-offs & pitfalls
Without the explicit infinity handling, this exact function would ship a plausible-looking, executes-without-error implementation that returns a silently wrong nan for a perfectly reasonable input (one infinite value among otherwise-finite ones) instead of raising or returning inf; this is precisely the kind of bug that a code-reading review would very likely miss, because Kahan summation's compensation formula reads as obviously correct arithmetic unless you specifically trace what happens when t becomes infinite.

Direct answer
Floating-point numbers cannot exactly represent most decimal fractions (0.1 in binary floating point is a repeating fraction, just as 1/3 is in decimal), so comparing floats with strict equality (==) is unreliable; the fix is to compare within a tolerance, using a RELATIVE tolerance for large-magnitude values and an ABSOLUTE tolerance as a floor for values near zero, since either one alone fails in a different regime.

Structured elaboration and worked example (executed)

import math

def approx_equal(a, b, rel_tol=1e-9, abs_tol=1e-12):
    return math.isclose(a, b, rel_tol=rel_tol, abs_tol=abs_tol)

print("0.1 + 0.2 == 0.3 ->", 0.1 + 0.2 == 0.3)
print("0.1 + 0.2 =", repr(0.1 + 0.2))
print("approx_equal(0.1+0.2, 0.3) ->", approx_equal(0.1+0.2, 0.3))

Running this:

0.1 + 0.2 == 0.3 -> False
0.1 + 0.2 = 0.30000000000000004
approx_equal(0.1+0.2, 0.3) -> True

The strict-equality check is False even though the values are 'the same' for any practical purpose, because 0.1 and 0.2 cannot be represented exactly in binary floating point and their sum accumulates a tiny representation error.

Why relative tolerance alone fails near zero (executed)

a, b = 1e-300, 2e-300
print("math.isclose(a,b) default (rel only) ->", math.isclose(a, b))
print("math.isclose(a,b, abs_tol=1e-12) ->", math.isclose(a, b, abs_tol=1e-12))

Actual output:

math.isclose(a,b) default (rel only) -> False
math.isclose(a,b, abs_tol=1e-12) -> True

Two numbers that are both astronomically small but differ by a factor of 2 (1e-300 vs 2e-300) fail a relative-tolerance-only check, correctly by relative-difference logic, but this is almost always NOT what a test author actually wants: near zero, tiny absolute differences are usually noise, not a meaningful failure, which is why an absolute tolerance floor is needed as a companion check.

Why absolute tolerance alone fails for large numbers (executed)

c, d = 1e15, 1e15 + 100
print("abs diff:", abs(c - d))
print("relative diff:", abs(c - d) / max(abs(c), abs(d)))
print("math.isclose(c, d, rel_tol=1e-9) ->", math.isclose(c, d, rel_tol=1e-9))
print("abs(c-d) < 0.01 ->", abs(c - d) < 0.01)

Actual output:

abs diff: 100.0
relative diff: 9.999999999999e-14
math.isclose(c, d, rel_tol=1e-9) -> True
abs(c-d) < 0.01 -> False

An absolute-tolerance-only check (e.g. abs(a - b) < 0.01) would FAIL this case, flagging two numbers that differ by only about one part in ten trillion as unequal, purely because their magnitude is large; relative tolerance correctly recognizes this as an insignificant difference.

Code-level best practices

1. Use a library function (math.isclose in Python, assertAlmostEqual/an epsilon-based custom matcher elsewhere) rather than hand-rolling abs(a-b) < 0.0001, since a hardcoded absolute epsilon silently breaks at both extremes shown above.
2. Always pass BOTH rel_tol and abs_tol explicitly rather than relying on library defaults, and choose abs_tol based on the smallest meaningful magnitude your domain actually produces (a physics simulation and a financial percentage calculation have very different notions of 'negligible').
3. Never use exact equality on any float that has passed through at least one arithmetic operation (addition, division, an accumulated sum); exact equality is only safe for a float that was directly assigned a literal and never recomputed.

Trade-offs & pitfalls
A tolerance that is too loose can mask a genuine regression (a calculation that is now systematically off by a small but real amount gets silently accepted), while a tolerance that is too tight reintroduces flaky test failures from ordinary floating-point noise across platforms or numeric library versions; the tolerance value itself is a design decision that belongs in code review, not a default nobody revisits.