Error Handling and Defensive Programming Questions

Direct answer
Unit-test retry logic by mocking the dependency to fail a controlled number of times then succeed (or fail permanently), asserting the FINAL result and call count are correct, and asserting backoff delays algorithmically (checking the computed delay VALUES passed to a mocked sleep function) rather than actually sleeping in the test, which would make the test both slow and flaky.

Structured elaboration

• Transient-then-success case: configure the mock's side_effect to raise on the first call(s) and return a value on a later call; assert the retry function's final return value matches the eventual success, and assert the mock was called the EXPECTED number of times (not just 'it eventually succeeded', which alone wouldn't catch a bug where it retried far more or fewer times than intended).
• Permanent-failure case: configure the mock to always raise; assert the retry function raises the SAME (or an appropriately wrapped) exception after exactly max_attempts calls, not more and not fewer.
• Backoff-delay assertion, algorithmic not wall-clock: inject a fake sleep function (a list-appending lambda, or a MagicMock) instead of real time.sleep, and assert the SEQUENCE of delay values passed to it matches the expected exponential progression; this is both instant (no real waiting) and deterministic (no flakiness from actual timing variance), which is exactly what the reproducibility rule against wall-clock-dependent tests requires.

Worked example (executed; all assertions passed)

from unittest.mock import MagicMock

mock_client = MagicMock()
mock_client.get.side_effect = [ConnectionError("timeout"), "success-response"]
result = retry(lambda: mock_client.get("/x"), max_attempts=5, base_delay=0.01)
assert result == "success-response"
assert mock_client.get.call_count == 2

mock_client2 = MagicMock()
mock_client2.get.side_effect = ConnectionError("still down")
try:
    retry(lambda: mock_client2.get("/y"), max_attempts=3, base_delay=0.001)
except ConnectionError:
    pass
assert mock_client2.get.call_count == 3

recorded_delays = []
mock_client3 = MagicMock()
mock_client3.get.side_effect = [ConnectionError()] * 3 + ["ok"]
retry(lambda: mock_client3.get("/z"), max_attempts=5, base_delay=0.1, sleep=recorded_delays.append)
assert recorded_delays == [0.1, 0.2, 0.4]

Verified: all three scenarios pass, including the exact exponential progression [0.1, 0.2, 0.4] for three consecutive failures with base_delay=0.1, confirmed algorithmically rather than by measuring real elapsed wall-clock time.

Trade-offs and pitfalls
A test that actually calls real time.sleep() for the backoff delays technically 'works' but is slow to run (compounding across a full test suite) and, worse, can become genuinely flaky if the retry logic includes JITTER (a randomized component), since asserting an EXACT delay value against a jittered implementation would fail intermittently; for a jittered version, assert the delay falls within the EXPECTED BOUNDS (0 <= delay <= computed_max) rather than an exact value, which is the adjustment needed when testing the full-jitter variant rather than this plain-exponential one.

Direct answer
Introduce team-level defensive-coding practices over a quarter through a phased rollout (define the practices, socialize and get buy-in, enforce via review/tooling, measure results) rather than a single mandate, since a practice imposed without buy-in or automated enforcement reliably decays back to old habits within weeks.

Structured elaboration

• Phase 1 (weeks 1-2): define and socialize: write down the SPECIFIC practices (not 'write better error handling' but concrete rules: no bare except, every public function validates its inputs, every resource-acquiring block uses a context manager) and discuss them WITH the team, incorporating their pushback, rather than presenting a finished mandate top-down.
• Testing quotas: pair the code-review rules with a lightweight, enforceable minimum (no PR touching error-handling code merges without at least one new test covering the failure path), tracked via a coverage-delta check in CI rather than left to reviewer memory, since 'sparse tests' was named alongside poor error handling as one of the two root problems and needs its own concrete lever, not just an assumption that better error-handling rules will incidentally produce more tests.
• Phase 2 (weeks 3-6): tooling and templates: back the practices with automated enforcement where possible (a lint rule catching the worst offenders) and a PR template/checklist reminding reviewers to check for the rest, so the practice doesn't rely purely on every individual remembering it every time.
• Phase 3 (weeks 7-10): review-driven enforcement: make the practices an explicit part of code review, with the manager (you) modeling the review comments initially so the team sees the calibration (how strict is too strict) rather than each reviewer independently guessing.
• Phase 4 (weeks 11-13): measure and adjust: track a concrete metric (incidents traced to the target failure classes, or a code-quality proxy like lint-rule violation trend) and share the result with the team AND with product, closing the loop on whether the investment paid off.
• Getting buy-in from product: frame the ask in terms product cares about (fewer firefighting-driven schedule disruptions, more predictable delivery) rather than purely as an engineering-quality initiative, and be explicit about the SHORT-TERM velocity cost (code review will be slightly slower initially) versus the medium-term payoff (fewer incidents pulling engineers off roadmap work).

Worked example
Week 1: propose 4 specific rules to the team in a design discussion, incorporating feedback that 2 of the originally-proposed rules were too strict for legacy code paths and should apply to new code only, and agree on the testing quota (one new test per error-handling fix); week 3: ship a lint rule catching bare-except patterns and a CI coverage-delta check enforcing the testing quota, both added as warnings (not yet blocking); week 7: flip the lint rule and the coverage-delta check to blocking for new code, with review-checklist backing for the harder-to-automate rules; week 13: present to the team and to product that incidents in the target category dropped from 3/month to 0/month over the quarter, with review turnaround time increasing by a measured (and acceptable) 10%, framing this as a concrete trade the data supports continuing.

Trade-offs and pitfalls
A rollout that skips the socialization/buy-in phase and goes straight to enforcement generates resentment and passive resistance (reflexive suppression comments, grudging compliance without real behavior change); a rollout that socializes endlessly without ever reaching enforcement never actually changes anything, since good intentions alone don't survive contact with a looming deadline. The phased structure exists specifically to avoid both failure modes.

Direct answer
Build alerting on SLOs (not raw error counts) so pages correlate with actual user-facing pain: use an error BUDGET to gate how aggressively you alert, distinguish symptom alerts (something is actually broken for users right now) from cause alerts (a likely contributing factor, lower urgency), and require a minimum sample size before firing so low-traffic noise doesn't page anyone.

Structured elaboration

• Metrics feeding the SLO: typically a ratio (successful requests / total requests) measured against an explicit target (99.9% success over a rolling window), NOT a raw error count, since raw counts don't distinguish '10 errors out of 100 requests' (10% error rate, bad) from '10 errors out of 1,000,000' (negligible).
• Symptom vs cause alerts: a symptom alert ('user-facing error rate exceeds SLO threshold') should page immediately, since it directly reflects user pain; a cause alert ('CPU usage is elevated') is a likely contributing factor but not itself proof of user impact, and should typically be visible on a dashboard or trigger a lower-urgency notification rather than a page, since elevated CPU alone doesn't necessarily mean anyone is affected.
• Error budgets gating alerts: track how much of the SLO's allowed error budget has been consumed over the current period; alert more aggressively (shorter time windows, lower thresholds) as the budget depletes, since burning through budget fast (a 'fast burn') threatens the SLO commitment much sooner than a slow, steady trickle of errors within otherwise-normal bounds.
• Minimum sample size: error_ratio > 1% computed over 5 requests is meaningless noise; requiring a minimum request count (e.g. 1000) before evaluating the ratio prevents low-traffic periods (overnight, for a low-volume service) from producing statistically meaningless alerts.
• Grouping/sampling for a noisy downstream integration: a specific downstream integration that is chronically flaky in a way that is well understood and already tolerated (a third-party API with a known baseline 2% error rate that the product already accounts for) will otherwise dominate a shared SLO's error budget and drown out signal from everything else calling into the same service; carve that dependency's calls into their OWN separate SLO/alert group (or exclude its known-tolerated error class from the primary SLO's numerator entirely) and sample its errors at a lower rate for logging/tracing purposes than a genuinely unexpected error type, so the noisy-but-understood integration doesn't consume alert budget or storage that a rare, unexpected failure needs.
• Combining static thresholds with anomaly detection: a static threshold (error ratio > 1%) is simple and predictable but can't distinguish 'normal daily variance' from a genuine anomaly for a service whose baseline error rate itself fluctuates; layering an anomaly-detection signal (current rate significantly deviates from the same time-of-day/day-of-week historical baseline) catches issues a fixed threshold misses without needing to hand-tune a threshold per time-of-day.

Worked example
PromQL-style expression: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.01 and sum(rate(http_requests_total[5m])) * 300 > 1000 fires only when the 5-minute error ratio exceeds 1% AND the 5-minute window saw at least 1000 total requests, avoiding a false alarm during a low-traffic window where a handful of errors could otherwise spike the ratio meaninglessly.

Trade-offs and pitfalls
An SLO-based alert threshold set too close to the actual SLO target itself pages on every minor blip that doesn't genuinely threaten the SLO commitment over its real measurement window; the fast-burn/slow-burn distinction (alert urgently on a fast burn rate that would exhaust the budget in hours, alert less urgently on a slow burn that would exhaust it over weeks) is what lets you page appropriately without either missing a real emergency or drowning on-call in noise from ordinary variance.

Direct answer
Coordinate retries across a whole fleet of clients (not just within one client) using either client-side randomized backoff (simple, no new infrastructure, statistically effective at scale but not deterministic), a centralized rate limiter (deterministic and tunable, but a new single point of failure and added latency), or a token-bucket approach (a middle ground: bounded, predictable throughput without a per-request round trip to a central service).

Structured elaboration

• Client-side randomized backoff: each client independently jitters its own retry timing; at fleet scale, the LAW OF LARGE NUMBERS smooths the aggregate retry rate even though no single client coordinates with any other. Simple and resilient (no new dependency), but gives you no hard guarantee on the aggregate retry rate, and a partial outage that resolves right as many clients' jittered windows happen to overlap can still produce a smaller thundering herd.
• Centralized rate limiter: every client checks in with a shared service before retrying, which gives a precise, enforceable cap on aggregate retry rate; the cost is the round trip's added latency on every retry decision, and the rate limiter itself becomes a new dependency that must be more available than the thing it's protecting, or you've just moved the single point of failure.
• Token bucket (typically local, refilled from a shared quota periodically, or fully distributed via a shared store): clients draw from a bucket that refills at a controlled rate; this bounds the aggregate rate without a synchronous round trip on every single retry decision, at the cost of the bucket's refill logic itself needing to be correctly distributed (or accepting some imprecision from local-only buckets, similar to the local-vs-shared circuit-breaker trade-off).
• Per-dependency policies: different downstream dependencies have different real capacity; a client library configuring one global backoff policy for every dependency it calls under-protects a fragile dependency and over-throttles a robust one, so policies should be keyed per dependency, ideally informed by that dependency's own observed error budget and latency (adapting backoff parameters dynamically rather than using a fixed constant).

Worked example
A payment gateway experiences a brief outage; 10,000 clients across a fleet all get a 503 within the same second. With pure client-side jitter alone (say, a 0-5 second uniform window), roughly 2,000 clients still retry within any given one-second slice, which can still be a meaningful spike; layering a centralized token bucket capped at 500 requests/second specifically for THIS dependency smooths that into a bounded, predictable ramp, letting the gateway recover without a second self-inflicted overload from the retry wave itself.

Trade-offs and pitfalls
The common mistake is treating this as an either/or choice: client-side jitter and a centralized (or token-bucket) rate limit are complementary, not competing, layers, since jitter reduces the SEVERITY of any coordination gap and the rate limiter provides the hard guarantee jitter alone can't. Operational complexity scales with the coordination mechanism chosen: pure jitter needs no new infrastructure at all, while a centralized limiter needs its own high-availability design, ironically often more demanding than the dependency it's protecting.

Direct answer
Avoid GPU out-of-memory during training or inference with a layered set of strategies (dynamic batch sizing, gradient accumulation, model sharding, mixed precision, activation offloading) chosen based on where the actual memory pressure originates (activations vs weights vs optimizer state), backed by monitoring that lets you detect pressure BEFORE it becomes a hard OOM crash.

Structured elaboration and per-strategy trade-offs

• Dynamic batch-size adaptation: reduce batch size when memory pressure is detected (measured via nvidia-smi/framework memory APIs) and increase it back when headroom allows; simple, broadly applicable, but changes training dynamics (smaller effective batch size) if not compensated for.
• Gradient accumulation: accumulate gradients over several smaller 'micro-batches' before applying an optimizer step, achieving a large EFFECTIVE batch size within a small ACTUAL per-step memory footprint; trades wall-clock speed (more forward/backward passes per optimizer step) for memory.
• Model sharding (data-parallel with sharded optimizer state, e.g. ZeRO/FSDP): splits the model's weights/optimizer state across multiple devices instead of replicating it fully on each; necessary once a single model no longer fits on one device at all, at the cost of added communication overhead between devices.
• Mixed precision: halves memory footprint for activations/weights (fp16/bf16 vs fp32) with the numeric-stability caveats covered in the companion survivor; usually the first, cheapest lever to pull.
• Activation offloading: move activations that aren't immediately needed to CPU memory (or recompute them on-demand via gradient checkpointing) rather than keeping every layer's activations on GPU simultaneously; trades compute (recomputation) or PCIe transfer time for GPU memory.
• Monitoring to detect pressure early: track GPU memory utilization as a first-class metric, alerting on a sustained high-watermark trend, not just on an actual OOM crash, so you can proactively adjust batch size or trigger a scale-out before a hard failure interrupts a long-running job.
• Safe failure handling when memory is still insufficient: catch the framework's OOM exception specifically, checkpoint the current training state immediately (before any further allocation attempts that might themselves fail), reduce batch size or clear cached memory, and retry rather than letting the whole job crash and lose hours of progress.

Worked example
A training job hits intermittent OOM on a subset of unusually large input sequences: rather than crashing the whole multi-hour job, the training loop catches the framework's OutOfMemoryError specifically around the optimizer step, calls the framework's cache-clearing function, halves the batch size for the next few steps, and logs the incident; if OOM persists even at the reduced batch size, it checkpoints and exits cleanly (rather than repeatedly crash-looping), letting an operator investigate rather than burning compute on a doomed retry loop.

Trade-offs and pitfalls
Each strategy trades a DIFFERENT resource for memory (gradient accumulation trades wall-clock time, sharding trades network communication, offloading trades PCIe bandwidth and compute for recomputation); picking the wrong one for your actual bottleneck (say, adding gradient accumulation when the real problem is a single oversized activation tensor that gradient accumulation doesn't touch at all) wastes engineering effort without solving the actual OOM. Diagnose WHERE the memory is going (weights, optimizer state, or activations, via a memory profiler) before choosing which lever to pull.