Error Handling and Defensive Programming Questions

Direct answer
Build alerting on SLOs (not raw error counts) so pages correlate with actual user-facing pain: use an error BUDGET to gate how aggressively you alert, distinguish symptom alerts (something is actually broken for users right now) from cause alerts (a likely contributing factor, lower urgency), and require a minimum sample size before firing so low-traffic noise doesn't page anyone.

Structured elaboration

• Metrics feeding the SLO: typically a ratio (successful requests / total requests) measured against an explicit target (99.9% success over a rolling window), NOT a raw error count, since raw counts don't distinguish '10 errors out of 100 requests' (10% error rate, bad) from '10 errors out of 1,000,000' (negligible).
• Symptom vs cause alerts: a symptom alert ('user-facing error rate exceeds SLO threshold') should page immediately, since it directly reflects user pain; a cause alert ('CPU usage is elevated') is a likely contributing factor but not itself proof of user impact, and should typically be visible on a dashboard or trigger a lower-urgency notification rather than a page, since elevated CPU alone doesn't necessarily mean anyone is affected.
• Error budgets gating alerts: track how much of the SLO's allowed error budget has been consumed over the current period; alert more aggressively (shorter time windows, lower thresholds) as the budget depletes, since burning through budget fast (a 'fast burn') threatens the SLO commitment much sooner than a slow, steady trickle of errors within otherwise-normal bounds.
• Minimum sample size: error_ratio > 1% computed over 5 requests is meaningless noise; requiring a minimum request count (e.g. 1000) before evaluating the ratio prevents low-traffic periods (overnight, for a low-volume service) from producing statistically meaningless alerts.
• Grouping/sampling for a noisy downstream integration: a specific downstream integration that is chronically flaky in a way that is well understood and already tolerated (a third-party API with a known baseline 2% error rate that the product already accounts for) will otherwise dominate a shared SLO's error budget and drown out signal from everything else calling into the same service; carve that dependency's calls into their OWN separate SLO/alert group (or exclude its known-tolerated error class from the primary SLO's numerator entirely) and sample its errors at a lower rate for logging/tracing purposes than a genuinely unexpected error type, so the noisy-but-understood integration doesn't consume alert budget or storage that a rare, unexpected failure needs.
• Combining static thresholds with anomaly detection: a static threshold (error ratio > 1%) is simple and predictable but can't distinguish 'normal daily variance' from a genuine anomaly for a service whose baseline error rate itself fluctuates; layering an anomaly-detection signal (current rate significantly deviates from the same time-of-day/day-of-week historical baseline) catches issues a fixed threshold misses without needing to hand-tune a threshold per time-of-day.

Worked example
PromQL-style expression: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.01 and sum(rate(http_requests_total[5m])) * 300 > 1000 fires only when the 5-minute error ratio exceeds 1% AND the 5-minute window saw at least 1000 total requests, avoiding a false alarm during a low-traffic window where a handful of errors could otherwise spike the ratio meaninglessly.

Trade-offs and pitfalls
An SLO-based alert threshold set too close to the actual SLO target itself pages on every minor blip that doesn't genuinely threaten the SLO commitment over its real measurement window; the fast-burn/slow-burn distinction (alert urgently on a fast burn rate that would exhaust the budget in hours, alert less urgently on a slow burn that would exhaust it over weeks) is what lets you page appropriately without either missing a real emergency or drowning on-call in noise from ordinary variance.

Direct answer
A safe graceful shutdown stops accepting new work first, then drains in-flight work within a bounded timeout, then exits, coordinated by trapping the termination signal and using Kubernetes readiness/liveness probes so the load balancer stops routing new traffic to this instance before the drain even begins.

Structured elaboration

• Stop accepting new requests: on receiving SIGTERM, immediately flip the readiness probe to unhealthy (so Kubernetes/the load balancer stops routing NEW traffic here within its own health-check interval) WITHOUT yet killing the process; this ordering matters, since killing the process before the load balancer notices the readiness change would drop in-flight AND newly-routed requests both.
• Signals to trap: trap SIGTERM specifically, since that is the signal Kubernetes (and orchestrators generally) send on pod termination as the standard 'please shut down gracefully' request; also trap SIGINT for local development, since Ctrl+C sends SIGINT, not SIGTERM, and a handler that only traps SIGTERM won't drain cleanly when a developer stops the process locally. SIGKILL cannot be trapped or handled at all, since it terminates the process immediately at the OS level, which is exactly why the app's own drain timeout must stay comfortably under Kubernetes's terminationGracePeriodSeconds: exceeding that ceiling means the orchestrator escalates to an untrappable SIGKILL, skipping the graceful sequence entirely.
• Drain in-flight requests: wait for currently-processing requests to complete naturally, up to a bounded timeout (commonly matching or slightly exceeding the load balancer's own deregistration delay), rather than killing them immediately.
• Background jobs: any background work (a queue consumer, a scheduled task) needs its own explicit shutdown signal, distinct from just letting HTTP requests drain, since it isn't tied to the request lifecycle at all.
• Liveness vs readiness: readiness controls whether traffic is routed here at all (flip this first, on SIGTERM); liveness controls whether Kubernetes considers the process healthy enough to keep running (should stay healthy throughout the drain, so Kubernetes doesn't forcibly kill the pod mid-drain via a liveness-triggered restart).
• Coordinating worker threads/asyncio tasks: signal each in-flight handler/task that shutdown has begun (an event/flag they check between logical steps) and await their completion with a timeout, force-terminating anything still running past the timeout rather than waiting indefinitely.

Worked example
Sequence on SIGTERM: (1) immediately set readiness probe to unhealthy (Kubernetes stops sending new traffic within its next health-check interval, typically a few seconds); (2) wait a short buffer (a few seconds) to let the deregistration actually propagate through the load balancer before the app itself starts refusing connections, avoiding a race where a request is routed here right as it stops accepting; (3) stop accepting new connections at the app layer; (4) wait up to 30 seconds for in-flight requests to complete, tracked via an active-request counter; (5) once the counter hits zero (or the 30s timeout expires, whichever first), exit cleanly.

Trade-offs and pitfalls
A drain timeout set too short forcibly kills legitimately-still-processing requests (a long-running upload, a slow downstream call), which produces exactly the abrupt failures graceful shutdown exists to avoid; too long delays deployments/scale-down events unnecessarily and risks Kubernetes eventually SIGKILLing the pod anyway once its own terminationGracePeriodSeconds elapses, which is a hard, non-negotiable ceiling on however generous your own drain timeout is, so the two need to be configured consistently (your app's drain timeout should always be comfortably shorter than Kubernetes's grace period, not equal to or longer than it).

Direct answer
Detect a crashed worker in distributed data-parallel training (including a SILENT failure like a stuck NCCL collective, which doesn't raise an exception but instead hangs the whole job) via a heartbeat/timeout mechanism on collective operations, checkpoint frequently enough to bound redo work, and ensure the RESUMED job's optimizer state (not just the model weights) stays consistent with the resumed parameters.

Structured elaboration

• Detecting a stuck collective op: a genuinely crashed process (killed, OOM'd) is usually detectable quickly via the process manager; a silently-hung process (alive, but stuck in a bad state, never calling into the collective op the rest of the group is waiting on) is harder, since the rest of the group just blocks waiting forever; a WATCHDOG timeout on collective operations (if an all-reduce hasn't completed within an expected bound, treat it as a failure and trigger recovery) is the standard defense, since a genuinely healthy job's collective ops should complete within a predictable time window.
• Checkpointing for resume: checkpoint model weights AND optimizer state (not weights alone) at a regular interval; Adam's moment estimates specifically need to be checkpointed and restored together with the weights, since resuming with fresh (zeroed) optimizer state after a long training run effectively restarts the optimizer's adaptive learning-rate behavior, which can visibly disrupt training dynamics even though the model weights themselves are correct.
• Synchronous vs asynchronous update trade-offs: synchronous (all-reduce, every worker waits for every other worker each step) gives clean, easy-to-reason-about consistency but means ONE slow or crashed worker blocks the entire group; asynchronous (each worker updates independently, tolerating stale gradients from slower peers) tolerates a slow/crashed worker without blocking everyone else, at the cost of training-dynamics complexity (stale gradients can, in some regimes, meaningfully hurt convergence) that synchronous training avoids.
• Master/coordinator failover: if using a parameter-server or coordinator-based architecture, the coordinator itself needs its own failover plan (a standby that can take over, informed by the same checkpoint the workers resume from), since a coordinator crash without failover stalls the whole job even if every worker is individually healthy.

Worked example
A 64-GPU training job uses synchronous all-reduce; worker 23 hits a driver-level hang (not a crash, no exception, no process exit) during a collective op. Without a watchdog, the other 63 workers block forever waiting for worker 23's participation in the all-reduce. With a watchdog timeout (say, 3x the median observed all-reduce duration), the job detects the stall, force-restarts worker 23's process (or the whole job, depending on the orchestration's granularity), and resumes all 64 workers from the last synchronized checkpoint, including optimizer state, so training dynamics continue smoothly rather than restarting the optimizer's adaptive state from scratch.

Trade-offs and pitfalls
Setting the watchdog timeout too aggressively (close to the median expected duration) produces false-positive restarts during normal, benign variance in collective-op timing (network jitter, a momentarily busy node); too generous delays detecting a genuine hang, wasting expensive GPU-hours idling. Tune the threshold from measured p99 (not median) collective-op latency under normal operation, with meaningful headroom above it.

Direct answer
A token-bucket rate limiter tracks a token count that refills continuously at a configured rate up to a maximum burst capacity, and tryAcquire() atomically checks and deducts the requested tokens, allowing a burst up to the bucket's full capacity while enforcing a steady sustained rate over time, implemented thread-safely with a lock (or a compare-and-swap loop) around the refill-and-deduct sequence.

Structured elaboration

• Refill logic: compute elapsed time since the last refill, add elapsed * refill_rate tokens (capped at the bucket's capacity), then check if enough tokens are available for this request; doing the refill calculation LAZILY (only when tryAcquire is called, based on elapsed real time) avoids needing a separate background thread just to periodically top up the bucket.
• Burst vs sustained rate: the bucket's CAPACITY governs how large a burst is allowed at once (a capacity of 5 lets 5 requests through instantly even if none have been made recently); the REFILL RATE governs the sustained long-term throughput once the initial burst is exhausted.
• Concurrency primitives: a single lock around the refill-check-deduct sequence is the simplest correct implementation; under high contention, a compare-and-swap loop on the token count (treating it as an atomic value) can reduce lock contention, at added implementation complexity, following the same simple-lock-first, atomic-primitive-if-profiling-justifies-it trade-off discussed in the LRU-cache survivor.
• Complexity: tryAcquire() is O(1) (a fixed amount of arithmetic plus a lock acquisition), making it cheap enough to call on every request even at high QPS.

Worked example (executed with a fake, controllable clock; all assertions passed)

class TokenBucket:
    def __init__(self, capacity, refill_rate, clock=time.monotonic):
        self.capacity = capacity
        self.refill_rate = refill_rate
        self._tokens = capacity
        self._last = clock()
        self._clock = clock
        self._lock = threading.Lock()

    def _refill(self):
        now = self._clock()
        elapsed = now - self._last
        self._tokens = min(self.capacity, self._tokens + elapsed * self.refill_rate)
        self._last = now

    def try_acquire(self, n=1):
        with self._lock:
            self._refill()
            if self._tokens >= n:
                self._tokens -= n
                return True
            return False

Verified with capacity=5, refill_rate=1.0: exactly 5 consecutive calls succeed immediately (the full burst), a 6th fails (bucket exhausted); after advancing the fake clock by 3 seconds (3 tokens' worth of refill at rate 1.0/s), exactly 3 more calls succeed and a 4th fails, confirming both the burst-capacity ceiling and the sustained refill rate behave as designed.

Trade-offs and pitfalls
A distributed system with MULTIPLE instances each running their own LOCAL token bucket effectively multiplies the aggregate allowed rate by the number of instances (each instance independently allows its own configured burst/rate), which may or may not be the intended behavior; if a GLOBAL rate limit across the whole fleet is required (not per-instance), the bucket state needs to live in a shared store (Redis, with the refill-and-deduct logic implemented as an atomic Lua script or similar), trading the simplicity and low latency of a local, in-memory bucket for a coordinated, fleet-wide guarantee.

Direct answer
Combine exponential backoff, jitter, a maximum attempt count, and a circuit breaker: backoff and jitter spread out retries from a single client to avoid overwhelming a recovering dependency; the circuit breaker stops retrying entirely once it's clear the dependency is down, protecting it from a fleet of clients retrying in aggregate even if each one individually backs off.

Structured elaboration

• Exponential backoff: each retry waits longer than the last (base * 2^attempt), so a transient blip gets a quick retry while a longer outage gets progressively less aggressive hammering.
• Jitter: without randomization, many clients that all failed at the same moment (a brief outage) will all retry at the same moment again, in near-lockstep, recreating the exact overload that caused the original failure; full jitter (random.uniform(0, delay)) spreads retries across the window instead of clustering them.
• Maximum attempts: bounds how long a single request can keep retrying, so a caller gets a definitive failure rather than retrying forever against a truly-down dependency.
• Circuit breaker integration: even with backoff and jitter, if the dependency is genuinely down, EVERY client independently retrying (each with its own reasonable backoff) still adds up to significant aggregate load; a circuit breaker that opens after a threshold of failures stops NEW attempts outright for a cooldown period, which backoff alone cannot do since backoff only slows one client's own requests, not the fleet's aggregate rate.

Worked example
func callWithResilience(fn, breaker): first checks breaker.state; if OPEN, fail immediately without even attempting a call (protecting the dependency and returning fast to the caller); if CLOSED or HALF_OPEN, call the retry-with-backoff-and-jitter wrapper around fn, and feed each attempt's success/failure back into the breaker's _on_success/_on_failure. This way, the fast path (dependency healthy) pays only the retry logic's minimal overhead, and the failure path (dependency down) fails fast for most callers once the breaker opens, rather than every caller separately discovering the outage through its own multi-attempt backoff sequence.

Trade-offs and pitfalls
A thundering herd can still occur even with per-client jitter if the jitter's RANGE is too narrow relative to the number of clients (thousands of clients jittering across only a 100ms window still cluster meaningfully); size the jitter window relative to the expected fleet size and the dependency's actual recovery time, not an arbitrary constant. Retrying at all is only safe for idempotent operations; wrapping a non-idempotent write (charging a card) in this same retry logic without an idempotency key turns a resilience pattern into a duplicate-side-effect bug generator.