Approach: use exponential backoff base * 2^attempt, clamp to max_delay, and apply jitter mode to spread retries. Provide sensible defaults (base=100ms, max_delay=10s, max_retries=5) to avoid thundering herd. Include modes: none, full, equal, decorrelated.python
# pseudocode / Python-like
import random, time
def backoff_delay(attempt, base=0.1, max_delay=10.0, cap_retries=5, jitter='full', prev_delay=None):
"""
attempt: 0-based attempt number
base: base delay seconds
max_delay: max single delay seconds
cap_retries: maximum attempts (caller should enforce)
jitter: 'none'|'full'|'equal'|'decorrelated'
prev_delay: previous delay for decorrelated jitter
returns (delay_seconds, next_prev_delay)
"""
expo = base * (2 ** attempt)
unjittered = min(expo, max_delay)
if jitter == 'none':
delay = unjittered
elif jitter == 'full':
# uniform between 0 and unjittered
delay = random.uniform(0, unjittered)
elif jitter == 'equal':
# half of unjittered plus uniform half
half = unjittered / 2.0
delay = half + random.uniform(0, half)
elif jitter == 'decorrelated':
# from "Exponential Backoff And Jitter" blog
# new = min(max(base, random(prev_delay * 3, unjittered)), max_delay)
if prev_delay is None:
prev = base
else:
prev = prev_delay
# choose random between base and prev*3, then clamp
candidate = random.uniform(base, prev * 3)
delay = min(candidate, max_delay)
else:
raise ValueError("unknown jitter")
return delay, delay # return delay and store as prev_delay for decorrelated
Key concepts and reasoning:- Jitter spreads retries across time windows to prevent synchronized retry spikes.- Modes: - none: deterministic, simple, but causes synchronized bursts. - full jitter: pick uniform[0,expo]; maximum spread, reduces worst-case contention. - equal jitter: deterministic half + random half — good balance between latency and smoothing. - decorrelated jitter: uses previous delay to avoid correlation and quickly decorrelate bursts while keeping average reasonable.- Caps: - max_retries prevents infinite loops. - max_delay bounds long waits and resource tying. - total_max_backoff (optional) bounds cumulative wait.- Sensible defaults to avoid thundering-herd: - base 100ms, max_delay 10s, max_retries 5, jitter='full' or 'equal' for production. - add random initial jitter for immediate requests.Unit testing under failure injection:- Deterministic RNG: inject a pseudo-random generator or seed to make outputs deterministic.- Tests: - verify delay formula without jitter (jitter='none'). - for full/equal/decorrelated, assert delay within expected ranges for several attempts. - statistical tests: run many trials and assert distribution properties (mean close to expected, low clustering). - concurrency test: simulate N clients failing simultaneously; assert retry timestamps are sufficiently spread (e.g., low probability of >K collisions). - edge cases: attempt beyond cap_retries raises/returns sentinel; max_delay enforced; decorrelated uses prev_delay correctly.- Use time mocking (fake clock) to avoid real sleeps; inject hooks to collect intended scheduled times.Trade-offs:- Full jitter minimizes contention but increases average latency. Equal/decorrelated balance latency vs spread.- Provide configuration for critical low-latency paths (smaller base, fewer retries) vs resilient background jobs (larger caps).