Definition:A retry budget is a deliberately limited pool of allowed retry attempts (extra requests) to flaky dependencies. It prevents retries from amplifying load during partial failures, protecting the overall system and preserving the primary traffic SLOs.Sizing:- Start from SLO/error budget: if downstream SLO allows X% errors per minute, convert that to allowed extra request volume. Example: incoming steady R requests/s and SLO error budget E% over window W → allowed_error_requests = R * W * E. Reserve a fraction f (e.g., 50–80%) of that as retry capacity to avoid exhausting SLO quickly.- Or set a max retry ratio α (typical 5–20%): allowed_retry_rps = α * R. Combine both and take the minimum: retry_budget_rps = min(allowed_error_requests/W, α * R).- Apply per-caller and global caps: per-instance cap = retry_budget_rps / N_callers (or weighted by traffic).Enforcement (client libraries):- Local token-bucket: each client instance maintains a retry token bucket sized from per-caller cap. A retry consumes a token; tokens refill at configured rps.- Backoff + jitter and respect Retry-After and idempotency: only retry idempotent ops by default; exponential backoff with full jitter.- Circuit breaker integration: if downstream error rate or latency exceeds threshold, stop retries and open circuit.- Metrics: expose metrics (retries_consumed, retry_denied, retry_latency) for alerting.Enforcement (service boundary / gateway):- Global token-bucket or leaky-bucket at API gateway or edge proxy to enforce global retry_rps. Reject excess retries with 429 + Retry-After or return a short-circuit error to callers.- Differentiate original requests vs retries using headers (X-Retry-Count, X-Original-Request-Id) to avoid counting originals toward retry quota.- Priority queuing: prioritize fresh traffic over retries during degraded states.Implementation example (token bucket pseudocode):python
# per-client token bucket
class RetryBucket:
def __init__(self, rate, capacity):
self.rate = rate
self.capacity = capacity
self.tokens = capacity
self.last = time.time()
def allow(self, cost=1):
now = time.time()
self.tokens = min(self.capacity, self.tokens + (now - self.last) * self.rate)
self.last = now
if self.tokens >= cost:
self.tokens -= cost
return True
return False
