Approach: create a decorator that retries a function on exceptions up to max_attempts, using exponential backoff (delay *= 2 each retry), clamped to max_delay, and add randomized jitter. Log attempt number and exception each time.python
import time
import random
import functools
import logging
logger = logging.getLogger(__name__)
def retry_with_backoff(max_attempts=3, initial_delay=0.5, max_delay=10.0, jitter=0.1):
"""
Retry decorator with exponential backoff + jitter.
- max_attempts: total attempts (initial try + retries = max_attempts)
- initial_delay: seconds before first retry
- max_delay: maximum delay between retries
- jitter: max random jitter in seconds added to delay (uniform 0..jitter)
"""
def decorator(fn):
@functools.wraps(fn)
def wrapper(*args, **kwargs):
attempt = 1
delay = initial_delay
while True:
try:
return fn(*args, **kwargs)
except Exception as e:
logger.warning("Attempt %d failed with exception: %s", attempt, e, exc_info=True)
if attempt >= max_attempts:
logger.error("All %d attempts failed; re-raising.", max_attempts)
raise
sleep_time = min(delay, max_delay) + random.uniform(0, jitter)
logger.info("Sleeping %.3fs before next attempt", sleep_time)
time.sleep(sleep_time)
attempt += 1
delay *= 2
return wrapper
return decorator
Key points:- jitter reduces synchronized retries (thundering herd).- initial try is attempt 1; retries up to max_attempts.- exc_info=True in logging preserves stack/trace.Where to use (Data Scientist context):- Wrapping transient feature-store reads, model-serving inference requests, or short-lived network calls that fail intermittently (timeouts, transient 5xx).- Useful in ETL pipelines or online feature fetch where occasional retries recover without manual intervention.When to prefer a circuit-breaker:- Use circuit-breaker when downstream service is failing repeatedly or is overloaded—circuit-breaker stops retrying after a failure threshold and trips to open state, preventing wasted resources and allowing the downstream service to recover. Combine both: circuit-breaker for protecting the system, retry-with-backoff for transient, occasional failures.