Rate Limiting, Throttling and Quota Management Questions

Protecting API capacity and enforcing fair use: rate-limiting algorithms (token bucket, leaky bucket, fixed/sliding window), per-client quotas, throttling responses (429 semantics, Retry-After), and tiered plan enforcement. Covers where to enforce limits (gateway vs. service), distributed counters, and graceful degradation under load.

MediumTechnical

72 practiced

Implement in Python (pseudocode is fine) a token-bucket rate limiter backed by Redis for an inference endpoint. Provide function prototype: def allow_request(user_id) -> bool. Explain how you ensure atomicity for refills and consumption, fairness across instances, and how this scales in production.

Sample Answer

Approach: use a Redis-backed token bucket per user with keys storing last_refill_ts and tokens. Perform refill and consume atomically with a Redis Lua script (ensures single-step update across distributed instances). allow_request(user_id) returns True if a token was consumed. python# pseudocode / Python with redis-py import time import redis r = redis.Redis(...) # bucket params CAPACITY = 10 # tokens REFILL_RATE = 1.0 # tokens per second # Lua script: atomically refill based on elapsed time then consume 1 token if available LUA = """ local key = KEYS[1] local now = tonumber(ARGV[1]) local capacity = tonumber(ARGV[2]) local refill_rate = tonumber(ARGV[3]) local data = redis.call('HMGET', key, 'tokens', 'last') local tokens = tonumber(data[1]) or capacity local last = tonumber(data[2]) or now -- refill tokens local delta = math.max(0, now - last) tokens = math.min(capacity, tokens + delta * refill_rate) last = now if tokens >= 1 then tokens = tokens - 1 redis.call('HMSET', key, 'tokens', tokens, 'last', last) redis.call('EXPIRE', key, math.ceil(capacity / refill_rate * 2)) return 1 else redis.call('HMSET', key, 'tokens', tokens, 'last', last) redis.call('EXPIRE', key, math.ceil(capacity / refill_rate * 2)) return 0 end """ allow_script = r.register_script(LUA) def allow_request(user_id: str) -> bool: now = time.time() key = f"tokenbucket:{user_id}" allowed = allow_script(keys=[key], args=[now, CAPACITY, REFILL_RATE]) return bool(allowed) Key points: Atomicity: Lua script executes atomically on a Redis node, so refill + consume cannot race across instances. Fairness across instances: All app instances call the same Redis script; Redis serializes updates per key. Use consistent key naming per user. Scaling: Redis handles high QPS; for extreme scale shard users across Redis clusters or use Redis Cluster. Use local in-memory token buckets as a first-check (leaky local cache) to reduce Redis load, with Redis as source-of-truth. Persistence/TTL: set EXPIRE to remove idle buckets. Edge cases: clock skew (use server time via Redis TIME or monotonic server-side timestamp), negative refill_rate, burst tuning. Alternatives: fixed-window or sliding-window counters, or Redis sorted-set leaky-bucket. Lua token-bucket is simple, low-latency, and widely used.

That is every published Rate Limiting, Throttling and Quota Management question for Machine Learning Engineer so far. Browse the other topics in this category, or practice this one interactively.

Browse APIs, Networking & Integration topics Start a practice session