Requirements & constraints:
- Functional: per-API-key rate limits supporting burst and sustained limits; load balancing to backends.
- Non-functional: p95 latency SLO < 200ms, high availability, resilience when rate-limiter datastore degrades.
High-level design:
API Gateway (edge, e.g., Envoy/NGINX/custom) -> Rate-limiter layer -> Load balancer -> Backend services
- Gateway handles TLS, routing, LB (consistent hashing or round-robin + health checks).
- Rate-limiter is executed at the gateway process (fast path) using a hybrid local+distributed token-bucket.
Rate limiting choices:
- Algorithm: Token Bucket (supports burst + sustained). Implement two buckets per API key: burst_bucket (large capacity, high refill short-term) and sustained_bucket (smaller capacity, long-term refill). Request allowed only if both buckets have tokens.
- Distributed counters/datastore: Redis Cluster (primary choice) with Lua scripts for atomic token consume + metadata, or a CRDT-based counter store if multi-datacenter strong availability required. Use consistent hashing of API key -> Redis shard to localize state and reduce cross-shard ops.
Where to apply limits:
- Primary enforcement at edge (API Gateway) — lowest latency, prevents backend overload.
- Secondary enforcement at service (defense-in-depth) — simplified checks (coarse quotas) to catch bypass or misconfiguration.
Performance & SLO strategies (keep p95 < 200ms):
- Fast-path: attempt local token consume from an in-memory token cache updated periodically (e.g., small leaky bucket tokens refreshed from Redis every 100–500ms). If local cache has tokens → accept immediately (no network call).
- If local cache insufficient → issue synchronous Redis Lua script to atomically deduct tokens. Keep Redis round-trip minimal (<5ms typical in same AZ). Instrument and monitor latency.
- Make logging/metrics asynchronous; do not block request on telemetry.
Resilience when rate-limiter datastore degrades:
- Degraded modes:
- Read-only fallback: use last-known local quotas and refill rates; enforce conservative limits (e.g., 50% of configured) to protect backend.
- Fail-open vs fail-closed policy configurable per API: high-trust internal keys may fail-open; public keys default to conservative fail-closed (strict throttling).
- Graceful degradation: apply purely local leaky-bucket limits (fixed-rate) per gateway instance when Redis unavailable — ensures fairness but less global accuracy.
- Circuit-breakers & healthchecks: monitor Redis latency/error rates; when thresholds exceeded, gateways switch to degraded mode and emit alerts/incident pages.
- Reconciliation: on recovery, gateways reconcile token usage via background sync to avoid burst/overuse.
Atomicity & correctness:
- Use Redis Lua script to check/refill/consume both buckets atomically. Script returns remaining tokens and decision.
- For the local cache, use monotonic token counters and periodic reconciliation to avoid double-spend across instances.
Scalability & capacity planning:
- Shard Redis by API-key; provision capacity for peak request rate; use pipelining & connection pooling.
- Autoscale gateway fleet behind LB. Use sticky session/consistent hashing if using local caches to improve hit rate.
Monitoring & alerting:
- Metrics: request latency p50/p95/p99, Redis RTT/error rate, reject rate per API key, degraded-mode switches.
- Alerts on SLO burn, high reject spikes, datastore latency.
Trade-offs:
- Local cache improves latency; sacrifices perfect global accuracy (eventual consistency).
- Strong consistency (single Redis) gives precise limits but increases latency/availability coupling.
- Using conservative degraded limits protects backends at cost of potential false throttling.
This design balances latency (fast local path) and accuracy (Redis atomic scripts), enforces burst+sustained using token buckets, applies limits at edge with defense-in-depth, and includes clear degraded-mode behavior to remain resilient when the rate-limiter datastore is degraded.
