Clarify goals & constraints
- Prevent noisy-neighbor and DoS, preserve low latency, support global regions, allow developer self-service and smooth upgrades, cost-bounded.
Tiered quotas & per-key limits
- Define tiers: Free, Developer, Business, Enterprise. Each tier includes:
- Monthly quota (requests/month, data egress)
- Steady-state RPS (requests/sec) and burst capacity
- Priority level (for graceful degradation)
- Per-key limits = (steady RPS, burst tokens, concurrent requests). Keys map to tenant + application.
Burst handling & throttling semantics
- Use token-bucket per key for bursts: bucket size = burst_capacity, refill rate = steady RPS.
- When tokens exhausted: return HTTP 429 with Retry-After header (seconds until next token) and descriptive body (current usage, tier, link to dashboard).
- Soft vs hard limits: soft throttles (429) for normal enforcement; hard caps for monthly quota (403 when exceeded) with billing/upgrade link.
Enforcement architecture
- Enforce at the edge (API Gateway / CDN) for low latency and reduced attack surface. Implement local per-key caches and token buckets in gateway instances.
- Central quota service for accounting, long-term quotas, billing, policy management, and global decisions.
- Edge-first; central authoritative. Edges operate independently using lease-based local allowances and periodic reconciliation.
Distributed quota sync
- Use hierarchical allowance model:
- Central allocator gives regional leases (e.g., N tokens/min) per key via consistent hashing or tenant sharding.
- Edge consumes from local bucket; when local allowance low, edge requests refill from central (non-blocking fallback).
- For cross-region failover, use optimistic local borrowing with background reconciliation and CRDT-based counters for monthly usage to tolerate partitions.
- Central store: strongly consistent DB for policy (e.g., Spanner/Cockroach) + distributed cache (Redis Cluster) for fast lookups. Use rate-limiter metadata with short TTLs.
Operational & security considerations
- DDoS protection: integrate WAF and anomaly detection (rate spikes per IP, geolocation), auto-block or escalate to manual review.
- Monitoring & alerting: per-tenant RPS, token consumption, rejects, error budgets; SLOs for 429 rates.
- Cost control: circuit breakers for extreme spikes and emergency kill-switch per tenant.
Developer self-service
- Usage dashboard showing live RPS, burst consumption, monthly quota, predicted exhaustion time.
- Self-serve upgrade: tier change via UI/API with immediate policy push and pro-rated billing.
- SDKs add client-side exponential backoff respecting Retry-After; sample code + best-practice docs.
- Webhooks/alerts for approaching quota thresholds.
Trade-offs
- Edge enforcement = low latency, complexity in consistency. Central-only = simpler but higher latency and single point of failure. Lease-based hybrid balances both.
This design provides low-latency protection at the edge, strong central accounting and global fairness, predictable burst behavior, and a developer-friendly upgrade and visibility experience.
