Rate Limiting, Throttling and Quota Management Questions

Token-bucket vs Leaky-bucket (brief)

• Token-bucket: tokens arrive at fixed rate; clients consume tokens to send bursts. Allows burstiness up to bucket capacity. Good when you want average rate control but allow short peaks (e.g., API clients that legitimately batch requests).
• Leaky-bucket: requests enter a queue and are processed at a fixed drip rate; excess are queued or dropped. Smooths traffic strictly, preventing bursts. Good when backend capacity must be steady (e.g., fixed-rate downstream systems).

Why choose which (security perspective)

• Use token-bucket for user-facing APIs where UX matters and occasional bursts are legitimate (login spike, batch sync).
• Use leaky-bucket at choke points (ingress proxies, legacy services) to enforce steady processing and protect finite resources.

IP-based vs User/account-scoped limits

• IP-based limits

  • Advantages: simple, effective against some volumetric abuse and unauthenticated traffic; cheap to implement at edge.
  • Evasion risks: NAT, proxies, botnets, IP rotation, mobile networks; blocks legitimate shared-IP users.
  • Use when: early-stage filtering at CDN/WAF to blunt basic floods or unauthenticated endpoints.

• User/account-scoped limits

  • Advantages: precise, fair per-actor throttling; aligns with business policies and privilege tiers; resilient to IP churn.
  • Evasion risks: credential stuffing, account creation (mitigate with verification), stolen tokens.
  • Use when: authenticated APIs, billing-sensitive operations, enforcing per-customer SLAs.

Distributed microservice considerations

• Centralize counters or use consistent hashing and sharded stores (Redis Cluster, token-bucket in Redis with Lua) to avoid race conditions.
• Employ hybrid approach: edge IP-based caps for volumetric defense + per-account token-bucket for fairness.
• Add telemetry and anomaly detection (per-IP and per-user) to detect evasion (rapid account creation, IP churn) and escalate (CAPTCHA, rate-limits, blocking).
• Design for eventual consistency: use short token TTLs, sliding windows, and conservative limits to avoid unexpected bursts when nodes disagree.

I would recommend a layered design: CDN/WAF IP throttles for volumetric protection, per-account token-bucket for business-level fairness, and strict leaky-bucket at sensitive backend queues.

Overview & goals
Design a global, low-latency (<5ms added) per-tenant rate limiter for 10k RPS peak (multi-region), scalable to 100k tenants, with burst handling and graceful degradation under datastore partitions. I propose a hybrid edge-local + centralized coordination approach using token-bucket at the edge and Redis for global quota enforcement and sync.

Algorithm choices

• Edge token-bucket (local, in-memory): immediate checks for low latency and burst absorption. Each API gateway instance keeps per-tenant tokens and refill logic.
• Periodic global reconciliation using a distributed counter (Redis) to enforce long-term quotas and prevent cross-region abuse.
• For strict global windows, use Redis LUA scripts (atomic) implementing sliding-window counters or a global leaky-bucket.

Data stores & topology

• Redis Cluster (sharded, persistence AOF/RDB) per-region for fast atomic ops; enable Redis Raft or Cluster with replicas for high availability.
• Config store (etcd) for tenant policies, burst sizes, risk posture.
• Use consistent-hash assignment of tenant -> shard to scale to 100k tenants.

Eventual consistency & correctness

• Primary enforcement at edge => correctness is optimistic; edges periodically (e.g., every 100–500ms) sync token usage to Redis.
• For strong enforcement (high-risk tenants), perform synchronous Redis check on each request (costly) — use risk-based policy.
• Use Redis LUA scripts for atomic decrements when doing synchronous checks; fallback to edge-only when Redis partitioned.

Graceful degradation

• Detect datastore partition via healthchecks/latency.
• Tiered fail mode:
  • Fail-closed for critical/high-risk tenants (deny if uncertain).
  • Fail-open for low-risk tenants (allow but mark telemetry and apply heavier post-facto throttles).
• Queue background reconciliation and rate-backoff if central counters resume.

Scaling & tenancy

• Shard tenants across Redis slots; push tenant policy to gateways via etcd watch to avoid per-request config fetch.
• Edge memory ~ O(active tenants per node). For 100k tenants, use many gateway instances and evict inactive tenant entries LRU.

Monitoring & metrics

• Per-tenant and per-region metrics: allowed, denied, dropped (due to datastore), sync lag, Redis latency, token consumption rate.
• Dashboards (Prometheus + Grafana), alerts for >1% failopen, Redis errors, sync lag >500ms.
• Audit logs for security investigations: decisions, tenant id, policy version, enforcement mode.

Trade-offs

• Hybrid reduces latency but accepts temporary cross-region quota overshoots; mitigated by reconciliation and risk-based synchronous checks.
• Strong consistency is possible at cost of latency; use selective strictness for high-risk tenants.

This design balances security posture, low latency, and operational resilience appropriate for a Security Architect: enforceable policies, risk-tiered strictness, and clear observability for incident response.