Application Programming Interface Design and Rate Limiting Questions

Focuses on designing scalable application programming interfaces that handle high request volumes while protecting infrastructure and preserving developer experience. Topics include API surface design and versioning, idempotency and retry semantics, authentication and authorization impacts, consistency and backward compatibility, and choosing protocols and payload formats. For scaling and protection discuss rate limiting and quota strategies such as token bucket, fixed window, sliding window, leaky bucket, per API key and per user limits, and hierarchical quotas. Cover backpressure, graceful degradation, circuit breakers, throttling responses and headers that communicate limits to clients, retry guidance, and strategies to avoid thundering herd effects. Also include operational concerns: monitoring and observability for request and error rates, metrics for usage and latency, metering and billing implications for usage based pricing, developer platform experience, documentation and developer tooling, testing at scale, and trade offs between strict protection and usability.

HardSystem Design

82 practiced

Design a distributed rate limiter using Redis clusters for a global API. Discuss algorithm choices (fixed-window with distributed counters, sliding window with sorted sets, token bucket), atomicity (Lua scripts), fairness, latency trade-offs, how to shard keys, and strategies to handle network partitions and clock skew.

Sample Answer

Requirements:- Global API rate limits per API key (e.g., 1000 req/min), burst support, low-latency (<10ms extra), strong-enough enforcement (soft consistency acceptable across regions), fairness across clients, fail-open/closed policy configurable.

High-level architecture:- Edge gateways in each region → local rate limiter middleware → Redis cluster shards (regional + central replica) → metrics/monitoring.- Prefer regional enforcement with occasional global coordination to reduce latency.

Algorithm choices & trade-offs:- Fixed-window counters: simple, low latency (INCR + EXPIRE), but suffers boundary spikes and coarse fairness. Good for coarse limits.- Sliding window with sorted sets: accurate per-request sliding window using ZADD + ZREMRANGEBYSCORE + ZCOUNT in a Lua script for atomicity. Higher memory/CPU cost, higher latency.- Token bucket (recommended): supports burst and steady rate. Implement tokens in Redis per key: store last_refill_ts and tokens. Each request uses a Lua script to compute tokens = min(capacity, tokens + rate*(now-last_ts)), consume if tokens>=1. Good balance: supports burst, lower operations than sorted sets.

Atomicity:- Use Redis Lua scripts to perform read-modify-write atomically (no rounds of network calls). Scripts should be small, use KEYS/ARGV, return remaining tokens and next-allowed timestamp. Keep scripts idempotent and versioned.

Fairness:- Enforce on key granularity (API key or user ID). For fairness across regions, choose one of: - Strong: central Redis or consistent hashing to a single owner (higher latency). - Probabilistic: regional enforcement with periodic reconciliation (sufficient for most APIs). - Weighted fairness: allocate tokens proportionally across regions.

Sharding keys:- Hash by API key to Redis cluster slots (use Redis Cluster or client-side consistent hashing). Keep per-key data colocated to a shard to avoid cross-shard transactions. For global counters (optional), use hierarchical counters: regional local + periodic aggregate to central.

Latency vs consistency:- Regional local enforcement => lowest latency, weaker global accuracy (possible over-usage up to N regions).- Centralized enforcement => strong global limits, higher latency and single-region failure impact.- Hybrid: local token bucket with a global background quota sync to bound global overuse.

Handling network partitions and failures:- Define fail-open vs fail-closed per endpoint. Default: fail-open with conservative local caps (small emergency bucket) to avoid total outage.- Use local caches of tokens/credits when Redis unreachable; allow short bursts until reconciliation.- Use leader election for cross-region reconciliation if strict global limits required.

Clock skew:- Use Redis server time via EVAL calling redis.call("TIME") to compute deltas inside Lua script — avoids dependence on client clocks.- Store monotonic timestamps from Redis TIME in token records.

Operational considerations:- Script caching (EVALSHA) and fallback to EVAL.- Metrics: per-key reject rate, latency, Redis CPU/memory.- Throttling headers: return remaining tokens, reset time.- Rate limit configuration via feature flags; rollout and canary.

Example token-bucket Lua outline:- KEYS[1]=key; ARGV=[capacity, rate_per_sec, now, tokens_requested]- Fetch token record, compute new_tokens = min(capacity, tokens + (now - last_ts)*rate)- If new_tokens >= requested: deduct and save; return allowed, remaining, ttl- Else return not allowed, wait_time.

This design balances latency and correctness: use token bucket + Lua per-region for low latency and burst support, add optional global coordination for strict global quotas.

EasyTechnical

89 practiced

Explain idempotency in HTTP APIs: define what idempotency means in practice, give examples of idempotent and non-idempotent operations (GET/PUT/POST), explain how idempotency keys work to make POST safe for retries, and describe both client and server responsibilities for reliable retry behavior.

MediumTechnical

90 practiced

Implement a single-process token bucket rate limiter in Python. Requirements: class TokenBucket(capacity, refill_rate_per_second) with method try_consume(n=1) -> bool. It must be thread-safe for concurrent callers, use O(1) time per call, and handle fractional refill using timestamps. Provide code and explain thread-safety and edge cases.

Sample Answer

Approach: maintain current_tokens (float) and last_refill_ts. On each try_consume call, compute elapsed time since last_refill_ts, add tokens = elapsed * refill_rate, cap at capacity, update last_refill_ts, then attempt to subtract n if enough tokens exist. Protect the read-modify-write with a threading.Lock for thread-safety. All operations are constant-time.

python

import time
import threading

class TokenBucket:
    def __init__(self, capacity: float, refill_rate_per_second: float):
        if capacity <= 0 or refill_rate_per_second < 0:
            raise ValueError("capacity must be >0 and refill_rate_per_second >= 0")
        self.capacity = float(capacity)
        self.refill_rate = float(refill_rate_per_second)
        self._tokens = self.capacity  # start full
        self._last_ts = time.monotonic()
        self._lock = threading.Lock()

    def try_consume(self, n: float = 1.0) -> bool:
        if n <= 0:
            raise ValueError("n must be positive")
        if n > self.capacity:
            return False  # cannot consume more than capacity

        now = time.monotonic()
        with self._lock:
            elapsed = now - self._last_ts
            if elapsed > 0 and self.refill_rate > 0:
                # fractional refill
                self._tokens = min(self.capacity, self._tokens + elapsed * self.refill_rate)
                self._last_ts = now
            # else: if elapsed <= 0 (clock jitter) or refill_rate == 0, don't change tokens/time

            if self._tokens + 1e-12 >= n:  # numeric tolerance
                self._tokens -= n
                return True
            else:
                return False

Thread-safety: a single Lock serializes updates to _tokens and _last_ts, preventing race conditions. Using time.monotonic avoids system clock jumps. All arithmetic and branching are O(1).

Edge cases:- n <= 0 raises error; n > capacity returns False.- refill_rate == 0: bucket never refills.- time monotonicity: if clock appears to go backward, elapsed <= 0 and we skip refill to avoid negative elapsed effects.- floating-point precision: small tolerance used.

Complexity:- Time: O(1) per call.- Space: O(1) additional memory.

HardTechnical

94 practiced

Design a metering and billing pipeline for usage-based API pricing. Requirements: ingest 100M events/day, deduplicate events, attribute usage to account/plan, compute charges daily with ability to dispute, and support staging/production pipelines. Describe event model, streaming stack, aggregation, reconciliation, and auditability.

Sample Answer

Framework / approach:- Goal: reliable, auditable daily charges from 100M events/day with dedupe, per-account attribution, staging/prod isolation, and dispute/reconciliation.

Requirements recap:- Ingest 100M/day (~1.2k events/sec average, bursts higher)- Deduplicate events- Attribute to account/plan, apply pricing rules, compute daily charges- Support disputes and reconciliation- Separate staging & prod pipelines

Event model:- Immutable JSON events with schema versioning (Avro/Protobuf): { event_id: UUID, source_ts: ISO8601, received_ts, customer_id, account_id, org_id, resource_id, usage_metric, usage_quantity, currency, region, signature, pipeline: prod|stg, schema_version }- event_id + signature used for idempotency and lineage.

Streaming stack:- Ingest: API gateway -> validated, enriched, and written to Kafka topics (separate topics per environment).- Schema registry (Avro) enforces contracts.- Stream processing: Apache Flink (or Spark Structured Streaming) for dedupe, enrichment, attribution, windowed aggregation. - Deduplication: stateful operator keyed by event_id with TTL (retain for 30 days) and bloom-filter cache for fast path. - Enrichment: join with user/account store (Redis or materialized view from a user DB via CDC).- Exactly-once semantics via Kafka + Flink checkpointing.

Aggregation & billing:- Real-time rolling aggregates (per-minute, per-hour) stored in a time-series store (ClickHouse/Timescale) and a daily rollup job (Flink/Spark) produces final daily usage per account/metric.- Pricing engine: declarative rules (rule service / policy engine) applied during daily rollup to compute charges; output an immutable invoice record with provenance links to contributing event_ids and aggregation buckets.

Reconciliation & dispute:- Persist raw events (cold storage S3) + processed events (Kafka compacted topics) + final invoices (DB).- Daily reconciliation job compares: sum(raw deduped usage) vs aggregated invoice per-account; generate mismatch reports.- Dispute flow: create dispute ticket referencing invoice_id, implicated event_ids and aggregate windows; allow reprocessing with corrected events (replay from S3 into a staging topic), approve changes -> create adjustment invoice and audit record.

Auditability & observability:- Store lineage: every invoice row includes provenance (list of event_id ranges or pointers to storage partitions), hash of contributing payloads, processing run_id, schema_version.- Append-only stores for invoices and adjustments (immutable ledger pattern). Use write-once S3 buckets + Glacier retention.- Monitoring: metrics (ingest rate, dedupe rate, processing lag), alerting, and dashboards; distributed tracing (trace id passed in events).- Access logging and role-based access control; cryptographic signing of invoices for non-repudiation.

Staging / Production:- Separate Kafka clusters (or tenant/topic prefixes) + isolated Flink namespaces; CI/CD pipeline that promotes processed test data to prod only after end-to-end checks. Use feature flags and canary replays.

Scale & trade-offs:- Flink + Kafka provides low-latency, exactly-once at scale. Storing raw events costs more but yields full auditability and replay ability. ClickHouse for analytical reads; RDBMS for ledger.- TTL for dedupe state balances memory vs window for late-arriving events.

This design ensures reliable ingestion, deterministic billing, full lineage for audits/disputes, and safe staging/prod separation.

MediumTechnical

99 practiced

As a platform owner, design SDK features and primitives that surface rate limit information and help client applications behave politely. Include automatic backoff strategies, callbacks or events developers can subscribe to, transparent retries for idempotent calls, and a small diagnostics API to inspect current quotas. Explain language-specific considerations for concurrency.

Sample Answer

Approach: expose a small, consistent surface in the SDK that (1) informs apps about limits, (2) lets them react (events/callbacks), (3) applies safe automatic behavior (backoff, retries) for idempotent calls, and (4) provides diagnostics for observability.

Primitives:- RateLimitInfo object: {limit, remaining, resetAt, window, resourceKey, scope} returned on every response and updated from X-RateLimit-* headers.- Events / Callbacks: - onRateLimitNearThreshold(callback: RateLimitInfo) - onRateLimitReached(callback: RateLimitInfo) - onBackoffStarted(callback: BackoffDetails) - onBackoffEnded(callback)- Automatic backoff & retry policy (configurable): - Default: exponential backoff with jitter: wait = base * 2^n + random(-jitter, +jitter) - Respect Retry-After header and ResetAt timestamps. - Only auto-retry idempotent methods (GET, PUT with idempotency key). Allow callers to opt-in per-call. - Circuit-breaker: soft-stop retries after N consecutive 429s for resourceKey.- Idempotency support: - Automatic idempotency-key header when SDK has request body and caller requests safe retries. - Transparent retry attempts are logged and surfaced via events.- Diagnostics API: - diagnostics.getQuotaState(): snapshot of RateLimitInfo per resourceKey. - diagnostics.getBackoffState(), diagnostics.getRecent429s() - lightweight, non-blocking, suitable for telemetry export.

Language-specific concurrency considerations:- Java: use ConcurrentHashMap for per-resource RateLimitInfo; ScheduledExecutorService for delayed retries; make events execute on caller-provided Executor to avoid blocking I/O threads.- Python: async/await with asyncio: store state in asyncio.Lock-protected structures; provide both sync and async clients to avoid mixing event loops; use asyncio.create_task for delayed retries.- JavaScript/Node: single-threaded event loop: use timers for backoff; keep state in plain objects; ensure SDK avoids long CPU work; provide promise-based hooks and optional EventEmitter.- C++: thread-safe primitives (std::atomic/ mutex); allow user to provide an io_context/worker for scheduling retries; minimize allocations for high-throughput clients.

Best practices:- Default conservative behavior; make policies configurable.- Surface metrics for observability.- Fail fast for non-idempotent calls rather than silently retrying.- Document defaults and per-language caveats clearly.

Unlock Full Question Bank

Get access to hundreds of Application Programming Interface Design and Rate Limiting interview questions and detailed answers.

Join thousands of developers preparing for their dream job.