Scalability Patterns and Techniques Questions

Practical scaling techniques and patterns for application and data layers. Topics include horizontal and vertical scaling strategies and the trade offs of each; caching topologies and strategies such as cache aside write through and write behind and approaches to cache invalidation and consistency; database scaling techniques including read replicas partitioning and sharding and rebalancing strategies; load balancing algorithms including round robin least connections consistent hashing and strategies for sticky sessions and service discovery; message queue and event streaming patterns for decoupling backpressure and asynchronous processing; content distribution using content delivery networks; connection pooling and resource management; rate limiting throttling retry strategies and approaches to avoid thundering herd problems; and how to combine patterns effectively given workload characteristics and operational constraints. Interviewers expect candidates to explain interactions between patterns and the operational pitfalls of each technique.

HardTechnical

29 practiced

Design strategies to avoid the thundering herd problem when many clients request the same expired cache key simultaneously. Evaluate techniques such as request coalescing (singleflight), distributed locks, grace caches (serve stale while revalidating), randomized TTLs, and queuing/coordinator patterns, and discuss trade-offs in latency and system complexity.

Sample Answer

Situation: Many clients simultaneously request an expired cache key, causing a surge of backend load — the thundering herd. Below are practical strategies, how they work, and their trade-offs in latency and complexity.

1) Request coalescing (singleflight)- How: First requester fetches from origin; others wait and receive the single result.- Pros: Minimizes redundant origin work, low extra latency for waiters.- Cons: Requires in-process or distributed coordination; in-process singleflight fails across multiple app instances.- Best when: Low request fan-out per node; origin calls are expensive.

2) Distributed locks- How: Acquire a lock (e.g., Redis SETNX) before refreshing cache; others either wait or serve fallback.- Pros: Works across instances, prevents duplicate refreshes.- Cons: Lock failures, deadlocks, slow lock acquisition adds latency; need TTL and robust failure handling.- Best when: Strict single-writer requirement for cache refresh.

3) Grace caches (stale-while-revalidate)- How: Serve stale value while kicking off async refresh.- Pros: Low client latency; origin load smoothed.- Cons: Clients see stale data for a window; more complex consistency semantics.- Best when: Slightly stale data is acceptable (e.g., feeds, product catalogs).

4) Randomized TTLs (jittered expirations)- How: Add random offsets to TTLs so expirations are spread.- Pros: Simple, cheap; reduces synchronized expiry.- Cons: Doesn’t prevent herd if TTL overlap large; still probabilistic.- Best when: Many keys with similar lifetimes; inexpensive to implement.

5) Queuing / coordinator patterns- How: Central coordinator or refresh queue serializes refresh requests.- Pros: Predictable control over refresh rate; can prioritize.- Cons: Single point of failure unless replicated; adds latency for queued requests and system complexity.- Best when: You need rate-limiting and operational control.

Hybrid recommendations- Combine jittered TTLs + stale-while-revalidate as default (low latency, low origin load, modest complexity).- Add distributed locks or a distributed singleflight for “hot” keys with strong consistency needs.- Use a coordinator/queue for very high-scale hotspots where you need throttling and observability.

Trade-offs summary- Latency: grace caches and coalescing offer lowest client latency; locks and queues can add wait time.- Complexity: randomized TTLs are simplest; distributed locking and queuing add operational complexity and failure modes.- Consistency: locks/singleflight give stronger freshness; stale-while-revalidate and TTL jitter favor availability and latency.

Monitoring & safety- Track miss rates, refresh latency, error rates; set circuit-breakers and fallbacks; instrument hot-key detection to apply targeted strategies per key.

Decision rule- If stale data acceptable → use stale-while-revalidate + jitter.- If freshness critical → use distributed singleflight/locks with careful timeout and fallback.- Always monitor and apply hybrid tactics for hot keys.

EasyTechnical

28 practiced

Describe the token bucket and leaky bucket algorithms for rate limiting. For an API that must allow bursts up to 200 requests and an average rate of 100 requests/sec per user, which model fits best and why? Explain how you'd instrument and test fairness across many clients.

EasyTechnical

24 practiced

Explain eventual consistency in distributed systems. Provide concrete examples where eventual consistency is acceptable (e.g., social feeds) and where it is not (e.g., financial transfers). How would you design conflict resolution and user-visible compensating actions in eventually-consistent systems?

MediumTechnical

44 practiced

Design connection pool management for a microservice that needs to serve 500 concurrent request handlers and makes DB and upstream HTTP calls. Describe how to size DB connection pools and HTTP client pools, choose timeouts and max lifetimes, integrate with circuit breakers, and test pool behavior under simulated production spikes.

Sample Answer

Approach summary: size pools based on concurrent handlers that actually need each resource, measured latencies, and safety factors; choose timeouts shorter than business SLA but long enough for tail latencies; use max lifetime and keep-alive to avoid stale sockets; couple pools with circuit breakers & bulkheads; validate with staged load tests and fault injection.

Sizing formulas (use real metrics from APM):- concurrent_users = 500- p_db = fraction of requests that perform a DB call concurrently (measure; example 30% => 150)- avg_db_latency = measured round-trip DB time (example 100ms = 0.1s)

Needed DB connections ≈ concurrent_db_calls * avg_db_latencyExample: 150 * 0.1 = 15 active connections. Add safety factor 1.5–3 => target pool 30–45. Cap to DB server limit and use connection queueing/backpressure.

HTTP client pool (per upstream host):- concurrent_up_calls = concurrent handlers * p_upstream (example 40% => 200)- If calls distributed across N hosts, per-host = 200/N.- For HTTP/1.1 set max_per_host ≈ per-host concurrent + safety (e.g., 2×); for HTTP/2 prefer fewer connections because multiplexing.

Timeouts and lifetimes:- Connect timeout: 200–1000ms depending on network locality (internal services shorter).- Socket/read/request deadline: set to a fraction of SLA. If end-to-end SLA = 500ms and your service needs 100ms processing, set upstream timeout = 200–300ms.- Idle timeout: 30–120s to free unused sockets.- Max connection lifetime: shorter than upstream/load balancer TCP timeout (e.g., 5–10 minutes) to avoid NAT/idle reaps; rotate to prevent long-lived stale sessions.- Pool queue/backpressure timeout: small (100–300ms) to fail fast rather than pile up latency.

Circuit breaker & bulkhead integration:- Put a circuit breaker per upstream and per DB host with sliding window (e.g., 10s or 100 requests) and thresholds: failure rate > 50% or consecutive failures > 5 → open.- When circuit open, immediately fail fast and return fallback or 503; use exponential backoff and limited half-open probes.- Bulkheads: isolate resources (separate DB pools/HTTP pools for critical vs non-critical paths) to prevent noisy neighbors.

Observability & metrics:- Track pool usage, queue length, wait time, connection creation failures, DNS errors, latency percentiles, circuit state transitions.- Alerts on high pool wait times or >X% exhausted connections.

Testing strategy:1. Measure real avg_db_latency and p_db in staging with production traffic replay.2. Load test with k6/Locust: ramp to 500 concurrent handlers, then spike 2×–5×. Monitor pool saturation and tail latencies.3. Chaos/fault injection: add latency and failures on DB/upstream (tc/netem or service mocks) to validate circuit breaker trips and fallback behavior.4. Soak test for max lifetime and connection leaks (run for >max_lifetime to see rotation).5. Test pool boundaries: reduce pool sizes to confirm graceful degradation (fail fast) and ensure no thread exhaustion.

Operational notes:- Start conservative: e.g., DB pool 30 for the example, HTTP per-host 50, observe and tune.- Automate autoscaling and health checks; ensure pool sizes are configurable via feature flags or env vars for rapid tuning.

HardSystem Design

31 practiced

Design a hybrid caching strategy for a social feed system using browser/CDN edges, per-service local in-memory caches, and a distributed cache (e.g., Redis cluster). Address feed fan-out on writes, cache invalidation when posts/comments arrive, freshness guarantees, reducing cold-starts, and handling hot keys and sudden popularity spikes.

Sample Answer

Requirements & constraints:- Low tail latency for feed reads globally- High write fan-out when popular users post- Strong-ish freshness for new posts/comments (seconds), but allowed small staleness to scale- Handle cold-starts and hot-key spikes gracefully

High-level architecture:Client browser <-> CDN edge (cacheable feed fragments) <-> API gateway -> service instances (local in-memory LRU) -> Redis cluster (distributed cache) -> Primary feed store + Fan-out workers / message queue

Core components and responsibilities:1. CDN/browser edges: cache rendered feed fragments or JSON pages with short TTLs (e.g., 5–30s) and support stale-while-revalidate to serve while refreshing.2. Per-service local in-memory caches: tiny LRU/MRU (~100s entries) for hottest requests, request coalescing (singleflight) to prevent thundering herd.3. Redis cluster: store precomputed feed slices per user (e.g., latest 100 IDs + metadata) with TTLs and version tokens. Use sharding by user-id; keep atomic updates for slices and counters.4. Fan-out subsystem: on write, publish event to message queue (Kafka). Workers perform hybrid fan-out: - For low-follower users: synchronous push to followers’ Redis slices (eager fan-out). - For high-follower users: lazy / pull model: append to author's timeline store and mark followers’ caches as “stale” via invalidation messages; followers recompute on next read or background refresh.5. Invalidation & freshness: - Workers publish invalidation messages (Redis pub/sub or Kafka) carrying user-id and version token. API instances subscribe to update or evict local caches and to trigger CDN purge/soft-invalidate (via surrogate-control headers). - Use versioned keys (feed:user:version) to atomically switch visible feed without extra invalidations. - Allow stale-while-revalidate windows: serve slightly stale content while background recompute updates caches.6. Cold-start reduction: - Populate Redis with synthetic seed content for new users (popular global posts) and precompute initial feed in background. - Use Bloom filter to avoid looking up non-existent small caches repeatedly.7. Hot keys & sudden spikes: - Detect hot keys (hit counters + decay). For hot keys: - Promote to in-memory pinned cache on service instances. - Route through dedicated hot-key partitions in Redis with higher capacity. - Apply request coalescing and rate-limit rebuilds; return cached stale content while enqueueing a single refresh job. - Use token-bucket throttling on fan-out workers for celebrity writes and move to lazy fan-out + partial push to active followers only.8. Consistency trade-offs: - Strong consistency for actions (likes, comments visibility) that must appear immediately: write-through to Redis and invalidate CDN edge immediately. - For regular posts, accept eventual consistency within configured SLA (e.g., 1–5s).

Data flow example (new post):1. User posts -> write to primary store -> emit event to Kafka.2. Fan-out worker examines follower count: - If small -> push post ID into followers’ Redis lists and trim; publish version bump. - If large -> append to author timeline; publish invalidation tokens for follower groups / mark followers’ caches stale.3. API receives invalidation -> evict local LRU and send soft-purge header to CDN; next client read triggers fetch: API checks local -> Redis -> recompute from timelines if miss -> rehydrate caches.

Operational notes:- Monitor failover: degrade to read-from-store if Redis unavailable; backpressure on writes.- Metrics: cache hit rates at CDN/Redis/local, invalidation latencies, fan-out queue lag, hot-key detection rates.- Security & privacy: per-user authorization enforced at API, never cache private feeds at CDN without auth.

Why this design:- Hybrid push/pull balances write cost vs read latency: eager for low-fanout, lazy for celebrities.- Multi-layer caches (browser/CDN, Redis, local) reduce latency and origin load.- Versioning, stale-while-revalidate, and singleflight reduce thundering herd and control freshness/availability trade-offs.

Unlock Full Question Bank

Get access to hundreds of Scalability Patterns and Techniques interview questions and detailed answers.

Join thousands of developers preparing for their dream job.