Scalability Patterns and Techniques Questions

Practical scaling techniques and patterns for application and data layers. Topics include horizontal and vertical scaling strategies and the trade offs of each; caching topologies and strategies such as cache aside write through and write behind and approaches to cache invalidation and consistency; database scaling techniques including read replicas partitioning and sharding and rebalancing strategies; load balancing algorithms including round robin least connections consistent hashing and strategies for sticky sessions and service discovery; message queue and event streaming patterns for decoupling backpressure and asynchronous processing; content distribution using content delivery networks; connection pooling and resource management; rate limiting throttling retry strategies and approaches to avoid thundering herd problems; and how to combine patterns effectively given workload characteristics and operational constraints. Interviewers expect candidates to explain interactions between patterns and the operational pitfalls of each technique.

MediumTechnical

36 practiced

A system uses cache-aside caches at the app tier while reads come from asynchronous database replicas. Describe operational pitfalls like stale cache entries caused by replica reads, race conditions between cache invalidation and replica lag, and propose mitigations such as read routing, versioned reads, and ordering guarantees between write commit and cache updates.

Sample Answer

Situation: You run an app-tier cache-aside pattern (app reads cache, on miss reads DB, then writes cache) while read traffic is served from async read replicas. This setup is vulnerable to stale reads and race conditions because replicas lag.

Pitfalls:- Stale cache entries caused by replica reads: an update committed to primary may not appear on replica yet; a cache miss routed to the lagging replica returns old data which gets written into cache — amplifying staleness.- Race between cache invalidation and replica lag: writer invalidates or updates cache after commit, but if reads hit a replica that hasn’t caught up, they may repopulate cache with stale data.- Write→replica ordering: asynchronous replication means readers can observe earlier versions; combined with concurrent writes, clients can see non-monotonic reads.- Hidden operational complexity: metrics, debugging, and SLOs become harder to reason about.

Mitigations (practical, architecture-level):- Read routing / consistency steering: - Route critical reads to primary or strongly-consistent nodes (read-after-write) for recently-updated keys; use sticky sessions or causal hints. - Use a “freshness window”: route reads for keys updated within N ms to primary.- Versioned reads and compare-and-swap: - Include a version/timestamp in DB and cache. On cache populate, store version; on reads reject stale versions or revalidate against primary if client requests latest.- Order guarantees between commit and cache updates: - Ensure cache invalidation/update happens only after transaction commit. Use DB change-streams (CDC) to drive cache updates, ensuring canonical ordering. - Prefer write-through or write-behind for critical items so cache reflects committed state deterministically.- Safe repopulation: - On cache miss, read primary if the key was recently written (track recent-writes set). - Use conditional cache set: only set cache if DB version >= expected.- Monitoring & operational controls: - Monitor replica lag, cache hit paradox (cache filled with old versions), and provide circuit (fallback to primary) when lag > threshold. - Instrument request traces with versions to debug staleness paths.Trade-offs: routing to primary increases latency/load; version checks add round trips. Combine approaches: use lightweight routing/thresholds for low-latency path and strict guarantees for critical operations.

MediumSystem Design

32 practiced

Design a caching topology for a read-heavy product catalog API serving 100k RPS globally with 10M products where 95% of requests are for the top 10% popular products. Describe cache layers (CDN, edge caches, application caches), cache keys and sharding, TTL strategies, eviction policies, cache invalidation for product updates, and approaches to prevent cache stampede on bursts.

Sample Answer

Requirements (clarify): read-heavy global API, 100k RPS, 10M products, 95% of traffic hits top 10% products → hot skew. Low latency, eventual consistency acceptable for some reads; product updates are less frequent but must propagate within seconds–minutes.

High-level topology:- Global CDN (multi-region POPs) for public API responses (JSON) and static assets.- Regional edge caches (Redis/KeyDB or managed edge KV like Cloudflare Workers KV / AWS Global Accelerator + Lambda@Edge) per region for fast reads and to absorb origin.- Application-level cache (local LRU in each app node + shared Redis/Memcached cluster per AZ) as fallback to origin DB.- Origin: read replicas of primary product DB and a write-through change-stream for invalidation.

Cache keys & sharding:- Key format: product:{product_id}:v{schema_version} or product:{product_id}:locale:{lc}:fields:{fieldset_hash}- Separate keys for full product vs lightweight listing/price view.- Shard Redis by product_id hash (consistent hashing) across cluster nodes; use replicas for failover.- Hot-key partitioning: reserve dedicated nodes or use Redis Cluster “hot-key” strategies (vertical scale or replica reads).

TTL strategies & eviction:- Hot products: longer TTL (5–30 min) with background refresh; cold products shorter TTL (30s–5min).- Use immutability versioning: include product_version in key to simplify invalidation.- Eviction: LRU on shared caches; for local app caches use small size with LRU. For CDN set Cache-Control with s-maxage and stale-while-revalidate.

Cache invalidation:- On product update: write to DB, increment product_version, publish invalidation event to message bus (Kafka/SNS).- Consumers (edge lambdas, Redis clients) either: - Purge CDN by key/pattern (fast for POPs) or issue soft-expire and background refresh. - Invalidate Redis key or increment version so old key becomes stale.- For near-real-time consistency critical fields (price/availability), use short TTL + write-through update to cache on write.

Preventing cache stampede:- Use randomized jittered TTLs to avoid simultaneous expiry for many replicas.- Cache-aside with locking: on cache miss, acquire distributed lock (RedLock) per key; other requests wait or serve stale value with stale-while-revalidate.- Serve stale (stale-while-revalidate): if cache expired, return stale immediately and trigger background refresh.- Pre-warming: proactively refresh hot keys before expiry (scheduler based on access metrics).- Rate-limit upstream refreshes and use request coalescing (singleflight).

Operational & monitoring:- Metrics: hit ratio per layer, tail latencies, hot-key amplification, lock contention.- Autoscale Redis clusters for throughput; use multi-AZ deployment, read replicas for burst reads.- Run chaos tests and simulate product updates and POP invalidation to validate propagation.

Trade-offs:- Versioned keys + event-driven invalidation: simpler and more scalable but requires storage for old keys and tombstone management.- Immediate CDN purge: lower staleness but higher cost and complexity.This topology balances latency, cost, and strong protection against stampedes while supporting global scale and hot-key skew.

MediumTechnical

33 practiced

For a social network user profile store with highly skewed user activity and several celebrity users generating hotspots, design a sharding strategy for horizontal partitioning. Compare hash-based, range-based, and directory-based sharding, explain the choice of shard key, and describe how you would handle hot users and rebalancing shards as user counts and activity patterns grow.

Sample Answer

Requirements & constraints:- Store user profiles and activity; read-heavy with highly skewed activity (a few celebrities generate hotspots).- Need horizontal scale, low latency for hot users, ability to rebalance as user count/growth changes.

Compare sharding strategies- Hash-based sharding: - Pros: Even distribution of users across shards; simple routing (hash(user_id) mod N); good for write/read load balancing when activity is uniform. - Cons: Breaks locality (range queries), migrating/adding shards requires rehashing or consistent-hash layer; hot users still concentrate on single shard causing hotspots.- Range-based sharding: - Pros: Good for range queries (e.g., alphabetical user lists); localized user groups on same shard; easier to split contiguous ranges when scaling. - Cons: Natural skew if activity correlates with key ranges (e.g., popular handles), causing overloaded ranges; requires active monitoring to split/merge ranges.- Directory-based (lookup) sharding: - Pros: Flexible — map any user to any shard using a metadata service; supports special-casing hot users and live reassignments without reshuffling data immediately. - Cons: Extra indirection and single metadata service (must be highly available & cached); operational complexity.

Shard key choice- Use user_id (stable, immutable). Prefer a numeric or UUID that’s the canonical identifier.- Combine user_id with a secondary attribute for composite keys only if queries require it (avoid if not needed).

Recommended design (hybrid)1. Base layer: consistent-hash of user_id to physical shard group to get even baseline distribution and reduce large-scale rebalancing pain.2. Directory layer (metadata service / sharding map): - Maintain mapping for exceptions: hot users, VIPs, migrated ranges. - Cache mappings in application/edge to reduce lookups; use TTL and invalidation on remap.3. Hot-user handling: - Read replicas: Promote multiple read-read replicas for hot-user shard; use follower reads for profile reads. - Fan-out caches: Cache celebrity profiles and popular derived content in CDN/edge cache and memory caches (Redis) keyed by user_id. - Dedicated shard or micro-shard: For extreme hotspots, move the hot user's profile/activity to a dedicated shard (via directory map) or split its data vertically (store heavy activity in a separate analytics store). - Rate-limiting and async writes: Buffer heavy write bursts via a write queue (Kafka) and apply asynchronously to main store to smooth spikes.4. Rebalancing strategy: - Monitor per-shard metrics (QPS, latency, CPU, storage). Automate split/merge actions. - For consistent-hash groups: add nodes to ring; use virtual nodes to minimize movement. - For ranges/exceptions: migrate small ranges using live data copy + dual-write + switch-over in directory map to avoid downtime. - Use rolling migrations with traffic steering and verify integrity; update caches/clients atomically via directory service change and eventing.5. Operational considerations & trade-offs: - Directory adds complexity but gives flexibility to isolate hotspots without full rehash. - Ensure metadata service is highly available, sharded and cached; provide metrics and alerts for remapping decisions. - Prefer eventual consistency for cached views; use strongly consistent operations for profile updates if business requires. - Test migrations in staging; provide rollback path.

This hybrid approach balances even distribution for most users (consistent hashing), operational flexibility to treat celebrities specially (directory), and multiple mitigation techniques (replication, caching, async writes) to handle hotspots and grow gracefully.

HardSystem Design

29 practiced

Design a global per-tenant rate limiting system for a multi-tenant API gateway that enforces quotas per tenant per minute and per month across geographic regions. Requirements include low enforcement latency, eventual consistency of counters, dynamic quota updates, and resilience to network partitions. Discuss algorithms, storage options, precision vs performance trade-offs, and techniques to handle clock skew.

Sample Answer

Requirements & clarifications- Enforce per-tenant: per-minute and per-month quotas, global across regions.- Low enforcement latency (ms), eventual consistency on counters, dynamic quota updates, resilient to partitions.

High-level design- API gateways in each region enforce requests locally (fast path). Each gateway maintains an in-memory local token-bucket per tenant for per-minute checks, and a monthly counter cache. Local actions are persisted asynchronously to a global backing store for reconciliation.- A global control plane stores authoritative quotas and aggregates counters. It pushes quota updates and receives periodic diffs from region agents.

Algorithms- Fast local enforcement: token bucket (for rate) and local decrement for monthly quota. Token refill is per-minute granularity for low-latency checks.- Global aggregation: use CRDT PN-counters (positive/negative) or monotonic per-region shards to allow commutative merges and tolerate partitions—sums represent global totals.- Reconciliation: each region periodically (e.g., every 1–5s) flushes its deltas to the global store; global counter = sum of region shards.

Storage options- Low-latency local: in-process memory + local Redis instance (for failover).- Durable/aggregated: distributed datastore supporting wide-area replication and merges: Cassandra or Dynamo-style stores, or CockroachDB if SQL guarantees needed. For CRDT support, storage or middleware can accept region-sharded writes and perform CRDT merges.- For hot tenants, use dedicated in-memory caches with persistence.

Precision vs performance trade-offs- Strong consistency = higher latency (global consensus). Not acceptable for per-request path.- Accept eventual consistency: local fast-path may temporarily allow slight overuse; global reconciliation enforces longer-term quotas and triggers throttling when aggregate exceeds quota.- Tunable parameters: local token allocation size (e.g., allow region to pre-allocate N tokens per tenant): larger allocations reduce remote calls but increase temporary overshoot. For monthly quotas, larger shards reduce coordination.

Handling dynamic quota updates- Control plane publishes updates via pub/sub to region gateways (e.g., Kafka / SNS). Gateways immediately adjust local token bucket capacity and reservation. For previously pre-allocated tokens, either (a) allow them to exhaust (graceful), or (b) fetch reclaim commands—configurable.

Resilience to partitions- During partition, gateways use last-known quotas and in-memory enforcement. Local deltas continue; CRDT shards allow later merge without data loss. To avoid uncontrolled overuse, conservative mode on partition: reduce pre-allocation or switch to strict local limits derived from global quota divided by number of regions.

Clock skew mitigation- Use monotonic timers for token buckets and local refills (do not rely solely on wall-clock). For monthly windows, use logical windows anchored to a trusted UTC epoch; allow small overlap tolerance (e.g., accept timestamps within ±clock_skew_allowance). Reconciliation uses region-write timestamps with vector clock / region-id to order operations where needed; CRDT eliminates strict ordering.

Operational considerations & metrics- Instrument per-tenant overshoot, reconciliation lag, and token pre-allocation utilization.- Alert on sustained overshoot or long reconciliation times.- Provide patterns: strict vs lenient tenant SLAs (different allocation sizes), emergency global hard-stop propagated by control plane.

Trade-offs summary- Low latency: achieved via local token buckets and caches.- Eventual consistency & resilience: achieved with CRDT or region-sharded counters and async flush.- Precision sacrificed for latency; tune token allocation and reconciliation frequency to meet business SLAs.- Use pub/sub for quota pushes; use monotonic clocks + vector/region IDs to reduce clock-dependency.

This design gives ms-level enforcement locally, cross-region correctness eventually, dynamic updates, and partition-tolerant aggregation.

MediumTechnical

28 practiced

Compare strong consistency and eventual consistency models in distributed systems. For each model list appropriate use cases, discuss how they affect latency and availability under network partitions, and explain techniques architects use to provide bounded staleness or read-your-writes guarantees on top of eventual systems.

Sample Answer

Strong consistency vs eventual consistency

Definition:- Strong consistency: after a successful write, all subsequent reads return that write (linearizability/serializability). System appears as a single up-to-date copy.- Eventual consistency: if no new updates occur, replicas will converge to the same state eventually; reads may return stale data.

Use cases:- Strong: financial transactions, inventory decrements, configuration that must be globally immediate (e.g., account balance, distributed locks).- Eventual: user profile replication, social feeds, caching, analytics, geo-replicated read-heavy workloads where low latency and high availability matter.

Latency and availability under partitions (CAP perspective):- Strong consistency favors correctness over availability: during network partitions you must either block operations or risk violating consistency — higher read/write latency or reduced availability.- Eventual consistency favors availability and low latency: nodes can serve reads/writes locally during partitions, but clients may see stale data until convergence.

Techniques to provide bounded staleness or read-your-writes on eventual systems:- Tunable quorums (e.g., Cassandra): choose R and W such that R+W>N to get strong-ish guarantees; lower latency by tuning.- Session/“sticky” consistency: clients bind to a replica and the system tracks session state so the client sees its own writes (read-your-writes, monotonic reads).- Version vectors / vector clocks and causal metadata: preserve causal order so reads respect causality (causal consistency).- Bounded staleness via TTL of divergence or maximum version lag: enforce max replicas-allowed-lag or time-based staleness windows.- Read repair and anti-entropy (background sync): accelerate convergence while keeping low-latency reads.- Leases and primary replicas for short windows: a leader offers linearizability for a lease term; fall back to eventual across regions.- Hybrid approaches (e.g., Causal+ or CRDTs): ensure monotonic and convergent updates for specific data types without global locking.

Trade-offs for architects:- Match consistency model to business risk: use strong where correctness is critical; use eventual + bounded guarantees where latency/availability and geo-distribution are priorities.- Provide clear SLAs to clients (staleness bounds, session guarantees).- Combine patterns: local eventual reads with conditional strong writes (compare-and-set), or strong consistency for control-plane operations and eventual for data-plane.

Unlock Full Question Bank

Get access to hundreds of Scalability Patterns and Techniques interview questions and detailed answers.

Join thousands of developers preparing for their dream job.