Caching Strategies and Distributed Caching Questions
Using caches to reduce latency and load: cache-aside, read-through, write-through, and write-behind patterns, TTLs, eviction policies, and distributed caches such as Redis or Memcached. Covers cache invalidation, stampede and thundering-herd protection, and the consistency tradeoffs of caching. Focuses on where and how to cache across tiers.
Describe a multi-level caching architecture for a web service: L1 in-process cache (per instance), L2 shared cache (Redis cluster), and a CDN in front of static assets. Explain read and write flows, benefits for latency and throughput, and the primary consistency and invalidation challenges for each layer.
Sample Answer
Direct answer
Layer the caches by how expensive a miss at that layer is and how often the content changes: content delivery network (CDN) edge for static/shared assets closest to the user, a regional shared cache (e.g., Redis) for personalized-but-cacheable data, and an application in-process cache for the hottest few keys where even a network hop to the regional cache is too slow, with each layer's time-to-live (TTL) and invalidation strategy sized to that layer's job.
Structured elaboration
- CDN/edge layer: caches fully static or long-TTL content (images, JS/CSS bundles, and cacheable HTML fragments for anonymous/public views) at points of presence near the user. This absorbs the largest fraction of read volume for almost no cost per request and should hold anything that does not vary per-user.
- Regional distributed cache (Redis/Memcached): serves personalized or frequently-changing data shared across all app instances in a region (product details, prices, computed recommendations). Requests per second (RPS) here are much lower than at the edge because the edge already absorbed the static traffic, but this layer must handle write-driven invalidation correctly since content changes.
- Application in-process (L1) cache: for the small set of extremely hot keys (a handful of top-selling products, a config flag read on every request), an in-process cache avoids even the network round-trip to the regional cache. It is the fastest layer and also the hardest to keep coherent, because every app instance has its own copy; use a short TTL or an event-driven invalidation signal (pub/sub) rather than relying on the regional cache alone.
- Database-side / materialized views: for expensive aggregate queries, a materialized view or a query-result cache sits closest to the database, reducing load on the primary store even when the higher layers miss.
- Invalidation strategy per layer: static assets at the edge use content-hashed URLs so "invalidation" is really just a new URL (no purge needed); the regional cache uses event-driven invalidation on write (an order/price update publishes an invalidation for that product's key); the in-process L1 layer uses a short TTL (seconds) plus a lightweight pub/sub signal, because coordinating a purge across every instance is expensive and slow.
- Sizing to a concrete target: pick numbers for the design (RPS, latency budget, SKU/product count) and work backward: at, say, 100,000 RPS globally with a 90 percent edge-cacheable static-asset ratio, only about 10,000 RPS reach the regional application tier, which then needs to comfortably serve that load with sub-50ms p95 latency from cache.
Worked example
For a product catalog with 1,000,000 stock-keeping units (SKUs) and 50,000 RPS globally: static images and category pages are edge-cached (roughly 35,000 RPS absorbed at essentially zero backend cost). The remaining ~15,000 RPS of personalized/price-sensitive reads hit the regional Redis cache; with a realistic 95 percent hit ratio there, the origin database sees roughly 750 RPS, which is a design a mid-sized read replica tier handles comfortably. Price changes (a write path) publish an invalidation event per SKU, propagated to the regional cache and any in-process L1 caches holding that SKU within roughly 100 to 500 ms, well inside a "price must update within seconds" product requirement.
Trade-offs and pitfalls
Adding more layers adds more places for staleness to hide; a change that only invalidates the regional cache and forgets the in-process L1 layer will show correct data to some app instances and stale data to others, which is a confusing bug class to debug. Do not put personalization-sensitive content at the CDN edge unless you are using edge compute (e.g., edge functions) that can vary the response per user; naively caching personalized HTML at a shared edge node leaks one user's data to another. Every additional layer is also an additional operational surface (its own metrics, its own failure mode, its own on-call runbook); do not add a layer unless the sizing math shows the layer above it cannot meet the latency or load target alone.
Design a multi-region caching strategy for a global application that requires sub-50ms read latency worldwide but strong consistency for user profile writes. Compare active-active replicated caches with conflict resolution, a global master for writes with local read caches, and CRDT-based approaches. Recommend an approach and justify trade-offs for latency, consistency, and operational complexity.
Sample Answer
Direct answer
There is no design that gives you both sub-50ms global reads and strong consistency for every write; you choose where the compromise sits. For most globally-distributed services the right default is regional read caches with a single-writer region per key (or per user) plus asynchronous replication, escalating to active-active with conflict resolution only for the specific keys that genuinely need multi-region writes.
Structured elaboration
- Global master with local read caches: writes always go to one authoritative region (chosen per key, e.g., the user's home region); that region's cache is updated synchronously, and other regions' caches are populated by asynchronous replication or event-driven invalidation. Reads in the writer's own region are strongly consistent; reads elsewhere are eventually consistent with a bounded lag (typically under a few seconds for a well-tuned replication pipeline). This is the simplest model to reason about and is the right default unless a specific workload proves it insufficient.
- Active-active with conflict resolution: every region can accept writes to the same key; conflicts are resolved with last-write-wins (using synchronized clocks or hybrid logical clocks), version vectors, or application-specific merge logic. This removes the single-writer bottleneck but pushes real complexity into conflict resolution and makes the failure modes harder to reason about; reserve it for keys where availability during a regional partition matters more than avoiding conflicts (e.g., a shopping cart, where a merge is acceptable).
- CRDT-based approaches: conflict-free replicated data types (structures with a mathematically well-defined merge function, such as a grow-only counter or a last-writer-wins register) let every region write locally and merge automatically without coordination. They are the cleanest fit when the data shape is naturally a CRDT (counters, sets, simple registers) but do not generalize to arbitrary application state.
- Choosing the read-latency-vs-consistency point per requirement: a common pattern is to segment data by freshness requirement. Authentication and permission state gets synchronous, single-region-authoritative reads (correctness matters more than latency). Profile pictures, preferences, and feed content get regional caches with eventual consistency (latency matters more, and staleness of a few seconds is invisible to users).
Worked example
For a service needing sub-50ms p95 reads worldwide with a single global-master design and 150ms average inter-region round-trip time: a user in Singapore reading data whose write-authoritative region is US-East would need a cross-region hop on every uncached read, blowing the 50ms budget by 3x. The fix is not "make replication faster" but "make the region the user reads from also the region their writes update" (their home region is the writer), with cross-region only used for failover. Under active-active with a 5-second bounded staleness target, replication lag has to be monitored per-region-pair and alerted when it exceeds roughly half the staleness budget (2.5s), leaving headroom for the merge/apply step.
Trade-offs and pitfalls
The most common design mistake is picking one consistency model for the whole service instead of per data class; forcing strong consistency on data that does not need it (e.g., a "last seen" timestamp) burns your entire latency budget for no user-visible benefit. Active-active without a clear conflict-resolution policy invites silent data loss (the losing write of a last-write-wins resolution simply disappears); make sure product owners have explicitly signed off on what "conflict" means for each data type before choosing this model. CRDTs remove coordination cost but restrict your data model; retrofitting CRDTs onto an existing schema is usually more work than designing around single-writer-per-key from the start.
Design an invalidation pipeline using Change Data Capture (CDC) (for example Debezium into Kafka) to keep caches updated across multiple services. Discuss topic design, ordering guarantees per key, consumer group design, retry semantics, and how to avoid over-invalidation or event storms.
Sample Answer
Direct answer
An event-driven or change-data-capture (CDC) invalidation pipeline turns every write in the source of truth into a message that downstream consumers use to correct their caches, which requires deliberate topic design, ordering guarantees per key, and idempotent handlers so a redelivered or reordered event does not corrupt cache state.
Structured elaboration
- CDC as the source of events: a tool like Debezium reads the database's own write-ahead log and publishes a change event (insert/update/delete) to a message broker (e.g., Kafka) automatically, which avoids the risk of the application forgetting to publish an invalidation on some code path; the database itself becomes the single source of truth for what changed.
- Message schema: include the key, a version or timestamp, and for deletes a tombstone marker rather than omitting the message; consumers need the version to detect and discard out-of-order or duplicate events, and an explicit tombstone rather than a missing message for deletes.
- Ordering guarantees per key: partition the topic by entity key so all events for the same entity land in the same partition and are processed in order by a single consumer; this avoids a newer update being overwritten by a late-arriving older event.
- Idempotency handling: because most message delivery is at-least-once, a consumer must be able to safely apply the same event twice without harm; using the event's version as a conditional check ("only apply if this version is newer than what's cached") makes redelivery safe.
- Consumer group design: scale consumers horizontally by partition, keeping the per-key ordering guarantee intact since Kafka guarantees order within a partition, not across the whole topic.
- Avoiding over-invalidation or event storms: a bulk update touching millions of rows produces millions of invalidation events; batch or debounce invalidations for the same key within a short window, and consider whether a coarser signal (e.g., "namespace version bumped") is more appropriate than a firehose of individual key invalidations for that specific case.
Worked example
A price-update job that touches 2 million product rows in a batch would, naively, produce 2 million invalidation events in a burst; consumers applying them all immediately would generate 2 million cache misses in a short window, an invalidation-triggered stampede against the very origin the cache exists to protect. Batching invalidations per consumer (apply up to N per second, or coalesce multiple updates to the same key within a short window into one) smooths this into a manageable, sustained rate instead of a spike.
Trade-offs and pitfalls
Relying purely on event-driven invalidation with no time-to-live (TTL) backstop means a lost or unprocessed event becomes permanent staleness with no self-correction; always pair this with a TTL ceiling. Partitioning by key for ordering can create a hot partition if one key is updated far more often than others, which needs the same hot-key mitigations as any other hot-partition problem.
Explain cache hit, miss, and eviction. Describe the metrics you would monitor to understand cache health in a production system and how you would measure them. Given a cache that receives 100k requests/min and returns 80k hits, calculate the hit rate and discuss implications for capacity sizing and SLOs.
Sample Answer
Direct answer
A cache hit is a read served from the cache; a miss is a read that had to go to the origin; the hit ratio is hits divided by total requests, and it is the single most important summary number for whether a cache is doing its job.
Structured elaboration
- Hit ratio formula: hit ratio=hits+misseshits. It should be tracked as a rolling metric (e.g., per minute) rather than a single lifetime number, since it reacts to traffic pattern changes, deploys, and cache flushes.
- Eviction: when the cache is full and a new item needs room, an eviction policy (least-recently-used, LRU; least-frequently-used, LFU; and others) picks an existing item to remove; eviction rate is a separate signal from hit ratio and matters because a high eviction rate on an undersized cache is a common root cause of a lower-than-expected hit ratio.
- Measuring it in production: most cache clients and cache servers (e.g., Redis's
INFOcommand) expose hit/miss counters directly; export them to your metrics system and alert on a sustained drop, not a single noisy data point. - What the number implies for capacity sizing: a hit ratio well below what the workload's theoretical repeat-read rate suggests usually means the cache is too small (working set does not fit, so useful entries get evicted before they are reused again) rather than a fundamental mismatch between the workload and caching.
- What it implies for service-level objectives (SLOs): because a miss is typically an order of magnitude or more slower than a hit, the hit ratio directly predicts your p95/p99 latency distribution; a drop in hit ratio should be treated as a leading indicator of a latency SLO breach, not just a cache-health curiosity.
Worked example
A cache receiving 100,000 requests per minute returns 80,000 hits. Hit ratio is 80,000/100,000=0.80, or 80 percent. If a hit takes 2ms and a miss takes 150ms, the blended average latency is 0.80×2ms+0.20×150ms=1.6ms+30ms=31.6ms. If the hit ratio drops to 60 percent with the same per-hit and per-miss costs, blended latency rises to 0.60×2+0.40×150=1.2+60=61.2ms, roughly doubling despite "only" a 20-point hit-ratio drop, which is why hit ratio is such a sensitive early-warning signal.
Trade-offs and pitfalls
A hit ratio number without context is misleading: 80 percent is excellent for a highly diverse, long-tail workload and mediocre for a workload with a small, stable, highly-repeated key set that should be closer to 99 percent. Always compare against the workload's own historical baseline and theoretical ceiling, not a generic industry number.
What is a cache stampede or thundering herd problem and how can it affect reliability? Name and briefly describe at least four practical prevention techniques.
Sample Answer
Direct answer
A cache stampede (also called a thundering herd) happens when a popular cache entry expires or is invalidated and many concurrent requests for that same key all miss at once, sending a burst of simultaneous load to the origin that it was never sized to absorb directly.
Structured elaboration
- Why it happens: caching's whole benefit is deduplicating repeated work; a stampede is exactly the moment that deduplication briefly stops working, because every one of the concurrent requests independently decides "I need to recompute this" at the same instant.
- Why it is dangerous: the origin (a database, an expensive computation, a third-party API) is usually sized assuming the cache absorbs most traffic; a stampede can multiply its load by the number of concurrent requesters for that one key, which for a genuinely hot key can be thousands of requests in a fraction of a second.
- Prevention techniques (name and describe at least four): locking (only the first requester acquires a lock and recomputes, others wait or serve stale data), request coalescing/singleflight (deduplicate concurrent identical fetches into one in-flight request), jittered expirations (randomize time-to-live, TTL, slightly so many keys written together do not expire at the exact same instant), and background/proactive refresh (recompute before expiry so the cache rarely actually goes empty for a hot key).
Worked example
A homepage configuration cached with a 60-second TTL, read 5,000 times per second, expires at exactly the 60-second mark with no protection: in the recomputation window (say 200ms), roughly 1,000 requests (5,000 times 0.2s) would all miss simultaneously and hit the origin. With locking or coalescing, exactly one of those 1,000 requests recomputes; the rest wait briefly or receive the prior value.
Trade-offs and pitfalls
Jitter alone does not fully solve a single very-hot key's stampede risk (it helps most when MANY keys expire together, i.e., a cache avalanche); locking or coalescing is needed for a single key read at very high concurrency. A lock without a timeout can deadlock the system if the process holding it crashes mid-recompute.
Unlock Full Question Bank
Get access to all 13 Caching Strategies and Distributed Caching interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.