Caching Strategies and Distributed Caching Questions
Using caches to reduce latency and load: cache-aside, read-through, write-through, and write-behind patterns, TTLs, eviction policies, and distributed caches such as Redis or Memcached. Covers cache invalidation, stampede and thundering-herd protection, and the consistency tradeoffs of caching. Focuses on where and how to cache across tiers.
Design an invalidation pipeline using Change Data Capture (CDC) (for example Debezium into Kafka) to keep caches updated across multiple services. Discuss topic design, ordering guarantees per key, consumer group design, retry semantics, and how to avoid over-invalidation or event storms.
Sample Answer
Direct answer
An event-driven or change-data-capture (CDC) invalidation pipeline turns every write in the source of truth into a message that downstream consumers use to correct their caches, which requires deliberate topic design, ordering guarantees per key, and idempotent handlers so a redelivered or reordered event does not corrupt cache state.
Structured elaboration
- CDC as the source of events: a tool like Debezium reads the database's own write-ahead log and publishes a change event (insert/update/delete) to a message broker (e.g., Kafka) automatically, which avoids the risk of the application forgetting to publish an invalidation on some code path; the database itself becomes the single source of truth for what changed.
- Message schema: include the key, a version or timestamp, and for deletes a tombstone marker rather than omitting the message; consumers need the version to detect and discard out-of-order or duplicate events, and an explicit tombstone rather than a missing message for deletes.
- Ordering guarantees per key: partition the topic by entity key so all events for the same entity land in the same partition and are processed in order by a single consumer; this avoids a newer update being overwritten by a late-arriving older event.
- Idempotency handling: because most message delivery is at-least-once, a consumer must be able to safely apply the same event twice without harm; using the event's version as a conditional check ("only apply if this version is newer than what's cached") makes redelivery safe.
- Consumer group design: scale consumers horizontally by partition, keeping the per-key ordering guarantee intact since Kafka guarantees order within a partition, not across the whole topic.
- Avoiding over-invalidation or event storms: a bulk update touching millions of rows produces millions of invalidation events; batch or debounce invalidations for the same key within a short window, and consider whether a coarser signal (e.g., "namespace version bumped") is more appropriate than a firehose of individual key invalidations for that specific case.
Worked example
A price-update job that touches 2 million product rows in a batch would, naively, produce 2 million invalidation events in a burst; consumers applying them all immediately would generate 2 million cache misses in a short window, an invalidation-triggered stampede against the very origin the cache exists to protect. Batching invalidations per consumer (apply up to N per second, or coalesce multiple updates to the same key within a short window into one) smooths this into a manageable, sustained rate instead of a spike.
Trade-offs and pitfalls
Relying purely on event-driven invalidation with no time-to-live (TTL) backstop means a lost or unprocessed event becomes permanent staleness with no self-correction; always pair this with a TTL ceiling. Partitioning by key for ordering can create a hot partition if one key is updated far more often than others, which needs the same hot-key mitigations as any other hot-partition problem.
How do you decide whether to introduce a cache for a given service endpoint? Describe the signals and measurements you would collect, the tests you would run (load, latency, profiling), and the criteria that justify adding an in-process cache, a shared cache (Redis), or a CDN. Include considerations for cost, operational complexity, and correctness.
Sample Answer
Direct answer
Decide whether to add a cache by measuring the actual read pattern (how often the same value is requested, how expensive it is to produce, and how much staleness is tolerable), not by defaulting to caching every endpoint; a cache with a low repeat-read rate or zero staleness tolerance is a cost with no real benefit.
Structured elaboration
- Signals to collect: request rate for the same key/query (does the same data actually get read repeatedly, or is nearly every read unique), the cost of producing the value (a fast, cheap lookup gains little from caching even if repeated), and the data's staleness tolerance (how quickly must a change be visible).
- Tests to run: a load test comparing latency and backend load with and without a proposed cache, and a profile of the actual query/computation to confirm it is genuinely a meaningful cost worth caching against.
- Criteria for choosing a cache tier: an in-process cache fits data that is cheap to duplicate per instance and does not need cross-instance consistency; a shared cache (Redis) fits data that benefits from being consistent across instances or too large to duplicate per instance; a content delivery network (CDN) fits public, non-personalized content that benefits from being close to users geographically.
- Cost: weigh the infrastructure and operational cost of adding a caching layer (a new dependency to monitor, secure, and keep available) against the actual load/latency benefit measured above; a marginal benefit may not justify the added complexity.
- Operational complexity: caching adds invalidation logic, a new failure mode (cache unavailable), and another thing to monitor; these costs are real even when the caching decision is otherwise sound, and should be weighed explicitly.
- Correctness: if the data's staleness tolerance is effectively zero (a value that must always reflect the absolute latest state, with no acceptable delay), caching adds risk without benefit, since any caching mechanism introduces at least a small window of potential staleness.
Worked example
An endpoint returning a real-time stock quote, requested uniquely per symbol per user with essentially no repeat reads within any meaningful window, and requiring zero staleness tolerance: this fails on both the "does the same value get read repeatedly" test and the "can staleness be tolerated" test, making it a poor caching candidate regardless of how expensive the underlying computation is. Contrast with a product description, read thousands of times per hour by different users for the same handful of popular items, changing rarely: this passes both tests clearly.
Trade-offs and pitfalls
Caching by default, without measuring the actual read-repetition rate, either wastes cache capacity on data that gets no benefit or, worse, introduces a staleness risk on data that could not tolerate it; always start from measurement, not habit. The decision is not binary per endpoint; the same service can have some data that benefits enormously from caching and other data (even on the same page) that should never be cached, and treating the whole endpoint uniformly misses that nuance.
What is a cache stampede or thundering herd problem and how can it affect reliability? Name and briefly describe at least four practical prevention techniques.
Sample Answer
Direct answer
A cache stampede (also called a thundering herd) happens when a popular cache entry expires or is invalidated and many concurrent requests for that same key all miss at once, sending a burst of simultaneous load to the origin that it was never sized to absorb directly.
Structured elaboration
- Why it happens: caching's whole benefit is deduplicating repeated work; a stampede is exactly the moment that deduplication briefly stops working, because every one of the concurrent requests independently decides "I need to recompute this" at the same instant.
- Why it is dangerous: the origin (a database, an expensive computation, a third-party API) is usually sized assuming the cache absorbs most traffic; a stampede can multiply its load by the number of concurrent requesters for that one key, which for a genuinely hot key can be thousands of requests in a fraction of a second.
- Prevention techniques (name and describe at least four): locking (only the first requester acquires a lock and recomputes, others wait or serve stale data), request coalescing/singleflight (deduplicate concurrent identical fetches into one in-flight request), jittered expirations (randomize time-to-live, TTL, slightly so many keys written together do not expire at the exact same instant), and background/proactive refresh (recompute before expiry so the cache rarely actually goes empty for a hot key).
Worked example
A homepage configuration cached with a 60-second TTL, read 5,000 times per second, expires at exactly the 60-second mark with no protection: in the recomputation window (say 200ms), roughly 1,000 requests (5,000 times 0.2s) would all miss simultaneously and hit the origin. With locking or coalescing, exactly one of those 1,000 requests recomputes; the rest wait briefly or receive the prior value.
Trade-offs and pitfalls
Jitter alone does not fully solve a single very-hot key's stampede risk (it helps most when MANY keys expire together, i.e., a cache avalanche); locking or coalescing is needed for a single key read at very high concurrency. A lock without a timeout can deadlock the system if the process holding it crashes mid-recompute.
Explain cache hit, miss, and eviction. Describe the metrics you would monitor to understand cache health in a production system and how you would measure them. Given a cache that receives 100k requests/min and returns 80k hits, calculate the hit rate and discuss implications for capacity sizing and SLOs.
Sample Answer
Direct answer
A cache hit is a read served from the cache; a miss is a read that had to go to the origin; the hit ratio is hits divided by total requests, and it is the single most important summary number for whether a cache is doing its job.
Structured elaboration
- Hit ratio formula: hit ratio=hits+misseshits. It should be tracked as a rolling metric (e.g., per minute) rather than a single lifetime number, since it reacts to traffic pattern changes, deploys, and cache flushes.
- Eviction: when the cache is full and a new item needs room, an eviction policy (least-recently-used, LRU; least-frequently-used, LFU; and others) picks an existing item to remove; eviction rate is a separate signal from hit ratio and matters because a high eviction rate on an undersized cache is a common root cause of a lower-than-expected hit ratio.
- Measuring it in production: most cache clients and cache servers (e.g., Redis's
INFOcommand) expose hit/miss counters directly; export them to your metrics system and alert on a sustained drop, not a single noisy data point. - What the number implies for capacity sizing: a hit ratio well below what the workload's theoretical repeat-read rate suggests usually means the cache is too small (working set does not fit, so useful entries get evicted before they are reused again) rather than a fundamental mismatch between the workload and caching.
- What it implies for service-level objectives (SLOs): because a miss is typically an order of magnitude or more slower than a hit, the hit ratio directly predicts your p95/p99 latency distribution; a drop in hit ratio should be treated as a leading indicator of a latency SLO breach, not just a cache-health curiosity.
Worked example
A cache receiving 100,000 requests per minute returns 80,000 hits. Hit ratio is 80,000/100,000=0.80, or 80 percent. If a hit takes 2ms and a miss takes 150ms, the blended average latency is 0.80×2ms+0.20×150ms=1.6ms+30ms=31.6ms. If the hit ratio drops to 60 percent with the same per-hit and per-miss costs, blended latency rises to 0.60×2+0.40×150=1.2+60=61.2ms, roughly doubling despite "only" a 20-point hit-ratio drop, which is why hit ratio is such a sensitive early-warning signal.
Trade-offs and pitfalls
A hit ratio number without context is misleading: 80 percent is excellent for a highly diverse, long-tail workload and mediocre for a workload with a small, stable, highly-repeated key set that should be closer to 99 percent. Always compare against the workload's own historical baseline and theoretical ceiling, not a generic industry number.
You are designing cache invalidation for a globally distributed service where reads are frequent and writes happen in a primary region: describe strategies to keep caches coherent across regions. Discuss trade-offs between consistency, staleness, cost, and complexity (push invalidation, TTL, versioned keys, fanout updates).
Sample Answer
Direct answer
Keeping caches coherent across regions means picking, per data class, between a pull model (short time-to-live, TTL, that self-heals without any messaging) and a push model (explicit invalidation events fanned out to every region), and being explicit about the bounded-staleness window you are willing to accept in exchange for lower cost and complexity.
Structured elaboration
- TTL (pull-based): the simplest option. Every region's cache entry expires on its own after a fixed window; no cross-region messaging is required. Staleness is bounded by the TTL itself, cost is near zero, and it self-heals from any missed update (a lost invalidation event just means the entry expires normally at worst-case TTL). The downside is that staleness is guaranteed, not just possible, for the full TTL window.
- Push invalidation via messaging: on write, the writing region publishes an invalidation event (key, and optionally a version or tombstone) to a durable message bus; consumers in every other region apply it to their local cache. This gets close-to-immediate propagation (bounded by replication lag of the message bus, typically under a second) but adds a moving part that can itself fail, reorder, or duplicate, so you still need a TTL as a backstop.
- Versioned keys: instead of invalidating, bump a version token that is part of the cache key (
profile:123:v42); old versions simply age out via normal eviction rather than needing an active delete. This avoids race conditions where an invalidation arrives before the write it corresponds to, at the cost of needing a place to look up "what's the current version" (which itself needs to be fast and consistent). - Fanout updates: instead of invalidating (forcing every region to re-fetch), push the new VALUE itself to every region's cache. This trades more network bandwidth (sending full payloads, not small invalidation markers) for lower read-side latency (no cache-miss round-trip to origin after the update lands), and is worth it for small, very hot objects.
- Bounded staleness as the actual requirement: whichever mechanism you choose, state the staleness bound explicitly (e.g., "under 5 seconds in 99% of cases") and design the propagation pipeline's monitoring around that number, rather than treating "eventually consistent" as sufficient specification.
Worked example
A profile-update event published in the primary write region reaches a message bus with typical cross-region replication lag of 200 to 500 ms, plus consumer processing time of tens of milliseconds per region; a realistic end-to-end propagation time is under 1 second for the median case, with a long tail driven by consumer backlog during traffic spikes. If the product requirement is "profile changes visible within 5 seconds globally," a push-based invalidation pipeline with a 5-second TTL backstop comfortably meets it even if a specific invalidation event is dropped, because the TTL guarantees the bound independently.
Trade-offs and pitfalls
Relying purely on push invalidation without a TTL backstop means any lost, delayed, or duplicated event becomes an unbounded staleness bug with no self-healing mechanism; always pair push invalidation with a TTL ceiling. Fanout-updates for large or rarely-read objects wastes bandwidth pushing data nobody will read in some regions; reserve it for small, universally-hot keys. Versioned keys solve the ordering race but shift complexity to "how fast and how consistent is the version lookup itself," which can become its own single point of failure if not designed carefully.
Unlock Full Question Bank
Get access to all 13 Caching Strategies and Distributed Caching interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.