Caching Strategies and Distributed Caching Questions
Using caches to reduce latency and load: cache-aside, read-through, write-through, and write-behind patterns, TTLs, eviction policies, and distributed caches such as Redis or Memcached. Covers cache invalidation, stampede and thundering-herd protection, and the consistency tradeoffs of caching. Focuses on where and how to cache across tiers.
What is a cache stampede or thundering herd problem and how can it affect reliability? Name and briefly describe at least four practical prevention techniques.
Sample Answer
Direct answer
A cache stampede (also called a thundering herd) happens when a popular cache entry expires or is invalidated and many concurrent requests for that same key all miss at once, sending a burst of simultaneous load to the origin that it was never sized to absorb directly.
Structured elaboration
- Why it happens: caching's whole benefit is deduplicating repeated work; a stampede is exactly the moment that deduplication briefly stops working, because every one of the concurrent requests independently decides "I need to recompute this" at the same instant.
- Why it is dangerous: the origin (a database, an expensive computation, a third-party API) is usually sized assuming the cache absorbs most traffic; a stampede can multiply its load by the number of concurrent requesters for that one key, which for a genuinely hot key can be thousands of requests in a fraction of a second.
- Prevention techniques (name and describe at least four): locking (only the first requester acquires a lock and recomputes, others wait or serve stale data), request coalescing/singleflight (deduplicate concurrent identical fetches into one in-flight request), jittered expirations (randomize time-to-live, TTL, slightly so many keys written together do not expire at the exact same instant), and background/proactive refresh (recompute before expiry so the cache rarely actually goes empty for a hot key).
Worked example
A homepage configuration cached with a 60-second TTL, read 5,000 times per second, expires at exactly the 60-second mark with no protection: in the recomputation window (say 200ms), roughly 1,000 requests (5,000 times 0.2s) would all miss simultaneously and hit the origin. With locking or coalescing, exactly one of those 1,000 requests recomputes; the rest wait briefly or receive the prior value.
Trade-offs and pitfalls
Jitter alone does not fully solve a single very-hot key's stampede risk (it helps most when MANY keys expire together, i.e., a cache avalanche); locking or coalescing is needed for a single key read at very high concurrency. A lock without a timeout can deadlock the system if the process holding it crashes mid-recompute.
Explain the purpose of caching in distributed systems. Define cache hit, miss, and hit ratio, and describe the typical benefits and tradeoffs of adding a cache to a service. Give concrete examples of workloads that benefit from caching (and why) and workloads where caching could be harmful.
Sample Answer
Direct answer
A cache is a fast, smaller copy of data kept close to where it is used, so repeated reads can be served without redoing expensive work (a database query, a network call, a heavy computation) every time. The core trade-off is speed and reduced load in exchange for the risk that the cached copy becomes stale relative to the real source of truth.
Structured elaboration
- What problems caching solves: latency (serving from memory or a nearby node is far faster than recomputing or fetching from a distant source), throughput/load (fewer requests reach the expensive backend), and cost (fewer database queries or external application programming interface (API) calls, which often cost money directly).
- Where caches typically live: client/browser (closest to the user, zero network cost on a hit), content delivery network (CDN) / edge (near the user but shared across many users), application in-memory (fast, but local to one process/instance), and a shared distributed cache like Redis or Memcached (shared across all instances in a region, one network hop away).
- Primary trade-offs: staleness (the cached copy may not reflect the latest write), added complexity (invalidation logic, cache-miss handling, monitoring another moving part), and memory cost (caches are not free storage).
- When caching helps: read-heavy workloads where the same data is requested repeatedly and can tolerate at least a little staleness, or where the underlying computation/fetch is expensive relative to a cache read.
- When caching is harmful: data that changes on every read (no repeated value to cache), workloads that are already write-heavy with low read repetition (cache churn without benefit), or correctness-critical data where any staleness is unacceptable and the added complexity of cache invalidation introduces more risk than the latency win is worth.
Worked example
An API endpoint that computes a dashboard aggregate over a large dataset in 800ms, requested 200 times per minute by the same handful of users, is a strong caching candidate: a 30-second time-to-live (TTL) cache serves nearly every request from cache after the first, cutting both latency (single-digit milliseconds instead of 800ms) and backend load by over 95 percent, for a staleness window most dashboard users will not notice. The same 30-second TTL applied to an account balance shown right after a deposit would be actively harmful, showing the user a stale, "wrong" number at the exact moment they are checking it.
Trade-offs and pitfalls
The most common mistake is caching by default rather than by evaluating whether the read pattern and staleness tolerance actually justify it; caching adds a second place data can be wrong (the cache disagreeing with the source of truth), and that failure mode does not exist at all if you never cache the data in the first place.
Compare TTL (time-to-live) based expiration with explicit eviction/purge. Provide examples of use-cases where TTL is sufficient and where explicit invalidation is necessary. Also list drawbacks of using long TTL values.
Sample Answer
Direct answer
Time-to-live (TTL) expiration is a passive, self-healing bound on staleness that requires no extra machinery; explicit invalidation (purge, versioned keys, or event-driven invalidation) actively corrects the cache the moment the source changes, at the cost of needing a reliable signal that a change happened.
Structured elaboration
- When TTL is sufficient: content where a bounded staleness window is genuinely acceptable (a homepage banner, a "trending now" list, most analytics dashboards); TTL requires no coordination and self-heals even if a write is missed entirely, because the entry expires on schedule regardless.
- When explicit invalidation is necessary: content where staleness has real consequences (permission revocation, inventory during checkout, a price correction that must take effect immediately); waiting out even a short TTL is unacceptable when correctness, not just freshness, is on the line.
- Versioned keys as a form of explicit invalidation: rather than deleting the old entry, bump a version token that is part of the key; the old version simply becomes unreachable and ages out naturally, avoiding races where a delete arrives before or after the write it corresponds to.
- Event-driven invalidation: a write publishes an event (directly or via change data capture, CDC) that consumers use to purge or update affected cache entries; this gets close to real-time correction but adds a dependency on the message delivery being reliable.
- A hybrid approach: use a moderate TTL as a safety net (so a lost invalidation event self-heals within a bounded window) combined with explicit invalidation for the fast path; this is the most common production pattern because it gets the responsiveness of explicit invalidation with the resilience of TTL.
Worked example
A user's permission level changes from "member" to "banned." Relying purely on a 5-minute TTL means the banned user could still access restricted content for up to 5 minutes; that is not acceptable, so this write triggers explicit invalidation of the user's cached permission entry, with a short TTL (e.g., 60 seconds) still applied as a safety net in case the invalidation event is somehow lost.
Trade-offs and pitfalls
Long TTLs used purely for "simplicity" on data that actually needs freshness is one of the most common production incidents in caching (a stale price, a stale permission, a stale feature flag reaching users for far longer than intended); size the TTL to the actual staleness tolerance of the data, not to whatever is convenient to implement. Relying on explicit invalidation alone, with no TTL backstop, means any lost or delayed invalidation event becomes unbounded staleness with no self-correction.
How do you decide whether to introduce a cache for a given service endpoint? Describe the signals and measurements you would collect, the tests you would run (load, latency, profiling), and the criteria that justify adding an in-process cache, a shared cache (Redis), or a CDN. Include considerations for cost, operational complexity, and correctness.
Sample Answer
Direct answer
Decide whether to add a cache by measuring the actual read pattern (how often the same value is requested, how expensive it is to produce, and how much staleness is tolerable), not by defaulting to caching every endpoint; a cache with a low repeat-read rate or zero staleness tolerance is a cost with no real benefit.
Structured elaboration
- Signals to collect: request rate for the same key/query (does the same data actually get read repeatedly, or is nearly every read unique), the cost of producing the value (a fast, cheap lookup gains little from caching even if repeated), and the data's staleness tolerance (how quickly must a change be visible).
- Tests to run: a load test comparing latency and backend load with and without a proposed cache, and a profile of the actual query/computation to confirm it is genuinely a meaningful cost worth caching against.
- Criteria for choosing a cache tier: an in-process cache fits data that is cheap to duplicate per instance and does not need cross-instance consistency; a shared cache (Redis) fits data that benefits from being consistent across instances or too large to duplicate per instance; a content delivery network (CDN) fits public, non-personalized content that benefits from being close to users geographically.
- Cost: weigh the infrastructure and operational cost of adding a caching layer (a new dependency to monitor, secure, and keep available) against the actual load/latency benefit measured above; a marginal benefit may not justify the added complexity.
- Operational complexity: caching adds invalidation logic, a new failure mode (cache unavailable), and another thing to monitor; these costs are real even when the caching decision is otherwise sound, and should be weighed explicitly.
- Correctness: if the data's staleness tolerance is effectively zero (a value that must always reflect the absolute latest state, with no acceptable delay), caching adds risk without benefit, since any caching mechanism introduces at least a small window of potential staleness.
Worked example
An endpoint returning a real-time stock quote, requested uniquely per symbol per user with essentially no repeat reads within any meaningful window, and requiring zero staleness tolerance: this fails on both the "does the same value get read repeatedly" test and the "can staleness be tolerated" test, making it a poor caching candidate regardless of how expensive the underlying computation is. Contrast with a product description, read thousands of times per hour by different users for the same handful of popular items, changing rarely: this passes both tests clearly.
Trade-offs and pitfalls
Caching by default, without measuring the actual read-repetition rate, either wastes cache capacity on data that gets no benefit or, worse, introduces a staleness risk on data that could not tolerate it; always start from measurement, not habit. The decision is not binary per endpoint; the same service can have some data that benefits enormously from caching and other data (even on the same page) that should never be cached, and treating the whole endpoint uniformly misses that nuance.
Compare the cache-aside, read-through, write-through, and write-behind caching patterns. For each pattern describe: (a) how reads and writes flow between cache and data store, (b) a typical use case, and (c) the main advantage and drawback. Give one example service type where each pattern is a good fit.
Sample Answer
Direct answer
Cache-aside, read-through, write-through, and write-behind differ in who is responsible for populating the cache and when a write becomes durable: cache-aside puts that responsibility on the application, read-through/write-through push it into the caching layer itself, and write-behind trades immediate durability for write throughput.
Structured elaboration
- Cache-aside (lazy loading): on read, the application checks the cache; on a miss, it reads from the datastore and populates the cache itself. On write, the application writes to the datastore and either invalidates or updates the cache entry. The application owns all the logic; the cache is a dumb key-value store. This is the most common pattern because it fails gracefully (if the cache is down, reads just go straight to the datastore) and only caches what is actually requested.
- Read-through: functionally similar to cache-aside from the caller's perspective, but the cache library/layer itself knows how to fetch from the datastore on a miss, so the application only ever talks to the cache. This centralizes the fetch logic but requires a caching layer that supports it.
- Write-through: every write goes to the cache first (or simultaneously), and the cache synchronously writes through to the datastore before acknowledging. Reads are always fresh because the cache is never behind the datastore, at the cost of write latency (you pay for both writes on every request) and caching data that may never actually be read.
- Write-behind (write-back): writes go to the cache and are acknowledged immediately; the cache asynchronously flushes to the datastore in the background (often batched). This gives the best write throughput and latency, at the cost of a durability window: a crash between the acknowledged write and the flush can lose data unless the write queue itself is durable.
- Picking one, by use case: a product catalog with heavy reads and occasional updates fits cache-aside well (simple, only caches what's actually browsed). A durability-sensitive write path (a payments ledger) generally avoids write-behind's data-loss window and prefers write-through or a cache-aside pattern with synchronous invalidation.
Worked example
For a product catalog service: cache-aside is a strong default. On a product-detail read, check Redis; on miss, query the database and populate Redis with a time-to-live (TTL); on a price update, write to the database and then delete (or update) the cached entry so the next read repopulates it. This avoids caching the 90+ percent of the catalog nobody is currently browsing, unlike write-through, which would populate the cache for every single write regardless of read demand.
Trade-offs and pitfalls
Cache-aside has a well-known race: a read that misses, starts fetching from the datastore, and finishes AFTER a concurrent write has already invalidated the cache, can re-populate the cache with the now-stale value it fetched before the write. Write-through eliminates staleness but adds write latency and can cache "dead weight" (data nobody reads). Write-behind's throughput win is real but its durability trade-off must be an explicit decision, not a default; never use write-behind for data where losing the last few seconds of writes is unacceptable.
Unlock Full Question Bank
Get access to all 7 Caching Strategies and Distributed Caching interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.