Caching Strategies and Distributed Caching Questions
Using caches to reduce latency and load: cache-aside, read-through, write-through, and write-behind patterns, TTLs, eviction policies, and distributed caches such as Redis or Memcached. Covers cache invalidation, stampede and thundering-herd protection, and the consistency tradeoffs of caching. Focuses on where and how to cache across tiers.
Compare cache placement options: client-side, CDN/edge, reverse-proxy (e.g., Varnish), application-level in-memory (e.g., Redis/Memcached), and database-side (materialized views or DB-level caching). For each option describe pros, cons, typical use cases, security/privacy considerations, and how TTLs and invalidation differ by placement.
Sample Answer
Direct answer
Cache placement is a ladder from "closest to the user, cheapest, hardest to invalidate precisely" to "closest to the source of truth, most expensive per request, easiest to keep correct": client-side, content delivery network (CDN)/edge, reverse proxy, application in-memory, and database-side.
Structured elaboration
- Client-side: browser cache, local storage, or a mobile app's local store. Zero network cost on a hit, but you have essentially no control once data leaves your servers; invalidation means waiting out a time-to-live (TTL) or changing a versioned URL.
- CDN/edge: shared across all users near a given geography, ideal for static or long-TTL, non-personalized content. Purge/invalidation is slower (can take seconds to propagate globally) and typically coarser-grained than a server-side cache.
- Reverse proxy (e.g., Varnish): sits in front of your application servers, caching full HTTP responses; good for reducing application-server load for cacheable pages without needing application code changes, but still shared and public unless carefully scoped per-user.
- Application-level in-memory (Redis/Memcached, or in-process): shared across your own service's instances (Redis/Memcached) or private to one instance (in-process); this is where most business-logic caching happens, because you have full control over invalidation and can cache personalized data safely.
- Database-side (materialized views, query result caching): closest to the source of truth, so almost always the most consistent option, at the cost of doing the least to reduce load on the database itself.
- Redis vs. CDN by use case: for static assets, a CDN wins outright (no reason to burn application-tier memory on data that never changes per-request). For personalized HTML fragments, a CDN only works with edge compute; otherwise application-tier Redis is the safe default. For frequently-read configuration flags, an in-process or small Redis cache with a short TTL beats a CDN, since the data is tiny and needs low latency, not global edge distribution. For large objects with varying TTLs (e.g., images), a CDN with per-object cache-control headers is the natural fit.
Worked example
A product page has a static hero image (CDN, long TTL, content-hashed URL so invalidation is just a new URL), a shared "similar products" block computed the same for all users (reverse proxy or application cache, medium TTL), and a personalized "recently viewed" section (application-level cache keyed per user, short TTL, never placed at a shared CDN/proxy layer).
Trade-offs and pitfalls
Placing personalized content at a shared caching layer (CDN or reverse proxy) without per-user cache keys is a serious privacy bug, not just a staleness inconvenience; one user's private data can be served to another. The further a cache sits from the source of truth, the cheaper it is per request and the harder it is to invalidate precisely; choose the placement based on how quickly and precisely that specific data needs to be corrected on write.
Compare TTL (time-to-live) based expiration with explicit eviction/purge. Provide examples of use-cases where TTL is sufficient and where explicit invalidation is necessary. Also list drawbacks of using long TTL values.
Sample Answer
Direct answer
Time-to-live (TTL) expiration is a passive, self-healing bound on staleness that requires no extra machinery; explicit invalidation (purge, versioned keys, or event-driven invalidation) actively corrects the cache the moment the source changes, at the cost of needing a reliable signal that a change happened.
Structured elaboration
- When TTL is sufficient: content where a bounded staleness window is genuinely acceptable (a homepage banner, a "trending now" list, most analytics dashboards); TTL requires no coordination and self-heals even if a write is missed entirely, because the entry expires on schedule regardless.
- When explicit invalidation is necessary: content where staleness has real consequences (permission revocation, inventory during checkout, a price correction that must take effect immediately); waiting out even a short TTL is unacceptable when correctness, not just freshness, is on the line.
- Versioned keys as a form of explicit invalidation: rather than deleting the old entry, bump a version token that is part of the key; the old version simply becomes unreachable and ages out naturally, avoiding races where a delete arrives before or after the write it corresponds to.
- Event-driven invalidation: a write publishes an event (directly or via change data capture, CDC) that consumers use to purge or update affected cache entries; this gets close to real-time correction but adds a dependency on the message delivery being reliable.
- A hybrid approach: use a moderate TTL as a safety net (so a lost invalidation event self-heals within a bounded window) combined with explicit invalidation for the fast path; this is the most common production pattern because it gets the responsiveness of explicit invalidation with the resilience of TTL.
Worked example
A user's permission level changes from "member" to "banned." Relying purely on a 5-minute TTL means the banned user could still access restricted content for up to 5 minutes; that is not acceptable, so this write triggers explicit invalidation of the user's cached permission entry, with a short TTL (e.g., 60 seconds) still applied as a safety net in case the invalidation event is somehow lost.
Trade-offs and pitfalls
Long TTLs used purely for "simplicity" on data that actually needs freshness is one of the most common production incidents in caching (a stale price, a stale permission, a stale feature flag reaching users for far longer than intended); size the TTL to the actual staleness tolerance of the data, not to whatever is convenient to implement. Relying on explicit invalidation alone, with no TTL backstop, means any lost or delayed invalidation event becomes unbounded staleness with no self-correction.
What is a cache stampede or thundering herd problem and how can it affect reliability? Name and briefly describe at least four practical prevention techniques.
Sample Answer
Direct answer
A cache stampede (also called a thundering herd) happens when a popular cache entry expires or is invalidated and many concurrent requests for that same key all miss at once, sending a burst of simultaneous load to the origin that it was never sized to absorb directly.
Structured elaboration
- Why it happens: caching's whole benefit is deduplicating repeated work; a stampede is exactly the moment that deduplication briefly stops working, because every one of the concurrent requests independently decides "I need to recompute this" at the same instant.
- Why it is dangerous: the origin (a database, an expensive computation, a third-party API) is usually sized assuming the cache absorbs most traffic; a stampede can multiply its load by the number of concurrent requesters for that one key, which for a genuinely hot key can be thousands of requests in a fraction of a second.
- Prevention techniques (name and describe at least four): locking (only the first requester acquires a lock and recomputes, others wait or serve stale data), request coalescing/singleflight (deduplicate concurrent identical fetches into one in-flight request), jittered expirations (randomize time-to-live, TTL, slightly so many keys written together do not expire at the exact same instant), and background/proactive refresh (recompute before expiry so the cache rarely actually goes empty for a hot key).
Worked example
A homepage configuration cached with a 60-second TTL, read 5,000 times per second, expires at exactly the 60-second mark with no protection: in the recomputation window (say 200ms), roughly 1,000 requests (5,000 times 0.2s) would all miss simultaneously and hit the origin. With locking or coalescing, exactly one of those 1,000 requests recomputes; the rest wait briefly or receive the prior value.
Trade-offs and pitfalls
Jitter alone does not fully solve a single very-hot key's stampede risk (it helps most when MANY keys expire together, i.e., a cache avalanche); locking or coalescing is needed for a single key read at very high concurrency. A lock without a timeout can deadlock the system if the process holding it crashes mid-recompute.
Explain the purpose of caching in distributed systems. Define cache hit, miss, and hit ratio, and describe the typical benefits and tradeoffs of adding a cache to a service. Give concrete examples of workloads that benefit from caching (and why) and workloads where caching could be harmful.
Sample Answer
Direct answer
A cache is a fast, smaller copy of data kept close to where it is used, so repeated reads can be served without redoing expensive work (a database query, a network call, a heavy computation) every time. The core trade-off is speed and reduced load in exchange for the risk that the cached copy becomes stale relative to the real source of truth.
Structured elaboration
- What problems caching solves: latency (serving from memory or a nearby node is far faster than recomputing or fetching from a distant source), throughput/load (fewer requests reach the expensive backend), and cost (fewer database queries or external application programming interface (API) calls, which often cost money directly).
- Where caches typically live: client/browser (closest to the user, zero network cost on a hit), content delivery network (CDN) / edge (near the user but shared across many users), application in-memory (fast, but local to one process/instance), and a shared distributed cache like Redis or Memcached (shared across all instances in a region, one network hop away).
- Primary trade-offs: staleness (the cached copy may not reflect the latest write), added complexity (invalidation logic, cache-miss handling, monitoring another moving part), and memory cost (caches are not free storage).
- When caching helps: read-heavy workloads where the same data is requested repeatedly and can tolerate at least a little staleness, or where the underlying computation/fetch is expensive relative to a cache read.
- When caching is harmful: data that changes on every read (no repeated value to cache), workloads that are already write-heavy with low read repetition (cache churn without benefit), or correctness-critical data where any staleness is unacceptable and the added complexity of cache invalidation introduces more risk than the latency win is worth.
Worked example
An API endpoint that computes a dashboard aggregate over a large dataset in 800ms, requested 200 times per minute by the same handful of users, is a strong caching candidate: a 30-second time-to-live (TTL) cache serves nearly every request from cache after the first, cutting both latency (single-digit milliseconds instead of 800ms) and backend load by over 95 percent, for a staleness window most dashboard users will not notice. The same 30-second TTL applied to an account balance shown right after a deposit would be actively harmful, showing the user a stale, "wrong" number at the exact moment they are checking it.
Trade-offs and pitfalls
The most common mistake is caching by default rather than by evaluating whether the read pattern and staleness tolerance actually justify it; caching adds a second place data can be wrong (the cache disagreeing with the source of truth), and that failure mode does not exist at all if you never cache the data in the first place.
You need a distributed cache topology for very large embedding tables (hundreds of GB) to reduce remote datastore hits during inference. Compare sharded in-memory caches, SSD-backed caches, and hybrid approaches. Discuss eviction policies, cache warming, cold-starts, and how to ensure graceful fallbacks when items miss the cache.
Sample Answer
Direct answer
Caching for machine learning (ML) inference has to account for a dimension plain application caching does not: model version, since a cached prediction or embedding is only valid for the model version that produced it, and serving a stale model's output as if it were current can silently degrade prediction quality without looking like a technical failure at all.
Structured elaboration
- Embedding tables: very large embedding tables (hundreds of gigabytes) rarely fit fully in a fast in-memory tier; a sharded in-memory cache for the hottest embeddings, backed by an SSD-backed or remote-store fallback for the long tail, balances cost against latency, with eviction policy tuned to actual embedding access skew (typically also a heavily skewed distribution, similar to other hot-key patterns).
- Model-output/prediction caching: caching an inference result keyed by input (and model version) avoids recomputing identical predictions repeatedly, valuable when the same input recurs often (a popular item's recommendation score, for example) but must invalidate on model updates, since a new model version's prediction for the same input can differ from the old one, and serving the old cached value would silently mix outputs from two different models.
- Feature caching: caching precomputed features (not just final predictions) is useful when features are expensive to compute but shared across multiple models or requests; freshness requirements here are usually tied to how quickly the underlying signal (user behavior, real-time context) needs to be reflected in the feature value.
- Per-tenant customization: different consumers of the same caching layer (different product surfaces, different customers) may need different TTLs or eviction priorities for what is otherwise the same underlying cached data, requiring the cache key or a policy layer to account for tenant-specific freshness/policy needs.
- Cache-warming to prevent thundering herd on deployment: rolling out a new model version invalidates (or requires re-keying) every cached prediction tied to the old version at once; without proactive warming of the hottest inputs against the new model before full traffic cutover, this produces exactly the kind of stampede covered elsewhere in this topic, now triggered by a model deploy rather than a data change.
- Graceful fallbacks on a cache miss: for latency-sensitive inference paths, a miss should fall back to either a faster, simpler model, or a cached PREVIOUS prediction with an explicit staleness flag, rather than blocking the request on a full, expensive inference call if that would violate a latency budget.
Worked example
A recommendation model deployed as version 47 caches predictions keyed by (user_id, item_id, model_version=47); when version 48 deploys, ALL previously-cached predictions are naturally excluded by the version-scoped key (a cache miss, not stale data served incorrectly), and a proactive warming pass precomputes predictions for the highest-traffic user/item pairs against version 48 before full traffic cutover, avoiding a cold-cache stampede against the inference service at deploy time.
Trade-offs and pitfalls
Caching predictions WITHOUT including model version in the key is a serious, easy-to-miss correctness bug: it silently serves a mix of predictions from different model versions with no visible error, which is far harder to detect than an outright cache miss, since predictions from a slightly different model version still "look like" plausible outputs. Warming every possible input against a newly deployed model is usually infeasible; prioritize by traffic volume (the same hot-key-driven approach used elsewhere in this topic) rather than attempting exhaustive coverage.
Unlock Full Question Bank
Get access to all 8 Caching Strategies and Distributed Caching interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.