Caching Strategies and Distributed Caching Questions
Using caches to reduce latency and load: cache-aside, read-through, write-through, and write-behind patterns, TTLs, eviction policies, and distributed caches such as Redis or Memcached. Covers cache invalidation, stampede and thundering-herd protection, and the consistency tradeoffs of caching. Focuses on where and how to cache across tiers.
Explain and compare strategies to partition (shard) cache data across multiple cache nodes. Discuss rebalancing costs, techniques to reduce key movement, how to mitigate hot shards, and failover behavior when nodes are added or removed.
Sample Answer
Direct answer
Consistent hashing places both cache nodes and keys on a conceptual ring (by hashing each to a point on the ring) and routes a key to the next node clockwise from it, so adding or removing a node only remaps the keys between it and its neighbor, not the entire keyspace the way naive modulo hashing would.
Structured elaboration
- Why modulo hashing fails at scale: routing a key with
hash(key) % Nmeans every node-count change (N) reshuffles almost every key's destination node, since the modulo result changes for nearly all keys; that means a near-total cache flush (and a stampede of misses to the origin) on every scale-out or node failure. - Consistent hashing's improvement: only the keys that fell between the added/removed node and its next neighbor on the ring need to move; on average, adding the Nth node to a ring of N nodes remaps roughly 1/N of the keyspace, not close to 100 percent.
- Virtual nodes: mapping each physical node to many points on the ring (e.g., 100 to 200 virtual nodes per physical node) smooths out the otherwise-uneven distribution you'd get from a small number of random hash points, and makes the fraction of keys remapped on a node change proportional to that node's actual share of capacity rather than being lumpy.
- Alternatives: rendezvous hashing (highest-random-weight) computes, for each key, a weighted score against every node and picks the highest, which achieves similar minimal-remapping properties without needing an explicit ring structure. A proxy layer (client-side hashing vs a routing proxy like Twemproxy) changes WHERE the hashing decision is made but not the underlying algorithm choice.
- Hot shards: consistent hashing distributes keys evenly in aggregate but does not know that some keys are read far more than others; a node can still become hot if it happens to own a popular key, which is a separate problem from rebalancing cost (see hot-key mitigation).
- Failover behavior: when a node fails, its portion of the ring is picked up by its neighbor (or, with virtual nodes, spread across several neighbors); this is graceful for read-through/cache-aside caches (a "miss" for a remapped key just refetches from origin) but can look like data loss for a cache that was the system of record for anything.
Worked example
A cluster grows from 10 nodes to 11. Under naive modulo hashing, roughly 91 percent of keys change their target node (1−10/11), effectively invalidating almost the whole cache at once. Under consistent hashing with virtual nodes, only the keys that fall in the ring segment now owned by the new node move, roughly 1/11≈9% of the keyspace, leaving the other 91 percent of cached entries untouched and avoiding a stampede.
Trade-offs and pitfalls
Too few virtual nodes per physical node produces an uneven distribution (some nodes end up owning noticeably more of the ring than others by chance); 100+ virtual nodes per physical node is a common starting point to smooth this out. Consistent hashing solves REBALANCING cost, not hot-key load; do not expect it to fix a single overloaded key. Client-side hashing requires every client to agree on the exact same ring state, which becomes an operational hazard during a rolling deploy where old and new client versions might briefly disagree on routing; a proxy-side approach centralizes that decision at the cost of an extra network hop.
Describe how to implement negative caching safely for non-existent resources and how to set TTLs to balance reduced DB load with the risk of false negatives. Explain mechanisms to detect and recover from incorrectly cached negatives and how to prevent poisoning of the cache.
Sample Answer
Direct answer
Negative caching (caching the fact that something does NOT exist) reduces repeated backend load for lookups that legitimately miss often, but needs a shorter time-to-live (TTL) than positive caching and explicit protection against caching a TRANSIENT error as if it were a permanent absence.
Structured elaboration
- When negative caching helps: an endpoint frequently queried for resources that legitimately do not exist (a sparse product search, a lookup by an ID that is often invalid or not-yet-created) benefits from caching "not found" so repeated identical misses do not all hit the backend.
- TTL sizing: negative entries should generally use a SHORTER TTL than positive entries, since a resource that does not exist now might be created moments later (a product about to be listed, a user about to be created), and a long negative-cache TTL would delay legitimate new data from becoming visible.
- Avoiding poisoning from transient errors: a backend timeout or a temporary error must NOT be cached as "not found"; only cache an explicit, confirmed "this does not exist" response (e.g., an actual 404 from a healthy backend), never an ambiguous failure, or a temporary outage will look like every queried resource permanently vanished.
- Detecting and recovering from an incorrectly cached negative: monitor the negative-cache hit rate for anomalies (a sudden spike suggesting something is being wrongly cached as absent), and provide an explicit purge mechanism for a specific key if a false negative is identified.
- Preventing cache poisoning attacks: without care, an attacker (or a misbehaving client) could probe many nonexistent IDs to fill the negative cache with entries, which is generally low-risk for negative caching specifically (it just wastes some memory) but worth being aware of if negative-cache capacity is shared with more sensitive cached data.
Worked example
A product-search endpoint receiving many queries for discontinued or misspelled product IDs: caching "not found" for 30 seconds (much shorter than the 5-minute TTL used for actual product data) absorbs repeated identical misses for the same nonexistent ID within that window, while still allowing a newly-listed product with that same ID to become visible within 30 seconds rather than being stuck behind a long negative-cache TTL.
Trade-offs and pitfalls
Caching a timeout or a 500-level error as a negative ("not found") result is the single most damaging mistake in negative caching; it turns a transient backend problem into an apparent, cache-durable "this doesn't exist" for every subsequent request until the negative TTL expires, actively making an outage worse and longer-lasting than it needed to be. Setting a negative TTL as long as the positive TTL "for simplicity" delays visibility of genuinely new data unnecessarily; size them independently based on how quickly a false negative needs to self-correct.
Design an invalidation pipeline using Change Data Capture (CDC) (for example Debezium into Kafka) to keep caches updated across multiple services. Discuss topic design, ordering guarantees per key, consumer group design, retry semantics, and how to avoid over-invalidation or event storms.
Sample Answer
Direct answer
An event-driven or change-data-capture (CDC) invalidation pipeline turns every write in the source of truth into a message that downstream consumers use to correct their caches, which requires deliberate topic design, ordering guarantees per key, and idempotent handlers so a redelivered or reordered event does not corrupt cache state.
Structured elaboration
- CDC as the source of events: a tool like Debezium reads the database's own write-ahead log and publishes a change event (insert/update/delete) to a message broker (e.g., Kafka) automatically, which avoids the risk of the application forgetting to publish an invalidation on some code path; the database itself becomes the single source of truth for what changed.
- Message schema: include the key, a version or timestamp, and for deletes a tombstone marker rather than omitting the message; consumers need the version to detect and discard out-of-order or duplicate events, and an explicit tombstone rather than a missing message for deletes.
- Ordering guarantees per key: partition the topic by entity key so all events for the same entity land in the same partition and are processed in order by a single consumer; this avoids a newer update being overwritten by a late-arriving older event.
- Idempotency handling: because most message delivery is at-least-once, a consumer must be able to safely apply the same event twice without harm; using the event's version as a conditional check ("only apply if this version is newer than what's cached") makes redelivery safe.
- Consumer group design: scale consumers horizontally by partition, keeping the per-key ordering guarantee intact since Kafka guarantees order within a partition, not across the whole topic.
- Avoiding over-invalidation or event storms: a bulk update touching millions of rows produces millions of invalidation events; batch or debounce invalidations for the same key within a short window, and consider whether a coarser signal (e.g., "namespace version bumped") is more appropriate than a firehose of individual key invalidations for that specific case.
Worked example
A price-update job that touches 2 million product rows in a batch would, naively, produce 2 million invalidation events in a burst; consumers applying them all immediately would generate 2 million cache misses in a short window, an invalidation-triggered stampede against the very origin the cache exists to protect. Batching invalidations per consumer (apply up to N per second, or coalesce multiple updates to the same key within a short window into one) smooths this into a manageable, sustained rate instead of a spike.
Trade-offs and pitfalls
Relying purely on event-driven invalidation with no time-to-live (TTL) backstop means a lost or unprocessed event becomes permanent staleness with no self-correction; always pair this with a TTL ceiling. Partitioning by key for ordering can create a hot partition if one key is updated far more often than others, which needs the same hot-key mitigations as any other hot-partition problem.
Compare TTL (time-to-live) based expiration with explicit eviction/purge. Provide examples of use-cases where TTL is sufficient and where explicit invalidation is necessary. Also list drawbacks of using long TTL values.
Sample Answer
Direct answer
Time-to-live (TTL) expiration is a passive, self-healing bound on staleness that requires no extra machinery; explicit invalidation (purge, versioned keys, or event-driven invalidation) actively corrects the cache the moment the source changes, at the cost of needing a reliable signal that a change happened.
Structured elaboration
- When TTL is sufficient: content where a bounded staleness window is genuinely acceptable (a homepage banner, a "trending now" list, most analytics dashboards); TTL requires no coordination and self-heals even if a write is missed entirely, because the entry expires on schedule regardless.
- When explicit invalidation is necessary: content where staleness has real consequences (permission revocation, inventory during checkout, a price correction that must take effect immediately); waiting out even a short TTL is unacceptable when correctness, not just freshness, is on the line.
- Versioned keys as a form of explicit invalidation: rather than deleting the old entry, bump a version token that is part of the key; the old version simply becomes unreachable and ages out naturally, avoiding races where a delete arrives before or after the write it corresponds to.
- Event-driven invalidation: a write publishes an event (directly or via change data capture, CDC) that consumers use to purge or update affected cache entries; this gets close to real-time correction but adds a dependency on the message delivery being reliable.
- A hybrid approach: use a moderate TTL as a safety net (so a lost invalidation event self-heals within a bounded window) combined with explicit invalidation for the fast path; this is the most common production pattern because it gets the responsiveness of explicit invalidation with the resilience of TTL.
Worked example
A user's permission level changes from "member" to "banned." Relying purely on a 5-minute TTL means the banned user could still access restricted content for up to 5 minutes; that is not acceptable, so this write triggers explicit invalidation of the user's cached permission entry, with a short TTL (e.g., 60 seconds) still applied as a safety net in case the invalidation event is somehow lost.
Trade-offs and pitfalls
Long TTLs used purely for "simplicity" on data that actually needs freshness is one of the most common production incidents in caching (a stale price, a stale permission, a stale feature flag reaching users for far longer than intended); size the TTL to the actual staleness tolerance of the data, not to whatever is convenient to implement. Relying on explicit invalidation alone, with no TTL backstop, means any lost or delayed invalidation event becomes unbounded staleness with no self-correction.
You operate a high-read service with current cache hit ratio 70% and database cost that scales linearly with queries. Present a cost-benefit analysis framework for deciding whether to (a) increase cache capacity, (b) tune cache TTLs, or (c) scale the database. What measurements, estimation models, and experiments would you run to make a data-driven decision?
Sample Answer
Direct answer
Compare the three options (grow cache capacity, tune TTLs, scale the database) by estimating the marginal cost and marginal latency/load benefit of each, since they are not mutually exclusive and the cheapest lever that closes the gap is usually the right first move.
Structured elaboration
- Growing cache capacity: if evictions are the bottleneck (the working set does not fit), more memory directly raises hit ratio; the cost is roughly linear in additional cache node memory/count, and the benefit is bounded once the cache is large enough to hold the full working set (beyond that point, more memory buys nothing).
- Tuning TTLs: if the working set fits but TTLs are shorter than necessary for the actual staleness tolerance, extending TTLs raises effective hit ratio for close to zero additional infrastructure cost; the ceiling here is the data's genuine staleness tolerance, not technical capacity.
- Scaling the database: if the cache is already well-sized and well-tuned but query volume is still too high (e.g., a low cache-hit-ratio workload with lots of genuinely unique reads), the database itself needs more capacity (read replicas, bigger instances); this is usually the most expensive lever, both in infrastructure cost and operational complexity.
- A framework for choosing: first check eviction rate (tells you if more cache memory would help), then check whether current TTLs are shorter than the data's actual staleness tolerance (tells you if a free tuning win exists), and only after those are exhausted, model out database scaling cost against the marginal load reduction remaining.
Worked example
A service with a 70 percent cache hit ratio and database cost scaling linearly with query volume: modeling the current state at, say, Q queries per second to the database and cost C=k⋅Q. Raising hit ratio to 85 percent (a plausible outcome of either more cache memory or longer TTLs, depending on root cause) cuts database query volume by 1−0.701−0.85=0.300.15=0.5, a 50 percent reduction in database cost, versus scaling the database itself, which would only add capacity without reducing the query volume driving that cost at all. If eviction rate is low (cache is not memory-constrained) but TTLs are conservative relative to the data's real staleness tolerance, extending TTLs achieves this improvement essentially for free; if eviction rate is high, the same improvement requires paying for more cache capacity instead.
Trade-offs and pitfalls
Scaling the database without first checking whether cache tuning could achieve the same load reduction for less money is a common and expensive mistake; always rule out the cheaper levers first with actual data (eviction rate, staleness tolerance), not assumption. Pushing TTLs longer than the data's genuine staleness tolerance to chase a cost win trades a real user-facing correctness problem for an infrastructure savings; the framework only works if the staleness ceiling is respected as a hard constraint, not a dial.
Unlock Full Question Bank
Get access to all Caching Strategies and Distributed Caching interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.