Caching Strategies and Distributed Caching Questions
Using caches to reduce latency and load: cache-aside, read-through, write-through, and write-behind patterns, TTLs, eviction policies, and distributed caches such as Redis or Memcached. Covers cache invalidation, stampede and thundering-herd protection, and the consistency tradeoffs of caching. Focuses on where and how to cache across tiers.
Explain the differences between Least Recently Used (LRU) and Least Frequently Used (LFU) eviction policies. For an in-memory cache storing user session objects that are frequently accessed soon after login then rarely, which policy is more appropriate and why?
Sample Answer
Direct answer
Least Recently Used (LRU) evicts based on recency of access and fits workloads where "recently touched" predicts "will be touched again soon"; Least Frequently Used (LFU) evicts based on access count and fits workloads where a stable set of items is popular over a long time, even if not accessed in the last few seconds.
Structured elaboration
- LRU mechanics and fit: LRU tracks the order items were last accessed and evicts the item that has gone the longest without a touch. It fits access patterns with strong temporal locality: a session object, for example, is heavily accessed right after login and then goes cold; LRU naturally keeps the currently-active sessions warm and evicts the ones nobody has touched recently.
- LFU mechanics and fit: LFU tracks how many times each item has been accessed (often with some decay over time to avoid permanently favoring old-but-once-popular items) and evicts the least-accessed item. It fits patterns with a stable long-tail of popular items that get accessed periodically but not necessarily continuously, such as a small set of perennially popular catalog items that get read constantly, interspersed with occasional cold reads of rare items; a pure LRU cache would wrongly evict a genuinely popular item just because it happened not to be touched in the last few seconds.
- LRU's weak spot: a single burst of one-time scans (e.g., a batch job reading every record once) can flush an LRU cache of genuinely hot items, because the scan makes every item "recently used" once, pushing out items that are accessed far more often over time.
- LFU's weak spot: new items start at zero frequency and can be evicted immediately even if they would become popular, because they have not yet accumulated enough hits to compete with established items (this is sometimes mitigated with a decay factor or a frequency-boosted admission window).
Worked example
For user session objects that are heavily read right after login and then rarely touched again: LRU is the right default, because the access pattern IS recency-driven, and LFU would keep old, no-longer-relevant sessions around simply because they accumulated a lot of hits while they were active. For a product catalog with seasonal hot items plus a long tail of rarely-viewed products: LFU (or a hybrid) does better, because a genuinely popular item accessed every few minutes should survive occasional bursts of one-time-scan traffic on unrelated items that would otherwise flush it from an LRU cache.
Trade-offs and pitfalls
LFU requires more bookkeeping (a counter per item, often with decay logic) than LRU's simple access-order list, so it costs more memory and CPU per operation; do not reach for LFU unless the access pattern genuinely benefits from it. Neither pure policy handles every real workload well, which is why production caches (Redis's allkeys-lfu, or an adaptive scheme like W-TinyLFU) increasingly blend recency and frequency signals rather than using either in isolation.
Your recommendation service caches per-user recommendations and participates in A/B experiments. During a cache outage you must serve fallbacks while preserving experiment randomization and avoiding biased metrics. Propose a design that provides safe fallbacks and explain how to measure and correct for any bias introduced during the outage.
Sample Answer
Direct answer
Preserving A/B-experiment integrity during a cache outage means the fallback path must still respect each user's already-assigned experiment variant and avoid systematically favoring one variant's typical response shape, or the outage silently corrupts your experiment's results even after the cache recovers.
Structured elaboration
- The risk: a naive fallback (serving a generic default recommendation to everyone during the outage) breaks randomization, since users in the treatment and control groups would temporarily receive the SAME fallback response, diluting or masking whatever real difference the experiment was measuring during that window.
- Persisted experiment assignments: a user's variant assignment must be stored durably (not just derivable from the now-unavailable cache), so the fallback path can still honor "this user is in treatment, serve a treatment-appropriate response" even while the personalized recommendation itself is unavailable.
- Deterministic origin sampling: if the origin can compute a reasonable substitute response directly (bypassing the cache but still respecting the user's assigned variant), that is preferable to a single generic fallback shared by everyone.
- Synthetic traffic shaping: where a true substitute response is not feasible during the outage, deliberately shape the fallback to avoid a SYSTEMATIC bias correlated with variant (e.g., do not have the fallback happen to resemble the control experience more closely than the treatment experience, which would bias the comparison even without meaning to).
- Measuring and correcting for any bias introduced: after the outage, explicitly flag the affected time window in the experiment's data, and either exclude that window from the analysis or model its impact explicitly, rather than silently including outage-period data as if it were normal.
Worked example
During a cache outage, users already assigned to the treatment group (personalized recommendations) receive a computed-on-the-fly, best-effort personalized fallback (even if slower and less sophisticated than the normal cached path), while control-group users continue receiving their normal non-personalized experience; both groups' EXPERIENCE during the outage still reflects their assigned variant's intent, even if degraded, rather than both groups converging to the same generic response.
Trade-offs and pitfalls
The most common mistake is treating "serve something, anything" as sufficient during an outage without checking whether that something respects experiment assignment; this quietly corrupts experiment data in a way that is easy to miss until someone notices an unexplained dip in the effect size for that period. Excluding the outage window from analysis after the fact is a reasonable mitigation but is not free; it reduces sample size and delays the ability to reach statistical significance, which is a real cost worth weighing against the effort of building a variant-aware fallback in the first place.
Design a caching and locking scheme for inventory management in a high-concurrency e-commerce checkout flow to prevent oversell. Discuss the role of distributed locks (e.g., Redis Redlock), optimistic concurrency with version checks, decrement semantics in cache versus DB, final correctness guarantees, and compensation strategies if oversells occur.
Sample Answer
Direct answer
Preventing oversell in a high-concurrency checkout flow means the cache can accelerate reads but cannot be the sole source of truth for a decrement; use optimistic concurrency (version checks) or a distributed lock scoped tightly around the actual decrement, with the database as the final arbiter of correctness.
Structured elaboration
- Distributed locks (e.g., Redis Redlock): acquire a short-lived lock on the specific inventory item before checking and decrementing its count, ensuring only one concurrent checkout can act on that item's count at a time; the lock must be scoped narrowly (just around the check-and-decrement) to avoid serializing unrelated checkout steps.
- Optimistic concurrency with version checks: instead of locking, read the current count and its version, and write the decrement conditionally ("update count = count - 1 where version = X"); if the version has changed since the read (someone else already decremented), retry the read-decrement cycle. This avoids lock contention entirely for the common case and only pays a retry cost when there is genuine concurrent contention on the same item.
- Decrement semantics in cache versus database: the cache can hold a fast-read approximation of available inventory for display purposes (product page "5 left in stock"), but the actual decrement that prevents oversell must be an atomic, conditional operation against the authoritative store (or a cache operation, like a Redis
DECR, that is ITSELF atomic and treated as the source of truth for that specific counter, with the database reconciled asynchronously). - Final correctness guarantees: whichever mechanism is used, the checkout flow must never allow a "read count, decide it is available, decrement" sequence to interleave across two concurrent requests without one of them detecting the conflict; that interleaving is precisely what causes oversell.
- Compensation strategies if oversells occur despite these mechanisms: detect the oversell condition (a count that went negative, or a reconciliation between cache and database that reveals a discrepancy) and trigger a defined business process (cancel one order with an apology and compensation, or fulfill both if inventory can be expedited) rather than leaving the discrepancy silently unresolved.
Worked example
At 100,000 transactions per second (TPS) peak during a flash sale, using an atomic Redis DECR on an inventory counter (rather than a read-then-write from the application) means the decrement operation itself is safe under concurrency without needing an application-level lock at all; the application only needs to check the RESULT of the decrement (did it go negative, meaning oversold) and, if so, immediately compensate by incrementing it back and rejecting that specific order, rather than trying to prevent the race with a slower, coarser-grained application lock.
Trade-offs and pitfalls
A cache-only decrement (no reconciliation with the database) risks the cache being the sole record of a critical business fact with no durable backup if that cache node is lost; periodically or asynchronously reconcile the cache counter with the database's own accounting. Distributed locks add latency and a new failure mode (a lock holder crashing mid-operation); prefer an atomic cache primitive (like DECR) over an explicit lock whenever the operation can be expressed that way, since it is both simpler and faster.
Design a monitoring dashboard and alerting strategy for a distributed Redis cache serving an internal read-heavy API. Include specific metrics to display, dashboard panels, and alert conditions that would indicate (a) cache degradation, (b) emergence of a hot key, and (c) eviction-related problems.
Sample Answer
Direct answer
Design the dashboard around the three failure modes an operator actually needs to distinguish quickly: general degradation (rising latency or falling hit ratio across the board), an emerging hot key (one node/shard diverging from the rest), and eviction-related problems (memory pressure forcing out data that should still be cached).
Structured elaboration
- Panel: cluster-wide health: aggregate hit ratio, miss ratio, and p50/p95/p99 (50th/95th/99th percentile) latency over a rolling window, with the current service-level objective (SLO) target overlaid so a degradation is visible at a glance, not just as a raw number.
- Panel: per-node/per-shard breakdown: CPU, request rate, and latency broken out by individual node, specifically because an emerging hot key looks fine in the aggregate panel but shows one node's line diverging sharply from the others; this panel is what actually catches a hot key before it becomes an incident.
- Panel: eviction and memory: eviction rate and memory usage as a percentage of the configured limit, ideally split by key class or namespace if the cache is shared across use cases, so a memory-pressure problem can be traced to which data is actually filling the cache.
- Alert conditions for degradation: hit ratio dropping below a sustained threshold (e.g., more than 15 points below the trailing 7-day baseline for more than 2 consecutive 5-minute windows) or p99 latency exceeding the SLO target for a sustained period.
- Alert conditions for a hot key: any single node's request rate or CPU exceeding a set multiple (e.g., 3x) of the cluster's median node, sustained for more than a short window (to avoid alerting on normal brief variance).
- Alert conditions for eviction problems: eviction rate rising sharply relative to its own recent baseline, especially combined with memory usage near the configured limit, distinguishing "the cache is working as intended, evicting genuinely cold data" from "the cache is thrashing, evicting data that will be needed again soon."
Worked example
A hot-key alert fires when node 7's request rate is 4x the cluster median for more than 60 seconds; the dashboard's per-node panel immediately shows node 7 as a clear outlier against otherwise-flat lines for the other nodes, letting the on-call engineer confirm within seconds that this is a hot-key event rather than a cluster-wide issue, and start the hot-key mitigation runbook instead of investigating a broader outage.
Trade-offs and pitfalls
A dashboard that only shows cluster-wide aggregates cannot distinguish a hot-key problem from general health, which is exactly the failure mode this design targets; do not skip the per-node breakdown panel to save dashboard space. Alerting on eviction rate alone without memory context conflates "healthy, expected eviction" with "the cache is undersized"; always pair eviction-rate alerting with memory-utilization context.
Design an invalidation pipeline using Change Data Capture (CDC) (for example Debezium into Kafka) to keep caches updated across multiple services. Discuss topic design, ordering guarantees per key, consumer group design, retry semantics, and how to avoid over-invalidation or event storms.
Sample Answer
Direct answer
An event-driven or change-data-capture (CDC) invalidation pipeline turns every write in the source of truth into a message that downstream consumers use to correct their caches, which requires deliberate topic design, ordering guarantees per key, and idempotent handlers so a redelivered or reordered event does not corrupt cache state.
Structured elaboration
- CDC as the source of events: a tool like Debezium reads the database's own write-ahead log and publishes a change event (insert/update/delete) to a message broker (e.g., Kafka) automatically, which avoids the risk of the application forgetting to publish an invalidation on some code path; the database itself becomes the single source of truth for what changed.
- Message schema: include the key, a version or timestamp, and for deletes a tombstone marker rather than omitting the message; consumers need the version to detect and discard out-of-order or duplicate events, and an explicit tombstone rather than a missing message for deletes.
- Ordering guarantees per key: partition the topic by entity key so all events for the same entity land in the same partition and are processed in order by a single consumer; this avoids a newer update being overwritten by a late-arriving older event.
- Idempotency handling: because most message delivery is at-least-once, a consumer must be able to safely apply the same event twice without harm; using the event's version as a conditional check ("only apply if this version is newer than what's cached") makes redelivery safe.
- Consumer group design: scale consumers horizontally by partition, keeping the per-key ordering guarantee intact since Kafka guarantees order within a partition, not across the whole topic.
- Avoiding over-invalidation or event storms: a bulk update touching millions of rows produces millions of invalidation events; batch or debounce invalidations for the same key within a short window, and consider whether a coarser signal (e.g., "namespace version bumped") is more appropriate than a firehose of individual key invalidations for that specific case.
Worked example
A price-update job that touches 2 million product rows in a batch would, naively, produce 2 million invalidation events in a burst; consumers applying them all immediately would generate 2 million cache misses in a short window, an invalidation-triggered stampede against the very origin the cache exists to protect. Batching invalidations per consumer (apply up to N per second, or coalesce multiple updates to the same key within a short window into one) smooths this into a manageable, sustained rate instead of a spike.
Trade-offs and pitfalls
Relying purely on event-driven invalidation with no time-to-live (TTL) backstop means a lost or unprocessed event becomes permanent staleness with no self-correction; always pair this with a TTL ceiling. Partitioning by key for ordering can create a hot partition if one key is updated far more often than others, which needs the same hot-key mitigations as any other hot-partition problem.
Unlock Full Question Bank
Get access to all Caching Strategies and Distributed Caching interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.