Caching Strategies and Distributed Caching Questions
Using caches to reduce latency and load: cache-aside, read-through, write-through, and write-behind patterns, TTLs, eviction policies, and distributed caches such as Redis or Memcached. Covers cache invalidation, stampede and thundering-herd protection, and the consistency tradeoffs of caching. Focuses on where and how to cache across tiers.
You're paged: the primary distributed cache cluster is down, causing high DB load and user-facing errors. Draft an incident response runbook for this scenario that includes immediate triage steps, short-term mitigations (circuit-breakers, rate limiting, serve stale, emergency config changes), recovery steps (restore cluster, failover, reshard), verification and post-incident analysis items, and communications to stakeholders.
Sample Answer
Direct answer
Treat a downed primary cache cluster as a load-shedding emergency first and a cache-recovery problem second: protect the database from the traffic the cache was absorbing, then restore the cluster, then verify correctness before declaring the incident closed.
Structured elaboration
- Immediate triage (first 2 to 5 minutes): confirm the blast radius (which services depend on this cluster, is it one shard or the whole cluster), check whether the database is already showing elevated load or error rates, and page in the on-call owners for any dependent services.
- Short-term mitigations: enable circuit breakers on the cache client so failed cache calls fail fast instead of timing out and piling up connections; apply rate limiting or admission control at the edge to shed non-critical traffic; where the application supports it, serve last-known-good stale data from a fallback source instead of hammering the database; if a config flag exists to bypass the cache entirely for low-value reads, use it to reduce database queries per second (QPS) rather than retry the dead cache.
- Recovery steps: restore the cluster (restart nodes, promote a healthy replica, or failover to a standby cluster depending on your topology); if the cluster needs to be rebuilt or resharded, do that with the mitigations from the step above still in place so the database stays protected during the rebuild, which can itself generate a burst of cache-miss traffic (a cold-cache stampede).
- Verification: confirm cache hit ratio has returned to its normal baseline, confirm database load and latency have returned to baseline, and spot-check a few known keys for correct values before declaring recovery complete; a cluster that is "up" but silently missing on every key is not actually recovered.
- Post-incident analysis: capture a timeline (detection time, mitigation time, full recovery time), the root cause, what would have caught it sooner, and what permanent fix (better health checks, automatic failover, capacity headroom) prevents recurrence.
- Communications: status-page or internal-stakeholder updates at defined intervals during the incident (not just at the start and end), and a plain-language summary for non-technical stakeholders once resolved.
Worked example
A cluster of 6 Redis nodes loses 2 nodes to an availability-zone network issue; hit ratio drops from a 95 percent baseline to near 0 percent for the affected shards within seconds. Database QPS, previously around 2,000 with the cache absorbing the rest, spikes toward the full unfiltered request volume of roughly 40,000 QPS, which the database was never sized for. The correct first action is not "wait for the cache to heal" but immediately enabling a circuit breaker so the app fails cache reads fast (single-digit milliseconds) and falls back to a rate-limited direct-database path, buying time to fail the affected shards over to healthy replicas.
Trade-offs and pitfalls
Retrying a dead cache aggressively (rather than failing fast) is one of the most common ways an outage becomes worse: retries add latency and connection pressure without adding capacity. Restoring the cluster all at once without a cold-start warming plan can immediately reproduce the outage as a stampede of cache misses across every key simultaneously; prefer a gradual traffic ramp-up or a targeted warm of the hottest keys first. Skipping the verification step and declaring victory the moment the cluster reports healthy risks missing a silent correctness bug (e.g., a failed-over replica serving stale data).
You led an incident where stale feature flags in cache caused 20% of users to see an outdated UI for 45 minutes. Draft a high-level incident postmortem: timeline, root cause analysis, immediate remediation, permanent fixes, monitoring changes, and team-level learnings that prevent recurrence.
Sample Answer
Direct answer
A postmortem for a stale-cache correctness incident needs to establish exactly when the staleness started, how long it took to detect, and whether the fix addresses the root cause (a specific invalidation gap) rather than just the symptom (manually flushing the cache once).
Structured elaboration
- Timeline: when the underlying data actually changed, when the cache should have reflected that change, when it was detected (ideally by monitoring, not a user report), and when it was fully resolved.
- Root cause analysis: distinguish "the invalidation mechanism failed" (a bug, a missed code path, a dropped message) from "there was no invalidation mechanism for this specific data at all" (a design gap); the fix looks very different depending on which it is.
- Immediate remediation: manually flush or correct the affected cache entries, and communicate impact and resolution to affected users or stakeholders if the incident was customer-visible.
- Permanent fixes: address the actual gap (add invalidation for the missed code path, add a time-to-live (TTL) backstop where none existed, fix the specific bug that dropped invalidation events) rather than only adding a manual runbook step for next time.
- Monitoring changes: add or tighten alerting so a similar staleness window is caught by automated detection (a hit-ratio or freshness-lag metric) rather than a user noticing first, next time.
- Team-level learnings: capture what made this hard to detect (was staleness silent because nothing monitored it, or because the monitoring existed but did not page anyone) and fix that detection gap specifically.
Worked example
Stale feature flags served from cache caused 20 percent of users to see an outdated UI for 45 minutes: the root cause was a feature-flag update that bypassed the normal invalidation code path (a batch admin tool wrote directly to the database, skipping the application code that normally triggers cache invalidation on flag changes). The permanent fix is not "remember to flush the cache after using the admin tool" but ensuring every write path for that data, including the admin tool, goes through the same invalidation trigger, plus adding a TTL ceiling (previously unset) so any future gap self-heals within a bounded window instead of persisting indefinitely.
Trade-offs and pitfalls
A postmortem that stops at "we manually flushed the cache" without identifying why the automated invalidation did not fire has not actually fixed anything; the same gap will recur. Blaming the person who used the admin tool, rather than the system design that allowed a write path to bypass invalidation entirely, produces a fix that only works until someone forgets the informal rule; fix the system, not just the process document.
A payments ledger requires strong correctness when updating balances. Compare write-through caching (synchronous write to cache and datastore) vs write-behind (asynchronous background writes). For each, discuss durability, read visibility immediately after write, failure modes, and techniques (idempotency, ordering) to preserve correctness. Which approach would you choose and why?
Sample Answer
Direct answer
For a durability-sensitive service like payments, prefer write-through (synchronous write to both cache and datastore) or cache-aside with synchronous invalidation over write-behind, because write-behind's asynchronous flush introduces a data-loss window that is unacceptable when the data is money or an authentication state.
Structured elaboration
- Write-through for payments: every write goes to the cache and the datastore together, synchronously, before acknowledging the caller; reads are always fresh, and there is no window where an acknowledged write could be lost, at the cost of higher write latency (paying for both writes on the request path).
- Cache-aside with synchronous invalidation for sessions: a session service that must reflect login/logout changes quickly can use cache-aside (only cache what is actually read) with an invalidation triggered synchronously on logout, rather than waiting for a time-to-live (TTL) to expire; this balances the simplicity of cache-aside with the responsiveness a security-sensitive change needs.
- Why write-behind is the wrong default here: write-behind acknowledges the write before it is durably applied to the datastore; a crash in that window loses the write entirely, which is an acceptable trade for a metrics counter and not acceptable for a payment or a security-relevant state change.
- Trade-off in consistency versus write amplification: write-through pays extra write latency and writes data that might never be read (caching every write regardless of read demand); cache-aside with synchronous invalidation only caches what is actually requested, trading a small amount of extra invalidation-path complexity for less wasted cache capacity.
- Failure modes to consider: with write-through, a failure writing to EITHER the cache or the datastore must be handled explicitly (does the whole operation fail, or does it proceed with just the datastore write and treat the cache write as best-effort); an inconsistent partial failure here is exactly the kind of subtle bug that durability-sensitive systems cannot tolerate silently.
Worked example
A session service where sessions must invalidate within seconds of logout: cache-aside with an explicit, synchronous invalidation call on logout (rather than relying on a TTL to eventually expire the session) meets that requirement directly; a subsequent read after logout misses the cache, goes to the datastore, finds the session revoked, and correctly denies access, all within the same request cycle rather than waiting out a TTL window during which a logged-out session token would still work.
Trade-offs and pitfalls
Choosing write-behind for its throughput benefit on a durability-sensitive path is a common and dangerous shortcut; always name the data-loss window explicitly and get an explicit sign-off that it is acceptable before using it, rather than defaulting to it for performance reasons alone. A write-through implementation that treats the cache write as best-effort (silently swallowing cache write failures) can mask a growing coherence problem between cache and datastore; log and monitor cache write failures even when they are non-fatal to the request.
Design a monitoring dashboard and alerting strategy for a distributed Redis cache serving an internal read-heavy API. Include specific metrics to display, dashboard panels, and alert conditions that would indicate (a) cache degradation, (b) emergence of a hot key, and (c) eviction-related problems.
Sample Answer
Direct answer
Design the dashboard around the three failure modes an operator actually needs to distinguish quickly: general degradation (rising latency or falling hit ratio across the board), an emerging hot key (one node/shard diverging from the rest), and eviction-related problems (memory pressure forcing out data that should still be cached).
Structured elaboration
- Panel: cluster-wide health: aggregate hit ratio, miss ratio, and p50/p95/p99 (50th/95th/99th percentile) latency over a rolling window, with the current service-level objective (SLO) target overlaid so a degradation is visible at a glance, not just as a raw number.
- Panel: per-node/per-shard breakdown: CPU, request rate, and latency broken out by individual node, specifically because an emerging hot key looks fine in the aggregate panel but shows one node's line diverging sharply from the others; this panel is what actually catches a hot key before it becomes an incident.
- Panel: eviction and memory: eviction rate and memory usage as a percentage of the configured limit, ideally split by key class or namespace if the cache is shared across use cases, so a memory-pressure problem can be traced to which data is actually filling the cache.
- Alert conditions for degradation: hit ratio dropping below a sustained threshold (e.g., more than 15 points below the trailing 7-day baseline for more than 2 consecutive 5-minute windows) or p99 latency exceeding the SLO target for a sustained period.
- Alert conditions for a hot key: any single node's request rate or CPU exceeding a set multiple (e.g., 3x) of the cluster's median node, sustained for more than a short window (to avoid alerting on normal brief variance).
- Alert conditions for eviction problems: eviction rate rising sharply relative to its own recent baseline, especially combined with memory usage near the configured limit, distinguishing "the cache is working as intended, evicting genuinely cold data" from "the cache is thrashing, evicting data that will be needed again soon."
Worked example
A hot-key alert fires when node 7's request rate is 4x the cluster median for more than 60 seconds; the dashboard's per-node panel immediately shows node 7 as a clear outlier against otherwise-flat lines for the other nodes, letting the on-call engineer confirm within seconds that this is a hot-key event rather than a cluster-wide issue, and start the hot-key mitigation runbook instead of investigating a broader outage.
Trade-offs and pitfalls
A dashboard that only shows cluster-wide aggregates cannot distinguish a hot-key problem from general health, which is exactly the failure mode this design targets; do not skip the per-node breakdown panel to save dashboard space. Alerting on eviction rate alone without memory context conflates "healthy, expected eviction" with "the cache is undersized"; always pair eviction-rate alerting with memory-utilization context.
You operate a high-read service with current cache hit ratio 70% and database cost that scales linearly with queries. Present a cost-benefit analysis framework for deciding whether to (a) increase cache capacity, (b) tune cache TTLs, or (c) scale the database. What measurements, estimation models, and experiments would you run to make a data-driven decision?
Sample Answer
Direct answer
Compare the three options (grow cache capacity, tune TTLs, scale the database) by estimating the marginal cost and marginal latency/load benefit of each, since they are not mutually exclusive and the cheapest lever that closes the gap is usually the right first move.
Structured elaboration
- Growing cache capacity: if evictions are the bottleneck (the working set does not fit), more memory directly raises hit ratio; the cost is roughly linear in additional cache node memory/count, and the benefit is bounded once the cache is large enough to hold the full working set (beyond that point, more memory buys nothing).
- Tuning TTLs: if the working set fits but TTLs are shorter than necessary for the actual staleness tolerance, extending TTLs raises effective hit ratio for close to zero additional infrastructure cost; the ceiling here is the data's genuine staleness tolerance, not technical capacity.
- Scaling the database: if the cache is already well-sized and well-tuned but query volume is still too high (e.g., a low cache-hit-ratio workload with lots of genuinely unique reads), the database itself needs more capacity (read replicas, bigger instances); this is usually the most expensive lever, both in infrastructure cost and operational complexity.
- A framework for choosing: first check eviction rate (tells you if more cache memory would help), then check whether current TTLs are shorter than the data's actual staleness tolerance (tells you if a free tuning win exists), and only after those are exhausted, model out database scaling cost against the marginal load reduction remaining.
Worked example
A service with a 70 percent cache hit ratio and database cost scaling linearly with query volume: modeling the current state at, say, Q queries per second to the database and cost C=k⋅Q. Raising hit ratio to 85 percent (a plausible outcome of either more cache memory or longer TTLs, depending on root cause) cuts database query volume by 1−0.701−0.85=0.300.15=0.5, a 50 percent reduction in database cost, versus scaling the database itself, which would only add capacity without reducing the query volume driving that cost at all. If eviction rate is low (cache is not memory-constrained) but TTLs are conservative relative to the data's real staleness tolerance, extending TTLs achieves this improvement essentially for free; if eviction rate is high, the same improvement requires paying for more cache capacity instead.
Trade-offs and pitfalls
Scaling the database without first checking whether cache tuning could achieve the same load reduction for less money is a common and expensive mistake; always rule out the cheaper levers first with actual data (eviction rate, staleness tolerance), not assumption. Pushing TTLs longer than the data's genuine staleness tolerance to chase a cost win trades a real user-facing correctness problem for an infrastructure savings; the framework only works if the staleness ceiling is respected as a hard constraint, not a dial.
Unlock Full Question Bank
Get access to all Caching Strategies and Distributed Caching interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.