Caching Strategies and Distributed Caching Questions
Using caches to reduce latency and load: cache-aside, read-through, write-through, and write-behind patterns, TTLs, eviction policies, and distributed caches such as Redis or Memcached. Covers cache invalidation, stampede and thundering-herd protection, and the consistency tradeoffs of caching. Focuses on where and how to cache across tiers.
You observe that total request latency rose despite a high cache hit rate. Outline a step-by-step debugging plan to identify the root cause, including what metrics and traces you would collect.
Sample Answer
Direct answer
When latency rises despite a healthy hit ratio, the cache is doing its job (finding data), so the investigation shifts away from "is data cached" toward "what is slow about serving a cached value": serialization cost, network hops, object size, or an inconsistent-key problem that is quietly generating hidden extra misses within an otherwise-healthy aggregate hit ratio.
Structured elaboration
- Serialization/deserialization cost: a growing or changed payload shape can make encoding/decoding on every cache hit meaningfully slower even though the read itself succeeded; check whether payload size or format changed recently.
- Object size: a larger cached object takes longer to transfer over the network even on a hit; compare current average object size against historical baselines for a regression.
- Network hops / cache network topology: a routing change (a new proxy layer, a cross-availability-zone hop that used to be same-zone) adds latency to every hit without affecting whether it is a hit at all; check whether the cache client's network path to the cache changed recently.
- Cache misses due to inconsistent keys: a subtle bug (a key that includes a field with slightly different formatting on different code paths) can cause SOME requests to always miss while the AGGREGATE hit ratio still looks acceptable if that subset is a small fraction of total traffic; segment hit ratio and latency by endpoint or request shape, not just in aggregate.
- Backend cascading delays: if the "cache hit" path itself calls out to something else after the cache read (a second lookup, an enrichment call), a slowdown there shows up as rising latency without touching the cache's own hit ratio at all.
- Debugging plan: start with distributed tracing to see where time is actually spent within a request (cache read, deserialization, post-cache-hit processing), compare payload sizes and network path over time, and segment metrics by endpoint/request-shape rather than relying purely on aggregate numbers that can hide a localized problem.
Worked example
A service's overall hit ratio holds steady at 92 percent, but p99 latency has crept up over two weeks; tracing shows the cache READ itself is still fast (single-digit milliseconds), but deserialization time for the cached payload has grown from 2ms to 40ms, correlating with a recent change that added several new fields to the cached object; the fix is either trimming the payload to only what is actually needed, or switching to a faster serialization format, not anything related to the cache layer itself.
Trade-offs and pitfalls
Assuming "hit ratio is fine, so the cache isn't the problem" and stopping the investigation there is a common mistake; a hit ratio number says nothing about how expensive it is to USE a cached value once found. Aggregate metrics can hide a localized problem affecting a small but meaningful subset of traffic; segment by endpoint or request type before concluding a metric is uniformly healthy.
Describe how HTTP caching works using Cache-Control, ETag, and Last-Modified headers. Explain how a CDN and a Service Worker might interact with those headers and describe a conditional GET flow including a 304 response. Provide one example where a Service Worker should bypass CDN semantics.
Sample Answer
Direct answer
HTTP caching is coordinated through response headers: Cache-Control tells any cache (browser, content delivery network, CDN) how long and under what conditions a response can be reused, and ETag/Last-Modified let a client revalidate a possibly-stale cached copy cheaply via a conditional request instead of re-downloading the full response.
Structured elaboration
- Cache-Control: directives like
max-age=<seconds>(how long the response is fresh),no-cache(must revalidate before use, but can still be stored),no-store(never cache at all), andprivate/public(whether shared caches like a CDN may store it) give fine-grained control over caching behavior. - ETag: an opaque identifier (often a hash) for a specific version of a resource; a client that has a cached copy sends
If-None-Match: <etag>on its next request. - Last-Modified: a timestamp-based alternative to ETag; a client sends
If-Modified-Since: <timestamp>. - Conditional GET / 304 flow: the client's cached copy has expired (its
max-ageelapsed) but might still be valid; instead of re-fetching the full body, it sends a conditional request withIf-None-MatchorIf-Modified-Since. If the resource has not actually changed, the server responds304 Not Modifiedwith no body, and the client's existing cached copy is revalidated as fresh; if it has changed, the server responds normally with the new body and headers. - CDN and Service Worker interaction: a CDN typically respects
Cache-Controldirectly, serving from its edge cache without contacting the origin untilmax-ageexpires. A Service Worker sits in the browser and can implement its own caching logic (including bypassing normal HTTP semantics entirely) via the Cache API andfetchevent interception, which is useful when the app needs offline support or caching behavior more sophisticated than standard headers allow. - When a Service Worker should bypass CDN semantics: for content the app knows is safe to serve instantly from a local cache regardless of what a content delivery network (CDN)'s time-to-live (TTL) says (e.g., app shell assets for instant repeat-visit loads), a Service Worker can serve from its own cache first and revalidate in the background (a stale-while-revalidate pattern implemented at the application layer), rather than waiting on the CDN's standard freshness check.
Worked example
A JS bundle served with Cache-Control: max-age=31536000, immutable and a content-hashed filename (app.a1b2c3.js) never needs revalidation at all, since a new deploy produces a new filename/URL; "invalidation" is simply pointing to the new URL. A frequently-changing API response served with Cache-Control: max-age=0, must-revalidate plus an ETag lets the client always check freshness cheaply (a 304 response, no body) without re-downloading the full payload on every request when nothing has actually changed.
Trade-offs and pitfalls
Setting Cache-Control: public on a response containing per-user or sensitive data lets shared caches (a CDN, a corporate proxy) serve one user's private data to another; always use private or no-store for personalized responses. Relying on Last-Modified timestamp granularity (often only second-level precision) can miss legitimately distinct versions of a resource that changed within the same second; ETag avoids this by tying validation to actual content, not a coarse timestamp.
Design a monitoring dashboard and alerting strategy for a distributed Redis cache serving an internal read-heavy API. Include specific metrics to display, dashboard panels, and alert conditions that would indicate (a) cache degradation, (b) emergence of a hot key, and (c) eviction-related problems.
Sample Answer
Direct answer
Design the dashboard around the three failure modes an operator actually needs to distinguish quickly: general degradation (rising latency or falling hit ratio across the board), an emerging hot key (one node/shard diverging from the rest), and eviction-related problems (memory pressure forcing out data that should still be cached).
Structured elaboration
- Panel: cluster-wide health: aggregate hit ratio, miss ratio, and p50/p95/p99 (50th/95th/99th percentile) latency over a rolling window, with the current service-level objective (SLO) target overlaid so a degradation is visible at a glance, not just as a raw number.
- Panel: per-node/per-shard breakdown: CPU, request rate, and latency broken out by individual node, specifically because an emerging hot key looks fine in the aggregate panel but shows one node's line diverging sharply from the others; this panel is what actually catches a hot key before it becomes an incident.
- Panel: eviction and memory: eviction rate and memory usage as a percentage of the configured limit, ideally split by key class or namespace if the cache is shared across use cases, so a memory-pressure problem can be traced to which data is actually filling the cache.
- Alert conditions for degradation: hit ratio dropping below a sustained threshold (e.g., more than 15 points below the trailing 7-day baseline for more than 2 consecutive 5-minute windows) or p99 latency exceeding the SLO target for a sustained period.
- Alert conditions for a hot key: any single node's request rate or CPU exceeding a set multiple (e.g., 3x) of the cluster's median node, sustained for more than a short window (to avoid alerting on normal brief variance).
- Alert conditions for eviction problems: eviction rate rising sharply relative to its own recent baseline, especially combined with memory usage near the configured limit, distinguishing "the cache is working as intended, evicting genuinely cold data" from "the cache is thrashing, evicting data that will be needed again soon."
Worked example
A hot-key alert fires when node 7's request rate is 4x the cluster median for more than 60 seconds; the dashboard's per-node panel immediately shows node 7 as a clear outlier against otherwise-flat lines for the other nodes, letting the on-call engineer confirm within seconds that this is a hot-key event rather than a cluster-wide issue, and start the hot-key mitigation runbook instead of investigating a broader outage.
Trade-offs and pitfalls
A dashboard that only shows cluster-wide aggregates cannot distinguish a hot-key problem from general health, which is exactly the failure mode this design targets; do not skip the per-node breakdown panel to save dashboard space. Alerting on eviction rate alone without memory context conflates "healthy, expected eviction" with "the cache is undersized"; always pair eviction-rate alerting with memory-utilization context.
Describe how to implement negative caching safely for non-existent resources and how to set TTLs to balance reduced DB load with the risk of false negatives. Explain mechanisms to detect and recover from incorrectly cached negatives and how to prevent poisoning of the cache.
Sample Answer
Direct answer
Negative caching (caching the fact that something does NOT exist) reduces repeated backend load for lookups that legitimately miss often, but needs a shorter time-to-live (TTL) than positive caching and explicit protection against caching a TRANSIENT error as if it were a permanent absence.
Structured elaboration
- When negative caching helps: an endpoint frequently queried for resources that legitimately do not exist (a sparse product search, a lookup by an ID that is often invalid or not-yet-created) benefits from caching "not found" so repeated identical misses do not all hit the backend.
- TTL sizing: negative entries should generally use a SHORTER TTL than positive entries, since a resource that does not exist now might be created moments later (a product about to be listed, a user about to be created), and a long negative-cache TTL would delay legitimate new data from becoming visible.
- Avoiding poisoning from transient errors: a backend timeout or a temporary error must NOT be cached as "not found"; only cache an explicit, confirmed "this does not exist" response (e.g., an actual 404 from a healthy backend), never an ambiguous failure, or a temporary outage will look like every queried resource permanently vanished.
- Detecting and recovering from an incorrectly cached negative: monitor the negative-cache hit rate for anomalies (a sudden spike suggesting something is being wrongly cached as absent), and provide an explicit purge mechanism for a specific key if a false negative is identified.
- Preventing cache poisoning attacks: without care, an attacker (or a misbehaving client) could probe many nonexistent IDs to fill the negative cache with entries, which is generally low-risk for negative caching specifically (it just wastes some memory) but worth being aware of if negative-cache capacity is shared with more sensitive cached data.
Worked example
A product-search endpoint receiving many queries for discontinued or misspelled product IDs: caching "not found" for 30 seconds (much shorter than the 5-minute TTL used for actual product data) absorbs repeated identical misses for the same nonexistent ID within that window, while still allowing a newly-listed product with that same ID to become visible within 30 seconds rather than being stuck behind a long negative-cache TTL.
Trade-offs and pitfalls
Caching a timeout or a 500-level error as a negative ("not found") result is the single most damaging mistake in negative caching; it turns a transient backend problem into an apparent, cache-durable "this doesn't exist" for every subsequent request until the negative TTL expires, actively making an outage worse and longer-lasting than it needed to be. Setting a negative TTL as long as the positive TTL "for simplicity" delays visibility of genuinely new data unnecessarily; size them independently based on how quickly a false negative needs to self-correct.
You must size an in-memory cache cluster for a dataset: total data size 200GB, expected working set 20GB, replication factor 2 for high availability, and 20% extra headroom for fragmentation and metadata. Explain how you would calculate node count and per-node memory, consider shard overhead, and account for future growth.
Sample Answer
Direct answer
Size a cache cluster from the actual working set, not the total dataset, then add replication and fragmentation headroom on top; node count and per-node memory come out of that total, not the other way around.
Structured elaboration
- Working set versus total data size: the total dataset (200 GB, gigabytes, in a typical example) is not what needs to fit in cache; the working set (20 GB, the subset actually read repeatedly within a relevant time window) is the real sizing driver, since the whole point of caching is serving the repeatedly-accessed subset, not the entire dataset.
- Replication factor: for high availability, each byte of the working set is stored on more than one node (replication factor 2 doubles the effective memory requirement, since every item exists on a primary and at least one replica).
- Fragmentation and metadata headroom: real-world memory usage is always somewhat higher than the raw data size, due to per-key overhead (metadata, pointers) and memory fragmentation from the allocator; a 20 percent headroom is a reasonable starting estimate, refined with actual measurement once running.
- Calculating node count and per-node memory: total required memory equals working set size times replication factor times (1 plus headroom fraction); node count times per-node memory must exceed that total, with per-node memory chosen based on available instance types and a preference for more, smaller nodes (better fault isolation, smaller blast radius per node loss) versus fewer, larger nodes (simpler operations, less network overhead).
- Accounting for future growth: size with headroom for expected growth over a planning horizon (e.g., 6 to 12 months), not just current working-set size, since scaling a cache cluster up later involves the same resharding/migration considerations covered elsewhere in this topic.
Worked example
Working set 20 GB, replication factor 2, 20 percent headroom: required memory is 20 GB×2×1.2=48 GB. Choosing nodes with 8 GB of usable cache memory each (leaving room for the node's own operational overhead), that requires ⌈48/8⌉=6 nodes. If 6-month growth projections suggest the working set could grow to 30 GB, planning for 30×2×1.2=72 GB, or 9 nodes at the same per-node size, avoids a resharding event shortly after initial rollout.
Trade-offs and pitfalls
Sizing off total dataset size instead of working set drastically overestimates the memory needed for most read-heavy workloads with meaningful access skew (a small fraction of data accounting for most reads); always validate the working-set assumption with real access-pattern data where available, not just an estimate. Under-provisioning headroom for fragmentation and metadata is a common source of "sized correctly on paper, still evicting more than expected in production" surprises; treat the headroom percentage as a starting estimate to refine with real measurement, not a fixed constant to trust blindly.
Unlock Full Question Bank
Get access to all Caching Strategies and Distributed Caching interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.