Caching Strategies and Distributed Caching Questions
Using caches to reduce latency and load: cache-aside, read-through, write-through, and write-behind patterns, TTLs, eviction policies, and distributed caches such as Redis or Memcached. Covers cache invalidation, stampede and thundering-herd protection, and the consistency tradeoffs of caching. Focuses on where and how to cache across tiers.
You must choose an eviction policy for a cache storing mixed-size objects and experiencing skewed access. Propose heuristics or hybrid policies to make eviction more effective, and describe how you would measure and adjust the policy in production.
Sample Answer
Direct answer
Tune the eviction policy to the access-pattern shape you actually have, not a generic default: mixed-size skewed objects need a policy aware of both size and frequency, a small set of suddenly-hot items needs fast adaptation, and approximating recency cheaply across a distributed cluster needs a bounded-overhead sampling scheme rather than tracking exact order.
Structured elaboration
- Mixed-size, skewed access: a plain least-recently-used (LRU) or least-frequently-used (LFU) policy treats every item as equally costly to evict; for genuinely mixed-size objects, a size-aware policy (evict based on a combined score of recency/frequency AND size, so a rarely-used large object is evicted before a frequently-used small one) uses memory more efficiently than pure recency or frequency alone.
- A small subset suddenly becoming hot: pure LRU adapts naturally (newly hot items get accessed and stay at the front), but if the cache is undersized relative to even this smaller hot set, consider a hybrid policy that gives newly-popular items a faster path to "protected" status rather than requiring them to accumulate history the way pure LFU would.
- Approximating LRU cheaply across a distributed cluster: exact global LRU across many nodes would require expensive coordination on every access; instead, each node maintains its own local approximate order (e.g., a clock/second-chance algorithm, or random sampling of a few candidates and evicting the oldest of the sample) which is a well-known, low-overhead approximation that gets most of LRU's benefit without global coordination cost.
- Measuring and adjusting in production: track hit ratio and eviction rate segmented by the object class you are tuning for (not just cluster-wide aggregates), and treat the eviction-policy choice as something to revisit as the workload's actual shape changes, not a one-time decision.
Worked example
A cache with a mix of small (1 KB, kilobyte) frequently-read config objects and large (500 KB) infrequently-read reports: pure LRU might evict a config object to make room for a large report that was touched once, wasting far more memory than it frees benefit; a size-weighted eviction score (evict the item with the worst benefit-per-byte, roughly recency/frequency divided by size) keeps the small, hot config objects resident far longer while letting large, cold reports get evicted first.
Trade-offs and pitfalls
A size-aware or hybrid eviction policy is more complex to implement and reason about than plain LRU/LFU; only add that complexity once measurement (not intuition) shows plain policies are genuinely underperforming for your specific mixed workload. Approximate distributed eviction schemes trade exact optimality for low coordination cost; validate with production metrics that the approximation is close enough, rather than assuming it matches single-node exact LRU behavior.
You want to measure the real-world impact of adding a caching layer on user-facing latency and backend load. Design an experiment (canary or A/B) and list the metrics you would collect, your sampling method, how you'd ensure statistical significance, and how to attribute improvements specifically to caching.
Sample Answer
Direct answer
Design a canary or A/B experiment that isolates the caching change as the only variable, measures user-facing latency percentiles and backend load together, and uses a large enough sample and duration to be statistically confident the observed improvement is real and attributable to caching, not to normal traffic variance.
Structured elaboration
- Traffic selection: split traffic randomly (not by time-of-day, which confounds with natural traffic pattern variance) into a control group (no new caching) and a treatment group (the new caching layer), ideally at the request or session level so the same user consistently lands in one group.
- Metrics to collect: client-observed p50/p95/p99 (50th/95th/99th percentile) latency (what users actually experience, not just server-side timing), backend requests per second (RPS) and CPU utilization (the load-reduction claim), and cache hit ratio (to explain WHY the latency/load changed, not just that it did).
- Sampling method: a consistent hashing of user or session ID into control/treatment buckets keeps assignment stable across a user's session, avoiding a user flip-flopping between groups mid-session, which would muddy the comparison.
- Statistical significance: with latency distributions typically non-normal, use a test appropriate for percentiles/distributions (bootstrap confidence intervals are common) rather than assuming a simple mean-based t-test applies cleanly; run long enough to cover a full daily and, ideally, weekly traffic cycle, since a short window can catch an unrepresentative slice of traffic.
- Attributing improvement specifically to caching: because the treatment group differs from control ONLY in the caching change (everything else held constant by the randomized split), a statistically significant difference in the treatment group's metrics can be attributed to the caching layer, rather than to some other simultaneous change, as long as no OTHER change ships selectively to one group during the experiment window.
Worked example
Running the experiment for two full weeks (covering weekday/weekend traffic patterns) with a 50/50 randomized split: if the treatment group's p95 latency is a statistically significant 40ms lower than control's (using a bootstrap confidence interval that excludes zero difference) and backend CPU utilization for the treatment group's traffic is meaningfully lower, both changes correlate with a treatment-group cache hit ratio of 85 percent, giving a coherent causal story (caching reduced backend calls, which reduced both load and the latency contributed by those calls).
Trade-offs and pitfalls
Running the experiment for too short a window (a few hours) risks capturing an unrepresentative traffic slice and either overstating or understating the real effect; always cover at least a full daily cycle, ideally longer. Shipping any OTHER change to the treatment group during the experiment window (even an unrelated one) breaks the attribution, since you can no longer isolate caching as the sole cause of any observed difference.
You must choose between Redis and Memcached to implement a session store for a web app. List trade-offs and recommend one choice. Consider persistence, data types, replication/HA, memory efficiency, eviction semantics, and operational features such as monitoring and backup.
Sample Answer
Direct answer
Choose Redis when you need richer data structures, persistence, or replication/high-availability (HA) built in; choose Memcached when you need the simplest possible pure key-value cache with the lowest per-operation overhead and multi-threaded read scaling out of the box.
Structured elaboration
- Data types: Redis supports strings, hashes, lists, sets, sorted sets, and more, useful when the cache itself needs to do more than store opaque blobs (e.g., a sorted set for a leaderboard). Memcached only stores simple key-value byte strings.
- Persistence: Redis can persist to disk (RDB snapshots, append-only file, AOF) so data can survive a restart; Memcached is purely in-memory with no persistence, so a restart is always a full cache flush.
- Replication / high availability: Redis has built-in replication and clustering (Sentinel for failover, Cluster for sharding); Memcached has no native replication, relying on the client or an external layer for HA.
- Memory efficiency: Memcached's simpler data model generally has lower per-key memory overhead for pure key-value use cases; Redis's richer data structures and features carry some additional overhead.
- Eviction semantics: both support least-recently-used (LRU) style eviction, but Redis offers more configurable policies (
allkeys-lru,volatile-ttl,allkeys-lfu, and others) versus Memcached's simpler slab-based LRU. - Operational features: Redis has a larger ecosystem for observability, Lua scripting for atomic multi-step operations, and pub/sub; Memcached's multi-threaded architecture can give it an edge on raw throughput for simple get/set at very high concurrency on a single node.
Worked example
For a session store: if sessions are pure key-value blobs, Memcached is a perfectly reasonable, simpler choice. If sessions need persistence across a restart (avoiding logging every user out simultaneously) or replication for HA, Redis with AOF persistence and Sentinel-managed failover is the safer choice, at the cost of a slightly more complex operational footprint.
Trade-offs and pitfalls
Choosing Redis "because it can do more" for a workload that is genuinely simple key-value adds operational surface area (persistence tuning, replication topology) without benefit; match the tool to the actual requirement. Memcached's lack of native replication means a node loss is a hard cache-miss event for everything that node held, with no automatic failover; that must be an explicit, accepted trade-off, not an oversight.
Design a caching approach for GraphQL APIs where clients can request arbitrary fields across entities. Discuss normalized client-side caches (e.g., Apollo cache normalization), server-side response caching, persisted queries, caching at field vs object level, and invalidation strategies for mutations that touch nested data.
Sample Answer
Direct answer
GraphQL's flexible field selection makes whole-response caching largely ineffective (every distinct query shape produces a distinct response), so effective GraphQL caching happens at a finer grain: normalized entity caching on the client, persisted queries to make server-side caching by query identity practical, and field- or object-level caching strategies that survive arbitrary client query shapes.
Structured elaboration
- Normalized client-side caches: a library like Apollo Client's cache normalizes responses by entity ID (a
Product:123object is stored once, regardless of which query fetched it, or which fields were requested across different queries), so overlapping queries share cached entity data even when their exact field selections differ, and a mutation affectingProduct:123can update every query result referencing it without a full refetch. - Server-side response caching: because two clients requesting even slightly different field sets produce different response bodies, naive whole-response caching (keyed on the raw query string) has poor hit rates; this works better when combined with persisted queries (see below), which constrain the space of distinct queries actually seen.
- Persisted queries: instead of sending the full query text on every request, the client sends a hash referencing a pre-registered query (registered at build time); this both reduces request size AND makes server-side response caching practical again, since the server now sees a small, known set of distinct queries rather than an unbounded space of ad-hoc client-constructed ones.
- Field vs. object-level caching: caching at the object level (an entity like a specific product, regardless of which query fetched it) is more reusable across different client queries than caching at the whole-response level; some GraphQL server implementations support field-level caching directives to fine-tune this further for expensive individual fields.
- Invalidation for mutations touching nested data: a mutation updating one entity can affect many different cached query results that reference it (directly, or through a relationship); a normalized cache's entity-level invalidation (update
Product:123once, every referencing query result updates) is what makes this tractable, versus trying to enumerate and invalidate every affected whole-response cache entry.
Worked example
An Apollo Client cache normalized by entity ID means a query fetching Product(id: 123) { name, price } and a separate query fetching Product(id: 123) { name, description } both read and update the SAME normalized Product:123 entry for the name field they share, and a mutation updating Product:123's price automatically reflects in both queries' results on their next render, without either query needing to know about the other or trigger an explicit refetch.
Trade-offs and pitfalls
Whole-response server-side caching for GraphQL without persisted queries tends to have a poor hit ratio in practice, because the space of distinct query shapes a flexible client can construct is effectively unbounded; persisted queries are usually a prerequisite for server-side caching to be worthwhile at all, not an optional add-on. Normalized client caching requires entities to have stable, consistent IDs across every query that returns them; a schema that returns the same logical entity with inconsistent ID fields across different query paths breaks normalization silently.
You are designing cache invalidation for a globally distributed service where reads are frequent and writes happen in a primary region: describe strategies to keep caches coherent across regions. Discuss trade-offs between consistency, staleness, cost, and complexity (push invalidation, TTL, versioned keys, fanout updates).
Sample Answer
Direct answer
Keeping caches coherent across regions means picking, per data class, between a pull model (short time-to-live, TTL, that self-heals without any messaging) and a push model (explicit invalidation events fanned out to every region), and being explicit about the bounded-staleness window you are willing to accept in exchange for lower cost and complexity.
Structured elaboration
- TTL (pull-based): the simplest option. Every region's cache entry expires on its own after a fixed window; no cross-region messaging is required. Staleness is bounded by the TTL itself, cost is near zero, and it self-heals from any missed update (a lost invalidation event just means the entry expires normally at worst-case TTL). The downside is that staleness is guaranteed, not just possible, for the full TTL window.
- Push invalidation via messaging: on write, the writing region publishes an invalidation event (key, and optionally a version or tombstone) to a durable message bus; consumers in every other region apply it to their local cache. This gets close-to-immediate propagation (bounded by replication lag of the message bus, typically under a second) but adds a moving part that can itself fail, reorder, or duplicate, so you still need a TTL as a backstop.
- Versioned keys: instead of invalidating, bump a version token that is part of the cache key (
profile:123:v42); old versions simply age out via normal eviction rather than needing an active delete. This avoids race conditions where an invalidation arrives before the write it corresponds to, at the cost of needing a place to look up "what's the current version" (which itself needs to be fast and consistent). - Fanout updates: instead of invalidating (forcing every region to re-fetch), push the new VALUE itself to every region's cache. This trades more network bandwidth (sending full payloads, not small invalidation markers) for lower read-side latency (no cache-miss round-trip to origin after the update lands), and is worth it for small, very hot objects.
- Bounded staleness as the actual requirement: whichever mechanism you choose, state the staleness bound explicitly (e.g., "under 5 seconds in 99% of cases") and design the propagation pipeline's monitoring around that number, rather than treating "eventually consistent" as sufficient specification.
Worked example
A profile-update event published in the primary write region reaches a message bus with typical cross-region replication lag of 200 to 500 ms, plus consumer processing time of tens of milliseconds per region; a realistic end-to-end propagation time is under 1 second for the median case, with a long tail driven by consumer backlog during traffic spikes. If the product requirement is "profile changes visible within 5 seconds globally," a push-based invalidation pipeline with a 5-second TTL backstop comfortably meets it even if a specific invalidation event is dropped, because the TTL guarantees the bound independently.
Trade-offs and pitfalls
Relying purely on push invalidation without a TTL backstop means any lost, delayed, or duplicated event becomes an unbounded staleness bug with no self-healing mechanism; always pair push invalidation with a TTL ceiling. Fanout-updates for large or rarely-read objects wastes bandwidth pushing data nobody will read in some regions; reserve it for small, universally-hot keys. Versioned keys solve the ordering race but shift complexity to "how fast and how consistent is the version lookup itself," which can become its own single point of failure if not designed carefully.
Unlock Full Question Bank
Get access to all Caching Strategies and Distributed Caching interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.