Caching Strategies and Distributed Caching Questions
Using caches to reduce latency and load: cache-aside, read-through, write-through, and write-behind patterns, TTLs, eviction policies, and distributed caches such as Redis or Memcached. Covers cache invalidation, stampede and thundering-herd protection, and the consistency tradeoffs of caching. Focuses on where and how to cache across tiers.
List and explain the key operational metrics you would monitor for cache health and performance. For each metric state what action you would take if it trends poorly.
Sample Answer
Direct answer
The core cache health metrics are hit ratio, eviction rate, and tail latency (p95/p99, 95th/99th percentile); each one answers a different question, and a regression in any of them should trigger a specific, different response.
Structured elaboration
- Hit ratio: hits divided by total requests. A sustained drop is the earliest, most sensitive signal that something is wrong (undersized cache, a cold cache after a deploy, or a bug in cache-key construction); it should page or alert before the resulting latency/database-load impact becomes user-visible.
- Eviction rate: how often items are being evicted to make room for new ones. A high eviction rate on a workload whose working set should fit comfortably usually means the cache is undersized relative to demand; the action is to increase capacity or investigate why the working set grew unexpectedly.
- Tail latency (p95/p99): because a miss can be an order of magnitude slower than a hit, tail latency for cache operations directly reflects the hit ratio and the underlying origin's own latency; track it separately from average latency, since averages hide the exact experience a meaningful fraction of users have.
- Memory usage: absolute memory consumption and headroom before hitting the configured limit; combined with eviction rate, this tells you whether you are evicting because of genuine memory pressure or because of a misconfigured limit.
- What action to take when each regresses: a hit-ratio drop triggers investigating recent deploys, key-format changes, or a genuine traffic-pattern shift; a rising eviction rate triggers a capacity review; rising tail latency (with a stable hit ratio) points at the origin or the network path, not the cache itself.
Worked example
A service with a stable 90 percent hit ratio and healthy p99 latency deploys a change that accidentally alters the cache key format (e.g., adding a new query parameter to the key without updating existing entries). Hit ratio drops to near zero for that endpoint's traffic instantly. Because hit ratio is watched as a leading indicator, this is caught and rolled back within minutes; without it, the team would only notice once database load and p99 latency alerts fired, by which time real user impact has already occurred.
Trade-offs and pitfalls
Watching only aggregate, cluster-wide metrics can hide a problem localized to one shard or one hot key; pair cluster-wide dashboards with the ability to drill into per-shard or per-key detail. Alerting on a single noisy data point rather than a sustained trend produces alert fatigue; use a rolling window (e.g., 5-minute average, sustained for 2+ consecutive windows) before paging.
For a multi-tenant platform using Redis as a shared caching layer, propose a secure architecture: cover access control, encryption in transit and at rest, tenant key isolation, key discovery and least-privilege, detection of key leakage, and an operational runbook for key compromise. Discuss performance implications.
Sample Answer
Direct answer
Securing a shared multi-tenant cache means treating it like any other multi-tenant datastore: authenticate and authorize every client, encrypt data in transit and at rest, isolate tenants' keys from each other, and have a plan for detecting and responding to a key leak or compromise.
Structured elaboration
- Access control: use Redis access-control lists (ACLs) or an equivalent mechanism to give each service or tenant the minimum permissions it needs (read-only where possible, scoped to its own key prefix), rather than a single shared credential with full access for every client.
- Encryption in transit and at rest: Transport Layer Security (TLS) between clients and the cache prevents network-level eavesdropping; encryption at rest (or encrypting sensitive fields before storing them) protects against a compromised disk or snapshot backup being readable.
- Tenant key isolation: three common patterns, each with different trade-offs: key prefixes (simplest, but relies entirely on application-level discipline to never cross prefixes), logical databases (Redis's numbered databases, a bit more isolation but still shared infrastructure), and separate clusters per tenant (strongest isolation, highest operational cost). Choose based on how sensitive the data is and how much a cross-tenant leak would cost you.
- Least-privilege key discovery: a compromised client credential should only be able to see or affect its own tenant's keys, never enumerate or read across tenants; this is what ACLs scoped to key prefixes are for.
- Detecting key leakage: monitor for access patterns that look like enumeration (a client rapidly scanning many keys outside its normal pattern) or access from an unexpected tenant's credentials to another tenant's prefix.
- Operational runbook for key compromise: rotate the compromised credential immediately, audit what that credential accessed during the suspected compromise window, and assess whether any cached sensitive data needs to be purged or whether downstream systems need notification.
Worked example
A platform serving hundreds of tenants through one Redis cluster uses key prefixes (tenant:{id}:...) plus per-tenant ACL rules restricting each service credential to ~tenant:{id}:* patterns only; a compromised credential for one tenant therefore cannot read or write any other tenant's keys, bounding the blast radius of that specific credential's compromise to one tenant's data.
Trade-offs and pitfalls
Key prefixes alone, without ACL enforcement, are a convention, not a security boundary; a bug or a malicious client can simply read any prefix if nothing actually enforces the restriction. Performance implications of encryption (TLS overhead, at-rest encryption/decryption cost) are usually small relative to network and compute costs elsewhere, but should be measured, not assumed, especially for very high-throughput, low-latency use cases.
You observe that total request latency rose despite a high cache hit rate. Outline a step-by-step debugging plan to identify the root cause, including what metrics and traces you would collect.
Sample Answer
Direct answer
When latency rises despite a healthy hit ratio, the cache is doing its job (finding data), so the investigation shifts away from "is data cached" toward "what is slow about serving a cached value": serialization cost, network hops, object size, or an inconsistent-key problem that is quietly generating hidden extra misses within an otherwise-healthy aggregate hit ratio.
Structured elaboration
- Serialization/deserialization cost: a growing or changed payload shape can make encoding/decoding on every cache hit meaningfully slower even though the read itself succeeded; check whether payload size or format changed recently.
- Object size: a larger cached object takes longer to transfer over the network even on a hit; compare current average object size against historical baselines for a regression.
- Network hops / cache network topology: a routing change (a new proxy layer, a cross-availability-zone hop that used to be same-zone) adds latency to every hit without affecting whether it is a hit at all; check whether the cache client's network path to the cache changed recently.
- Cache misses due to inconsistent keys: a subtle bug (a key that includes a field with slightly different formatting on different code paths) can cause SOME requests to always miss while the AGGREGATE hit ratio still looks acceptable if that subset is a small fraction of total traffic; segment hit ratio and latency by endpoint or request shape, not just in aggregate.
- Backend cascading delays: if the "cache hit" path itself calls out to something else after the cache read (a second lookup, an enrichment call), a slowdown there shows up as rising latency without touching the cache's own hit ratio at all.
- Debugging plan: start with distributed tracing to see where time is actually spent within a request (cache read, deserialization, post-cache-hit processing), compare payload sizes and network path over time, and segment metrics by endpoint/request-shape rather than relying purely on aggregate numbers that can hide a localized problem.
Worked example
A service's overall hit ratio holds steady at 92 percent, but p99 latency has crept up over two weeks; tracing shows the cache READ itself is still fast (single-digit milliseconds), but deserialization time for the cached payload has grown from 2ms to 40ms, correlating with a recent change that added several new fields to the cached object; the fix is either trimming the payload to only what is actually needed, or switching to a faster serialization format, not anything related to the cache layer itself.
Trade-offs and pitfalls
Assuming "hit ratio is fine, so the cache isn't the problem" and stopping the investigation there is a common mistake; a hit ratio number says nothing about how expensive it is to USE a cached value once found. Aggregate metrics can hide a localized problem affecting a small but meaningful subset of traffic; segment by endpoint or request type before concluding a metric is uniformly healthy.
Explain the trade-offs between client-side caching (in-process or browser) and a server-side shared cache (Redis). From an operations standpoint, what concerns differ between the two approaches?
Sample Answer
Direct answer
Client-side (in-process or browser) caching is faster and needs no network hop but is invisible and hard to invalidate from the server; a server-side shared cache (Redis) is slower per read (a network hop) but is centrally observable, controllable, and consistent across every consumer.
Structured elaboration
- In-process caching: fastest possible access (no network at all), but exists as N independent copies across N instances, invisible to any centralized monitoring or invalidation mechanism unless you specifically build one (e.g., pub/sub to every instance).
- Browser caching: similarly fast for the end user, but the server has essentially no direct control once data leaves it; "invalidation" really means waiting out a time-to-live (TTL) or changing a versioned resource identifier.
- Server-side shared cache (Redis): a single, centrally-managed copy; invalidation is a single operation that immediately affects every consumer, and its state is directly observable via standard monitoring, at the cost of a network round-trip on every access.
- Operational concerns for a shared server-side cache: capacity planning, replication/failover for availability, and security (access control, encryption) are all first-class operational concerns that a purely in-process or browser cache does not have in the same way, since those are either per-instance (in-process, trivially "available" as long as the instance is up) or entirely outside your infrastructure (browser).
- Service-mesh sidecar caching: a variant worth naming, a sidecar proxy running alongside each service instance can cache responses transparently at the network layer; this gets some of in-process caching's speed benefit without embedding caching logic in application code, but invalidation and consistency questions are similar to any other local-cache design (each sidecar has its own copy).
- A hybrid, fast-tier-plus-persistent-tier design: combining a fast in-memory tier for hot items with a persistent, larger tier for warm items (promotion/demotion between them based on access) gets some of both worlds' benefits, at the cost of managing two tiers' worth of operational complexity.
Worked example
An application caching a rarely-changing configuration value both in-process (near-zero latency access on every request) and in Redis (as the source the in-process cache warms from, and as what gets explicitly invalidated on a config change) gets the speed benefit of in-process access for the common case while still having a centrally controllable invalidation point; the in-process copies use a short TTL as a backstop in case a pub/sub-based invalidation signal is missed by a given instance.
Trade-offs and pitfalls
Relying purely on in-process caching for data that changes and needs prompt, reliable invalidation across many instances is a common design mistake; without a centralized signal (or at least a short backstop TTL), some instances can serve stale data indefinitely after a change. From an operations standpoint, a shared server-side cache is a dependency you must monitor, secure, and plan capacity/failover for, in a way an in-process cache simply is not; that operational surface is a real cost, not just a technical detail.
You must choose an eviction policy for a cache storing mixed-size objects and experiencing skewed access. Propose heuristics or hybrid policies to make eviction more effective, and describe how you would measure and adjust the policy in production.
Sample Answer
Direct answer
Tune the eviction policy to the access-pattern shape you actually have, not a generic default: mixed-size skewed objects need a policy aware of both size and frequency, a small set of suddenly-hot items needs fast adaptation, and approximating recency cheaply across a distributed cluster needs a bounded-overhead sampling scheme rather than tracking exact order.
Structured elaboration
- Mixed-size, skewed access: a plain least-recently-used (LRU) or least-frequently-used (LFU) policy treats every item as equally costly to evict; for genuinely mixed-size objects, a size-aware policy (evict based on a combined score of recency/frequency AND size, so a rarely-used large object is evicted before a frequently-used small one) uses memory more efficiently than pure recency or frequency alone.
- A small subset suddenly becoming hot: pure LRU adapts naturally (newly hot items get accessed and stay at the front), but if the cache is undersized relative to even this smaller hot set, consider a hybrid policy that gives newly-popular items a faster path to "protected" status rather than requiring them to accumulate history the way pure LFU would.
- Approximating LRU cheaply across a distributed cluster: exact global LRU across many nodes would require expensive coordination on every access; instead, each node maintains its own local approximate order (e.g., a clock/second-chance algorithm, or random sampling of a few candidates and evicting the oldest of the sample) which is a well-known, low-overhead approximation that gets most of LRU's benefit without global coordination cost.
- Measuring and adjusting in production: track hit ratio and eviction rate segmented by the object class you are tuning for (not just cluster-wide aggregates), and treat the eviction-policy choice as something to revisit as the workload's actual shape changes, not a one-time decision.
Worked example
A cache with a mix of small (1 KB, kilobyte) frequently-read config objects and large (500 KB) infrequently-read reports: pure LRU might evict a config object to make room for a large report that was touched once, wasting far more memory than it frees benefit; a size-weighted eviction score (evict the item with the worst benefit-per-byte, roughly recency/frequency divided by size) keeps the small, hot config objects resident far longer while letting large, cold reports get evicted first.
Trade-offs and pitfalls
A size-aware or hybrid eviction policy is more complex to implement and reason about than plain LRU/LFU; only add that complexity once measurement (not intuition) shows plain policies are genuinely underperforming for your specific mixed workload. Approximate distributed eviction schemes trade exact optimality for low coordination cost; validate with production metrics that the approximation is close enough, rather than assuming it matches single-node exact LRU behavior.
Unlock Full Question Bank
Get access to all Caching Strategies and Distributed Caching interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.