Caching Strategies & In-Memory Optimization Questions
Designing cache layers to cut redundant work and speed up reads, and the correctness costs that come with them. Covers cache placement across client, CDN, application, and in-memory store tiers, including platform-specific caching on mobile (in-memory and disk caches on iOS and Android, secure on-device token storage) and in the browser (localStorage, IndexedDB, Service Workers). Covers eviction policies, TTLs, write-through vs write-back, warming, and invalidation, plus cache stampede and thundering-herd mitigation when a hot key expires under load. Also covers using an in-memory store like Redis directly rather than only as a cache: distributed locks, counters, rate limiters, and leaderboards. Includes distributed and sharded cache architecture (replication, failover, sizing, migration) and the operational side of running one: health monitoring, alerting, and incident runbooks, plus the security and privacy risks caching introduces, such as cache poisoning and caching PII across shared tenants. Emphasizes hit-rate reasoning and the staleness/consistency trade-offs caching introduces.
Your company wants to roll out caching across multiple backend services to reduce DB costs, but leaders worry about correctness risk. Prepare a short plan (technical + rollout) describing how you'd pilot caching, measure success, handle invalidation, implement circuit-breakers, and provide rollback. Include what success metrics you'd track and how to mitigate data-staleness issues.
Sample Answer
Plan: pilot first, prove correctness before chasing savings
Cost reduction is the business goal, but correctness risk is the actual blocker leadership named, so the plan is sequenced to retire the risk before scaling the savings: pilot on one low-risk service, instrument it heavily, and only widen the rollout once the pilot's own metrics show it's safe.
Pilot selection and rollout mechanism
Pick a read-heavy, non-financial service first, a product-catalog read endpoint rather than a payments or inventory-count endpoint where a stale read has a direct financial consequence, and roll it out with a deterministic canary: bucket a percentage of TRAFFIC (not a percentage of time) into the caching path, so cached and uncached behavior can be compared side by side on the same day and traffic mix, not before/after across two different days.
import random
import hashlib
random.seed(7)
def canary_bucket(request_id: str, rollout_pct: int) -> bool:
# Deterministic: the SAME request_id always lands in the same bucket.
# Python's built-in hash() is randomized per process by design, so it
# is NOT safe here; a stable hash keeps the bucket assignment the same
# across restarts.
digest = hashlib.md5(request_id.encode()).hexdigest()
return (int(digest, 16) % 100) < rollout_pct
class CacheCircuitBreaker:
# CLOSED = caching allowed. OPEN = bypass cache, read DB directly.
# HALF_OPEN = probe with a single request to check for recovery.
def __init__(self, window=20, error_threshold=0.3, cooldown_calls=15):
self.window = window
self.error_threshold = error_threshold
self.cooldown_calls = cooldown_calls
self.results = []
self.state = "CLOSED"
self.calls_since_open = 0
self.trips = 0
def allow_cache(self) -> bool:
if self.state == "OPEN":
self.calls_since_open += 1
if self.calls_since_open >= self.cooldown_calls:
self.state = "HALF_OPEN"
return True
return False
return True
def record(self, ok: bool):
if self.state == "HALF_OPEN":
self.state = "CLOSED" if ok else "OPEN"
if self.state == "OPEN":
self.calls_since_open = 0
self.trips += 1
else:
self.results = []
return
self.results.append(ok)
if len(self.results) > self.window:
self.results.pop(0)
if len(self.results) == self.window:
error_rate = 1 - sum(self.results) / self.window
if error_rate >= self.error_threshold:
self.state = "OPEN"
self.calls_since_open = 0
self.trips += 1
def call_cache(healthy: bool) -> bool:
fail_rate = 0.02 if healthy else 0.6
return random.random() > fail_rate
breaker = CacheCircuitBreaker(window=20, error_threshold=0.3, cooldown_calls=15)
db_reads, cache_reads, cache_errors = 0, 0, 0
service_healthy = True
for i in range(160):
req_id = f"svc-orders-{i}"
in_rollout = canary_bucket(req_id, rollout_pct=40)
if i == 40:
service_healthy = False # inject a cache-tier incident
if i == 90:
service_healthy = True # incident resolved
if in_rollout and breaker.allow_cache():
ok = call_cache(service_healthy)
breaker.record(ok)
if ok:
cache_reads += 1
else:
cache_errors += 1
db_reads += 1 # cache errored -> fall back to DB
else:
db_reads += 1
if i in (39, 60, 89, 120, 159):
print(f"req {i:>3} | breaker={breaker.state:<9} trips={breaker.trips} "
f"| cache_reads={cache_reads} cache_errors={cache_errors} db_reads={db_reads}")
total_requests = 160
total_reads = cache_reads + cache_errors + db_reads
print(f"\nfinal: {total_requests} requests -> {total_reads} read ops "
f"(a cache error costs 2: the failed cache read + the DB fallback it triggers). "
f"cache hit-path={cache_reads} ({cache_reads/total_requests:.0%} of requests), "
f"DB reads={db_reads} ({db_reads/total_requests:.0%} of requests), "
f"breaker tripped {breaker.trips} time(s) during the injected incident")
Output:
req 39 | breaker=CLOSED trips=0 | cache_reads=18 cache_errors=0 db_reads=22
req 60 | breaker=OPEN trips=1 | cache_reads=20 cache_errors=6 db_reads=41
req 89 | breaker=OPEN trips=1 | cache_reads=20 cache_errors=6 db_reads=70
req 120 | breaker=CLOSED trips=1 | cache_reads=31 cache_errors=6 db_reads=90
req 159 | breaker=CLOSED trips=1 | cache_reads=48 cache_errors=6 db_reads=112
final: 160 requests -> 166 read ops (a cache error costs 2: the failed cache read + the DB fallback it triggers). cache hit-path=48 (30% of requests), DB reads=112 (70% of requests), breaker tripped 1 time(s) during the injected incident
The breaker did its job automatically: when the simulated incident pushed the cache's error rate over 30% in a rolling 20-request window (around request 41 to 60), it opened and every canary request fell through to the database directly, with zero human intervention, until the cooldown period let it try a probe request and confirm recovery, closing again by request 120.
Success metrics
Track, per canary bucket versus the same-day control bucket: database query volume (the actual cost-reduction signal), p50 and p99 latency, error rate, and cache hit rate. Comparing against a same-day control bucket is what makes the metric trustworthy, database load varies hour to hour, so "queries dropped 30%" only means something measured against a control group seeing the same traffic pattern, not against yesterday's baseline.
Invalidation
Default to TTL-based expiry, bounded staleness with no invalidation logic to get wrong, for data where a short delay is acceptable, and use explicit delete-on-write invalidation only where staleness has real cost, for example a product's cache entry the moment its price changes. Explicit invalidation is the higher-risk mechanism, a missed invalidation path is a correctness bug that's easy to miss until a customer hits it, so the pilot should start TTL-only and add explicit invalidation for specific fields only once that simpler path has proven safe.
Rollback
Two levers, cheapest first: turn the canary percentage back to zero via the same flag used to ramp it up, no deploy required; and if the caching code path itself is suspect, a full rollback to the pre-caching deploy. The circuit breaker above is a third, automatic layer that acts faster than either human-triggered lever, which is why it's built into the pilot rather than bolted on afterward.
Mitigating data staleness
Set TTLs based on how quickly the underlying data actually changes, minutes for a product description, seconds for an inventory count, and for anything a customer would notice directly, such as an address update, use read-your-own-writes: invalidate that specific key immediately after the write, or route that user's next read past the cache for a short window, rather than lowering the TTL for everyone to cover one user's edge case.
Trade-off
Starting with TTL-only invalidation and a conservative canary percentage is slower to reach the cost target than caching everything with explicit invalidation from day one, but it directly answers the stated worry: it makes the riskiest mechanism, explicit invalidation, something added deliberately once the simpler path is already proven safe, not something the whole rollout is bet on from the start.
You need to convince a non-technical executive to fund a project to implement a multi-layer caching solution to reduce 99th percentile latency and improve conversions. Draft the outline of a pitch that includes the problem statement, measurable objectives, expected benefits, estimated costs, key risks, and a phased delivery plan with quick wins.
Sample Answer
Framing
This pitch is for an executive who cares about revenue, cost, and risk, not cache internals. Lead with business impact and hold the mechanism (multi-layer caching) as the "how," not the headline.
Problem statement
Name the pain in the executive's terms: checkout page p99 (the 99th percentile load time, the slow end of the distribution most users never notice but a meaningful minority hits every day) is currently well above target, and slow pages correlate with cart abandonment. Anchor it to something the executive already tracks, conversion rate or a support-ticket trend, rather than a purely technical number.
Measurable objectives
Separate the technical objective (reduce p99 latency to a stated target) from the business objective (reduce cart abandonment attributable to slow loads by a stated amount, or move conversion rate toward a target), and state explicitly that the link between the two is a modeled estimate to be confirmed by the pilot, not a guaranteed number.
Expected benefits, backed by a simple model
# every input is a labeled assumption to be replaced with real measured numbers
monthly_visits = 2_000_000
current_conversion_rate = 0.021
avg_order_value_usd = 46.0
p95_latency_ms_before = 480
p95_latency_ms_after = 180
latency_removed_ms = p95_latency_ms_before - p95_latency_ms_after
# deliberately conservative illustrative rule of thumb, not a measured constant
relative_lift = min(0.03, 0.003 * (latency_removed_ms / 100))
new_conversion_rate = current_conversion_rate * (1 + relative_lift)
extra_orders = monthly_visits * (new_conversion_rate - current_conversion_rate)
extra_revenue = extra_orders * avg_order_value_usd
Running this: modeled relative conversion lift 0.90%, conversion rate 2.10% to 2.119%, about 378 extra orders per month, about $17,388 in modeled extra monthly revenue.
Estimated costs
A Redis cluster at roughly $3,200 per month in run cost, and a one-time build cost of about six engineer-weeks (roughly $28,800 at a typical loaded cost). Net of infra cost, the model above pays back the one-time build cost in about two months, a number worth stating precisely because payback period is the number most executives will actually remember.
Key risks
A new cache layer is a new source of stale-data bugs if invalidation is not built carefully, name this briefly rather than hiding it. There is real operational risk, new infrastructure to run and staff on-call for. And there is a risk in NOT doing it, continued abandonment and a competitor with a faster checkout, worth stating to balance the pitch rather than only listing downside risk of the proposal itself.
Phased delivery plan with quick wins
Phase 0 (about two weeks): a small, low-risk pilot on one high-traffic, easy-to-cache endpoint, this both de-risks the approach and builds credibility with real numbers before asking for the full investment. Phase 1: extend to the next highest-traffic endpoints, replacing the illustrative model above with real measured hit-rate and latency numbers from the pilot. Phase 2: full multi-layer rollout, edge, application, and database read-offload, justified by the pilot's actual results rather than the initial estimate.
Which metrics to lead with
Lead the room with exactly one technical metric (p99 latency, the SLO, service level objective, the target you hold that metric to, stakeholders can hold the team accountable to) and one business metric it is modeled to move (conversion or revenue). Save operating metrics like hit rate and eviction counts for the team's own dashboards and the post-pilot report, where they become evidence rather than noise in the pitch itself.
You're responsible for migrating multiple teams from per-service local caches to a shared Redis cluster. Outline a governance and operational plan: tenancy model, quotas, key naming conventions, SLAs, monitoring, on-call responsibilities, and a migration checklist that minimizes regressions.
Sample Answer
Framing
Local per-service caches (each service keeping its own in-process or single-node cache) can't share warm data across services, and there's no consistent way to reason about capacity, ownership, or blast radius. Moving to a shared cluster fixes reuse but introduces a new risk: one team's bug or spike can now degrade every other team sharing the cluster. The plan below is built around containing that risk.
Tenancy model
Two realistic choices: logical tenancy (every team shares one Redis cluster, isolated only by key namespace and per-tenant quotas) or physical tenancy (each team, or risk tier, gets its own cluster). Logical tenancy is cheaper and easier to reuse hot keys across teams, but one misbehaving client can starve everyone else. Physical tenancy costs more to operate but bounds blast radius. A workable middle ground: logical tenancy for low-risk, low-QPS teams, and a dedicated cluster for the handful of teams whose traffic is large enough to threaten a shared one.
Quotas
Every team gets a hard memory quota (bytes) and a request-rate quota (ops/sec), enforced at the client-connection or proxy layer, not just monitored after the fact. Enforcement matters because a shared maxmemory-policy will evict everyone's keys under LRU (least-recently-used eviction, which drops whichever key has gone longest unused) once ONE tenant fills the cluster; a proxy that rejects or throttles the offending tenant protects everyone else. Start quotas conservative, sized from each team's current local-cache footprint plus headroom, and revisit after a month of real traffic.
Key naming conventions
Enforce a fixed prefix scheme, <team>:<service>:<entity>:<id>, applied by a shared client library rather than left to each team, so quotas can be measured per prefix, an incident can be triaged to "which team's keys are growing," and a team's keys can be bulk-purged without touching anyone else's. Reserve a version segment (<team>:<service>:v2:<entity>:<id>) so a breaking schema change doesn't require a coordinated flush.
SLAs
Publish a cluster-wide latency and availability SLA (service level agreement, a formal commitment on latency/availability), but also publish PER-TENANT budgets (max ops/sec, max memory): once a tenant exceeds their agreed budget, the SLA no longer covers their own requests, they get throttled instead of degrading the whole cluster. That gives every team a concrete number to design against.
Monitoring
Track four things per tenant, not just cluster-wide: hit rate, ops/sec, memory used, and evictions/sec. Cluster-wide, track replication lag, memory used versus max, and command latency percentiles. Alert on any tenant seeing evictions while staying under quota (a symptom that a DIFFERENT tenant blew past quota and is evicting everyone), and alert at 80% of a tenant's own quota so on-call has lead time before a hard throttle.
On-call responsibilities
The platform team owns the cluster itself: failover, replication health, capacity, upgrades. Each consuming team owns their own cache-miss behavior, what their service does when Redis is slow or unavailable. That split has to be explicit in the runbook, because "the cache is down" is really the platform team's incident plus, potentially, N teams' incidents if their fallback path was never tested.
Migration checklist
- Ship a shared client library with naming convention, TTL defaults, and quota-aware error handling before any team migrates.
- Migrate the lowest-traffic, lowest-risk team first as a pilot; validate quotas and monitoring against real traffic.
- Shadow-read against both the old local cache and the shared cluster for one team, comparing hit rate and error rate before cutting over.
- Cut over reads behind a per-team feature flag, with an instant rollback to the old local cache if error rate or latency regresses.
- Decommission a team's local cache only after their shared-cluster metrics have been stable for an agreed burn-in window.
- Never migrate two large teams in the same week, so a regression has an unambiguous owner.
Trade-off
This trades migration speed for safety: one team at a time with a rollback flag is slower than a big-bang cutover, but a shared cluster's real failure mode (one team's regression hurting everyone) is exactly the risk leadership is worried about, so the checklist surfaces that risk on the smallest blast radius first.
Explain the difference between cache-aside, write-through, and write-behind (write-back) caching patterns. For each pattern list one ideal use case and one significant downside that a Solutions Architect must communicate to stakeholders.
Sample Answer
Direct answer
These are three different answers to "who writes to the cache, and when." Cache-aside (lazy
loading): the application checks the cache first, and on a miss, reads the database itself and
populates the cache; the cache only ever holds what's actually been asked for. Write-through:
every write updates the cache and the database together, synchronously, before the write is
considered done, so the cache is never stale but every write pays the database's latency.
Write-behind (write-back): a write lands in the cache immediately and is acknowledged right
away, while the database update happens afterward, asynchronously, which is fast but leaves a
window where a crash can lose data that was never actually written to the database.
Structured elaboration
| Pattern | How it works | Ideal use case | Downside to tell stakeholders |
|---|---|---|---|
| Cache-aside | App reads the cache first; on a miss it reads the database itself and populates the cache; the app also decides when to invalidate on writes. | Read-heavy data that's expensive to compute or fetch and doesn't change every second, e.g. a product catalog page. | The FIRST request for anything always pays full database latency, and if the app forgets to invalidate on a write, users can see stale data with no warning at all. |
| Write-through | Every write updates the cache and database in the same operation before returning success. | Data the cache must never lie about, e.g. an account balance shown right after the customer changes it. | Every write is now as slow as the SLOWER of the two stores, so users feel a database-latency tax on writes that a cache was supposed to help avoid. |
| Write-behind | The write updates the cache immediately and returns; the database write happens later, batched or on a delay. | Very high write volume where a little data-loss risk is acceptable, e.g. incrementing a page-view counter. | If the service or cache node fails before the delayed database write happens, that data is gone. This is a real, board-level risk to name explicitly, not a hypothetical edge case. |
Worked example
An e-commerce platform:
- Product description and price page: cache-aside. The first shopper to view a product pays a
real database read; every shopper after that, for the cache's freshness window, gets it from
the cache. - A payment ledger entry shown back to the customer on the confirmation screen: write-through,
because showing someone a confirmation for a charge that technically failed to persist is a
trust and compliance problem, not just a performance one. - A "times this listing was viewed" counter: write-behind, batched into the database every few
seconds; if a handful of increments are lost during a rare crash, nobody notices or cares.
Trade-offs and pitfalls
- Cache-aside's classic race: two requests miss at the same moment, both read the database, both
write the cache; harmless for pure reads, but if a WRITE happens between those two reads, one
of them can overwrite the cache with a now-stale value. Mitigated with a short expiry and, at
real scale, request coalescing or locking around the miss path. - Write-through doubles the worst-case write latency (cache write plus database write, in
series), which shows up directly in how fast a product feels; worth quantifying with real
numbers from your own systems for a stakeholder conversation, not an assumed figure. - Write-behind's failure mode is silent, nothing errors out, the data is simply gone, which is
exactly why it should never be used for financial or legally-required records regardless of
how attractive the write throughput looks in isolation.
That is every published Caching Strategies & In-Memory Optimization question for Engineering Manager so far. Browse the other topics in this category, or practice this one interactively.