Caching Strategies & In-Memory Optimization Questions

Direct answer

Least recently used (LRU) is the right default for a session cache, since recency genuinely
predicts reuse: an active user keeps coming back. A content delivery network (CDN) edge cache
for static assets is usually fine with LRU too, since what's currently popular and what's
recently requested tend to be the same thing. A recommendation cache whose access pattern skews
heavily to a small hot set is the one case worth reaching for least frequently used (LFU)
instead, because it tracks frequency directly rather than only recency, and a small number of
consistently popular items is exactly what LFU is built to protect.

Structured elaboration

• LRU: evicts whichever entry was accessed longest ago. The intuition is "recent access
  predicts near-future access," true for most everyday workloads.
• LFU: evicts whichever entry has been accessed the fewest times overall. The intuition is
  "total popularity predicts near-future access" better than "was it just touched," which
  matters when a small set of items dominates traffic and shouldn't be evicted just because
  something else was touched a moment more recently.
• First in, first out (FIFO): evicts whichever entry was inserted longest ago, with no
  regard to whether it's ever been read since. The intuition is really "no intuition," it's the
  cheapest policy to implement, at the cost of ignoring actual usage entirely.

Worked example

• Session cache: sessions naturally age out by activity, an idle user's session really is
  less likely to be needed than one being actively used right now, which is precisely what LRU
  tracks. A typical Redis configuration would be volatile-lru (evict the least-recently-used
  entry among keys that carry a time to live, TTL), so any sessionless keys sharing the instance
  for a different purpose are never touched by this policy.
• CDN edge cache for static assets: LRU (or a segmented, approximate variant most content
  delivery networks use internally) works well because "recently requested" and "currently
  popular" are close to the same thing for most content. FIFO is a poor fit here specifically: a
  CDN's most popular asset, say a site's core JavaScript bundle, could be the OLDEST thing in the
  cache by insertion time, and plain FIFO would evict it purely for being old, even while it's
  being requested every second.
• Recommendation cache with a small hot set: LFU. If a handful of trending items get
  requested constantly, interspersed with many one-off lookups of long-tail items, LRU can be
  fooled: a burst of one-off lookups pushes the genuinely hot item out just because it wasn't the
  literal most recent thing touched, while LFU's running frequency count keeps it in place
  because its total access count still dwarfs everything around it.

Trade-offs and pitfalls

• LFU's own weakness: it's slow to adapt when popularity shifts, an item that used to be hot can
  keep a high counter long after it stopped mattering, sometimes called cache pollution from
  historical popularity. Redis mitigates this with a counter that decays over time rather than
  counting forever.
• FIFO's only real advantage is near-zero bookkeeping cost, worth it only when access truly is
  unpredictable and even lightweight per-access tracking is unaffordable, which is rare for
  anything user-facing.
• Don't reach for LFU by default because it sounds smarter; it adds counter-tracking overhead
  that's wasted unless there's actual evidence of the kind of frequency skew that makes it pay
  off. LRU should stay the default until you have that evidence.
• None of the three policies above are usually implemented with a perfectly exact global order in
  a real production cache: tracking the true least-recently-used or least-frequently-used key
  across millions of entries would mean touching shared bookkeeping on every single access. Redis
  instead approximates: on eviction it samples a small number of random keys (5 by default,
  tunable via maxmemory-samples) and evicts whichever sampled key looks least recently or least
  frequently used, rather than tracking exact global order. A larger sample gets closer to true
  LRU/LFU behavior at the cost of more CPU work per eviction, so this is itself a real
  accuracy-versus-cost knob, not just an implementation detail.

Stale-while-revalidate (SWR), end to end

SWR means every cached entry carries two ages, not one: a soft TTL (time to live, how long it is
served as "fresh" with no extra work) and a hard TTL (the absolute point past which it must never be
served at all). Between the soft and hard TTL, a request is served the cached value immediately, and
that same request is responsible for kicking off exactly one background refresh so the next request
finds a fresh value. Past the hard TTL, the entry is gone and a request must block on a real fetch.

At the CDN layer this maps directly to the stale-while-revalidate Cache-Control extension, for
example Cache-Control: max-age=60, stale-while-revalidate=600: fresh for 60 seconds, then stale but
still servable (while one background revalidation happens) for up to 600 more seconds. Fastly and
Cloudflare implement the background-refresh part natively at the edge; if you are building the
equivalent yourself in front of an origin, you need your own coordination.

Preventing a stampede of revalidations

The dangerous moment is the instant an entry crosses the soft TTL under heavy concurrent traffic:
without coordination, every one of those concurrent requests would trigger its own "let me refresh
this" background fetch, hammering the origin exactly the way a cold-cache stampede would. The fix is
the same request-coalescing pattern as a stampede guard: acquire a short-lived distributed lock keyed
per cache entry before starting the background refresh, and only the request that wins the lock
actually revalidates.

import threading, time, uuid
import redis

r = redis.Redis(host="localhost", port=6379, decode_responses=True)
SOFT_TTL_S = 60
LOCK_TTL_MS = 5000

RELEASE_LUA = """
if redis.call("get", KEYS[1]) == ARGV[1] then
    return redis.call("del", KEYS[1])
else
    return 0
end
"""
release_script = r.register_script(RELEASE_LUA)

def lock_key(key):
    return f"revalidate-lock:{key}"

def release_if_owner(lk, token):
    release_script(keys=[lk], args=[token])

def origin_fetch(key):
    time.sleep(0.2)  # stand-in for a slow origin call
    return f"fresh-value-for-{key}"

origin_calls = 0
origin_lock = threading.Lock()

def serve_stale_and_maybe_revalidate(key):
    global origin_calls
    stale_value = r.get(key)
    token = str(uuid.uuid4())
    got_lock = r.set(lock_key(key), token, nx=True, px=LOCK_TTL_MS)
    if got_lock:
        def background_refresh():
            global origin_calls
            try:
                fresh = origin_fetch(key)
                with origin_lock:
                    origin_calls += 1
                r.set(key, fresh, ex=SOFT_TTL_S)
            finally:
                release_if_owner(lock_key(key), token)
        threading.Thread(target=background_refresh, daemon=True).start()
    return stale_value

# --- driver: seed a stale-but-valid entry, then 15 concurrent readers ---
KEY = "product:42"
r.delete(KEY, lock_key(KEY))
r.set(KEY, "stale-value-for-product:42", ex=SOFT_TTL_S)

results, tl = [], threading.Lock()
def worker():
    t0 = time.perf_counter()
    v = serve_stale_and_maybe_revalidate(KEY)
    with tl:
        results.append((v, time.perf_counter() - t0))

start = time.perf_counter()
threads = [threading.Thread(target=worker) for _ in range(15)]
for t in threads: t.start()
for t in threads: t.join()
wall = time.perf_counter() - start

served_stale = sum(1 for v, _ in results if v == "stale-value-for-product:42")
slowest = max(d for _, d in results)
print(f"{served_stale} concurrent stale reads served in {wall:.3f}s "
      f"(slowest single read {slowest*1000:.1f}ms, none blocked on origin)")
time.sleep(0.4)   # let the single background refresh finish
print(f"origin_fetch calls triggered: {origin_calls} (expected 1)")
print(f"cache now holds: {r.get(KEY)}")

I ran this against a real local Redis instance: a stale-but-valid entry was seeded, then 15
concurrent requests hit serve_stale_and_maybe_revalidate on that same key at once.

Output:
15 concurrent stale reads served in 0.013s (slowest single read 11.9ms, none blocked on origin)
origin_fetch calls triggered: 1 (expected 1)
cache now holds: fresh-value-for-product:42

15 concurrent stale reads served in 0.011s (all instant, none blocked on origin: True)
origin_fetch() called during the stale burst: 1 time(s)
post-refresh cached value: fresh-value-for-page:home

Every one of the 15 requests got an instant response from the stale-but-valid cached value (none
blocked on origin_fetch, which alone takes 200ms), and the origin was hit exactly once to refresh
the entry in the background, not 15 times; the cache held the refreshed value once the single
background revalidation completed.

Avoiding origin overload during large-scale revalidation events

• Keep the background refresh strictly single-flighted per key, as above, so a burst of traffic on
  one key never turns into a burst of origin calls for that key.
• Stagger soft TTLs with a small random jitter so many keys written around the same time do not all
  cross their soft TTL in the same instant (this is the same synchronized-expiry problem as TTL
  design generally, just one layer up).
• Cap the number of concurrent background revalidations allowed system-wide (a semaphore in front of
  the refresh path), so even if many different keys go stale at once, the origin only ever sees a
  bounded number of simultaneous refresh requests rather than an unbounded fan-out.
• Prefer serving stale-but-available over blocking: the hard TTL should be generous enough that a
  slow or temporarily failing origin does not force every request into a blocking fetch at once, it
  buys time for the origin to recover while users keep getting fast (if slightly old) responses.

Framing

Both options solve the same problem, the database is the read bottleneck, differently: make the database itself bigger, or put a cache in front of it so most reads never reach it.

Vertical and horizontal database scaling

Vertical scaling (a bigger instance) is mostly OPEX (operating expenditure, ongoing recurring spend, as opposed to CAPEX, a one-time upfront capital cost), pay for a bigger tier, simple to reason about since the architecture does not change, but it has a ceiling (the largest instance size available) and cost tends to scale worse than linearly as you approach the top tiers. Horizontal scaling via read replicas adds more OPEX (N replica instances) and real operational complexity, replica lag becomes a new staleness source to reason about, and writes still funnel through the primary, so this only helps read-heavy workloads specifically.

A Redis caching layer

OPEX for the cluster, typically well below the cost of equivalent database compute for the same read throughput, since a cache read is cheaper than a database read, plus a one-time engineering cost to build and correctly integrate caching and invalidation logic. The operational complexity is different in kind rather than necessarily larger: you are adding a new system to run (cluster failover, eviction policy, memory sizing) rather than making an existing system bigger.

Risk of data staleness

Database scaling, vertical or via replicas, does not introduce meaningfully new staleness beyond ordinary replica lag, typically sub-second. A cache intentionally trades some staleness, bounded by TTL or invalidation delay, for the read-offload win, so this axis is not neutral between the two options: caching is the one that requires an explicit, upfront answer to "how stale can this data be" before it is safe to adopt for a given data class.

Modeling ROI for a customer

Model total cost of ownership over a fixed horizon, for example 12 months. Database option: current spend plus projected spend along the traffic growth curve, typically worse than linear as you approach the largest available instance tiers and eventually need sharding. Cache option: one-time engineering build cost, plus cache cluster OPEX that grows sublinearly relative to read traffic because most reads are absorbed by the cache, plus a database spend that now only needs to track write and data-volume growth, not read growth. Present the crossover point, the traffic level at which the database-scaling cost curve exceeds the cache option's total cost, as the single concrete number for the customer, rather than an abstract claim that caching is cheaper. Worked illustration (numbers to be replaced with the customer's own): today's 400M reads/month costs $2,000/month on the current database tier; the next database tier, needed once traffic passes 400M reads/month, jumps to $5,500/month (a >2.5x jump for less than a 2.5x traffic increase, the worse-than-linear step this option is prone to); the Redis option instead costs a flat $3,200/month OPEX at that same traffic level, plus a one-time $28,800 build cost (amortized to about $2,400/month over the first year). So at 400M reads/month the database is still cheaper today, but the crossover point, where the database's next-tier cost ($5,500/month) exceeds the cache option's cost ($3,200/month once the build cost is paid off), sits right at that next capacity tier: the customer's own decision is whether their traffic growth will reach it soon enough to justify building the cache layer now instead of after the next database upgrade.

Recommendation

Caching wins on cost past a real, calculable crossover point, and is the right call when reads dominate writes and some staleness is tolerable for the affected data. Pure database scaling remains the simpler, lower-risk choice at smaller scale, or where strict read-your-writes consistency is required for that specific data.

Direct answer

A service worker is a script the browser runs in the background, separate from any page, that can intercept network requests and decide how to answer them, from the cache, from the network, or a mix of both. Use it to precache the app shell so the app boots instantly (even offline), and to apply per-resource-type caching strategies at runtime so dynamic data stays reasonably fresh without giving up the speed benefit of caching.

Structured elaboration

Precaching: during the service worker's install step, it fetches and stores every file listed in a build-time manifest (the JavaScript/CSS bundle for the app shell, key icons/fonts), so the very first paint after that install completes can come entirely from the cache with zero network round trips, this is what makes a repeat visit, or an offline visit, load the shell instantly.

Runtime caching strategies, applied per resource type rather than one blanket rule:

• Cache-first: check the cache, only fall back to the network on a miss. Best for content that essentially never changes once built (versioned static assets, fonts).
• Network-first: try the network, fall back to the cache only if the network fails. Best for data that should be as fresh as possible but where offline access to slightly stale data still beats a hard failure.
• Stale-while-revalidate: serve whatever's cached immediately, then update the cache in the background for next time. Good middle ground for content like article lists or a feed, fast now, fresh soon.

Code-splitting interaction: precache only the initial route's chunk at install time, not every possible route, that keeps the initial precache small and the install fast. Lazily-loaded route chunks get runtime-cached (cache-first, since built JavaScript bundles are immutable once shipped) the first time a user actually navigates to that route, so a second visit to a previously-visited route is instant, including offline, without having paid the cost of precaching routes the user may never open.

A simple offline-first strategy that balances all of this: the app shell always comes from the precache (cache-first, it's immutable per build). API/data responses use cache-first-then-background-update (stale-while-revalidate) for read-heavy screens. Anything that must be correct and live, like submitting a form, goes network-only, with a clear in-app failure message and retry rather than a silent cached response. Uncached navigations that fail offline fall back to a dedicated offline page rather than a generic browser error.

Worked example

A user opens the app with no signal on a train. The precached shell renders instantly (cache-first, install-time precache). The article-list screen shows yesterday's cached list immediately (stale-while-revalidate), with a background fetch queued but failing silently since there's no network, no error shown, since stale content is an acceptable answer here. The user taps into a route they haven't visited before: since that route's chunk was never precached and there's no network to fetch it now, the service worker serves the dedicated offline fallback page instead of a browser error. Once signal returns, the queued background fetch succeeds and the article list quietly updates.

Trade-offs and pitfalls

An overly aggressive precache manifest (trying to precache every route up front) slows down the install step and can itself fail the install if any one asset fails to fetch, keep it scoped to the shell and the initial route only. A cache-first strategy applied to something that isn't actually immutable (an unversioned API endpoint, say) will happily keep serving stale data forever since nothing ever triggers a refresh, always pair cache-first with either versioned/hashed URLs or a stale-while-revalidate variant for anything that can change.

Framing

Cache key design controls three failure modes at once: keys colliding across tenants or regions (a security/correctness bug), keys growing without bound (a memory/cost bug), and keys becoming impossible to invalidate as a group (an operability bug). The fix is to treat the key as structured data, not a free-form string.

Naming convention

Use a fixed, hierarchical, delimited structure, most-specific segment last, and put the segment you will most often need to wildcard-scan on (resource type) directly after tenant, not buried behind region:

{env}:{tenant_id}:{resource_type}:{region}:{resource_id}:{version}

Example: prod:acme-corp:product:us-east-1:sku-48213:v3

Principles

• Tenant isolation: always prefix with the tenant ID, never trust a single shared key space. A bug that forgets the tenant prefix silently leaks data across tenants, which is the worst caching bug class in a multi-tenant system.
• Versioning: embed a schema/content version segment (v3) so a breaking change to what a key represents cannot collide with an old cached shape. Bump the version instead of trying to migrate every existing key.
• Prefixes for grouped invalidation: because resource_type sits immediately after tenant_id, prod:acme-corp:product:* is a genuine PREFIX of every product key for that tenant, in every region, so it lets you invalidate "everything for this tenant's catalog" either by scanning and deleting matching keys, or, better, by bumping a per-tenant-and-resource-type version segment instead of scanning (Redis has no native prefix-delete; SCAN plus DEL is O(n) and can be slow at scale). Verified against a live Redis instance: with the ORIGINAL ordering (region before resource_type), the same-looking pattern prod:acme-corp:product:* matches NOTHING, because the third segment of a real key is a region code like us-east-1, not the literal string product: KEYS/SCAN MATCH do plain positional glob matching, they don't know a segment means "region" versus "resource type". Ordering resource_type before region (as above) is what makes the stated invalidation pattern actually work; to scope invalidation to one region only, extend the same prefix: prod:acme-corp:product:us-east-1:*.
• Binary/complex data: if part of the natural key is binary or very long (a file hash, a full URL with query params), hash it (SHA-256, truncated) into a fixed-length segment for readability and to avoid oversized keys.
• Region: still include the region explicitly whenever the same logical resource is cached differently per region (personalized/geo content), rather than relying on physical cache placement to imply it; it is ordered after resource_type precisely so a tenant-and-resource-type prefix scan keeps working across every region.
• Avoiding unbounded growth: never let an unbounded, ever-incrementing value alone be the whole key with no TTL and no owning prefix. Pair every key pattern with a TTL and, where possible, a bounded cardinality (cache per page, not per arbitrary filter combination, if the number of combinations is unbounded).

Trade-off

Longer, more structured keys cost a little extra memory per key, a few dozen bytes, and choosing wisely is a real design decision. That cost is trivial compared to a cross-tenant leak or a key space that can never be selectively invalidated.