Caching Strategies & In-Memory Optimization Questions

Architecture

A two-tier design: each worker in the fleet keeps a small, fast, per-process in-memory LRU cache
(least recently used, meaning that when the cache is full, the item that hasn't been accessed in the
longest time is the one evicted to make room) as its L1, and Redis serves as a shared L2 that every
worker in the fleet can read from and write to.

Consistency model: eventual, deliberately

Strong consistency across every worker's L1 (guaranteeing all workers agree on a value at every
instant) would require a synchronous invalidation broadcast on every write, reaching every worker
before the write is considered complete, which is real coordination overhead for a workload where the
cache exists purely to speed up repeated computation, not to serve as a system of record. Eventual
consistency, where a worker's L1 can be briefly stale relative to Redis or another worker's L1, is the
practical choice: a worker occasionally recomputing something that another worker already produced a
moment earlier is wasted work, not a correctness bug, as long as the shared L2 (Redis) is the one
place treated as authoritative for anything that must be correct.

Eviction coordination

L1 eviction needs no cross-worker coordination at all: each worker manages its own memory budget
independently, evicting its own least-recently-used entries with no awareness of what any other
worker's L1 currently holds. L2 (Redis) eviction is handled by Redis's own maxmemory-policy
(allkeys-lru or allkeys-lfu depending on whether recency or frequency better predicts what's worth
keeping for this workload), and doesn't need to know anything about individual workers' L1 contents
either, since it's a shared resource independent of any one worker.

Cache-aside versus write-through

Cache-aside (a worker checks L1, then L2, and on a miss computes or loads the value itself and
populates both tiers) fits a data-processing pipeline well when writes are comparatively rare relative
to reads, which is typical when most cached values are derived/computed results rather than frequently
mutated state. Write-through (write to L2 synchronously as part of producing a new result, rather than
waiting for a later reader to populate the cache) is the better fit when one worker's output is meant to
be immediately available to other workers, since it removes the gap where another worker could redo the
same computation before the producer's result ever reaches the shared cache.

Warm-up and rehydration

A restarted worker (deploy, crash-restart, autoscale-up event) starts with an empty L1, causing a burst
of L2 reads until it re-warms naturally from traffic, which is generally fine precisely because L2
already holds the data, that's the point of having a shared tier at all. Where the cold-start latency
dip matters, pre-populate the worker's L1 at startup with the top-N most-frequently-accessed keys:
track access counts in a Redis sorted set with ZINCRBY access:counts 1 <key> on every L2 read, then
at startup fetch the top N with ZREVRANGE access:counts 0 N-1 (or, on Redis 6.2+, the unified
ZRANGE access:counts 0 N-1 REV). Plain ZRANGE returns members in ascending score order, so it
would hand back the LEAST-accessed keys, the opposite of what a warm-up needs; I confirmed the
direction against a real Redis instance with a small sorted set (keyD at score 100, keyB at 50,
keyC at 10, keyA at 5, keyE at 1):

ZRANGE access:counts 0 2 WITHSCORES     -> keyE 1, keyA 5, keyC 10      (ascending: least-accessed first)
ZREVRANGE access:counts 0 2 WITHSCORES  -> keyD 100, keyB 50, keyC 10   (descending: most-accessed first)

ZREVRANGE (or ZRANGE ... REV) is the one that actually produces a useful warm-up set; using plain
ZRANGE would silently pre-populate L1 with the coldest keys in the whole keyspace and defeat the
point of pre-warming.

Minimizing cross-worker misses and network overhead

• Size L1 to capture each worker's own hot working set, measured by tracking each worker's own L1 hit
  rate rather than guessing a size.
• Where the workload allows it, route the same keys consistently to the same worker (partitioning a
  batch job by key range, or consistent hashing across workers) so a given worker's L1 accumulates a
  genuinely relevant hot set instead of a random slice of the whole keyspace, structurally raising L1
  hit rate rather than relying on chance.
• Batch L2 reads (a single MGET for many keys) instead of many sequential round trips when a worker
  needs several keys at once, since round-trip latency, not Redis's own per-command cost, is usually
  the dominant overhead at that scale.

Direct answer

Cache-aside (the app checks Redis first on a read; on a miss it reads the database directly and writes the result into Redis so the next read is a hit) with Redis in front of the aggregation query is the right starting point: the results are read-heavy, computed (not raw rows), and can tolerate a little staleness far better than something like a live price would.

Location

Put Redis in the same region and, ideally, the same availability zone as the API servers. At a 50ms p95 (95th-percentile response time) budget, network round-trip cost matters: an intra-AZ Redis GET typically costs under a couple of milliseconds, leaving most of the budget for the rest of the request; a cross-region call alone could burn a large fraction of that 50ms before any actual work happens. A CDN/edge cache is worth adding on top only if many different callers request the exact same query shape (for example a public leaderboard everyone sees identically); if each caller supplies personalized filter parameters, a shared edge cache won't help much, and Redis stays the primary layer.

Sizing

Estimate the working set before worrying about hardware. If there are roughly 5,000 distinct aggregation queries requested regularly and each cached result is about 4KB, that's 5,000 x 4KB = 20MB, trivial for a single Redis node's memory. Even padding generously for less-common queries, larger payloads, and Redis's own per-key overhead, this stays well under a gigabyte. Sizing memory is not the binding constraint for this workload, throughput and latency are.

Invalidation

For read-heavy aggregated analytics, TTL-based expiration (for example 30 to 60 seconds) is usually enough, and far simpler to implement correctly than event-driven invalidation. If the underlying data updates on a known cadence (say, an hourly ETL job), either set the TTL to roughly match that cadence, or explicitly clear the relevant keys when the job finishes, whichever is simpler to wire into the existing pipeline without adding a new failure mode.

Meeting >5k RPS at 50ms p95

With a healthy hit ratio, and aggregation queries repeating across many callers should comfortably clear 90%+, the large majority of the 5,000+ requests per second never reach the origin computation at all. Size the expensive aggregation path only for the miss traffic (well under 500 RPS at a 90% hit ratio), which is a far easier target to hit reliably than the full 5,000+ RPS.

Trade-offs

• TTL-based staleness (viewers can see data up to the TTL old) versus the complexity of building exact invalidation for marginal freshness gain on a workload that doesn't need it.
• A single Redis instance's simplicity versus needing to shard or add replicas once traffic or dataset size grows past one node's comfortable capacity.
• Stampede risk: many callers recomputing the same expensive aggregation at once, right as a hot key's TTL expires. Mitigate with a small jittered spread on TTLs (so hot keys don't all expire in the same instant) or a lightweight lock/single-flight around recomputation for the handful of genuinely hot keys.