Caching Strategies & In-Memory Optimization Questions

Consistent hashing versus client-side sharding

These solve the same problem at different layers. Redis Cluster's built-in consistent hashing partitions the keyspace into 16,384 fixed hash slots (CRC16(key) mod 16384), tracks slot ownership inside the cluster itself, and hands clients MOVED/ASK redirects automatically; resharding is a first-class cluster operation. Client-side sharding instead has the APPLICATION compute which of several independent, non-clustered Redis instances owns a key (commonly a simple hash(key) % shard_count, or a custom hash ring implemented in application code). Client-side sharding gives total control and works with any Redis version or a mix of vendors, but the application now owns rebalancing logic itself; a naive modulo scheme remaps nearly the whole keyspace on every shard-count change unless the application ALSO implements a consistent-hash ring at that layer, in which case you've effectively rebuilt what Redis Cluster already gives you for free. For a 10x growth arc like this one, Redis Cluster's built-in mechanism is the better default; client-side sharding mainly makes sense when you need Redis instances that Cluster mode itself doesn't support (e.g. certain managed-service configurations).

Cluster slot math for this specific growth

16,384 total slots. At 10 nodes, average slots per node is 16384 / 10 ~ 1,638. At 100 nodes, it's 16384 / 100 ~ 164, about a tenth as many per node. That tenfold drop means this is a large rebalancing project, not a routine single-node addition, and it should be planned as several smaller rounds.

Resharding steps

1. Add new, empty nodes to the cluster (CLUSTER MEET); they start owning zero slots.
2. Decide the target slot distribution for the new total node count, aiming for roughly even ownership.
3. For each slot being moved: mark it MIGRATING on the source node and IMPORTING on the destination, move its keys one at a time with MIGRATE, then update slot ownership cluster-wide with CLUSTER SETSLOT <slot> NODE <destination> once every key for that slot has moved.
4. Automate this with a tool (redis-cli --cluster reshard or equivalent) rather than issuing these commands by hand across dozens of new nodes; a 10-to-100 migration is not something to script ad hoc under time pressure.
5. Add and rebalance in cohorts (for example, 10 nodes at a time, verify application error rates and latency stay flat, then continue) rather than attempting one 90-node jump.

Moving hot keys with minimal downtime and latency impact

A single hot key doesn't get helped by slot-level rebalancing alone; the whole slot it lives in can still be overloaded even after a clean, even slot distribution. Use hash tags ({user123}.profile) only when multiple RELATED keys genuinely need to land in the same slot for a multi-key operation; pinning a hot key deliberately concentrates load and should be avoided unless there's a specific multi-key reason for it. Before migrating a hot key's slot, consider absorbing its read traffic with a short-TTL, local in-process cache in front of Redis, so the brief unavailability of that slot during migration doesn't dump a traffic spike onto the newly-importing node all at once. MIGRATE itself is synchronous per key, so a very large or unusually hot key's value being moved can briefly block the source node's event loop; for outsized values, split them into smaller keys (for example, hash field partitioning) before undertaking a 100-node migration, so no single MIGRATE call stalls processing for other keys sharing that node.

Direct answer

Four layers, each doing a different job: an edge content delivery network (CDN) serves static
assets essentially forever; the CDN also caches public HTML and JSON with short, content-aware
freshness windows; regional Redis caches sit in front of the database for dynamic,
per-request data that isn't cacheable at the shared edge; and the database remains the single
source of truth everything else eventually falls back to. The design problem is not "add caching
everywhere," it's choosing the right time to live (TTL), invalidation approach, and cache-hit
requirement for each layer so that, at 100,000 requests per second (RPS) across 20 regions, the
vast majority of traffic never reaches the database at all, which is the only way to hit a
95th-percentile (P95) latency target of 200 milliseconds for public pages.

Architecture

flowchart LR
    Client[Client request] --> CDN[Edge CDN node, nearest region]
    CDN -- cache hit: static or HTML or JSON --> Client
    CDN -- cache miss --> App[Regional app or API layer]
    App --> Redis[Regional Redis cache]
    Redis -- hit --> App
    App -- populate --> CDN
    Redis -- miss --> Lock[Redis-backed stampede lock]
    Lock --> DB[(Source-of-truth database)]
    DB --> Redis
    App --> Client

A request first hits the nearest edge CDN node. A CDN hit returns immediately, no further hop.
A CDN miss goes to the regional application layer, which checks the regional Redis cache. A
Redis hit returns to the app, which also repopulates the CDN for the next request at that edge.
A Redis miss goes through a stampede-prevention lock (the same distributed-locking mechanism
used to stop a hot key from overwhelming an origin on expiry) before reading the database, so a
regional traffic spike on one key doesn't turn into thousands of simultaneous database reads.

Cache key namespaces and TTL strategy

Namespaced keys separate content types so each can carry its own TTL policy and be invalidated
independently, without an expensive scan across unrelated data:

• static:{content-hash}, immutable assets (images, bundled scripts), effectively infinite TTL,
  since a content change produces a new hash and therefore a new key entirely.
• page:{region}:{locale}:{path}, public HTML pages, a moderate TTL (minutes), since page
  content changes more often than a static asset but rarely on every request.
• api:{resource}:{id}:{version}, public JSON API responses, TTL tuned per resource.
• inv:{sku}:{region}, inventory counts in Redis, a short TTL (well under a minute), since
  showing "in stock" a little late is a UX annoyance, not a correctness failure, for a product
  PAGE (see the inventory/checkout distinction below).
• price:{sku}:{market}, price data, the shortest TTL of anything in this design, discussed
  below as its own case.

Per-content-type TTL choices follow directly from how expensive being wrong is for that content:
images can be wrong for a year for free (they're immutable by construction); inventory can be
wrong for tens of seconds at real but bounded cost; price can be wrong for only a few seconds
before it becomes a trust and, in some jurisdictions, a compliance problem.

A/B testing angle: any cached response that differs by experiment variant must fold the variant
identifier into the cache key, page:{region}:{locale}:{path}:{variant}, or it becomes a
correctness bug: caching upstream of variant assignment means one experiment arm's response can
leak to users who should see a different arm, a subtle bug that's easy to introduce by caching at
the edge before the app layer has even decided which variant applies.

Assume the business sets a 5-second freshness target for price specifically, since price is the field on this page where being wrong costs the most, in lost trust and, in some jurisdictions, as a compliance problem. The 5-second price-update SLA (service-level agreement, a commitment on how fast this must be). A price change must be reflected everywhere within 5 seconds. A TTL alone can't guarantee
that without either being so short (well under 5 seconds) that it hammers Redis and the database
at 100,000 RPS worth of demand, or being event-driven instead: on a price change, the write path
explicitly invalidates price:{sku}:{market} at Redis AND triggers a scoped CDN purge for any
page embedding that price, rather than waiting for TTL expiry. Because that explicit invalidation
still creates a miss that many concurrent readers could hit at once, the stampede lock from the
architecture above is wired in specifically for the price key, not the whole page, since price is
the one piece of data on this page where "cache aggressively" and "must be exactly right" collide
hardest.

Invalidation and origin protection

Invalidation strategy is layered by how urgently correctness matters: TTL-only for slow-changing
content (static assets, page shells), explicit event-driven purge for genuinely urgent changes
(the price example above), and version-suffixed namespaces for larger structural changes, bumping
a namespace like catalog:v7:... to catalog:v8:... lets consumers start requesting the new
version immediately while old-version entries simply age out of Redis and the CDN naturally,
avoiding an expensive mass-delete scan across millions of keys.

Origin protection has three parts: per-route rate limiting at the app or edge layer so a single
misbehaving client or bot can't monopolize database capacity; the Redis-backed stampede lock,
applied at every Redis-miss path, not just the price key, so a regional spike on any hot key
funnels through one database read instead of thousands; and a circuit breaker that serves the
last-known-good, expired cache entry (clearly distinguishable internally as stale) rather than
passing failures straight through to the database when the database itself is unhealthy, trading
a bounded amount of staleness for not making an active incident worse.

Personalization, experiments, and what never gets cached at the edge

Signed URLs, a URL containing a cryptographic token proving the request is authorized to fetch a
specific, often time-limited resource, let genuinely access-gated content still be served from
the shared edge cache: the CDN caches the PUBLIC bytes once, and the signature (verified at the
edge or the origin) is what gates who's allowed to request that URL at all, rather than the cache
needing a separate copy per user. The Vary header (tells a cache to keep a separate cached copy
per distinct value of some other request header, Vary: Accept-Language caches one version per
language) is the safe way to cache content that differs along a KNOWN, BOUNDED axis, language,
device type, or the A/B variant above, without serving the wrong variant to the wrong visitor.

The explicit rule this design enforces: authenticated endpoints, anything whose content depends
on WHICH logged-in user is asking, are never cached at the shared edge. Vary can enumerate a
handful of languages or experiment arms; it cannot safely enumerate "every possible user."
Authenticated responses carry Cache-Control: private, no-store (or bypass the CDN layer
entirely) and, if caching is still wanted for performance, use the regional Redis layer instead,
keyed explicitly by user, where per-user granularity is the expected shape of the data rather
than a leak risk.

Cache warm-up, degradation, and a lower-traffic calibration point

Warm-up and stale-inventory avoidance. After a deploy, a region restart, or a cold Redis
cluster, EVERY key in that region is simultaneously missing, which looks exactly like the
stampede scenario above but at the scale of an entire regional dataset, not one key. Mitigate
with a deliberate warm-up step, pre-populating the most-requested keys from recent access
patterns before routing live traffic to a newly started region or cluster, and a gradual,
weighted traffic ramp rather than an instant cutover. Stale inventory specifically must not be
served through the normal stale-while-revalidate style path at the moment of checkout: the
product PAGE's general "in stock" indicator can tolerate the cache's usual staleness window, but
the final inventory check at checkout should bypass the cache (or use a very short, coalesced
TTL) and hit the source of truth directly, since showing "in stock" from a slightly stale cache
on an item that just sold out is the one inventory-staleness failure that actually costs money
and trust.

Origin-fallback behavior. If Redis itself is unreachable in a region, the app layer falls
back to reading the database directly (through the same stampede-protection path, since a
regional Redis outage concentrates exactly the traffic the lock was built to spread out), rather
than failing the request outright; if the database is also degraded, the circuit breaker above
takes over and serves the last cached value with a staleness flag instead of hard-failing.

A lower-traffic calibration point. The same product-detail-page problem at 10,000 RPS instead
of 100,000, one tenth the load, spread across the same 20 regions, works out to 500 RPS per
region instead of 5,000. At that volume, this full design is usually over-built: a simpler tier
would collapse to 3 to 5 regions instead of 20, a single regional Redis (or a couple of paired
regions with simple replication) instead of a fully active-active topology per region, and the
stampede lock, while still worth keeping since it's cheap, matters far less in practice, since
500 concurrent requests on one expiring key is a much smaller pileup than 5,000. This is a useful
calibration point for when the full 20-region, lock-protected design is actually justified by
traffic, versus when it's premature infrastructure for the volume actually being served.

Worked example: sizing the 100k RPS / 20-region design

100,000 requests per second spread evenly across 20 regions is 100,000 / 20 = 5,000 RPS per
region on average, real regions won't be perfectly even, but this is the right order of magnitude
to design around.

Meeting a 200 millisecond P95 for public pages is fundamentally a HIT-RATE problem, not a raw
horsepower problem: a CDN cache hit is dominated by network distance to the nearest edge node, a
small fraction of the 200 millisecond budget. A CDN miss that hits a regional Redis cache adds a
sub-millisecond-to-low-single-digit-millisecond lookup, still comfortably inside budget. A full
miss down to the database, application render plus a real database round trip, is the expensive
path, and it's the one path that must stay a small minority of the 5,000 RPS a region sees for
the P95 target to hold, since a P95 target is violated by whatever the slowest 5% of requests do,
and database-miss requests are by far the slowest tier in this design. Concretely: at 5,000 RPS
per region, even a modest single-digit percentage of requests falling all the way through to the
database is hundreds of database reads per second per region without the stampede lock in place,
exactly why origin protection above is not an optional hardening step at this traffic level, it's
load-bearing for the SLA itself.

Trade-offs and pitfalls

• Namespace and version-based invalidation trades a small amount of "wasted" cache space,
  old-version keys aging out naturally instead of being deleted immediately, for avoiding an
  expensive mass-scan delete; a good trade at this request volume.
• The 5-second price SLA is the single place in this design where "cache aggressively" and "must
  be exactly right" collide hardest. Getting it wrong either serves stale prices, a trust and
  possibly legal problem, or hammers the database every few seconds across 100,000 RPS worth of
  demand, exactly what the targeted stampede lock on that one key exists to prevent.
• Vary on too many axes, or on a header with too many distinct values such as personalization
  by user identifier, turns a shared cache into a private one in all but name, exploding the
  number of cached variants and collapsing hit rate. That's precisely why per-user data must
  never rely on Vary, it belongs behind private, no-store and the regional Redis layer
  instead.
• Warm-up and gradual ramp-up add real deployment complexity and time; skipping them is fine for
  a low-traffic region but genuinely risky at 5,000 RPS into a cold cache, worth budgeting as a
  deliberate operational cost, not an afterthought.

Framing

A CDN (content delivery network, a network of edge servers positioned close to users) caches HTTP responses close to the person requesting them. An application-level in-memory cache, such as Redis or Memcached, sits next to your application servers, typically in one region near the database. They shorten different distances: a CDN shortens the hop to the USER, an app-level cache shortens the hop to the DATABASE.

Read-heavy static asset (a product image, a JS bundle)

Use the CDN. The content is identical for every user, changes rarely and predictably (on deploy), and the biggest cost is network latency to a geographically distant user, exactly what edge caching removes. TTL here can be long, hours to days, since invalidation ties to a deploy event rather than to arbitrary writes.

Dynamic personalized page (a logged-in user's dashboard)

Use the app-level cache, or don't cache the full page at all, only the expensive PARTS (a database query result, a computed recommendation list) behind the personalization. A CDN can't usefully cache a response that differs per user without either not caching it (the cache key includes the user, hit rate collapses) or accidentally serving one user's page to another, a real security bug class known as cache poisoning via an unkeyed header. TTL here is short, seconds, and invalidation is driven by the underlying write rather than a deploy.

Geo latency and locality

A CDN's value is placing a byte physically close to the request; an app-level cache's value is placing it logically close to the code that needs it, a low-microsecond hop in the same data center. A CDN miss still has to travel back to an origin, so CDN hit rate matters enormously for a global user base, while app-level cache locality mostly matters for keeping database load down.

CDN caching of dynamic API responses

It's possible to CDN-cache a dynamic API response if you're careful: bound the cache key to exactly the request dimensions that vary the response (method, path, the specific query params and headers that affect output), use a short TTL, and use origin shielding, where a single CDN region fetches from origin and fans out to other edge regions, so a spike in edge misses doesn't become a spike of origin requests. This sits between pure static CDN caching and pure app-level caching, and is worth it specifically when the same dynamic response is requested by many different users (a public leaderboard, a popular product's stock count), not when it's per-user.

Privacy and cost

Caching anything personalization-aware at a CDN edge raises a real privacy question: if the cache key isn't exactly right, one user's personalized response can leak to another. The safer default is to keep personalized responses out of CDN caching entirely and rely on the app-level cache, a controlled environment you operate directly, reserving CDN caching for genuinely shared content. Cost-wise, CDN caching also reduces egress (data leaving your cloud or network region out to the internet, which cloud providers typically bill per GB) from your own origin/database region, which is often billed, while app-level caching mainly reduces database compute and I/O; a system with both a slow database and expensive egress benefits from layering both rather than picking one.

Framing

Personalized feeds can't be edge-cached wholesale since every user's feed differs and changes continuously, so this design leans on app-layer caching plus precomputation rather than the CDN doing the heavy lifting. "Fanout" means how a new post from one user reaches all of their followers' feeds. A "hot key" is a single cache key receiving disproportionate traffic, for example a celebrity's feed fragment, hit far more than an average user's.

Cache placement, layer by layer

• Client: cache the user's last-fetched feed page locally so reopening the app or scrolling back doesn't refetch; invalidate on pull-to-refresh or a push signal that new content exists.
• Edge/CDN: limited use for the personalized feed response itself, but the static assets it references (images, thumbnails) cache extremely well at the edge, and a short, single-digit-second edge cache on the feed API response can absorb accidental client retry storms without going stale enough to matter.
• Regional Redis: the main workhorse. Each region keeps precomputed feed fragments per user, served from the region nearest the requester to hit the p99 target (p99 latency: the 99th-percentile response time, the slow tail almost all requests beat).
• App-layer, in-process: a small local cache per app server for the hottest keys, shaving a network round trip for the small number of keys dominating traffic; see hot-key handling below.

Precomputation vs on-read composition

Fanout-on-write: when a user posts, push that post id into the precomputed feed entry of every follower, so a read is one Redis lookup. Cheap reads, but expensive writes for accounts with millions of followers.

Fanout-on-read: store only each user's own post list; a read fetches recent posts from everyone the requester follows and merges them on the fly. Cheap for accounts you follow that post rarely, but a read now costs "fetch from N followees and merge," which gets expensive following thousands of accounts, exactly where read latency risks blowing the 150ms p99 budget.

Hybrid (what real feed systems do): fanout-on-write for the common case, fanout-on-read for a small set of very-high-follower accounts, where fanning out every post would be a write-amplification disaster. A read merges the precomputed feed with a small on-the-fly lookup bounded to a handful of high-follower accounts.

Storage of precomputed feed vs incremental updates

Store the precomputed feed as a bounded, ranked list of post ids (not full content) per user, updated incrementally: a new post from a followee is inserted or appended and periodically re-ranked, rather than recomputing from scratch on every write. Cap the list length and evict the oldest or lowest-ranked entries.

Hot-key handling

A celebrity account or a viral post can single-handedly saturate one shard even though the cluster overall has capacity. Mitigations: replicate the hot key across multiple shards or replicas and round-robin reads across the copies; add a short-lived local cache in front of Redis specifically for keys detected as hot; and for the fanout-on-read celebrity case, cache the celebrity's own post list aggressively, since many followers' reads hit it independently.

Graceful degradation during bursts

When the cache layer is saturated, degrade rather than fail: serve a slightly stale precomputed feed (accept extra minutes of staleness rather than falling through to full on-read composition for every user, which would multiply load on the database during exactly the burst already stressing it), and shed the expensive on-read merge path first since it's the costliest per request.

Scale, restated

At 100M monthly active users and a 200k QPS peak, this is the same design problem whether it's described as "200k QPS peak" or as "millions of queries per second with a cache keyed per user": the per-user precomputed feed cache above IS the per-user cache that framing is pointing at. The architecture doesn't change with the exact peak number; what changes is how many regional shards and replicas you provision to hit the latency budget at that volume.

Trade-off

The hybrid fanout model trades implementation complexity (two code paths) for keeping both write amplification and read latency bounded; a pure fanout-on-write or pure fanout-on-read design is simpler but breaks at one end of the follower-count distribution.

The two options

In-process (local, "L1") caches like Caffeine or Guava (popular Java in-memory caching libraries) live inside the application's own process. Remote/shared ("L2") caches like Redis live outside it, reached over the network and shared by every instance.

Comparison

Guidance

Use a local cache for small, very-hot, mostly-read data where a little staleness across instances is tolerable, feature flags, small config or lookup tables, things where all instances converging on the same value within a few seconds is fine. Use a remote cache when the data set is too large to duplicate across every instance, needs a consistent view across instances, or changes often enough that per-instance staleness would actually matter.

Two-level strategy

Put a small local L1 in front of a shared L2 (Redis). L1 absorbs the very highest-QPS keys at sub-microsecond latency and takes load off L2; L2 remains the shared source of truth across instances and survives individual instance restarts and deploys, which also softens a cold rolling-deployment start, since a freshly started instance only has an empty local L1 and still serves from the already-warm shared L2 instead of hitting the origin directly. The risk is that L1 can go stale relative to L2 after a write, until L1's own entry expires. Mitigate that by keeping L1 TTLs deliberately short (seconds, not minutes) or by broadcasting an invalidation (pub/sub) to every instance's L1 the moment L2 is invalidated, so writes propagate quickly instead of waiting out L1's TTL.