Caching Strategies & In-Memory Optimization Questions

Framing

Local per-service caches (each service keeping its own in-process or single-node cache) can't share warm data across services, and there's no consistent way to reason about capacity, ownership, or blast radius. Moving to a shared cluster fixes reuse but introduces a new risk: one team's bug or spike can now degrade every other team sharing the cluster. The plan below is built around containing that risk.

Tenancy model

Two realistic choices: logical tenancy (every team shares one Redis cluster, isolated only by key namespace and per-tenant quotas) or physical tenancy (each team, or risk tier, gets its own cluster). Logical tenancy is cheaper and easier to reuse hot keys across teams, but one misbehaving client can starve everyone else. Physical tenancy costs more to operate but bounds blast radius. A workable middle ground: logical tenancy for low-risk, low-QPS teams, and a dedicated cluster for the handful of teams whose traffic is large enough to threaten a shared one.

Quotas

Every team gets a hard memory quota (bytes) and a request-rate quota (ops/sec), enforced at the client-connection or proxy layer, not just monitored after the fact. Enforcement matters because a shared maxmemory-policy will evict everyone's keys under LRU (least-recently-used eviction, which drops whichever key has gone longest unused) once ONE tenant fills the cluster; a proxy that rejects or throttles the offending tenant protects everyone else. Start quotas conservative, sized from each team's current local-cache footprint plus headroom, and revisit after a month of real traffic.

Key naming conventions

Enforce a fixed prefix scheme, <team>:<service>:<entity>:<id>, applied by a shared client library rather than left to each team, so quotas can be measured per prefix, an incident can be triaged to "which team's keys are growing," and a team's keys can be bulk-purged without touching anyone else's. Reserve a version segment (<team>:<service>:v2:<entity>:<id>) so a breaking schema change doesn't require a coordinated flush.

SLAs

Publish a cluster-wide latency and availability SLA (service level agreement, a formal commitment on latency/availability), but also publish PER-TENANT budgets (max ops/sec, max memory): once a tenant exceeds their agreed budget, the SLA no longer covers their own requests, they get throttled instead of degrading the whole cluster. That gives every team a concrete number to design against.

Monitoring

Track four things per tenant, not just cluster-wide: hit rate, ops/sec, memory used, and evictions/sec. Cluster-wide, track replication lag, memory used versus max, and command latency percentiles. Alert on any tenant seeing evictions while staying under quota (a symptom that a DIFFERENT tenant blew past quota and is evicting everyone), and alert at 80% of a tenant's own quota so on-call has lead time before a hard throttle.

On-call responsibilities

The platform team owns the cluster itself: failover, replication health, capacity, upgrades. Each consuming team owns their own cache-miss behavior, what their service does when Redis is slow or unavailable. That split has to be explicit in the runbook, because "the cache is down" is really the platform team's incident plus, potentially, N teams' incidents if their fallback path was never tested.

Migration checklist

1. Ship a shared client library with naming convention, TTL defaults, and quota-aware error handling before any team migrates.
2. Migrate the lowest-traffic, lowest-risk team first as a pilot; validate quotas and monitoring against real traffic.
3. Shadow-read against both the old local cache and the shared cluster for one team, comparing hit rate and error rate before cutting over.
4. Cut over reads behind a per-team feature flag, with an instant rollback to the old local cache if error rate or latency regresses.
5. Decommission a team's local cache only after their shared-cluster metrics have been stable for an agreed burn-in window.
6. Never migrate two large teams in the same week, so a regression has an unambiguous owner.

Trade-off

This trades migration speed for safety: one team at a time with a rollback flag is slower than a big-bang cutover, but a shared cluster's real failure mode (one team's regression hurting everyone) is exactly the risk leadership is worried about, so the checklist surfaces that risk on the smallest blast radius first.

Direct answer

Treat this as two separate questions that happen to share a symptom: why is memory pressure
high, which INFO and CONFIG answer directly, and why are specific requests slow, which needs
looking at individual key SIZE and access pattern, not just aggregate memory. Frequent evictions
under an allkeys-lru (LRU, least recently used) or allkeys-lfu (LFU, least frequently
used) policy, evicting whichever entry qualifies, are often working exactly as designed once memory is
genuinely full. The real question is whether the memory ceiling and eviction policy fit this
workload, or whether the cache is oversized for its allotted memory, undersized for its dataset,
or fragmented in a way that hides real free memory from Redis.

Structured elaboration

1. Confirm memory pressure is real, and check the policy. INFO memory for used_memory,
   used_memory_rss (actual operating-system-level resident memory), maxmemory, and
   mem_fragmentation_ratio (the ratio of RSS to used_memory). CONFIG GET maxmemory-policy to
   confirm eviction is even configured to do something sensible, noeviction would explain write
   ERRORS, not silent evictions.
2. Check eviction volume and hit ratio together. INFO stats for evicted_keys (a rising,
   non-zero counter under sustained load is the direct signature of memory pressure) alongside
   keyspace_hits and keyspace_misses. A rising eviction count paired with a FALLING hit ratio
   means the working set genuinely no longer fits; a rising eviction count with a STABLE hit
   ratio can just mean the cache has settled into its normal steady state.
3. Find hot keys and oversized keys separately, they're different problems.
   redis-cli --hotkeys (needs an LFU maxmemory-policy for its counters to mean anything)
   surfaces keys with disproportionate access counts; redis-cli --bigkeys scans the keyspace
   for the largest value per type. A hot key is a load-concentration risk, one shard doing too
   much work. A big key is a memory-bloat AND a latency risk, since a large value read or written
   on Redis's effectively single-threaded command path can block everything else on that shard
   for the duration of that one command. That's the exact mechanism behind large keys correlating
   with tail-latency spikes: a handful of oversized keys, if they cluster around a particular time
   window such as a nightly batch job writing large aggregates, can each briefly stall the shard,
   and every OTHER request queued behind them inherits that latency, showing up as a 95th- or
   99th-percentile latency problem that has nothing to do with the eviction policy at all.
4. Check for allocator fragmentation as a distinct root cause from "too much data."
   mem_fragmentation_ratio meaningfully above 1 on a large, long-running instance means the
   memory Redis is actually holding at the operating-system level (used_memory_rss) is bigger
   than the memory it thinks it's using (used_memory). This happens when a write-heavy, churny
   workload, many differently-sized keys created and deleted over time, leaves the memory
   allocator holding fragmented, hard-to-reuse free blocks instead of returning them to the
   operating system. This is a genuinely different root cause from "the working set is too big":
   the fix for a too-big working set is more memory or shorter TTLs; the fix for fragmentation is
   usually an allocator choice (production Redis typically ships with jemalloc specifically
   because it returns memory to the operating system better than a plain libc allocator under
   this pattern) or a controlled restart to reclaim the fragmented space.
5. Application-side mitigations, once the above narrows down the actual cause: shrink
   oversized values (store a reference plus a separate blob store instead of a huge inline
   value), add or tighten TTLs so cold data doesn't compete with hot data for space, split one
   large logical dataset across multiple shards so a single node's fragmentation or hot-key
   problem doesn't take down everything, and only then consider raising maxmemory if the
   working set is legitimately larger than what's provisioned.

Worked example

Executed evidence for two of the steps above, run against a fresh local redis-server 8.x
instance, with the driving script included so these numbers are reproducible rather than
asserted.

Eviction under a fixed 3 megabyte ceiling with allkeys-lru (redis-server --maxmemory 3mb --maxmemory-policy allkeys-lru), loading 30,000 keys of 200 bytes each, roughly 6 megabytes of
logical value data into a 3 megabyte budget:

import redis
r = redis.Redis(port=6399)
val = 'x' * 200
for i in range(30000):
    r.set(f'key:{i}', val)
print('DBSIZE:', r.dbsize())
print('used_memory:', r.info('memory')['used_memory'], 'maxmemory:', r.info('memory')['maxmemory'])
print('evicted_keys:', r.info('stats')['evicted_keys'])

Actual output:

DBSIZE: 7870
used_memory: 3145776 maxmemory: 3145728
evicted_keys: 22316

7,870 of the 30,000 writes survived; 22,316 evictions were recorded keeping used_memory pinned
right at the 3 megabyte ceiling, exactly the eviction-under-pressure signature from step 2. Note
that surviving keys (7,870) plus evicted_keys (22,316) comes to 30,186, a little over the 30,000
writes issued: evicted_keys counts eviction OPERATIONS, and because Redis evicts in a loop until
back under the memory ceiling on whichever write pushed it over, a single write can trigger more
than one eviction, so this is not a strict per-write subtraction identity. Exact surviving-key and
eviction counts vary with Redis version, client protocol and per-key hash-table overhead: rerunning
this scenario five times here (single SET calls, pipelined writes, and redis-cli --pipe) gave
7,075 to 7,961 surviving keys and sums of 30,093 to 30,186, so treat these as illustrative of the
eviction-under-pressure SHAPE (a large majority of writes evicted to hold used_memory at the
ceiling), not as fixed constants to expect on a different build.

Fragmentation from write and delete churn, isolated from eviction (no memory ceiling set here, so
nothing was evicted, this is purely allocator behavior): loading 200,000 keys of random size (50
to 500 bytes), then deleting every other one, a churn pattern typical of a cache with a lot of
turnover:

import redis, random
r = redis.Redis(port=6399)  # no maxmemory set, so no eviction pressure here
random.seed(42)
for i in range(200000):
    r.set(f'k:{i}', 'x' * random.randint(50, 500))
im1 = r.info('memory')
print('after load:', im1['used_memory'], im1['used_memory_rss'], im1['mem_fragmentation_ratio'])
for i in range(0, 200000, 2):
    r.delete(f'k:{i}')
im2 = r.info('memory')
print('after delete every other:', im2['used_memory'], im2['used_memory_rss'], im2['mem_fragmentation_ratio'], r.dbsize())

Actual output:

after load: used_memory=71225008 used_memory_rss=76808192 mem_fragmentation_ratio=1.08
after delete every other: used_memory=37288800 used_memory_rss=76906496 mem_fragmentation_ratio=2.06 dbsize=100000

used_memory (what Redis thinks it's using) roughly halved after the deletions (71.2MB to
37.3MB), as expected, but used_memory_rss (what the operating system actually sees Redis
holding) barely moved (76.8MB to 76.9MB), so mem_fragmentation_ratio roughly doubled (1.08 to
2.06). That gap between the two numbers, not the eviction counter, is the fragmentation signature,
and it would silently make Redis look like it's using twice the memory its live data actually
needs. This instance used the libc allocator (Redis's default build here); production Redis on
Linux typically ships with jemalloc, which is designed to return memory to the operating system
better under this kind of churn, so real fragmentation behavior should be measured against your
actual production allocator, not assumed from this number alone.

Trade-offs and pitfalls

• Don't treat a rising evicted_keys counter as automatically bad; on a cache intentionally
  sized smaller than its full addressable dataset, eviction under a good policy is the system
  working as intended. Pair it with the hit ratio before concluding anything is actually wrong.
• Chasing fragmentation with a full restart reclaims memory but drops everything currently
  cached, or requires warming from a persisted snapshot, a real availability trade-off to
  schedule deliberately, not a knee-jerk mid-incident fix.
• Don't fix a big-key latency problem by adding capacity; a single huge value blocking a single
  shard's command path is a design problem, that value shouldn't live in one key, that more
  memory doesn't solve.
• --hotkeys needs the LFU policy active with real traffic behind it to produce meaningful
  counters; running it right after a policy switch, or on a mostly idle instance, mostly just
  shows every key's baseline counter, not genuine hot keys.

Persistence trade-offs

The same RDB (point-in-time snapshot) versus AOF (append-only file, a log of every write command replayed on restart) trade-off applies here, but the stakes are higher. As a pure cache, losing Redis's contents just means a cold-cache period while it refills from the real source of truth. As the primary store, losing Redis's contents means the data is actually gone, there is no other copy to refill from.

Durability and RPO/RTO

• RPO (Recovery Point Objective: how much data you can afford to lose, measured in time) with AOF everysec: up to about one second of writes lost on a crash. As a primary store that's real, permanent data loss for those writes, not a cache-refill inconvenience.
• RTO (Recovery Time Objective: how long you can afford to be down while recovering): depends on dataset size and whether you're replaying the AOF log or loading an RDB snapshot; a large primary dataset can mean a slow restart at exactly the moment fast recovery matters most.

Memory constraints

Redis keeps essentially everything in RAM, even with persistence enabled, since RAM is the active working set. As primary storage that caps total data size at what fits in memory across the cluster, expensive at scale compared to a disk-based primary store, and a hard ceiling a pure cache doesn't have to worry about the same way (a cache can just evict under pressure; a primary store cannot, silently losing data on eviction would be a correctness bug, not a performance one).

Eviction behavior

A pure cache is deliberately configured to evict, for example allkeys-lru (least-recently-used: drop the entry that hasn't been read in the longest time) when memory fills up, that's the intended behavior. As primary storage, eviction must be disabled (noeviction, or maxmemory set high enough to never realistically hit), because evicting a leaderboard entry or a session that's the only copy of that data is data loss, not a cache miss.

Operational complexity

Backups, failover, and replica promotion all get more serious as primary storage: real backup verification (can you actually restore from that RDB file), a tested failover runbook, and monitoring that treats a Redis outage as a full incident rather than a "traffic falls back to the database, nobody's paged" event. Running Redis as a cache is operationally much lighter precisely because the worst case is "temporarily slower," not "data loss."

Bottom line: workable for use cases like session storage and leaderboards that are inherently ephemeral or reconstructable, provided persistence, eviction policy, and backup and failover are deliberately configured for durability rather than left at cache-friendly defaults. Treat it as a real database decision with real operational cost, not a shortcut around standing up a proper datastore.

Start from the honest arithmetic

100 million keys times roughly 1KB of JSON each is about 100,000,000 x 1,024 bytes = 102.4 GB of raw
payload (roughly 95 GiB using the 1024-based unit), before Redis's own per-key bookkeeping is counted
at all. That per-key overhead (the dictionary entry, the object header, pointers) is commonly cited as
tens of bytes per key for Redis; even a conservative ballpark of 50 to 70 bytes of overhead per key
adds another 5 to 7 GB on top at 100 million keys. I'm flagging that overhead figure as an estimate,
not a measured number for your exact Redis version and configuration, but it is large enough at this
key count that it belongs in the sizing conversation, not just the raw payload size.

Levers, roughly in order of effort

1. Shrink the payload itself. If the JSON is verbose (repeated field names, human-readable
   numbers), switch to a compact binary serialization like MessagePack or Protocol Buffers, which
   typically cuts size meaningfully versus JSON text for the same data, and/or apply compression
   (gzip/zstd) on the value before storing it. Trade-off: every read now pays a small CPU cost to
   decompress/decode, so this helps most when you're memory-bound, not CPU-bound.
2. Reduce structural (per-key) overhead by consolidating. If many of the 100 million keys are
   small pieces of a related entity (say, several fields of the same logical object stored as separate
   string keys), storing them as fields of a single Redis hash instead of separate top-level keys
   amortizes the per-key overhead across fewer actual keys, and Redis encodes small hashes very
   compactly (the listpack/ziplist-style encoding) as long as they stay under the configured field
   count and value size thresholds. Trade-off: you lose the ability to set a TTL on an individual
   field, only on the whole hash, so this only works cleanly if those fields genuinely expire
   together.
3. Tighten TTLs where staleness is cheap. Shrinking TTLs reduces the steady-state working set, at
   the cost of more cache misses and more backend load; this only helps net-positive if the backend
   can absorb the extra miss traffic.
4. Tune Redis's own memory behavior rather than just the data model:
   • maxmemory-policy (for example allkeys-lru or allkeys-lfu) (allkeys-lru evicts the least-recently-accessed key first; allkeys-lfu evicts the least-frequently-accessed key first) decides what gets evicted once
     you're at the memory ceiling; picking the wrong one (or leaving it at the noeviction default)
     turns memory pressure into write errors instead of graceful eviction.
   • mem_fragmentation_ratio from INFO memory tells you whether the problem is actually the data
     itself or fragmented allocator overhead; a high ratio (well above 1) with active defragmentation
     turned on can reclaim real memory without touching the data model at all.
   • lazyfree-lazy-eviction (and the related lazyfree-lazy-expire) moves the actual freeing of a
     big evicted key's memory to a background thread instead of blocking the main event loop, which
     matters more for latency than for total memory, but is cheap to turn on and worth checking.
5. Scale horizontally (Redis Cluster) as the fallback, once the per-key and per-payload levers are
   exhausted, rather than as the first move, since resharding and rebalancing carry their own
   operational cost.

How I'd weigh the options

Serialization/compression is usually the highest-leverage first move because it attacks the largest
share of the footprint (the payload, not the per-key overhead) and needs no data-model change, at the
cost of CPU on every read and write. Hash consolidation is the next move if many keys really are
fragments of the same entity, since it removes structural overhead essentially for free, but it is a
real data-model change and only fits data that genuinely shares a TTL. The maxmemory-policy and
fragmentation knobs are close to free to check and should happen regardless of which of the above you
choose, since a misconfigured eviction policy can make even a well-sized cache behave badly. Buying
more memory or sharding out is the right call only after these are exhausted, since it doesn't reduce
the underlying footprint, just defers dealing with it.

Approach: sliding-window log, not fixed window

A fixed window (count resets every T seconds on a clock boundary) allows up to 2x the intended rate right at the boundary, a burst just before the window resets plus a burst just after. A sliding-window LOG avoids that by tracking the actual timestamp of every recent request in a Redis sorted set (ZSET) and counting only the ones still inside the trailing T-second window, no matter where "now" falls relative to any clock boundary.

Data structure

One sorted set per user, ratelimit:{userId}, where each member is a unique per-request identifier and its score is the request's timestamp in milliseconds. To check and record a request: remove members older than now - windowMs (ZREMRANGEBYSCORE), count what's left (ZCARD), and if under the limit, add the new request (ZADD).

Atomicity: why this has to be a single Lua script

Redis commands are individually atomic, but a SEQUENCE of them (remove-old, count, then decide, then add) is not atomic across separate round trips: two concurrent requests for the same user could both read a count of 2 (under a limit of 3), both decide "allowed," and both add themselves, letting 4 requests through a limit of 3. Wrapping the whole remove-count-decide-add sequence in a Lua script and running it with EVAL makes Redis execute it as one atomic step; no other command for that key can interleave in the middle, because Redis is single-threaded for command execution. This is the concrete mechanism behind "atomic": not a lock the application manages, but the server executing the whole script as one unit of work.

Returning remaining quota and reset time

The script also computes the remaining quota (limit - count) and a reset time (the score of the OLDEST request still in the window, plus the window length, since that's the moment the window will next have room), so a caller can build a standard X-RateLimit-Remaining / X-RateLimit-Reset response without a second round trip.

Runnable code, executed against a real local Redis instance

const { createClient } = require('redis');

// Atomic sliding-window-log rate limiter.
// One Redis sorted set per user: member = unique request id, score = request
// timestamp (ms). The whole check-and-increment runs as a single Lua script, so
// Redis executes it as one atomic step and no other client can interleave a read
// and a write in between (the classic race in a naive GET-then-SET limiter).
const SCRIPT = `
local key        = KEYS[1]
local now        = tonumber(ARGV[1])
local windowMs    = tonumber(ARGV[2])
local limit       = tonumber(ARGV[3])
local member      = ARGV[4]

local windowStart = now - windowMs

-- 1. Drop entries older than the window (lazy cleanup, keeps the set small).
redis.call('ZREMRANGEBYSCORE', key, '-inf', windowStart)

-- 2. Count what is still inside the window.
local count = redis.call('ZCARD', key)

local allowed = 0
if count < limit then
  redis.call('ZADD', key, now, member)
  allowed = 1
  count = count + 1
end

-- 3. TTL so an idle user's key is reclaimed instead of living forever.
redis.call('PEXPIRE', key, windowMs)

-- 4. Reset time: when the OLDEST request currently in the window falls out of it.
local oldest = redis.call('ZRANGE', key, 0, 0, 'WITHSCORES')
local resetAt = now + windowMs
if oldest[2] ~= nil then
  resetAt = tonumber(oldest[2]) + windowMs
end

return { allowed, limit - count, resetAt }
`;

function makeRateLimiter(client, { limit, windowMs }) {
  let seq = 0;
  return async function checkLimit(userId) {
    const now = Date.now();
    seq += 1;
    const member = `${now}-${seq}-${Math.random().toString(36).slice(2, 8)}`;
    const [allowed, remaining, resetAt] = await client.eval(SCRIPT, {
      keys: [`ratelimit:${userId}`],
      arguments: [String(now), String(windowMs), String(limit), member],
    });
    return {
      allowed: allowed === 1,
      remaining: Math.max(0, remaining),
      resetAt, // epoch ms when the quota next has room
    };
  };
}

async function main() {
  const client = createClient({ socket: { port: 6399 } });
  await client.connect();
  await client.flushAll();

  const checkLimit = makeRateLimiter(client, { limit: 3, windowMs: 1000 });

  console.log('Firing 5 requests back-to-back for user "u1" (limit=3 per 1000ms):');
  for (let i = 1; i <= 5; i++) {
    const result = await checkLimit('u1');
    console.log(`  request ${i}: allowed=${result.allowed} remaining=${result.remaining} resetInMs=${result.resetAt - Date.now()}`);
  }

  console.log('\nFiring 20 concurrent requests for user "u2" (limit=3 per 1000ms), to prove atomicity:');
  const results = await Promise.all(Array.from({ length: 20 }, () => checkLimit('u2')));
  const allowedCount = results.filter((r) => r.allowed).length;
  console.log(`  allowed=${allowedCount} out of 20 concurrent requests (expected exactly 3)`);

  console.log('\nWaiting 1100ms for the window to roll forward, then trying user "u1" again:');
  await new Promise((r) => setTimeout(r, 1100));
  const after = await checkLimit('u1');
  console.log(`  request after window reset: allowed=${after.allowed} remaining=${after.remaining}`);

  await client.quit();
}

main().catch((err) => {
  console.error(err);
  process.exit(1);
});

Output (run against a real local Redis instance):

Firing 5 requests back-to-back for user "u1" (limit=3 per 1000ms):
  request 1: allowed=true remaining=2 resetInMs=999
  request 2: allowed=true remaining=1 resetInMs=997
  request 3: allowed=true remaining=0 resetInMs=995
  request 4: allowed=false remaining=0 resetInMs=993
  request 5: allowed=false remaining=0 resetInMs=991

Firing 20 concurrent requests for user "u2" (limit=3 per 1000ms), to prove atomicity:
  allowed=3 out of 20 concurrent requests (expected exactly 3)

Waiting 1100ms for the window to roll forward, then trying user "u1" again:
  request after window reset: allowed=true remaining=2

What the output demonstrates

The first block shows a single user firing 5 requests against a limit of 3: the first 3 are allowed with remaining counting down to 0, and requests 4 and 5 are correctly denied. The second block is the atomicity proof: 20 truly concurrent requests (Promise.all, not a loop with awaits) for the same user against the same limit of 3 result in exactly 3 allowed, not more, which is exactly the race a naive multi-command implementation would fail. The third block confirms the sliding window actually slides: waiting past the 1000ms window lets a new request through again, with a fresh remaining count.

Performance note

This is O(log N) per call for the ZSET operations, where N is the number of requests within the window for that user, which stays small (bounded by limit) for any sane rate limit. For very high-cardinality users (an API key shared by many callers) or very large windows, a token-bucket implementation using a single counter plus a timestamp (two fields instead of one member per request) trades a small amount of burst-boundary precision for O(1) memory per key; the sliding-window log here is the right choice specifically because the question calls for returning an accurate reset time, which a simple counter can't give you as precisely.