Caching Strategies & In-Memory Optimization Questions

Write-through

Every write goes to the cache AND the durable store synchronously, as one logical operation, before the write is acknowledged to the caller.

• Durable the instant the write succeeds: a crash right after cannot lose the comment.
• Cache and database are never out of sync.
• Slightly higher write latency, since the database write sits on the critical path, though a comment system's write volume is nowhere near enough to make this matter in practice.

Write-back (write-behind)

The write lands in the cache immediately, fast acknowledgment to the caller, and is flushed to the durable store asynchronously, later, usually batched.

• Lowest possible write latency.
• Data-loss window: if the cache node crashes before the async flush runs, the comment is gone, even though the author was told it succeeded.

Recommendation: write-through

The requirement is explicit: durability matters, and the comment must be visible immediately to the author and followers. Write-back's entire selling point, absorbing high write volume by delaying the durable write, doesn't apply: comment write volume for a typical feature is nowhere near what would justify trading away durability for latency.

Guaranteeing immediate visibility to readers

Durability and visibility are two different guarantees; write-through alone only buys the first.

1. Write-through the comment to cache and database.
2. Have the read path for "comments on this post" invalidate or append into its own cached list at write time (or skip caching that specific list at all, if reading it live is cheap enough given real comment volumes), rather than waiting out that list's TTL.
3. For the author's own view specifically, render optimistically from the write response itself rather than re-fetching from a cache that might not have caught up in the same millisecond.

Full write-through durability plus immediate list invalidation is strictly more work than either piece alone, but for a feature where "my comment silently didn't save" and "I can't see my own comment" are both trust-breaking bugs, that is the right trade to make.

The design in one sentence

Redis holds two things per counted item: a fast authoritative total for reads, and an unflushed delta since the last durable write. A background worker periodically claims that delta, applies it to the relational database (shown below as SQLite standing in for PostgreSQL, since the pattern is the same transactional semantics regardless of which relational store backs it), and only clears it from Redis after the database write is confirmed durable.

import itertools
import sqlite3
import redis

_batch_ids = itertools.count(1)  # deterministic ids for this demo; a real system would use a UUID
r = redis.Redis(host="localhost", port=6399, decode_responses=True)
r.flushall()

db = sqlite3.connect(":memory:")
db.execute("CREATE TABLE likes (post_id TEXT PRIMARY KEY, count INTEGER NOT NULL DEFAULT 0)")
db.execute("INSERT INTO likes (post_id, count) VALUES ('post-42', 0)")
db.execute("CREATE TABLE flush_log (batch_id TEXT PRIMARY KEY, post_id TEXT NOT NULL, "
           "delta INTEGER NOT NULL)")
db.commit()


def like(post_id: str):
    # Cache-layer write: O(1), atomic, never touches the database.
    pipe = r.pipeline()
    pipe.incr(f"count:{post_id}")       # authoritative value for reads
    pipe.incr(f"unflushed:{post_id}")   # delta not yet durable in the DB
    pipe.execute()


def read_count(post_id: str) -> int:
    return int(r.get(f"count:{post_id}") or 0)


def claim_batch(post_id: str):
    # Atomically rename unflushed:{id} -> inflight:{id}. Because Redis is
    # single-threaded, any INCR racing this RENAME either lands fully
    # before it (in the claimed batch) or fully after it (starts a fresh
    # unflushed:{id} at 1) -- no increment is ever silently dropped.
    key, inflight_key = f"unflushed:{post_id}", f"inflight:{post_id}"
    try:
        r.rename(key, inflight_key)
    except redis.exceptions.ResponseError:
        return None  # nothing pending
    batch_id = r.get(f"inflight_id:{post_id}") or f"batch-{next(_batch_ids)}"
    r.set(f"inflight_id:{post_id}", batch_id)
    return batch_id, int(r.get(inflight_key))


def apply_to_db(post_id, batch_id, delta, crash_before_ack=False):
    # Idempotent via UNIQUE(batch_id): a replayed flush cannot double-apply.
    already = db.execute("SELECT 1 FROM flush_log WHERE batch_id=?", (batch_id,)).fetchone()
    if not already:
        db.execute("UPDATE likes SET count=count+? WHERE post_id=?", (delta, post_id))
        db.execute("INSERT INTO flush_log (batch_id, post_id, delta) VALUES (?,?,?)",
                   (batch_id, post_id, delta))
        db.commit()
    if crash_before_ack:
        return "CRASHED"  # DB write is durable; Redis cleanup never ran
    r.delete(f"inflight:{post_id}")
    r.delete(f"inflight_id:{post_id}")
    return "OK"


def recover():
    # Any leftover inflight:{id} key means a flush was claimed but never
    # acknowledged. Replaying it with the SAME batch_id is safe: the
    # UNIQUE(batch_id) constraint makes a repeat apply a no-op.
    recovered = []
    for key in r.keys("inflight:*"):
        post_id = key.split(":", 1)[1]
        batch_id = r.get(f"inflight_id:{post_id}")
        delta = int(r.get(key))
        apply_to_db(post_id, batch_id, delta)
        recovered.append((post_id, batch_id, delta))
    return recovered


# --- run the scenario ---
post = "post-42"
for _ in range(7):
    like(post)
print(f"after 7 likes -> cache reads: {read_count(post)}")

batch_id, delta = claim_batch(post)
print(f"claimed batch {batch_id} delta={delta}")

for _ in range(3):            # 3 MORE likes arrive WHILE the flush is in flight
    like(post)
print(f"3 more likes arrive mid-flush -> cache reads: {read_count(post)}")

status = apply_to_db(post, batch_id, delta, crash_before_ack=True)
print(f"flush applied delta={delta} to db, then CRASHED before Redis ack (status={status})")

recovered = recover()
print(f"process restarts, recovery finds {len(recovered)} orphaned key(s): {recovered}")

final_db = db.execute("SELECT count FROM likes WHERE post_id=?", (post,)).fetchone()[0]
pending = int(r.get(f"unflushed:{post}") or 0)
print(f"after recovery: db count={final_db}, cache count={read_count(post)}, pending={pending}")

Output (against a real local Redis instance and a real SQLite connection):

after 7 likes -> cache reads: 7
claimed batch batch-1 delta=7
3 more likes arrive mid-flush -> cache reads: 10
flush applied delta=7 to db, then CRASHED before Redis ack (status=CRASHED)
process restarts, recovery finds 1 orphaned key(s): [('post-42', 'batch-1', 7)]
after recovery: db count=7, cache count=10, pending=3

Reading the result

The database ends at exactly 7, only the claimed batch, applied exactly once, even though recovery replayed it. The cache read stays correct throughout at 10 (7 plus the 3 that arrived mid-flush). The 3 mid-flush likes are neither lost nor double-counted, they sit in pending for the next flush cycle.

That is the bounded error this design guarantees, stated precisely. Between flush cycles, when no batch is currently claimed, the true total is db count + pending, which is exactly what this run shows after recovery completes: 7 plus 3 equals 10. While a flush is actually in progress, meaning right after claim_batch has renamed unflushed to inflight but before apply_to_db has committed, the claimed batch's own delta has to be counted too, since it has left pending but has not yet landed in the database: true total = db count + pending + inflight. In this run, at the moment just after the batch was claimed and before it was applied, that is 0 (nothing durable yet) plus 3 (the mid-flush likes accumulating in the fresh unflushed key) plus 7 (the claimed delta sitting in inflight), which is 10, the same total the cache shows. Either way the gap between what the cache shows and what the database has durably recorded is at most one flush interval's worth of increments, never permanently wrong and never double-applied, because the idempotency key makes a crash-and-replay safe.

Trade-off

This buys exactly-once durability at the cost of real complexity, an idempotency table, a rename-based claim step, and an explicit recovery scan on startup, versus a naive "increment Redis, periodically overwrite the database with the current total" approach that's simpler but loses any increments landing between the read and the write of that overwrite. For a metric where a small, bounded staleness window is acceptable but silent data loss or double-counting is not, likes on a post being the canonical example, this is the right amount of engineering for the guarantee it buys.

What caching is

Caching is storing a copy of data somewhere faster or closer to where it's needed, so a repeat request can be answered from that copy instead of redoing the original expensive work (a network round trip, a database query, a heavy computation). For a web frontend, "faster or closer" usually means physically closer to the user's browser and cheaper to reach than the original source.

The typical layers, from closest to the user to furthest

1. Browser: the browser's own HTTP cache stores responses (scripts, styles, images, sometimes API responses) directly on the user's device; a cache hit here costs zero network time.
2. CDN/edge: a content delivery network, a network of servers geographically distributed to sit close to users, caches responses so a user in another country doesn't have to reach your single origin server for something already cached nearby.
3. Origin server: the application server itself may cache in front of its own downstream calls (for example, caching a rendered page fragment).
4. Application: an in-process cache inside the running application (a plain in-memory map, valid only for that one process's lifetime).
5. Data store: a dedicated cache like Redis or Memcached sitting in front of the database, so repeated identical queries don't re-hit the database.

Three key benefits

• Lower latency for users: a cache hit is served from something fast and nearby instead of round-tripping to a database or a distant server, which is the difference users actually feel as "the app feels fast."
• Reduced load on the backend: every request served from cache is one fewer request the database or origin server has to handle, which matters directly for cost and for surviving traffic spikes.
• Better resilience: a CDN absorbs a sudden traffic spike that would otherwise overwhelm the origin, and in some designs a cache can keep serving a slightly stale-but-working response even if the origin is briefly unavailable, rather than the user seeing an outright error.

One common downside when caching is misconfigured

Serving stale or, worse, wrong data. The most damaging version in a frontend context is a personalization leak: caching a page that includes one user's private data (their name, their cart, their account details) at a SHARED layer like a CDN, so a completely different user requesting the same URL gets served User A's cached, personal data. This happens when a response varies per-user but the cache configuration doesn't know that (missing or wrong Cache-Control / Vary headers), and it is one of the more serious classes of bug caching can introduce, not just an inconvenience.

Framing

Dashboards need sub-second response over what is usually a huge underlying event volume. The standard answer is precompute, aggregate ahead of time rather than scanning raw events per request, trading some flexibility and freshness for speed.

Cache key schema for aggregates

Encode the dimensions and the time bucket directly in the key: agg:{metric}:{dimension_values}:{granularity}:{bucket_start}, for example agg:pageviews:country=US:hour:2026-09-08T14. Precompute multiple granularities in parallel, minute, hour, day, rather than only the finest grain, so a "last 30 days" query reads 30 daily keys instead of tens of thousands of minute keys. This is the single biggest lever for both memory use and query speed.

Freshness and retention windows

Finer buckets (minute) only need short retention, dashboards mostly want recent granular detail and roll up to coarser buckets for anything older. Coarser buckets (day) can retain much longer at low memory cost since there are far fewer of them. The current, still-open bucket is necessarily incomplete since events are still arriving for it, mark it explicitly (a partial: true flag alongside the value) so the dashboard can show "as of a few minutes ago" instead of implying a final, closed number.

Incremental updates for streaming data

Do not recompute a bucket from scratch on every new event, apply each event as a small delta to the existing aggregate, increment a counter, update a running sum or count for an average, merge into a sketch (a small, probabilistic data structure that trades a bit of exact accuracy for using far less memory) like HyperLogLog for approximate distinct counts. This keeps per-event update cost O(1) regardless of how much history that bucket already holds. A stream-processing job consuming the event stream writes these incremental updates directly into the aggregate store.

Handling ad-hoc queries

Precomputed keys only cover the dimensions and granularities planned for in advance. For a genuinely ad-hoc slice, a dimension combination nobody predicted, fall back to the raw event warehouse, slower but general, and treat that as an accepted trade-off rather than trying to precompute every possible combination, which explodes combinatorially. Optionally cache the RESULT of a popular ad-hoc query with a short TTL, so a repeated ad-hoc slice becomes cheap on its second request without needing to have been pre-aggregated at all.

Precomputation memory versus query flexibility

Precompute the small number of dimension and granularity combinations that cover the dashboard's known, common views, that is most of the real traffic. Keep a general but slower path for the long tail of ad-hoc exploration, rather than trying to make everything equally fast. That is the trade-off this design makes explicit rather than papering over.

Framing

Instead of a hand-tuned TTL (time to live) per key or per key-class, a model predicts a good TTL (or a caching decision more generally) from features about the key's observed behavior. This is worth the added complexity specifically when the keyspace is large and heterogeneous enough that no small number of hand-picked TTL tiers captures the real variation well, and it needs strong operational guardrails because a misbehaving model choosing bad TTLs at scale can cause real user-facing staleness or real origin-load spikes.

Feature extraction

Per key (or per key-class, for very large keyspaces where per-key tracking is too expensive), track:

• Recency and frequency of access: time since last read, rolling read rate, read-interval distribution.
• Object size: larger objects cost more to keep cached (memory) and often more to recompute, both relevant to the TTL trade-off.
• Compute cost to regenerate: latency and resource cost of a cache miss (a cheap key lookup versus an expensive join or an external API call), directly relevant because a higher miss cost should push toward a longer TTL, all else equal.
• Update frequency / volatility: how often the underlying data actually changes, which bounds how long a TTL can safely be before staleness risk becomes unacceptable.
• Business-defined staleness sensitivity: a label or tag (not purely learned) for how much this data class tolerates staleness, since some staleness costs (e.g. showing a stale price) are qualitatively different from others (e.g. a stale "last seen" timestamp) in a way raw access statistics alone can't capture.

Training pipeline

• Offline, to start: log the features above alongside the OUTCOME of different TTL choices (via a small held-out slice of keys running varied TTLs, essentially a controlled experiment, or via historical logs if TTL has varied naturally over time), and train a model (even something as simple as a regression or a small gradient-boosted tree predicting a good TTL from the features performs well here, this doesn't need to be a large or complex model since the feature set is small and structured) to predict a TTL that would have minimized a defined cost (origin load plus a staleness penalty, similar in spirit to the cost-function approach used for simpler TTL auto-tuning, just with a learned function instead of a hand-derived one).
• Online, once validated offline: periodically retrain (e.g. daily) as access patterns drift, and consider a lightweight online update (adjusting predictions based on very recent observed outcomes) for keys whose behavior changes faster than a daily retrain would catch.

Evaluation metrics

• Hit rate: does the model's TTL choice actually reduce cache misses versus the prior fixed-TTL baseline, for a comparable staleness profile.
• Origin load: total request volume and cost hitting the backend, the thing a good TTL choice is directly trying to reduce.
• Staleness distribution: not just "did hit rate go up" but "how old was the data, on average and at the tail, when clients actually read a cache hit," since a naive optimizer could trivially maximize hit rate by choosing enormous TTLs and silently accept unacceptable staleness; staleness has to be tracked and bounded as its own metric, not inferred from hit rate.

Safe rollout

• Shadow mode first: run the model's predictions alongside the current TTL policy without actually applying them, logging what it WOULD have chosen, and compare its predicted cost against the actual baseline cost before it ever controls a live TTL.
• Canary on a small, low-risk slice of keys (or key-classes) once shadow evaluation looks good, watching the real hit-rate/origin-load/staleness metrics on that slice specifically before expanding.
• Ramp gradually (percentage of keys, or percentage of traffic) rather than a single cutover, so a bad model version affects a bounded blast radius while it's being caught.

Fallback safeguards

• Hard bounds: clamp every model-predicted TTL to a [min, max] range appropriate to that key's data-sensitivity class, so the model can never push a compliance-sensitive key's TTL past a legally or contractually required ceiling, regardless of what it predicts.
• A circuit breaker: if the model's predictions start diverging wildly from recent history (a sudden shift suggesting a bad deploy, a feature-pipeline bug, or model drift), fall back automatically to the last-known-good static TTL policy for the affected keys rather than trusting an anomalous prediction.
• Treat the model as an optimizer layered on top of a safe default, never as the sole source of the TTL; a key that's never been scored by the model, or whose score looks implausible, should fall back to its class's default TTL, not to an unclamped model output.

Worked example

A content-delivery cache with 10 million distinct keys currently uses 3 hand-picked TTL tiers (1 minute, 10 minutes, 1 hour), and the "10 minute" tier holds 2 million of those keys. Shadow-mode evaluation on a held-out sample shows the hand-picked tier running at a 76.2% hit rate (23.8% miss rate) for a large sub-population of stable keys within it, while the trained model's predicted TTLs project an 83.7% hit rate (16.3% miss rate) at the same staleness profile, a (23.8 - 16.3) / 23.8 = 31.5% relative reduction in origin-hitting requests for that slice. Canaried on 2% of the tier's keys (40,000 keys), the observed hit rate over one week comes in at 83.4%, within 0.3 points of the shadow-mode projection, and the p99 (99th-percentile) staleness-at-hit metric, capturing the slow/stale tail rather than the average, holds at 9.6 minutes versus the hand-picked baseline's 9.9 minutes, both comfortably inside the tier's 10-minute ceiling, confirmed before wider rollout.

Trade-offs

The added complexity (feature pipeline, training/retraining, shadow and canary infrastructure, fallback logic) is real operational cost; it's justified specifically when a keyspace is heterogeneous enough that hand-picked tiers are provably leaving meaningful hit-rate or cost improvement on the table, which is worth demonstrating with the simpler cost-function auto-tuning approach first before reaching for a learned model at all.