Caching Strategies & In-Memory Optimization Questions

Approach

The risky window is between reading the database and writing the cache: a slow reader can read a value, get preempted, and only write it to the cache AFTER a concurrent writer has already updated the database and invalidated the cache, leaving a stale value sitting in the cache with nothing left to refresh it until TTL expiry. Add a monotonic version column so every write bumps a version, and only let a cache write land if it is not older than what is already true in the database (a compare-and-set check), instead of just clobbering the cache unconditionally.

-- version is the freshness token the cache write will check against
ALTER TABLE orders ADD COLUMN version INTEGER NOT NULL DEFAULT 1;

-- bump it in the SAME transaction as the data change
BEGIN;
UPDATE orders
   SET status = 'shipped', version = version + 1
 WHERE id = 42;
COMMIT;

def cache_cas_write(row_id, version, data, cache, db_read):
    current_version, _ = db_read(row_id)
    if version < current_version:
        return False              # reject: this write is older than what's already committed
    cache[row_id] = (version, data)
    return True

To confirm this actually closes the window rather than just sounding right, force the exact race with a small harness: a reader reads the row at version 1, gets preempted, a concurrent writer then commits version 2 and invalidates the cache, and only THEN does the reader's delayed write land.

def naive_cache_write(row_id, data, cache):
    # no version check at all: whichever write lands last wins, even if stale
    cache[row_id] = data
    return True

def run_race(use_cas):
    db_state = {42: (1, "status=processing")}
    def db_read(row_id):
        return db_state[row_id]
    cache = {}

    # reader reads at version 1, then is preempted before writing to cache
    reader_version, reader_data = db_read(42)

    # concurrent writer commits version 2 and invalidates the cache
    db_state[42] = (2, "status=shipped")
    cache.pop(42, None)

    # reader's delayed write finally lands, using its now-stale snapshot
    if use_cas:
        accepted = cache_cas_write(42, reader_version, reader_data, cache, db_read)
    else:
        accepted = naive_cache_write(42, reader_data, cache)
    return accepted, dict(cache), db_state[42]

Running run_race(use_cas=False): the delayed write is accepted, and the cache ends up holding {42: 'status=processing'} while the database is already at (2, 'status=shipped'), a genuinely stale cache entry that will sit there until TTL expiry. Running run_race(use_cas=True): the delayed write is rejected (accepted is False), the cache stays empty, and the next real reader repopulates it from the current, correct row. Both runs were executed, not just reasoned about, so the fix is confirmed to close the specific window described above, not just plausible.

Key points

• The bug is not in the SQL, it is in the ordering between the DB read and the cache write, which invalidate-on-write alone does not fix.
• A version number, not a timestamp, avoids any dependence on clock synchronization across processes.
• The check happens at the moment of the cache write, not the moment of the DB read, that is what closes the window.

Complexity

O(1) per read, write, and version check, no additional round trips beyond what cache-aside already does.

Edge cases

Multiple concurrent writers need the version bump to be atomic, a single UPDATE ... SET version = version + 1 inside one transaction is safe under standard SQL isolation; a read-modify-write done as two separate statements is not. A second reader that misses right after invalidation is not a problem, it will read the now-current row and write a correct value. Do not use a wall-clock timestamp in place of the version, clock skew across app servers can make a genuinely newer write look older.

Two layers of measurement

Treat "is the cache working" and "does it matter to the business" as two separate questions that you connect deliberately, not one number that does both jobs.

Instrumentation

• Tag every request span with cache.hit (true/false) and cache.key_prefix, propagated through the same trace id from edge to database fallback. That lets you filter p50/p95/p99 latency (the 50th/95th/99th-percentile response time: the value below which that percentage of requests complete) by hit versus miss instead of averaging them together.
• Emit a hit/miss counter event per cache lookup (cheap, every request) separate from full distributed traces (expensive, sampled). Metrics answer "how often," traces answer "why this one request was slow."
• Collect data two ways: real user monitoring (RUM), timing beacons sent from actual users' browsers or devices, which reflects real network and device conditions but is noisy and sparse on low-traffic pages; and synthetic monitoring, scripted probes hitting the endpoint from fixed locations on a fixed schedule, which is consistent and catches regressions before a single real user does, but doesn't reflect true variance. Use synthetic as an early-warning tripwire and RUM as the source of truth for what users actually experienced.

Dashboards

Build panels for: latency percentiles split by cache hit/miss, hit ratio over time, and business metrics (conversion rate, add-to-cart rate) over the same time axis, so a reviewer can visually line up a latency change with a business change instead of trusting a claimed correlation.

Tying it to an SLO

Define a formal service level objective (SLO), a target like "95% of homepage requests complete under 200ms," with an error budget, the allowed amount of time you can miss that target before it's treated as an incident. A latency improvement from caching then has a concrete, stakeholder-legible target it's moving toward, instead of a vague "it feels faster."

Experiment, not just before/after

A pure before/after comparison is confounded by anything else that changed that week (traffic mix, a marketing push). Run a randomized A/B test instead: route a control group through the uncached path and a treatment group through the cached path at the same time, then compare conversion rate between groups. For example, suppose control sees a 500ms p95 and converts at 3.10%, while treatment sees a 150ms p95 and converts at 3.35%: that 0.25 percentage point difference, about an 8% relative lift, is attributable to the latency change because both groups saw the same traffic conditions at the same time. Multiply that lift by average order value and traffic volume to translate it into a revenue estimate for stakeholders. Treat any single experiment's exact numbers as illustrative until you've run it for real and checked statistical significance, not as something to promise in advance.

Framing

The core risk in a planned traffic spike isn't steady-state load, it's the transition: a cold cache meeting a sudden concurrency spike on a small set of assets is exactly the shape of a cache stampede (many concurrent requests all miss the cache at once and all hit the origin simultaneously, overloading it), amplified because it's predictable and can be prepared for in advance.

Plan

• Prefetch/pre-warm: before the campaign goes live, proactively populate the CDN (content delivery network) edge caches and any app-level cache for the known campaign assets, hero images, the landing page, the specific API responses the campaign page calls, by issuing synthetic requests through each edge region rather than waiting for real user traffic to trigger the first, slow, stampede-prone fetch.
• Origin shielding: configure a single CDN region, or a small shield layer, to be the only thing allowed to fetch from origin on a miss, with every other edge region fetching from the shield rather than origin directly. This caps origin request concurrency to roughly the number of shield regions instead of the number of edge regions times concurrent users.
• Cache-busting readiness: if the campaign assets might need a last-minute content fix, use content-hashed filenames so a fix doesn't require purging anything, it's just a new URL referenced by an updated page.
• High concurrency at origin: put a per-key single-flight lock, or request coalescing at the CDN/shield layer, in front of the origin so an unexpected cache miss during the spike collapses concurrent identical requests into one origin fetch instead of thousands.
• Fallback behavior: define what happens if origin genuinely can't keep up, serve a stale-but-still-valid cached version (stale-while-revalidate) rather than an error page, and have a static, CDN-only fallback page ready, with no origin dependency at all, for the worst case.

Worked timeline

T minus 24h: pre-warm campaign assets across all edge regions and app-level cache, and verify hit rate is already high before real traffic arrives. T zero, launch: monitor origin request rate and cache hit ratio in real time; origin shielding should keep origin requests roughly flat even as edge traffic spikes 10 to 50 times. T plus 48h, campaign ends: let TTLs expire naturally rather than actively purging, since the traffic drop-off removes the urgency.

Risks to watch

A pre-warm that hits origin too aggressively, all regions warming simultaneously, can itself cause a mini-stampede, so stagger it. And a fallback page that's "static but references a live API" isn't actually origin-independent, so verify the fallback truly has zero origin dependency before trusting it as the worst-case plan.

Strategies

• Versioned filenames (a content hash baked into the filename, such as app.a1b2c3.js): the browser/CDN (content delivery network) can cache this file forever, since a content change always produces a new filename, there is nothing to invalidate, and the old URL simply stops being referenced.
• Query string versioning (app.js?v=42): simpler to wire up, no build-step filename rewriting, but some intermediate caches have historically treated the query string inconsistently for caching decisions, so it's a weaker guarantee than a true filename change.
• Asset manifest: the build produces a manifest mapping a logical name (app.js) to the current hashed filename; the HTML, which is not long-cached, references the manifest's current filenames, so a deploy only needs the manifest and HTML to update, not every cached asset.

Concrete header example

For an aggressively cached, filename-versioned static asset:

Cache-Control: public, max-age=31536000, immutable
ETag: "a1b2c3d4"

And for the HTML document that references it, which must NOT be long-cached since it's what points at the current hashed filenames:

Cache-Control: no-cache
ETag: "html-etag-value"
Vary: Accept-Encoding

no-cache here means "always revalidate with the server before using a cached copy," using If-None-Match against the ETag to get a fast 304 response if unchanged, not "never cache." Vary: Accept-Encoding tells any shared/CDN cache to keep separate cached copies per encoding, gzip vs brotli vs none, rather than serving one client's compressed response to a client that didn't ask for compression.

Pros/cons for CDNs and browser caching

• Versioned filenames: pro, zero-coordination invalidation, nothing to purge, old files just age out; con, every deploy technically leaves old hashed files on the CDN/origin until a cleanup job removes them.
• Query string: pro, low tooling complexity; con, weaker cache-correctness guarantee across all intermediate caches.
• Manifest with a short HTML TTL: pro, a single small point of coordination; con, the HTML itself becomes the thing that must never be served stale.

Coordinating deploy, purge, and rollback safely

Deploy the new hashed assets to the CDN/origin first, since they're new files nothing references yet, no risk, then flip the manifest/HTML last, so there's never a window where the HTML references an asset that doesn't exist yet. For rollback, keep the previous deploy's hashed assets available for a grace period rather than deleting on every deploy, so rolling the HTML/manifest back still finds its assets. This means you rarely need to actively purge a CDN for static assets at all; purging is mainly for the short-TTL HTML/manifest itself, or for a client holding a stale HTML it hasn't revalidated yet.

Framing

A CDN (content delivery network) sits in front of your origin and enforces caching rules per content class. The design decision for each class is really three linked choices: what goes in the cache key, how long it's cached, and who's allowed to see a cached copy, shared/public versus private/per-user.

Rules by content class

Origin shielding

Configure one shield region/layer between all edges and origin so a multi-region miss on the same key doesn't turn into as many simultaneous origin requests as there are edge regions. This matters most for the public API class, where misses are more frequent than for long-lived static assets.

Purge strategy

Static assets rely on content-hashed filenames and rarely need active purging. Public API responses should purge by tag or surrogate key if the CDN supports it, invalidating "everything tagged product:sku-123" on a price change, rather than purging by exact URL, since the same data can be reachable through multiple URL/query combinations. Never rely on purging for authenticated content; it shouldn't be cached at the shared layer in the first place.

Cache-busting for deploys

Ship new static assets under new hashed filenames before flipping the HTML/manifest that references them. Public API responses aren't tied to a deploy the same way; their TTL and tag-based purging handle change independently of the deploy pipeline.