Caching Strategies & In-Memory Optimization Questions

Situation / goal clarification:

• Product rolled out aggressive caching for content pages and we now see a 5% drop in ad impressions / revenue. My goals as SRE are to determine if caching caused the drop, find root cause, propose mitigations that balance performance and revenue, and validate fixes with measurable evidence.

Investigation plan (quick triage then deep dive):

1. Confirm signal and timeline

• Correlate timestamps: when cache change deployed vs drop in ad metrics (impressions, eCPM, clicks, fill-rate).
• Use dashboards (Grafana/Datadog) and BI (BigQuery/Redshift) to compare pre/post windows and segment by country, device, browser, user-agent.

2. Hypotheses to test

• Ads being served from cache (stale ad tags or ad targeting headers not forwarded) -> lower dynamic ad calls.
• Cache key/Vary misconfiguration causing same ad payload to be served to multiple users.
• Missing ad-request headers (cookies, X-User-ID, consent) filtered at CDN/edge.
• Edge TTL too long causing cached ad snippets instead of fresh calls.
• Client-side ad library behavior changes due to HTML changes (e.g., async load suppressed).

3. Concrete data collection and checks

• Sample full request/response traces (edge logs / origin logs / CDN headers). Check for Cache-Control, Age, X-Cache, Vary, Set-Cookie, and any ad-related headers.
• Compare origin logs to CDN edge logs to see if ad-request endpoints were hit less frequently.
• Instrument a few production pages with synthetic users to capture network waterfall (browser devtools HAR) to see whether ad calls are being requested, whether they're served from cache, or blocked.
• Run SQL queries: impressions_by_cache_status, ad_requests_by_edge_cache_hit, revenue_by_region/device, before/after.
• Check ad vendor logs: are ad calls failing, returning empty bids, or returning cached creative?

Determining causality:

• If we observe a drop in origin ad-request rates correlated with increased cache hits and responses containing cached ad HTML/creatives, caching is likely cause.
• If ad requests are same but fill-rate/eCPM dropped, cause may be ad partner or targeting mismatch.
• Use A/B (canary) test: route X% of traffic to previous caching behavior (or bypass) and compare ad metrics. If bypass group recovers impressions/revenue, that shows causality.

Mitigations balancing performance and revenue:

1. Short-term, low-risk

• Reduce TTL for ad-related fragments or pages (e.g., set Edge TTL=0 for ad zones).
• Honor Cache-Control/Vary: ensure responses that depend on user headers (cookies, consent, device) are Vary-ing those headers or excluded from caching.
• Implement cache bypass for pages with ad placeholders (via cookie or query param) or for known advertiser-sensitive paths.

2. Medium-term

• Fragmented caching: cache static parts (header/footer, article body) but exclude or server-side render ad slots on each request (Edge Side Includes / ESI).
• Use stale-while-revalidate / stale-if-error for non-ad HTML while forcing fresh ad calls.
• Add edge logic to preserve and forward ad-targeting headers and cookies to origin/ad endpoints.

3. Long-term

• Introduce deterministic edge rendering for ad wrappers that always call ad servers client-side, or prefetch ad tags without caching creatives.
• Add observability hooks in ad pipeline (tracing for ad calls).

Validation and rollout:

• Implement A/B or canary rollout using feature flags or CDN config to route small % to new behavior.
• Define success metrics: impressions, revenue per page view, ad request rate, page latency, cache hit ratio. Set thresholds and required statistical significance (p<0.05) over a minimum sample size.
• Monitor real-time dashboards and alerts for regressions. Run owner-approved surge tests.
• If metrics improve for canary, progressively ramp to 100% with rollback plan.
• Post-rollback/post-deploy: run postmortem documenting root cause, remediation, metrics, and update runbooks to prevent recurrence.

Communication:

• Notify product/ads teams early with findings and proposed temporary mitigations. Use clear impact estimates and planned timeline.
• After fix, share validation results and update SLOs and release checklist to include ad-impact testing for caching changes.

This approach isolates cause quickly, provides safe mitigations that tradeoff minimal performance, and validates fixes with A/B and metric-driven rollouts.

Below are the most important metrics/KPIs for caching services (Redis/Memcached/CDN), why they matter, how I’d visualize each on a dashboard, and example alert thresholds.

1. Cache hit rate (hits / (hits+misses))

• Why: Primary indicator of effectiveness — higher = fewer origin requests.
• Visual: Time-series line + 1h/24h rolling average and target band.
• Alert: warn if < 95% for 5m; critical if < 90% for 2m.

2. Cache miss rate and origin request rate

• Why: Shows load shifted to origin and potential cost/latency spikes.
• Visual: Stacked area (misses vs hits) and origin QPS chart.
• Alert: origin QPS > baseline*2 for 5m or miss rate > 10% sustained.

3. Latency (p50/p95/p99 for cache lookups; CDN edge latency)

• Why: User-perceived performance and backend pressure.
• Visual: Multi-percentile latency lines + SLA threshold band.
• Alert: p95 > 50ms for Redis / >200ms for CDN edge for 5m.

4. Error rate (connection errors, timeouts, 5xx)

• Why: Indicates service degradation or networking issues.
• Visual: Error count per minute + error ratio (%) heatmap.
• Alert: error rate > 1% for 5m or >0.1% critical for production traffic.

5. Evictions / OOM events

• Why: Cache too full or wrong eviction policy -> data loss and cache thrash.
• Visual: Counter of eviction events with correlated memory usage spikes.
• Alert: any OOM or >10 evictions/min for 5m.

6. Memory utilization and fragmentation

• Why: Capacity planning and performance (fragmentation hurts throughput).
• Visual: Gauge for % used, line for fragmentation ratio over time.
• Alert: memory used > 80% (warn), > 95% (critical); fragmentation ratio > 1.5.

7. CPU utilization and threads/connection count

• Why: High CPU/too many connections cause high latency/errors.
• Visual: CPU% and active connections time-series with thresholds.
• Alert: CPU > 70% sustained 10m or connections > capacity-10% for 5m.

8. Throughput (ops/sec / bandwidth)

• Why: Capacity and billing (CDN egress).
• Visual: Time-series ops/sec and bytes/sec, correlate with cache hit rate.
• Alert: sustained throughput > planned capacity or bandwidth > budgeted limit.

9. TTL distribution and stale content ratio

• Why: Detects misconfigured TTLs causing early expiration or stale serving.
• Visual: Histogram of TTLs and ratio of served-stale responses (CDN).
• Alert: >20% of keys TTL < expected baseline or stale ratio > 1%.

10. Top keys / key churn

• Why: Hot keys causing uneven load; churn indicates ineffective caching patterns.
• Visual: Top-N keys by requests and churn rate heatmap.
• Alert: single key > 50% of ops (hotspot) or churn rate > threshold.

Visualization tips: combine related charts (hit rate + origin QPS + latency) and show annotations for deploys/config changes. Use sensible rolling windows, and implement escalation (warn -> critical) with runbook links. Thresholds above are starting points; tune per workload and SLOs.

Requirements & constraints:

• Zero downtime, consistent reads during rollout, support fast rollback, avoid cache stampedes, work across CDN + multiple Redis clusters.

Rollout & invalidation plan (high level):

1. Use versioned cache keys and feature-flagged deploys
   • New code reads new-key namespace (v2) while old remains (v1). That enables fast rollback by switching traffic/flag back.
2. Canary traffic shift
   • Deploy app version to a small % of hosts; route small % of traffic (or internal users) to canary. Validate correctness and performance before broader rollouts.
3. Coordinated CDN + Redis strategy
   • CDN: issue targeted, rate-limited PURGE by path/pattern for canary traffic only. Prefer soft-control: reduce max-age + serve stale-while-revalidate to avoid sudden misses.
   • Redis: avoid global flush. For each Redis cluster, atomically switch namespace pointer (e.g., a namespace token in config) or begin writing to v2 keys while leaving v1 reads available.

Warm-up (cache priming) vs purge:

• Warm-up first for canary subset: background jobs (workers) generate and write critical heavy keys to v2 Redis and pre-warm CDN by making authenticated requests that populate edge caches (simulate real requests). Rate-limit warmers to avoid thundering on origin.
• After warm canary validation, progressively increase warm-up coverage and CDN prefetch for additional routes.
• Purge only after a warmed v2 exists or when rolling back: perform targeted CDN purges (by URL or surrogate keys) with rate limits and backoff. For Redis, instead of FLUSH, expire v1 keys gracefully (set short TTLs) or remove namespace token so reads start pulling from v2.

Avoiding cache stampedes:

• Implement request coalescing (singleflight) on origin; first miss triggers recompute while others wait briefly or get stale-while-revalidate content.
• Use jittered TTLs and probabilistic early refresh (if remaining TTL < threshold, background refresh).
• Locking keys for recompute with short lease and fallback to stale content if recompute fails.

Rollback support:

• Because reads can be toggled via namespace token or feature flag, rollback is immediate: switch namespace back to v1, stop writing v2, and purge any warmed CDN entries for v2 if necessary. Canary-first rollout makes issues visible early with small blast radius.

Operational details & monitoring:

• Automate with deployment pipeline: steps = deploy canary app -> warm caches -> run integration checks -> shift traffic via load balancer/feature flag -> gradually increase traffic -> final purge of old namespace TTLs.
• Throttle purges (e.g., 1000 keys/min) and monitor origin load, cache hit-rate, error rates, latency, and Redis CPU/memory with alerts.
• Add playbook for failures: pause rollout, increase stale-while-revalidate window, scale origin, or rollback namespace flag.

Trade-offs:

• Versioned keys increase memory usage (temporary duplication) but give safe rollbacks and predictable invalidation.
• Full global purge is fastest but risky; progressive warm-then-purge minimizes origin spikes.

This plan gives zero-downtime behavior by reading from stable keys, prevents stampedes via coalescing and staged warm-up, and supports fast rollback through namespace/version switches and feature flags.

Situation: We need to size Redis for 50k RPS of short reads, occasional writes, average cached value 2 KB, with 2x headroom and replication for durability. I'll show formulas and a concrete example so you can adapt to different assumptions.

Assumptions / constants

• Avg value = 2 KB
• Key + metadata overhead (Redis object, pointers) ≈ 100–200 B → use 0.1 KB
• Effective item size S = 2.1 KB
• Working set (N) — number of distinct cached items; you must supply or estimate. I’ll show calculation for N = 10M as an example.
• Headroom factor H = 2.0
• Replication factor R = 2 (master + 1 replica)
• Usable RAM per node U = 48 GB (example: 64 GB instance, keep ~25% free for OS/frag/eviction)

Memory sizing (per cluster)

• Dataset size D = N * S
• After headroom: D_h = D * H
• Accounting replication (total cluster RAM) D_total = D_h * R

Example (N = 10,000,000)

• D = 10,000,000 * 2.1 KB = 21,000,000 KB ≈ 21 GB
• D_h = 21 GB * 2 = 42 GB
• D_total = 42 GB * 2 = 84 GB total RAM across cluster

Shards / nodes

• Number of master shards M = ceil(D_h / U) = ceil(42 / 48) = 1 master shard (fits), but for operational HA and throughput we usually split, so choose M = 2 masters
• Total nodes = M * R = 2 * 2 = 4 nodes (2 masters, each with 1 replica)
• If using M = 1 master, you'd still need 2 nodes (master+replica) but throughput risk; prefer >1 master for fault isolation.

Throughput / network

• Read bandwidth = RPS_read * value_size ≈ 50,000 * 2 KB = 100,000 KB/s ≈ 100 MB/s (~800 Mbps)
• Add ~10% protocol overhead → ~110 MB/s (~880 Mbps) outbound from masters/replicas depending on client distribution
• Per-shard bandwidth ≈ (total read BW) / M. With M=2 → ~50 MB/s per master
• Replication bandwidth depends on write rate. If writes = 1k RPS: write BW = 1,000 * 2 KB = 2 MB/s; replication per master→replica similar
• Connections: estimate concurrent clients C (e.g., 10k), plus inter-node replication connections (M*(R-1)). Redis handles many connections but monitor file-descriptor limits; plan 64k+ FD per node for safety.

Operational notes / trade-offs

• Tune maxmemory-policy, eviction, and client pipelining to reduce latency at 50k RPS.
• If using cluster mode, shard count equals M; cluster slot balancing and resharding ease capacity changes.
• If dataset larger, increase M proportionally; recalc using U for chosen instance sizes.
• Monitor commands/sec, CPU (single-threaded Redis), and network; consider read replicas for read scaling or Redis Cluster to distribute CPU-bound load.

This sketch gives formulas you can apply to any N, U, H, R and concrete numbers for a 10M-key example yielding ~84 GB total RAM and a 4-node (2 masters + 2 replicas) deployment with ~800–900 Mbps aggregate network for reads.

Approach summary: keep DB as source of truth, ensure cache invalidation happens after DB commit, and avoid the window where a reader sees stale cache by using one or more patterns: 1) cache-versioning (store a row version in cache and DB), 2) transactional outbox/async invalidation, or 3) “double-delete” + short TTL as fallback. Below are examples.

SQL (add a version column and update in a transaction):

BEGIN;
UPDATE items
SET data = $1, version = version + 1, updated_at = now()
WHERE id = $id;
-- Optionally insert into outbox for async invalidation
INSERT INTO outbox (topic, key, version) VALUES ('cache_invalidate','item:'||$id, (SELECT version FROM items WHERE id=$id));
COMMIT;

Application-level pseudocode (cache-aside with version check + outbox consumer):

# Writer: run DB transaction shown above, then return new_version

# Reader:
def get_item(id):
    cache_key = f"item:{id}"
    cached = cache.get(cache_key)
    if cached:
        # cached includes payload and version
        payload, cached_version = cached.payload, cached.version
        # fetch only version from DB to detect staleness (cheap)
        db_version = db.query("SELECT version FROM items WHERE id = ?", id)
        if db_version == cached_version:
            return payload
        # else fall through to reload
    row = db.query("SELECT data, version FROM items WHERE id = ?", id)
    cache.set(cache_key, (row.data, row.version))
    return row.data

# Outbox consumer (reliable async invalidation)
def outbox_worker():
    for msg in outbox.poll():
        cache.delete(msg.key)                 # safe after DB commit
        mark_outbox_processed(msg)

Patterns & reasoning:

• Versioning avoids race: reader validates cached version against DB version; only hits DB for a small version check.
• Outbox ensures invalidation happens after commit; reliable consumer deletes cache.
• Double-delete (delete, sleep short, delete again) is weaker: use when outbox not available.
• Always combine with short TTL and monitoring to bound staleness.
  Trade-offs: version check requires a cheap DB read; outbox adds complexity but scales and is reliable.