Comprehensive knowledge of caching principles, architectures, patterns, and operational practices used to improve latency, throughput, and scalability. Covers multi level caching across browser or client, edge content delivery networks, application in memory caches, dedicated distributed caches such as Redis and Memcached, and database or query caches. Includes cache design and selection of technologies, defining cache boundaries to match access patterns, and deciding when caching is appropriate such as read heavy workloads or expensive computations versus when it is harmful such as highly write heavy or rapidly changing data. Candidates should understand and compare cache patterns including cache aside, read through, write through, write behind, lazy loading, proactive refresh, and prepopulation. Invalidation and freshness strategies include time to live based expiration, explicit eviction and purge, versioned keys, event driven or messaging based invalidation, background refresh, and cache warming. Discuss consistency and correctness trade offs such as stale reads, race conditions, eventual consistency versus strong consistency, and tactics to maintain correctness including invalidate on write, versioning, conditional updates, and careful ordering of writes. Operational concerns include eviction policies such as least recently used and least frequently used, hot key mitigation, partitioning and sharding of cache data, replication, cache stampede prevention techniques such as request coalescing and locking, fallback to origin and graceful degradation, monitoring and metrics such as hit ratio, eviction rates, and tail latency, alerting and instrumentation, and failure and recovery strategies. At senior levels interviewers may probe distributed cache design, cross layer consistency trade offs, global versus regional content delivery choices, measuring end to end impact on user facing latency and backend load, incident handling, rollbacks and migrations, and operational runbooks.
EasyTechnical
100 practiced
Explain cache warming and prepopulation strategies before deploying a new service or a new cache cluster. Include step-by-step approaches, how to avoid creating a thundering herd, and risks during warmup such as stale data and traffic spikes.
Sample Answer
Approach (step-by-step):1. Clarify scope & requirements: identify key keys/paths, acceptable staleness, capacity targets, and SLOs for latency during warmup.2. Prepare data set: generate a prioritized list of hot keys from access logs / telemetry (top N%, recent high-QPS endpoints).3. Prepopulate safely: - Bulk-write hot keys via a controlled job (batch writer) that uses the same serialization/TTL as production. - Or replay recent requests against the new cache in a staging mode (idempotent reads) to populate realistic entries.4. Controlled rollout: - Canary the new cache cluster for a small % of traffic. - Ramp traffic gradually (e.g., 5% → 25% → 100%) while monitoring latency, miss-rate, and backend load.5. Post-warm validation: smoke tests, sample requests to verify correctness and freshness.Avoiding thundering herd:- Use singleflight / de-duplication so only one upstream fetch per missed key.- Apply jittered staggered warmup scheduling (randomize per-key start times).- Rate-limit prepopulation writes and cap parallelism; use exponential backoff on upstream errors.- Use token bucket or concurrency limits on workers to bound QPS to origin.Risks and mitigations:- Stale data: set conservative TTLs during warmup, version keys (include schema/version), and refresh critical keys after go-live.- Traffic spikes and backend overload: ramp traffic, cap prepopulation rate, employ circuit breakers and autoscaling for origin.- Cache poisoning / inconsistency: sign/validate data, only warm from trusted canonical sources; run integrity checks.- Increased latency during initial misses: monitor miss-rate, latency, error budget and be ready to rollback or slow rollout.Monitoring & runbooks:- Track cache hit-rate, origin QPS, error rates, p95/p99 latency, CPU/memory.- Have alerts for origin saturation and automated rollback steps.- Document warmup job idempotency and safe re-run procedure.Example patterns/tools:- Use singleflight libraries (Go singleflight), Redis Lua scripts for atomic fills, message queues for paced warmup jobs, and feature flags for gradual traffic shift.
EasyTechnical
93 practiced
List and explain the key metrics an SRE should monitor to assess cache health and effectiveness, such as hit ratio, miss rate, eviction rate, memory usage, and tail latency. For each metric suggest why it matters and propose reasonable alerting triggers or thresholds.
Sample Answer
Below are key cache health metrics an SRE should monitor, why each matters, and suggested alerting triggers/thresholds (examples — tune to workload and baseline).1) Hit ratio (hits / total requests)- Why: Primary indicator of cache effectiveness; higher = less backend load and lower latency.- Alert: sustained drop >10–20% from 7‑day baseline for >5m OR absolute hit ratio <70% for >10m.2) Miss rate (misses / total requests)- Why: Complement to hit ratio; high miss rate means requests go to origin, increasing load/cost.- Alert: sustained increase >2x baseline for >5m OR miss rate >30% for high-traffic caches.3) Eviction rate (evictions/sec or % of operations)- Why: High evictions indicate insufficient size or poor TTL/eviction policy; evicted keys may be hot.- Alert: sustained evictions/sec above baseline by >3x OR >10% of items evicted in last 5m.4) Memory usage / utilization- Why: Approaching capacity causes evictions, OOM risks, degraded performance.- Alert: memory >85–90% for >10m; sudden jump >10% in 1m.5) Tail latency (p95/p99 request latency)- Why: Average latency can hide outliers; tail affects user experience and downstream SLAs.- Alert: p99 > X ms (e.g., >100ms for in‑process cache, >500ms for remote) or p95 increased >2x baseline for >5m.6) Error rate (failed ops, connection errors)- Why: Errors mean functional problems (network, auth, resource exhaustion).- Alert: error rate >1% or >5 errors/min sustained for 5m; any authentication failures.7) Fill rate / cold-start rate (new keys written / reads)- Why: High fill rate suggests cache warming or ineffective prepopulation; spikes increase origin load.- Alert: fill rate >2x baseline sustained 10m.8) TTL distribution and expirations- Why: Short TTLs cause churning; unexpected expirations point to misconfig.- Alert: fraction of expirations > expected by >20% or sudden TTL policy changes.9) Client-side metrics (client miss rate, retries)- Why: Distinguish cache vs client issues; helps diagnose misconfiguration.- Alert: client retry > baseline, or mismatch between client and cache hit ratios.Practical guidance:- Combine absolute thresholds with anomaly detection (baseline/seasonality).- Use multi-window alerts (e.g., 1m spike vs sustained 10m) and paging only for sustained/high-impact cases; send notifications for transient anomalies.- Correlate cache metrics with backend latency/CPU and origin request rate to find root cause before scaling or resizing.
MediumTechnical
135 practiced
Implement a request coalescing mechanism (singleflight) in Go or Python that ensures concurrent cache-miss requests for the same key trigger only one backend computation, while others wait for the result. Provide code or pseudocode and describe timeout and deadlock considerations.
Sample Answer
To coalesce concurrent cache-miss requests for the same key, use a map from key → call state that tracks in-progress work, result, and waiters. The first goroutine starts the backend call; others wait on a channel. Ensure proper locking, context-based timeouts, and cleanup to avoid deadlocks.
go
package main
import (
"context"
"errors"
"sync"
"time"
)
type result struct {
val interface{}
err error
}
type call struct {
wg sync.WaitGroup
res result
once sync.Once
}
type Singleflight struct {
mu sync.Mutex
m map[string]*call
}
func New() *Singleflight { return &Singleflight{m: make(map[string]*call)} }
// Do ensures only one backendFunc runs per key concurrently.
// ctx controls timeout/cancellation.
func (sf *Singleflight) Do(ctx context.Context, key string, backendFunc func(context.Context) (interface{}, error)) (interface{}, error) {
sf.mu.Lock()
if c, ok := sf.m[key]; ok {
// another goroutine is computing; wait
sf.mu.Unlock()
done := make(chan struct{})
go func() {
c.wg.Wait()
close(done)
}()
select {
case <-ctx.Done():
return nil, ctx.Err()
case <-done:
return c.res.val, c.res.err
}
}
// create call and mark in-progress
c := &call{}
c.wg.Add(1)
sf.m[key] = c
sf.mu.Unlock()
// ensure cleanup and mark done exactly once
defer func() {
c.wg.Done()
sf.mu.Lock()
delete(sf.m, key)
sf.mu.Unlock()
}()
val, err := backendFunc(ctx)
c.res = result{val: val, err: err}
return val, err
}
func main() {
// usage example omitted
_ = New()
}
Key points:- Use context for timeouts/cancellation; waiting goroutines return on ctx.Done().- Protect map with mutex; each call has WaitGroup so waiters block without busy-wait.- Cleanup map entry in defer to avoid memory leaks.Deadlock/timeouts:- If backendFunc ignores ctx cancellation, the call may continue while waiters time out — they get ctx.Err() but the backend still runs; ensure backend respects ctx.- Always delete the call entry in a defer so future requests aren't blocked.- Limit concurrent backend work using worker pool or semaphore if backend is heavy.
MediumTechnical
99 practiced
Design a cache key naming scheme that supports schema evolution, multi-tenancy, and versioned content. Explain how to include prefixes, version tokens, tenant identifiers and TTLs in keys and how to handle migrations when schema changes require key format updates.
Sample Answer
Approach: define a consistent, explicit key format that encodes purpose, tenant, schema version, resource id and optional env/region; keep separators predictable and safe for your cache backend. Use version tokens to enable schema evolution and migration strategies (double-write, lazy-migrate, backfill, and TTL-based eviction).Recommended key format examples:- Prefix pattern: svc:env:tenant:v{schema_version}:resource_type:resource_id- Example: orders:prod:tenant_42:v2:user_profile:12345- For long IDs, hash trailing part: orders:prod:tenant_42:v2:invoice:sha1(very-long-id)Design details:- Prefix (svc/env): isolates services and deploy stages to avoid cross-contamination.- Tenant identifier next: keeps multi-tenancy explicit and allows per-tenant invalidation.- Version token (vN): integer or semantic tag; bump when schema or serialization changes.- Resource type then ID: human-readable and good for tooling.- TTL: store TTL in cache metadata or encode intended TTL in instrumentation (not in key). If using per-key TTL variants, keep canonical TTL mapping per vN in config.Migration strategy:1. Backwards-compatible read: service reads vN key first, if miss then try previous vN keys (read-fallback).2. Double-write: on write, write both vN and vN+1 keys until migration complete.3. Lazy migration: on read-miss of vN, fetch authoritative store, write new vN key (no full backfill).4. Backfill job: for heavy-read items, run controlled bulk migration to vN format using workers with rate-limiting and monitoring.5. Finalize: once traffic shows negligible reads to old versions, retire old keys by marking deprecated and then evicting.Operational practices:- Monitor hit/miss per version & per-tenant; alert on spikes.- Provide admin tooling to list & bulk-delete by prefix (tenant or version).- Limit key length, use hashing for long IDs, and avoid embedding secrets.- Use consistent separators (:) and document format in runbook.Trade-offs:- Double-write increases write cost but reduces latency for reads.- Lazy migration simplifies rollout but may spike origin load; mitigate with rate-limited backfill.This scheme gives clear isolation for multi-tenancy, explicit versioning for schema evolution, and multiple migration paths to balance availability and operational cost.
MediumSystem Design
120 practiced
Design a caching strategy for a read-heavy analytics endpoint that takes around 1 second of CPU per request. Clients tolerate 5% staleness up to 1 minute. Propose TTLs, background proactive refresh, cache-aside or read-through pattern, and how to handle cache misses and failures gracefully.
Sample Answer
Requirements & constraints:- Read-heavy analytics endpoint: expensive CPU ~1s per request.- Clients tolerate up to 5% staleness and <=60s staleness.- Goal: low tail latency, high availability, predictable CPU usage.High-level approach:- Use cache-aside with proactive background refresh (hybrid): cache-aside for correctness and simple miss handling; proactive refresh to avoid thundering herd and reduce user-visible latency on cold/expired keys.TTL strategy:- Serve cached values with a hard TTL = 60s (max staleness).- Also attach a soft-TTL = 30s. After soft-TTL expires, value is still served, but entry is marked “staleable” and eligible for background refresh.- Keep metadata: last-refresh timestamp, in-flight-refresh flag, and version/hash.Proactive refresh:- When a request sees a key older than soft-TTL and in-flight-refresh is false, trigger an asynchronous refresh job (non-blocking) that: - Sets in-flight-refresh = true (atomic CAS in cache/sidecar) - Computes fresh result (1s CPU) - Writes new value with updated timestamps, resets in-flight flag- If multiple nodes try to refresh, CAS prevents duplication; if CAS not available, use distributed lock with short lease (e.g., Redis SETNX with TTL 5s).Cache miss & read path:- On miss, perform a synchronous compute (blocking) but first attempt to acquire short lock (to prevent stampede). If lock acquired, compute and populate cache; if lock not acquired, wait up to 100–200ms for cache fill, then if still empty, compute as fallback to avoid long waits.Failure handling & graceful degradation:- If background refresh fails, keep serving stale value until hard TTL. If hard TTL passes and compute fails: - Return last-known stale value (if any) with warning header and 503 fallback header if strict freshness required. - Implement circuit-breaker: if >X% refreshes fail over Y minutes, stop attempting background refreshes and alert SRE.- Use exponential backoff for retrying failed refreshes.Monitoring, SLOs & operational:- SLOs: 95th latency <200ms (cache hit), availability 99.9%, freshness SLO: >=95% requests <=60s age.- Metrics: hit/miss ratio, refresh success rate, refresh latency, in-flight refresh counts, cache age distribution.- Alerts: drop in hit rate, increased compute CPU, elevated refresh failures, many concurrent lock contends.- Autoscale worker pool for refresh tasks based on queue length and CPU.Trade-offs:- Soft+hard TTL reduces user latency and avoids frequent recompute at cost of serving slightly stale data within SLA.- Cache-aside gives control and easier consistency with asynchronous refresh; read-through could simplify but couples cache provider and compute logic.- Use strong locking to avoid stampedes but keep low lock time to prevent blocking.Implementation notes:- Store cached item as JSON: {value, last_refresh, soft_ttl, hard_ttl, in_flight_flag}.- Use Redis or in-memory sidecar + Redis for distributed state. Background refresh can be executed by request threads (spawned) or dedicated refresh workers depending on load.This design minimizes exposed compute latency, keeps staleness within 60s for >=95% requests, and provides operational observability and graceful failure modes.
Unlock Full Question Bank
Get access to hundreds of Caching Strategies and Patterns interview questions and detailed answers.
