Scalability Patterns and Techniques Questions

Practical scaling techniques and patterns for application and data layers. Topics include horizontal and vertical scaling strategies and the trade offs of each; caching topologies and strategies such as cache aside write through and write behind and approaches to cache invalidation and consistency; database scaling techniques including read replicas partitioning and sharding and rebalancing strategies; load balancing algorithms including round robin least connections consistent hashing and strategies for sticky sessions and service discovery; message queue and event streaming patterns for decoupling backpressure and asynchronous processing; content distribution using content delivery networks; connection pooling and resource management; rate limiting throttling retry strategies and approaches to avoid thundering herd problems; and how to combine patterns effectively given workload characteristics and operational constraints. Interviewers expect candidates to explain interactions between patterns and the operational pitfalls of each technique.

MediumTechnical

24 practiced

During a deployment, a popular API endpoint experiences a thundering herd when caches are invalidated and many clients re-request the resource simultaneously. List mitigation techniques (including request coalescing, randomized TTLs, cache pre-warming, locks, and graceful degradation) and explain operational trade-offs for each approach.

Sample Answer

Approach: enumerate practical mitigations, describe how they work, and call out operational trade-offs so an SRE can choose a mix for a deployment.

1) Request coalescing (single-flight)- What: If many requests for the same key arrive, only one fetches origin; others wait for the result.- Trade-offs: Greatly reduces origin load and latency variance. Requires in-process or distributed coordination; in-process is simple but fails across replicas. Adds short wait latency to second+ requests and complexity to implement (cache of inflight promises). Must handle failures/timeouts carefully to avoid head-of-line blocking.

2) Randomized TTL / cache jitter- What: Add ±random variance to TTLs so many keys don’t expire simultaneously.- Trade-offs: Very low complexity and effective at smoothing expirations. Causes slight unpredictability in cache freshness; needs distribution tuning. Not sufficient alone for a mass invalidation caused by config change.

3) Cache pre-warming- What: After deploy, proactively populate caches (bulk fetch or background workers) for hot keys.- Trade-offs: Eliminates cold-start thundering if you can identify hot keys. Cost: extra origin load during warm-up window; scheduling must be rate-limited to avoid creating a different surge. Requires reliable key list and orchestration tied to deployments.

4) Locks / distributed single-writer fetch (mutex)- What: Use a lightweight distributed lock (Redis/etcd) so one worker refreshes a key while others serve stale or wait.- Trade-offs: Prevents duplicate refreshes but introduces lock management complexity and risk of deadlocks/lock storms if misconfigured. Can be combined with serving stale-on-lock to avoid blocking.

5) Graceful degradation / stale-while-revalidate- What: Serve stale cached responses while triggering background refreshes; optionally return degraded responses when origin overloaded.- Trade-offs: Improves availability and reduces origin pressure, but serves potentially out-of-date data. Must be acceptable for user experience and clearly bounded by SLAs. Requires clear metrics on staleness and business acceptance.

6) Rate limiting / ingress throttling- What: Limit client request rate at edge, with 429 and Retry-After.- Trade-offs: Effective to protect origin; impacts availability for some clients. Needs careful SLO/priority rules (e.g., allow internal/paid clients higher rate).

7) Retry/backoff with jitter and client-side caching- What: Encourage clients to use exponential backoff with jitter and honor cache headers/304s.- Trade-offs: Lowers repeated hammering but requires client changes and good observability to ensure retries don’t amplify issues.

8) CDN and multi-level caches- What: Push caching to edge/CDNs so origin sees much lower QPS.- Trade-offs: Costlier and adds cache invalidation complexity; CDN TTLs/jitter strategies still apply.

Operational considerations:- Combine strategies: e.g., cache jitter + pre-warm + single-flight for best coverage.- Instrumentation: track cache hit ratio, origin QPS, inflight fetches, 429s, latency percentiles. Alert on origin QPS spikes and cache miss storms.- Deployment practices: staged rollouts, feature flags, throttled warm-ups, and chaos testing for cache invalidation scenarios.- Trade-off summary: simplicity (jitter, client backoff) vs stronger guarantees (coalescing, locks) which add complexity and potential single points. Choose based on tolerance for stale data, implementation cost, and origin capacity.

EasyTechnical

52 practiced

What is consistent hashing and why is it useful for distributed caches and sharded storage systems? Explain the basic algorithm, how virtual nodes help, and what happens to key distribution when nodes are added or removed.

EasyTechnical

33 practiced

Compare message queues and event streams (e.g., RabbitMQ vs Kafka) for decoupling microservices. Discuss delivery semantics, ordering, retention, consumer scaling, and appropriate use cases such as task queues, pub/sub for notifications, and long-term event storage.

Sample Answer

High-level summary: both message queues (e.g., RabbitMQ) and event streams (e.g., Kafka) decouple services but target different patterns. Choose by delivery semantics, ordering needs, retention, scaling model and operational trade-offs.

Delivery semantics- RabbitMQ: typically at-most-once or at-least-once depending on acks; deduplication requires app logic. Good for tasks where immediate ack/visibility is needed.- Kafka: at-least-once by default; supports exactly-once semantics in producers+consumers (with transactions) for specific flows; dedupe/idempotency often simpler.

Ordering- RabbitMQ: queue-level ordering, preserved if single consumer; competing consumers break global ordering.- Kafka: ordering guaranteed per-partition. Designing partition keys controls ordering scope.

Retention- RabbitMQ: messages usually removed after successful ack (short-lived). Can persist but not designed as long-term store.- Kafka: time/size-based retention with configurable compaction—built for long-term event retention and replay.

Consumer scaling- RabbitMQ: scale via multiple queues or competing consumers; one queue’s messages distributed among consumers (good for work-queue patterns).- Kafka: scale by increasing partitions; consumer groups map 1:1 to partitions, so partition count limits parallelism.

Appropriate use cases- Task queues / background jobs: RabbitMQ excels (low-latency dispatch, worker pooling).- Pub/sub notifications: RabbitMQ or Kafka; RabbitMQ for low-latency fan-out, Kafka when you need replay, analytics, or many consumers.- Long-term event storage / event sourcing / stream processing: Kafka is the clear choice (replay, retention, high throughput).

SRE considerations- Monitoring: track consumer lag (Kafka), queue length and unacked messages (RabbitMQ), broker resource usage, and GC/IO.- Backpressure and durability: Kafka tolerates high-throughput bursts; RabbitMQ can back up memory if consumers are slow—require flow control.- Ops complexity: Kafka (Zookeeper/KRaft, partition management) is operationally heavier but scales better; RabbitMQ is simpler to run for smaller workloads.- Reliability: enforce idempotent consumers, set appropriate acks/replication, tune retention and retention compaction, and plan partitioning strategy to meet SLOs.

Recommendation: use RabbitMQ for task processing and low-latency fan-out; use Kafka when you need durable, replayable streams, high throughput, and analytics/stream-processing integration.

MediumTechnical

26 practiced

You operate a service with strong diurnal traffic and occasional unpredictable spikes. How would you size and configure autoscaling policies for stateless frontends and stateful backends? Discuss metrics (CPU, latency, queue depth), cooldowns, predictive scaling (using historical data), and safety limits to avoid oscillation.

Sample Answer

Clarify requirements first: SLOs (p95 latency target, error budget), traffic patterns (peak hour, spike frequency), and recovery time objectives. Then treat stateless frontends and stateful backends differently.

Stateless frontends- Primary metrics: request rate (RPS), p95/p99 latency, CPU, and active connections. Use RPS-per-instance derived from load testing to convert target RPS→desired instances.- Autoscale policy: scale-out on RPS > threshold or p95 latency > target; scale-in when RPS and latency comfortably below threshold.- Cooldowns: short scale-out cooldown (30–60s) to react to spikes; longer scale-in cooldown (5–10m) to avoid oscillation.- Predictive scaling: schedule capacity from historical diurnal patterns (increase before expected morning peak) and use short-term forecasts to pre-warm for anticipated spikes.- Safety limits: min pool to handle cold starts, max instances to cap cost; step-size limits (e.g., +25% per action) and stabilization window (hysteresis) to prevent thrash.

Stateful backends- Primary metrics: queue depth/consumer lag, write latency, CPU, memory, disk I/O; for databases also replication lag.- Prefer conservative scaling: scale-out when queue depth or lag exceeds threshold (e.g., consumer lag > X messages or p95 write latency > SLO). Use gradual increments (add 1–2 nodes), ensure automated draining and rebalancing with health checks.- Cooldowns: longer (10–30m) to allow rebalancing; require successful readiness checks before routing traffic.- Predictive scaling: schedule additional replicas/replicasets ahead of known peaks; for DBs prefer vertical scaling or read-replicas for read-heavy loads.- Safety limits: max node count, maximum per-minute scaling rate, automated rollback on unhealthy join.

Avoid oscillation- Use combined signals (RPS + latency + queue depth) rather than single noisy metric.- Use percentile metrics (p95/p99) and short moving averages to filter noise.- Implement hysteresis (different thresholds for scale-out vs scale-in), stabilization windows, and scale step caps.- Reserve a small buffer (10–20%) above expected peak inside policies and hold warm instances (or use warm pools) for fast recovery.- Add alerting when autoscaling is triggered frequently (investigate root cause) and run game days to validate behavior.

Finally, test with chaos/scale experiments, validate cost vs reliability trade-offs, and iterate thresholds based on observed incidents and SLOs.

MediumTechnical

33 practiced

Compare write-behind and write-through caches with respect to data durability, failure recovery, and operational observability. For a payment processing workflow, which would you choose and how would you mitigate the main risks?

Sample Answer

Write-through vs write-behind — short comparison (SRE lens)

- Data durability - Write-through: Every cache write is synchronously persisted to backing store before acknowledging. Stronger durability guarantees: once client gets success, data is durably stored (modulo DB durability config). - Write-behind: Cache acknowledges immediately and flushes to store asynchronously. Higher risk of lost writes on cache failure unless you add persistence (e.g., on-disk queue, WAL).

- Failure recovery - Write-through: Recovery is simpler — cache can be repopulated from authoritative store; no in-flight writes are lost. - Write-behind: Must replay in-flight buffers. Recovery requires durable write buffers, exactly-once/at-least-once semantics handling, and de-dup/idempotency to avoid double-application.

- Operational observability - Write-through: Easier to reason about latency and errors (client errors surface immediately). Metrics: cache hit/miss, DB latency, error rates. - Write-behind: More moving parts: flush queue size, flush latency, retry/backoff stats, persistent buffer health. Observability must cover queue backlog, dropped writes, retry counts, and divergence between cache and store.

Recommendation for payment processing workflow- Choose write-through for core payment state (transactions, balances) because correctness and durability trump latency. Payments require strong ordering, no lost or duplicated money transitions, and clear audit trail.

If write-behind is considered for performance-sensitive non-critical paths (analytics, non-authoritative views), mitigate risks:- Use durable write buffer (append-only WAL on disk or replicated queue like Kafka).- Enforce idempotent operations or use deduplication keys (transaction IDs) so replays are safe.- Implement transactional or two-phase commit boundaries where needed (cache write + enqueue flush in same durable operation).- Instrument aggressively: metrics for queue depth, flush latency, retry/error rates, and end-to-end reconciliation gaps; set alerts on thresholds.- Periodic reconciliation job that diffs cache vs authoritative store and triggers fixes; store an immutable audit log for forensic recovery.- Chaos test failure modes (cache crash, partial flush, network partitions) in staging to validate recovery playbooks.

Operational practices (SRE priorities)- Make durability SLAs explicit in SLOs and error budgets.- Automate failover and replay procedures; document runbooks for incidents involving lost or duplicated writes.- Prefer simplicity for payment-critical paths: observable synchronous persistence (write-through) minimizes blast radius and reduces incident complexity.

Unlock Full Question Bank

Get access to hundreds of Scalability Patterns and Techniques interview questions and detailed answers.

Join thousands of developers preparing for their dream job.