Practical scaling techniques and patterns for application and data layers. Topics include horizontal and vertical scaling strategies and the trade offs of each; caching topologies and strategies such as cache aside write through and write behind and approaches to cache invalidation and consistency; database scaling techniques including read replicas partitioning and sharding and rebalancing strategies; load balancing algorithms including round robin least connections consistent hashing and strategies for sticky sessions and service discovery; message queue and event streaming patterns for decoupling backpressure and asynchronous processing; content distribution using content delivery networks; connection pooling and resource management; rate limiting throttling retry strategies and approaches to avoid thundering herd problems; and how to combine patterns effectively given workload characteristics and operational constraints. Interviewers expect candidates to explain interactions between patterns and the operational pitfalls of each technique.
HardTechnical
25 practiced
You are investigating data corruption in an order service caused by cache invalidation races: concurrent writes and cache updates produced stale or missing order state. Perform a root-cause analysis listing likely race conditions (lost-updates, read-after-write), and propose fixes including write-through caching, versioned objects / optimistic compare-and-swap, single-flight cache refresh, or moving to an append-only event store. Explain pros/cons for each fix and a safe rollout plan.
Sample Answer
**Situation & brief RCA**Concurrent API requests were updating orders while cache invalidation/refreshes ran asynchronously. Likely race conditions:- Lost-updates: two clients read-modify-write; last writer overwrites intermediate change.- Read-after-write (stale read): writer updates DB but cache still has old value until invalidation completes.- Cache stampede & double-writes: concurrent cache-miss triggers multiple DB reads/overwrites during refresh.**Fix options (what, pros, cons)**- Write-through cache - What: write to cache synchronously on successful DB commit. - Pros: eliminates window between DB and cache; reads see latest. - Cons: higher write latency; must ensure atomicity if write fails to cache.- Versioned objects / optimistic CAS - What: attach version/timestamp; update only if version matches. - Pros: prevents lost-updates without locking; scales well. - Cons: requires client retry logic; extra complexity for merges.- Single-flight cache refresh - What: ensure only one refresh per key (mutex or in-memory singleflight). - Pros: prevents cache stampede and redundant DB reads. - Cons: needs distributed coordination for multi-instance systems.- Append-only event store - What: store changes as events; materialize current state from stream. - Pros: immutable source of truth, auditability, easier concurrency (idempotent reducers). - Cons: significant architecture change and migration cost.**Safe rollout plan**1. Add metrics/logging for races (versions, cache-miss timing).2. Start with single-flight for hot keys (low-risk).3. Introduce optimistic versioning for write paths with retries; run in shadow mode.4. Move to write-through for critical endpoints if latency acceptable.5. Long-term: pilot event store for one bounded context; migrate incrementally.6. Use canary deploys, feature flags, and data consistency tests; monitor error/latency/ divergence metrics and rollback if regressions appear.
HardTechnical
26 practiced
Provide a comprehensive design for mitigating thundering herd problems across a globally-distributed cache and service fleet. Include preventative techniques (randomized TTLs, staggered renewals), reactive techniques (request coalescing/singleflight, leader-based warmers, circuit breakers, rate-limits), connection pooling and DB limits, and monitoring/alerting. Discuss implementation details, failure cases, and trade-offs between complexity and protection.
Sample Answer
**Situation & goal**Design a global mitigation strategy for thundering herd across caches and services so cache misses or expirations don’t overload origin DBs or downstream services.**Preventative techniques**- Randomized TTLs: add jitter to TTLs (e.g., base 60s ± 5–20%) so expiries spread.- Staggered renewals: refresh hot keys proactively before expiry using background refresh windows per region.- Hierarchical caching: local per-AZ cache → regional shared cache → origin to reduce cross-region spikes.**Reactive techniques**- Request coalescing / singleflight: on miss, only one goroutine/worker fetches and others wait with a timeout; implement per-key locks with sharded lock table to avoid contention.- Leader-based warmers: elect a regional leader (consensus or simple lease) to pre-warm hot keys on known expiry windows.- Circuit breakers & rate-limits: when origin latency/error rate spikes, open circuit to serve stale cache or 503; token-bucket rate-limiters per region to protect DB.- Backoff and retry: exponential with jitter for origin calls.**Connection pooling & DB limits**- Enforce max DB connections per region; use queuing layer or proxy (pgbouncer) with admission control; prefer pooled async workers to avoid connection storms.**Monitoring & alerting**- Metrics: cache hit ratio, miss storm rate (misses/sec per key), queue lengths, origin error-rate, connection saturated, request latencies.- Alerts: sudden surge in miss/sec, low hit ratio, DB connection pool saturation, circuit open events.- Dashboards: top-miss keys, regional heatmaps, per-key waiters.**Implementation details**- Sharded in-memory per-process singleflight keyed by hash; fallback to distributed coalescing (Redis locks with short TTL) for cross-process; use atomic counters to observe waiters.- Leader warmers use ephemeral leases (etcd/consul/Redis SETNX with TTL) to avoid split-brain.- Serve stale-on-error: keep stale value with served-by timestamp and a decreasing staleness penalty.**Failure cases & mitigations**- Lock storms if coalescing fails: add timeout and fallback to origin with circuit breaker to prevent deadlock.- Leader loss during warm: design idempotent warmers and accept transient extra origin load; use shorter leader lease to failover quickly.- Distributed lock latency: use local singleflight first; only escalate to distributed locks when necessary.**Trade-offs**- Complexity vs protection: local singleflight + TTL jitter gives high ROI with low complexity. Distributed locks, leader elections, and stale-serving add robustness at cost of latency, operational complexity and potential stale data.- Strong protection requires more infra (coordination systems, warmers, proxies) and testing; choose based on SLOs and cost.Overall, combine inexpensive local techniques (TTL jitter, singleflight) first, add regional protections (leader warmers, rate-limits, circuit breakers), and only add global coordination when traffic patterns justify the operational cost.
EasyTechnical
31 practiced
Explain eventual consistency versus strong consistency in distributed systems. Provide concrete backend service examples where each model is appropriate (e.g., profiles, shopping carts, payments), discuss trade-offs (latency vs correctness), and describe techniques used to reason about correctness under eventual consistency such as vector clocks, version numbers, read-repair, and bounded staleness.
Sample Answer
**Definition & trade-offs**Strong consistency: every read reflects the latest write (linearizability). Good for correctness-critical ops (payments, inventory decrement). Trade-off: higher latency and reduced availability during partitions (synchronous replication, locks).Eventual consistency: replicas converge over time; reads may return stale data briefly. Good for high-throughput, low-latency features (user profiles, activity feeds, caches). Trade-off: lower latency and higher availability but requires conflict resolution and careful reasoning about correctness.**Concrete mappings**- Payments / ledger updates: strong consistency (ACID DBs, distributed transactions, Paxos/Raft).- Shopping cart: often eventual or session-scoped strong (optimistic merging, per-user partitioning); require merge rules.- Profiles/preferences: eventual consistency acceptable; background sync to converge.**Techniques to reason about correctness**- Version numbers / Lamport timestamps: simple ordering for last-write-wins.- Vector clocks: detect concurrent updates and enable application-level merge.- Read-repair: on reads, detect stale replicas and update them to converge.- Bounded staleness (e.g., R/W quorum tuning, max replication lag): offers latency/correctness SLA (e.g., read-after-write within X seconds).- Quorum configs (R + W > N): provide tunable consistency (stronger if R and W large).As a backend developer I'd pick the model per operation: prioritize strong consistency for money/state transitions; use eventual with clear merge semantics and monitoring for user-facing, high-scale data.
HardSystem Design
25 practiced
Design service discovery and global load-balancing for a multi-region microservice deployment. Requirements: fast failover between regions, supported health checks, minimal DNS propagation issues, and robustness to partial WAN failures. Consider DNS TTLs, Anycast, global LBs (GSLB), service mesh, client-side discovery, and trade-offs between control-plane complexity and operational reliability.
Sample Answer
**Clarify requirements & goals**- Fast cross-region failover (<30s), health-checked endpoints, minimal DNS propagation issues, tolerant to partial WAN failures, operable by backend team.**High-level design**- Hybrid: Anycast + regional ingress + GSLB + client-aware fallback + optional service mesh for intra-region discovery.- Public entry: Anycasted edge (Cloudflare/AWS Global Accelerator) routes to nearest healthy region. GSLB (route53 latency-weighted + health checks) maintains DNS as a secondary/longer-term control.- Regional: Each region has an ingress LB (ALB/NGINX) and regional control-plane running service mesh (optional: Istio/Consul) for service-to-service discovery.**Components & responsibilities**- Anycast/global edge: fast routing based on BGP to nearest POP; performs TLS termination and active health checks.- Regional ingress: enforces regional health, auth, metrics; circuit-breakers for degraded backends.- GSLB/DNS: short TTL (30s) for service endpoints when using client-side fallback; longer TTL (300s) for stable records to reduce churn. Health checks drive DNS weights.- Client-side resolver: SDK implements fallback logic — prefer edge endpoint; on failure, query GSLB to get alternate region; exponential backoff and sticky session via cookies or JWT with region claim.- Control plane: centralized but with regional caches to reduce WAN dependency.**Health checks & failover**- Multi-layer health: POP → regional ingress → service probe. POP Anycast withdraws route when region unreachable; GSLB removes region via health checks; clients detect failed requests and switch to alternate DNS/IP quickly (respecting short TTL).**Partial WAN failures**- Anycast routes may still reach a degraded region; combine active health and client-side retries to avoid blackholing. Regional caches in control plane allow discovery even when cross-region control-plane is flaky.**Trade-offs**- Anycast + edge gives fastest failover but less fine-grained control and harder to debug.- Short DNS TTLs enable fast recovery but increase DNS traffic and risk of inconsistent caches; mitigated with client-side fallback and longer secondary TTLs.- Service mesh improves intra-region resilience and observability but increases control-plane complexity and resource use.- Client-side logic increases client complexity but gives fastest and most reliable failover behavior.**Operational notes**- Instrument end-to-end (SLOs, synthetic tests), run chaos (cable-cut) drills, and document rollback paths. Start with GSLB + regional LBs, add Anycast/mesh incrementally.
HardSystem Design
25 practiced
Design a sharding and rebalancing strategy for a multi-tenant user database with tenants that vary by orders of magnitude. Explain shard key selection, the mapping layer (lookup vs algorithmic), online rebalancing (split and merge), minimizing downtime, handling cross-shard transactions, data migration steps, and instrumentation to detect imbalance and trigger rebalancing.
Sample Answer
**Context & goals**Design a scalable multi-tenant sharding + online rebalancing scheme that handles tenants from tiny to huge with minimal downtime and correct cross-shard semantics.**Shard key selection**- Primary: tenant_id for strong tenant isolation.- Hybrid key: for very large tenants, use (tenant_id, user_id) or range-shard the tenant into sub-shards. Small tenants can be hashed together (tenant_id -> shard via namespace hashing).- Rationale: tenant_id keeps tenant data co-located; sub-sharding large tenants prevents “hot” tenant hotspots.**Mapping layer**- Hybrid approach: - Algorithmic routing (consistent hashing with virtual nodes) for small/medium tenants for O(1) routing and easy scale. - Lookup table (metadata DB / config service) for large tenants and split ranges so routing can point to multiple sub-shards.- Store mapping in highly-available config service (etcd/Consul) with cached local copies and versioned leases.**Online rebalancing (split & merge)**- Split: create new target shard, start background incremental copy of key-range or tenant set; maintain dual-write to source+target during sync; use change-stream/log-based CDC to catch deltas; when caught up switch mapping atomically and stop writes to source.- Merge: pause new writes to merging shard, copy remaining data to target, update mapping, then garbage collect.- Ensure idempotent operations and maintain monotonic mapping versions.**Minimizing downtime**- Use CDC (logical replication/WAL/Change Streams) for incremental sync.- Dual-write or write-through proxy that records pending migrations.- Atomic mapping swap using mapping service version; clients fetch mapping with short TTL and retry on mismatch.- Use leader/lease tokens so only one rebalancer mutates mapping at a time.**Cross-shard transactions**- Prefer compensating Sagas for business workflows spanning shards.- For strict ACID across shards: two-phase commit (2PC) with a transaction coordinator; avoid for high throughput due to blocking and complexity.- Use idempotent operations, well-defined retry semantics, and per-shard optimistic locking where needed.**Data migration steps**1. Plan: choose split/merge targets based on metrics.2. Provision target shard(s).3. Start bulk copy (snapshot).4. Start CDC to replicate mutations.5. Validate checksums / row counts.6. Enter dual-write or write-forwarding mode.7. Atomically update mapping service (single-version switch).8. Drain and verify no pending writes; stop CDC; decommission source or mark read-only.9. Clean up and update monitoring.**Instrumentation & triggers**- Metrics: per-shard throughput (reqs/sec), latency, CPU, storage size, active connections, tenant distribution (users/rows).- Derived metrics: skew ratio (largest shard / median shard), hot-tenant indicators.- Alerts & thresholds: skew ratio > X, shard storage > capacity, sustained high latency.- Automated rebalancer: policy engine to propose splits/merges, human approval for large moves, dry-run mode.- Audit logs, migration traces, and end-to-end consistency checks.**Trade-offs**- Lookup table = flexibility, extra hop and state; algorithmic = simple, faster routing but harder to handle large-tenants.- 2PC = strong consistency, higher latency; Sagas = eventual consistency, simpler at scale.This design balances fast routing for most tenants with explicit handling for giant tenants, supports online split/merge with CDC and mapping swaps, and promotes safe cross-shard behavior with observable triggers for rebalancing.
Unlock Full Question Bank
Get access to hundreds of Scalability Patterns and Techniques interview questions and detailed answers.
