Scalability Patterns and Techniques Questions

Practical scaling techniques and patterns for application and data layers. Topics include horizontal and vertical scaling strategies and the trade offs of each; caching topologies and strategies such as cache aside write through and write behind and approaches to cache invalidation and consistency; database scaling techniques including read replicas partitioning and sharding and rebalancing strategies; load balancing algorithms including round robin least connections consistent hashing and strategies for sticky sessions and service discovery; message queue and event streaming patterns for decoupling backpressure and asynchronous processing; content distribution using content delivery networks; connection pooling and resource management; rate limiting throttling retry strategies and approaches to avoid thundering herd problems; and how to combine patterns effectively given workload characteristics and operational constraints. Interviewers expect candidates to explain interactions between patterns and the operational pitfalls of each technique.

HardTechnical

44 practiced

Design an idempotent event-processing pipeline for at-least-once delivery (e.g., consuming from Kafka) at high throughput. Describe deduplication strategies (idempotency keys, compacted store, bloom filters), storage and TTL for dedupe state, handling of large state growth, and how to maintain throughput and correctness.

Sample Answer

**Approach summary**Design a dedupe layer that is fast, distributed, and eventually evicts old keys. Combine strong determinism (idempotency keys / sequence numbers) + compacted authoritative store + lightweight probabilistic filters to preserve throughput.

**Core components**- Producer/Message: include an idempotency key (UUID, producer-id + sequence) or monotonic sequence per entity.- Consumer pipeline: partitioned consumers (Kafka partitions → consumer group) so same key goes to same consumer for local de-duplication.- Authoritative dedupe store: Kafka compacted topic OR local embedded KV (RocksDB) with periodic sync to compacted topic for recovery.

**Deduplication strategies**- Idempotency keys / sequence numbers: deterministic, zero false positives.- Compacted store (Kafka compacted topic): durable, replicated, scales with Kafka; can rebuild state after restart.- Bloom filters (or Cuckoo filters) in front: fast in-memory test to avoid expensive store reads; accept small false positives (causes rare extra drops) but never false negatives.- Two-tier check: first in-memory bloom filter → if miss, check local RocksDB → if not present, process and write key to RocksDB and append to compacted topic asynchronously.

**Storage & TTL**- Store a timestamp with each key.- TTL policies: - Short TTL for high-cardinality, low-value keys (e.g., 24–72 hours). - Long TTL for keys that must be retained longer per business SLAs.- Eviction: background compaction job removes expired keys from RocksDB and writes tombstones to compacted topic to free space.

**Handling large state growth**- Shard state by natural partition key; add more consumers/partitions to scale horizontally.- Use disk-backed RocksDB per consumer to hold large state with memory caching.- Periodic snapshot + compaction: compacted topic tombstones let Kafka reclaim space.- Tiered storage: keep recent keys in memory, older in RocksDB, archive very old to cold storage for audit if needed.

**Maintaining throughput & correctness**- Batch writes to dedupe store and compacted topic; use idempotent appends and backpressure controls.- Asynchronous writes with write-ahead: mark “in-flight” in local store before external side effects; commit after durable append to compacted topic (or use transactional append).- Use Bloom filter sizing tuned for acceptable false-positive rate; rebuild periodically from compacted topic.- Monitor metrics: false-positive rate, store size, write latency, consumer lag. Autoscale partitions/consumers when lag increases.- Failure recovery: rebuild local RocksDB from compacted topic; use consumer-group offsets and compacted topic's last known key timestamp to avoid reprocessing beyond TTL window.

**Trade-offs**- Bloom filters reduce load but can drop true messages (false positives) — only acceptable if business tolerates rare drops; otherwise rely solely on deterministic keys + durable store.- Compacted topic gives durability and easy rebuilds but increases write amplification and storage usage.

Example: partitioned Kafka topic per customer-id; consumer uses per-partition RocksDB + bloom filter; on message arrival check bloom → RocksDB; if new, append key to compacted topic asynchronously (or use Kafka transactions) then perform side-effect. Evict keys older than 48h; scale consumers when consumer-lag > threshold.

This design keeps correctness via deterministic keys and durable compacted store while preserving throughput through in-memory filters, batching, and sharding.

HardTechnical

34 practiced

Compare consistent hashing and range-based partitioning for a large-scale datastore where complex queries and joins across ranges are common. Explain pros/cons for query locality, rebalancing, and ease of scaling. Propose a hybrid partitioning approach that supports complex queries without creating hotspots.

Sample Answer

**High-level comparison**

- **Consistent hashing** - Pros: excellent even load distribution, low-data-movement on node changes (virtual nodes), simple scaling. - Cons: poor range locality — range scans and range joins require scatter/gather; complex queries across sorted ranges are expensive.

- **Range-based partitioning** - Pros: strong query locality for range scans and joins; sequential storage benefits (compression, SSTables). - Cons: hotspot risk (skewed key distributions), expensive rebalancing when splits/merges move large contiguous ranges, harder autoscaling.

**Query locality, rebalancing, scaling**

- Query locality: range- partitioning wins for range queries and joins because related keys colocate. Consistent hashing forces multi-node reads.- Rebalancing: consistent hashing minimizes per-node data moved; range partitioning may need bulk moves to rebalance skew.- Ease of scaling: adding capacity is smoother with consistent hashing; range-partitioned systems need careful split planning or automated split services.

**Hybrid approach (practical, systems-engineer view)**

Goal: retain range-locality for complex queries while avoiding hotspots and making scaling manageable.

Design:- Top-level range partitioning by coarse attribute that preserves most join locality (e.g., time window, tenant id range).- Within each range shard, sub-partition using consistent hashing (or hash prefix of secondary key) into virtual sub-shards spread across nodes. This prevents single-range hotspots.- Maintain a centralized/router service (lightweight metadata) mapping range -> list of sub-shard endpoints and preferred replicas.- Use adaptive split/merge: monitor per-sub-shard load; if a sub-shard exceeds thresholds, split its range and redistribute only that range’s sub-shards (small data movement).- Co-locate replicas for common join partners using placement hints; use Bloom filters and per-node min/max keys to prune scatter/gather.- Support distributed range-join execution: planner routes most work to nodes holding matching coarse ranges and coordinates inner-hash lookups in parallel.

Trade-offs:- More metadata and routing complexity, but dramatically reduced hotspots and manageable rebalancing. Suits large-scale datastores needing both complex joins and elastic scaling.

EasyTechnical

28 practiced

Differentiate between message queues and event streaming platforms. For each describe durability, retention, ordering guarantees, consumer models (competing consumers vs pub/sub), and typical use-cases (task queues, audit logs, change data capture). Give one example where streaming is a better fit than a simple queue.

HardTechnical

27 practiced

Design a mitigation strategy for a thundering herd caused by mass cache invalidation during a flash sale where millions of product keys are invalidated simultaneously. Include cache architecture, warming, staggered expiry, request coalescing, and traffic shaping. Provide a step-by-step operational runbook to enact before the sale.

Sample Answer

**Overview (goal)** Prevent thundering herd when millions of cache keys are invalidated at flash-sale start by combining resilient cache architecture, proactive warming, staggered expiry, request coalescing, and traffic shaping. Below is architecture + pre-sale operational runbook.

**Cache architecture**- Use multi-layer cache: CDN edge -> regional Redis/Memcached read-replicas -> authoritative DB.- Redis clusters with client-side routing and consistent hashing to avoid hot-shard risk.- Enable TTL and versioned keys (product:v2:{id}) to allow gradual rollouts.

**Techniques**- Cache warming: precompute top N products’ HTML/JSON and bulk MSET into Redis + pre-populate CDN via purge+prefill APIs.- Staggered expiry: randomize small jitter to TTLs for non-critical keys; on global invalidation, perform version bump per product-group over short windows (e.g., 1s–30s buckets).- Request coalescing: enable singleflight/mutex in API layer so only one backend fetch per key-per-region triggers DB load; others wait.- Traffic shaping: rate-limit new cache-miss requests per region, queue/backoff with exponential jitter, and return stale-while-revalidate responses when safe.

**Operational runbook (before sale)**1. 48h: Identify top 1M keys; precompute responses and push to Redis + CDN.2. 6h: Enable singleflight in services; deploy staggered-versioning script.3. 60m: Increase Redis connection pools; enable read-replicas; set monitoring alerts.4. 10m: Activate traffic shaping config (rate-limits, SLO thresholds).5. T-1m: Start rolling version bump job that updates product-group versions in 30s buckets; verify coalescing counts and error rates.6. During sale: monitor cache-miss rate, DB CPU, tail latency; if surge, increase TTL jitter window, raise rate-limit, and enable serving stale content.7. Post-sale: drain staged changes, revert temporary limits, and run retrospective.

**Why this works**Combines prevention (warming), controlled invalidation (staggered/versioned), suppression of duplicate backend load (coalescing), and graceful degradation (traffic shaping/stale-while-revalidate), minimizing DB spikes and latency.

HardSystem Design

36 practiced

Design an end-to-end image upload and processing pipeline that must serve millions of uploads per day with thumbnails and multiple sizes, low latency for end users, and cost constraints. Include ingress, storage (object store), processing (sync/async choices), CDN usage, queues and worker autoscaling, cache strategies, invalidation, and security considerations.

Sample Answer

**Clarify requirements & constraints**- Millions uploads/day, low-latency user reads (thumbnails / multiple sizes), cost-sensitive, durable storage, secure. Assume eventual-consistency acceptable for derived images.

**High-level flow**1. Client -> Ingress (signed PUT URL) -> Object Store (primary) 2. Upload event -> Message Queue -> Worker pool -> generate sizes -> store derived objects -> CDN 3. Read path: CDN edge -> origin (object store) fallback

**Ingress**- Issue short-lived signed upload URLs (S3 presigned, GCS signed URLs) via an API gateway (Auth + rate limits, upload metadata validation).- Support multipart/resumable uploads for large files.

**Storage**- Primary durable object store (S3/GCS) with lifecycle rules (archive older originals).- Store originals + derived under stable keys and content-addressed or versioned keys for idempotency.

**Processing (sync vs async)**- Synchronous for small immediate UX-critical thumbnails (generate via edge Lambda@Edge or CloudFront Functions for tiny transforms).- Asynchronous for full-size variants: emit event to queue after upload; workers pull and process. This balances latency vs cost.

**Queue & Worker Autoscaling**- Use a durable queue (SQS, Pub/Sub) with visibility timeouts and DLQ.- Autoscale workers based on queue length + processing time; use spot/ephemeral instances & burst capacity with fallback to on-demand.- Design idempotent workers (check if derived exists, use object metadata/version tags).

**CDN & Cache Strategy**- Serve derived images via CDN (CloudFront/Cloudflare); set Cache-Control and immutable content headers for long TTLs.- Use origin shielding and tiered-caching to reduce origin egress.- For mutable updates, use content versioning (filename or query param) to avoid frequent invalidations.

**Invalidation**- Prefer cache-busting (new key) for most updates. Use CDN invalidation API sparingly for urgent rollbacks (costly).- Provide short-lived signed URLs for private content and set appropriate cache headers.

**Security**- Authenticate users issuing signed URLs; enforce ACLs on object store with least privilege roles.- Scan uploads for malware using async scanning pipeline before making public; quarantine suspicious objects.- Enforce size/type checks at ingress; use WAF and rate-limiting; encrypt at rest and in transit (TLS, SSE).- Audit logs, KMS for keys, rotate credentials, and monitor for anomalous upload patterns.

**Cost & Optimization**- Use serverless for low steady-state cost (Lambda for small transforms), batch workers for heavy loads on spot instances.- Lifecycle policies to move originals to cheaper tiers; compress/WEBP newest formats; use multi-size generation strategy to avoid redundant variants.

**Reliability & Observability**- Metrics: upload rate, queue depth, worker errors, processing latency, CDN hit ratio.- Alerts on rising queue depth, error rates. Health-check endpoints, graceful shutdown, backpressure to clients.

This design balances low-latency reads via CDN/edge, cost via async processing + lifecycle, and security via signed URLs, scanning, and least-privilege storage.

Unlock Full Question Bank

Get access to hundreds of Scalability Patterns and Techniques interview questions and detailed answers.

Join thousands of developers preparing for their dream job.