Design and operational patterns for reducing latency and decoupling components using caching layers and asynchronous communication. For caching, understand when to introduce caches, cache placement, eviction policies, cache coherence, cache invalidation strategies, read through and write through and write behind patterns, cache warming, and trade offs between consistency and freshness. For asynchronous processing and message driven systems, understand producer consumer and publish subscribe patterns, event streaming architectures, common brokers and systems such as Kafka, RabbitMQ, and Amazon Simple Queue Service, and the difference between queues and streams. Be able to reason about delivery semantics including at most once, at least once, and exactly once delivery, and mitigation techniques such as idempotency, deduplication, acknowledgements, retries, and dead letter queues. Know how to handle ordering, partitioning, consumer groups, batching, and throughput tuning. Cover reliability and operational concerns such as backpressure and flow control, rate limiting, monitoring and alerting, failure modes and retry strategies, eventual consistency and how to design for it, and when to choose synchronous versus asynchronous approaches to meet performance, scalability, and correctness goals.
HardTechnical
47 practiced
Design an architecture that uses Redis as a cache and leverages Lua scripts to perform atomic composite updates (e.g., decrement inventory and record reservation) in a distributed environment. Explain how Lua in Redis ensures atomicity, limitations in cluster mode, and fallback patterns when scripts fail or a Redis node is partitioned.
Sample Answer
High-level design:- Clients → API layer (stateless services / API Gateway) → Redis cluster (caching + atomic ops) + backing DB (primary durable store, e.g., PostgreSQL)- API services use optimistic read-through cache: read inventory from Redis; on write/reservation, call EVALSHA Lua script in Redis to perform composite update atomically; if script succeeds, publish event to Kafka for downstream processing and async DB reconciliation.Lua atomicity in Redis:- Redis executes each Lua script as a single Redis command under the server’s global GIL for that shard — no other commands interleave while the script runs. This guarantees atomicity of composite operations (decrement inventory and create reservation metadata) on that shard, avoiding race conditions and partial state.Limitations in cluster mode:- Redis Cluster shards keys by hash slot. A Lua script that touches multiple keys must have all keys in the same hash slot (use a hash tag {inventory:123} to co-locate). Cross-slot atomicity is not supported; multi-key transactions across different nodes cannot be made atomic with a single script.- Scripts executing long-running logic can block the node (affecting latency) and may be interrupted on failover.Fallback and resilience patterns:1. Single-shard co-location: design key schema to colocate related keys (inventory, reservations) with hash tags to preserve atomicity.2. Pessimistic fallback: if cluster constraints prevent colocation, implement a distributed lock (Redlock with multiple Redis masters) around multi-shard updates — accept higher latency and complexity.3. Compensating transactions: if script fails mid-flow (e.g., node crash), rely on idempotent operations and a reconciler job that consumes reservation events and compares DB vs cache to repair state.4. Circuit-breaker + graceful degradation: if Redis node unavailable, API can: - Reject reservation requests with clear 5xx/429, or - Fall back to DB-only synchronous transaction (strong consistency but higher latency) — record in DB and then populate cache asynchronously.5. Monitoring & safety: set script execution timeouts, monitor Redis blocked clients, use alerts for long-running scripts, and enable AOF/RDB persistence with careful RPO/RTO planning.Trade-offs:- Using Lua gives low-latency atomic ops but requires careful key design and node sizing to avoid long scripts. Distributed locks add safety but complexity and possible deadlocks. Compensating reconciler simplifies runtime logic at cost of eventual consistency.This architecture balances performance (fast atomic cache ops) and durability (DB reconciliation), with clear fallback paths when cluster or script constraints are hit.
MediumSystem Design
70 practiced
Design a caching strategy for a product catalog service expected to handle 100k reads/sec and 1k writes/sec. The catalog includes static images and frequently-read metadata with occasional price updates. Specify placement (edge CDN for images, app-level caches), cache-aside vs read-through choices, TTLs, invalidation, cache warming approach, eviction policy, and metrics to validate correctness and freshness.
Sample Answer
Requirements recap:- 100k reads/sec, 1k writes/sec- Catalog: static images + frequently-read metadata; occasional price updates require freshnessHigh-level architecture:- Images: serve via edge CDN (CloudFront/Cloudflare) with origin (S3) and cache-control headers.- Metadata: two-tier cache — global distributed cache (Redis Cluster / Memcached) + local in-app LRU for microsecond reads. Primary DB (RDS/NoSQL) as source of truth.- Messaging: pub/sub (Kafka/Redis Streams/SNS) for invalidation events.Cache strategy and rationale:- Metadata: cache-aside (application checks cache first, loads from DB on miss, writes only to DB). On writes (price update), writer updates DB then publishes invalidation event so caches evict/refresh. Cache-aside keeps control in app and avoids accidental stale writes; supports high read volume and bursts.- Alternative: read-through with a cache provider can simplify but couples app to cache; choose if you need centralized TTL/refresh control.TTL and freshness:- Images: Cache-Control: public, max-age=86400-604800 (1 day–1 week); use CDN purge API on image update.- Metadata: short TTLs for price-sensitive fields (5–30 seconds), longer for static fields (5–30 minutes). Implement conditional GETs (ETag/If-Modified-Since) for origin fetches if supported.Invalidation:- On write: DB transaction -> publish invalidation event with object id and version/timestamp -> consumers (app instances, regional caches) evict or refresh entry. For critical price updates, perform immediate cache write-after-update (write-through variant) for that key to reduce staleness window.- CDN images: purge specific path or use versioned filenames (best practice) to avoid mass purge.Cache warming:- Pre-warm hot catalog items at deploy/scale using: - Recent access logs / top-N hot items to bulk populate Redis. - Predictive warming before sales/events using business rules. - On scale-up, background threads fetch keys from DB to local caches rather than serving cold misses synchronously.Eviction policies and sizing:- Global Redis: size by working set; use maxmemory-policy volatile-lru or allkeys-lru depending on whether we set TTLs for all keys. Prefer LFU if access skew with long-tail items.- App-level: small LRU caches with fixed entry count to bound memory.- Monitor memory usage, adjust sharding and replication.Metrics to validate correctness & freshness:- Cache hit rate and miss rate (global + per-region)- Read latency (P50/P95/P99) from cache vs origin- Origin DB QPS (should drop after cache)- Stale-read rate: fraction of requests that served data older than last DB update (compare timestamp/version)- Invalidation latency: time from DB write to cache eviction/refresh- Error rates and 429s from cache/CDN- CDN hit ratio and purge countsOperational considerations and trade-offs:- Strong consistency vs latency: full synchronous write-through on every update increases write latency; targeted immediate refresh for price updates balances freshness and performance.- Use versioned keys (id:version) to make stale reads impossible if you can atomically increment version on write.- Ensure observability and a safe fallback path (circuit-breaker to origin) if cache fails.This design balances high read throughput with acceptable freshness for price updates while minimizing origin load and enabling predictable invalidation and monitoring.
MediumTechnical
42 practiced
A client asks you to recommend between Kafka, Amazon SQS, and RabbitMQ given requirements: durable event storage, long retention, consumer replay, low-latency processing, and moderate ops complexity. Compare the three in terms of ordering guarantees, retention/replay, scaling model, operational cost, and typical use-cases (audit log, task queue, pub/sub notification).
Sample Answer
Recommendation summary: Given durable event storage, long retention, consumer replay, low-latency processing, and moderate ops complexity, Apache Kafka is the best fit in most cases. RabbitMQ and Amazon SQS have strengths for task queues and simple pub/sub but trade off replay/retention or ordering at scale.Comparison:- Ordering guarantees - Kafka: Strong ordering per partition (strict FIFO within a partition). Good for ordered event streams if you pick partition key. - RabbitMQ: Ordering best-effort per queue; can be impacted by clustering/high-availability and consumers acking. - SQS: Standard queue = at-least-once, best-effort ordering; FIFO queues provide strict ordering but limited throughput and additional cost.- Retention / Replay - Kafka: Built for long retention (configurable), cheap replay by consumers using offsets — ideal for event sourcing, audit logs. - RabbitMQ: Designed for transient messages; persistence exists but long-term retention and replay are awkward (not native—requires custom persistence or dead-lettering). - SQS: Messages are ephemeral (max 14 days). No native arbitrary replay; you can re-inject messages but managing retention/replay is manual.- Scaling model - Kafka: Partition-based horizontal scale for throughput; consumers scale by partition count (must plan partitioning upfront). - RabbitMQ: Node/queue sharding and federation; scaling can be operationally trickier and may impact ordering. - SQS: Fully managed, virtually infinite scaling with minimal ops. FIFO queues scale less than standard.- Operational cost / complexity - Kafka: Higher ops (cluster management, Zookeeper/Controllers, monitoring), unless using managed services (MSK, Confluent) which trade cost for simplicity. - RabbitMQ: Moderate ops; easier to run small clusters but gets complex for multi-datacenter durability. - SQS: Lowest ops—fully managed; predictably priced but costs accumulate with high volume and long retention via other AWS storage.- Typical use-cases - Audit log / event store: Kafka (durable, replayable, long retention) - Task queue (work distribution): RabbitMQ or SQS (RabbitMQ for complex routing/ack patterns; SQS for simple, highly-available managed queues) - Pub/sub notifications: Kafka (streaming, durable) or SQS/RabbitMQ for simple fan-out (SNS+SQS in AWS for managed pub/sub)Guidance: Choose Kafka when you need durable long-term storage, consumer replay, and low-latency stream processing and you can accept higher ops or use a managed Kafka. Choose SQS when you want minimal ops and a simple reliable queue (but limited replay/retention). Choose RabbitMQ when you need flexible routing, complex ack/visibility semantics, and moderate retention but not long-term replay.
HardTechnical
39 practiced
Design an adaptive batching algorithm for workers consuming from a stream that balances throughput and latency. Specify metrics to observe (backlog size, avg processing time, SLA latency), control loop logic to grow/shrink batch sizes, safe upper/lower bounds, and strategies to avoid oscillation or starvation under variable load.
Sample Answer
Requirements & goals:- Maximize throughput while keeping per-item latency below SLA (e.g., 95th percentile < L_sla).- Adapt to varying load and processing cost.Observed metrics (per worker / per shard, windowed e.g., 10s):- backlog_size: queued items awaiting processing- batch_size_current: current batch size- avg_proc_time_per_item (μ): exponential moving average of processing time per item- avg_batch_overhead (o): fixed overhead per batch (serialize/deserialize, network)- p95_latency: measured 95th percentile end-to-end latency- arrival_rate λ: items/sec (optional; estimate from backlog delta)Control model (closed-loop PID-like with safety):1. Compute target batch size B_target that optimizes throughput vs. latency: - Estimated per-batch latency = o + B * μ - Want p95_latency ≈ o + B*μ ≤ L_sla_target (use margin α, e.g., 0.9*L_sla) - So B_latency_limit = floor((α*L_sla - o)/μ)2. Use backlog to boost throughput when safe: - If backlog_size > high_watermark H_high and B_latency_limit> B_current → increase - If backlog_size < low_watermark H_low → decrease to reduce latency3. Smooth adjustments: - ΔB = clamp(Kp*(B_target - B_current) + Ki*integral_error + Kd*derivative, -max_step, +max_step) - Apply integer steps and ratelimit changes (e.g., once per control_interval = 1s–5s)Safe bounds:- B_min = 1 (or small lower bound for efficiency)- B_max = min(system_limit, B_latency_limit * safety_margin e.g., 0.8) to avoid hitting SLA- Absolute cap based on memory / processing constraints.Anti-oscillation & stability:- Hysteresis between increase/decrease thresholds (separate H_high/H_low).- Exponential moving averages for μ and p95 to avoid reacting to spikes.- Min dwell time between opposite direction changes (cooldown).- Use conservative integral gain to avoid wind-up; reset integral on large load changes.- When p95 crosses SLA, immediately shrink batch aggressively (emergency decrement) and backoff growth rate.Starvation & variable load handling:- If arrival_rate drops to near-zero but backlog persists, allow B to decay slowly to B_min to reduce latency.- If arrival_rate spikes, opportunistically increase B up to B_latency_limit and B_max; if workers idle frequently, increase B to amortize overhead.- Add deadline-aware priority: if oldest item wait_time > wait_threshold, force immediate dispatch even if batch small.Operational notes:- Per-shard tuning preferred (heterogeneous processing).- Expose metrics and knobs (α, H_high/H_low, control gains) for ops and SRE.- Simulate with representative traces before production rollout; implement canary rollout with telemetry and automatic rollback if SLA breached.
HardSystem Design
44 practiced
For a large event-streaming ecosystem with many producers and consumers, design a schema evolution strategy that supports backward and forward compatibility, safe deployments, and consumer upgrades. Include schema registry use, compatibility modes (backward, forward, full), topic versioning, migration patterns, and automated testing strategies to detect breaking changes.
Sample Answer
Requirements:- Support many producers/consumers, enable backward+forward compatibility, safe deployments, rolling consumer upgrades, automated detection of breaking changes, low latency and high throughput.High-level strategy:1. Central Schema Registry: enforce all schemas (Avro/Protobuf/JSON Schema) in a registry (e.g., Confluent Schema Registry or AWS Glue Schema Registry). All producers/consumers must fetch schemas by ID; store canonical schema + metadata (owner, topic, version, change notes, compatibility mode).2. Compatibility modes: default to "backward" for producers (new schema can read old data) and "forward" for consumers where necessary; use "full" only for tightly controlled topics. Allow per-subject (topic) override and staged mode changes.Topic & versioning model:- TopicName.vN for major-breaking changes. Minor non-breaking changes stay on same topic with schema version increments (embedded schema ID in message header).- Major version triggers a new topic (Topic.v2) with write-forward proxy/dual-writes during migration.Migration patterns:- Non-breaking changes: add optional fields with defaults, enums append values; producers deploy first, then consumers optionally read new fields. Use feature flags for producers to start emitting values.- Backward-incompatible change (semantic): publish new topic version. Implement dual-write in producers (to old and new topics) or use bridge service to replicate and transform messages.- Consumer upgrade: consumers should be able to ignore unknown fields; use tolerant deserialization libraries. Roll upgrades gradually; monitor for errors.Safe deployment practices:- CI gate: schema registry API call to validate compatibility before merge. Require schema owner approval.- Deploy order: (1) register compatible new schema, (2) deploy producers, (3) deploy consumers. For incompatible, create new topic and perform dual-write + consumer migration.- Feature flags and canary producers/consumers for small-scale testing.Automated testing & detection:- Schema contract tests in CI: run synthetic producer/consumer integration tests that serialize with new schema and deserialize with each consumer schema (cross-compat matrix). Fail on incompatibility.- Fuzz testing and property-based tests to detect edge cases (nulls, default values).- Backwards/forwards compatibility matrix job: whenever a schema changes, run automated matrix: old-producer -> new-consumer, new-producer -> old-consumer, etc.- End-to-end staging environment with replay capability to validate consumer behavior under new schemas.Observability & governance:- Monitor serialization/deserialization errors, message rejection rates, and schema usage.- Audit logs in registry, automated changelog and rollback plan.- Governance board for "full" compatibility flips or major topic version approvals.Trade-offs:- Topic versioning increases operational overhead (more topics) but isolates breaking changes.- Dual-write adds cost/complexity but enables smooth migrations.This approach balances backward/forward compatibility, safe rollouts, and automated detection, while giving teams clear operational patterns for schema evolution.
Unlock Full Question Bank
Get access to hundreds of Caching and Asynchronous Processing interview questions and detailed answers.
