Design and operational patterns for reducing latency and decoupling components using caching layers and asynchronous communication. For caching, understand when to introduce caches, cache placement, eviction policies, cache coherence, cache invalidation strategies, read through and write through and write behind patterns, cache warming, and trade offs between consistency and freshness. For asynchronous processing and message driven systems, understand producer consumer and publish subscribe patterns, event streaming architectures, common brokers and systems such as Kafka, RabbitMQ, and Amazon Simple Queue Service, and the difference between queues and streams. Be able to reason about delivery semantics including at most once, at least once, and exactly once delivery, and mitigation techniques such as idempotency, deduplication, acknowledgements, retries, and dead letter queues. Know how to handle ordering, partitioning, consumer groups, batching, and throughput tuning. Cover reliability and operational concerns such as backpressure and flow control, rate limiting, monitoring and alerting, failure modes and retry strategies, eventual consistency and how to design for it, and when to choose synchronous versus asynchronous approaches to meet performance, scalability, and correctness goals.
HardSystem Design
42 practiced
You must integrate a synchronous primary write (user profile update) with asynchronous fan-out to several downstream systems (search index, analytics). Architect a reliable sync+async integration that guarantees the primary write is durable and the fan-out is delivered at-least-once while minimizing duplication and ensuring observability. Discuss outbox, change-data-capture (CDC), and transactional options.
Sample Answer
Requirements & constraints (clarify):- Primary write (user profile update) must be synchronous and durable before returning success.- Fan-out to search/analytics is asynchronous but must be delivered at-least-once.- Minimize duplicate side-effects, provide observability, be scalable and operationally simple for SRE ownership.High-level recommendation (summary):Use the transactional outbox pattern (app writes profile + outbox row in one DB transaction) + durable message broker (Kafka) as the async transport. Optionally use CDC (Debezium) to produce to Kafka instead of an application poller if you want zero app changes. Ensure consumer-side idempotency and observability; use DLQs, monitoring, tracing, and backpressure controls.Architecture components:1. Primary service + relational DB - Handles synchronous profile update; in same DB transaction, insert an outbox record (topic, payload, status, idempotency_key, created_at).2. Outbox publisher - Either an in-process publisher that reads committed outbox rows and writes to Kafka using transactional producer OR a separate outbox-service/poller (safer operational separation). - Mark outbox rows as published in same DB transaction or use Kafka transaction to avoid loss.3. Kafka (durable log) - Acts as buffer, supports partitions, retention, replay.4. Downstream consumers - Search indexer, analytics loader subscribe to topics, apply messages, commit offsets after successful apply. Implement idempotency (see below).5. Observability & Ops - Metrics (outbox queue depth, publish latency, publish failures, consumer lag) - Distributed tracing (trace id passed in payload) - Alerts: growing outbox, consumer lag > threshold, DLQ events - DLQ and replay tools for manual reprocessingData flow (typical):- API: receive update → begin DB transaction → update user row + insert outbox row → commit → return 200.- Outbox publisher: poll for unpublished rows → publish to Kafka (optionally in Kafka transaction) → update outbox row to published OR rely on CDC to avoid second update.- Consumers: read from Kafka → apply to target (search/analytics) → acknowledge/commit. On failures, retry with exponential backoff; after N attempts send to DLQ.Why outbox vs CDC vs 2PC:- Two-phase commit across DB and messaging is complex, brittle, and not recommended at scale.- Outbox pattern: simple, transactional consistency between DB state and intent to publish. Application needs to add outbox writes and a publisher component. Guarantees at-least-once delivery (if publisher retries); duplicates possible, so consumers must be idempotent.- CDC (Debezium/binlog -> Kafka): no app changes; captures committed DB changes and produces to Kafka reliably. It’s robust but operationally heavier (binlog access, schema changes handling), and you must map row-level changes to business events. CDC can replace the outbox publisher; it still results in at-least-once delivery semantics (dedupe required).- Trade-offs: Outbox gives explicit domain events and easier schema control. CDC is lower-app-effort and reduces developer mistakes but needs careful mapping and evolution.Minimizing duplication / achieving effective idempotency:- Give each event a globally unique event_id (UUID) and include user_id + sequence_number or version (increment on update).- Consumers should store last-applied event id or last version per entity in the downstream system (e.g., search index metadata store) and reject duplicates (if event_id seen) or ignore older versions (if version <= applied_version).- Use upserts with conditional writes (compare-and-swap) where supported (Elasticsearch partial updates with version, analytics store dedupe with unique key).- If downstream is not transactional, make consumer operations idempotent by design (idempotent writes, dedupe tables).Delivery guarantees and Kafka features:- Kafka provides durable at-least-once delivery by default. For stronger semantics: - Use producer acks=all and retries. - Use Kafka transactions and idempotent producers to avoid duplicates from publisher restarts (helps exactly-once semantics between producer and Kafka). - Consumers should commit offsets only after successful application.Failure modes & handling:- Publisher crash after DB commit but before publish → outbox still has row → publisher will pick it up and publish.- Publish succeeded but outbox update failed → if using Kafka transaction to atomically write to Kafka and mark outbox, choose one of: - Use Kafka transaction to include a tombstone message that indicates published state OR - Let publisher idempotently mark outbox published via update; if that update fails, monitoring and retry will handle.- Consumer apply fails after consuming → message will be retried; idempotency protects correctness. After repeated failures, send to DLQ and alert.Scalability and performance:- Batch outbox publishes for throughput.- Partition Kafka by user_id or tenant to preserve ordering where required.- Horizontal scale publishers and consumers. Monitor partition assignment and rebalancing impact.- Use rate-limiting/backpressure to protect downstream systems (throttle, backlog metric).Observability and SRE controls:- Dashboards: outbox size by topic, publish rate, publish latency, Kafka produce failures, consumer lag, retries, DLQ counts.- Traces: propagate trace_id in event metadata; instrument publisher and consumers.- Alerts: outbox backlog > SLO, consumer lag > threshold, DLQ spike, publish error rate > threshold.- Runbooks: steps for replay (re-publish outbox rows, reprocess DLQ), leader-election for publisher, schema migration procedures.- SLOs: define success percent for async delivery within time window (e.g., 99.9% delivered within 10 min), error budget for DLQ rate.Concrete deployment choices (example):- DB: PostgreSQL for user and outbox tables.- Publisher: sidecar service or cron/poller in Kubernetes; use Kafka transactional producer.- Broker: Kafka cluster (or managed: Confluent/MSK) with replication factor 3.- CDC alternative: Debezium reading Postgres WAL -> Kafka topics (if you prefer no app changes).- Consumers: Kubernetes deployments with liveness/readiness probes, circuit breakers for downstream failures, and persistent offset commit storage (Kafka consumer groups).Summary of trade-offs:- Outbox + Kafka: robust, explicit control, easier to reason about events and schema; small app change required.- CDC: minimal app changes, strong durability, operational complexity around binlog and schema mapping.- 2PC: avoid—complex and fragile.- Always design consumers to be idempotent and instrument the system for real-time visibility and replay operations under SRE control.
EasyTechnical
49 practiced
You must decide between making an action synchronous (client waits for DB write) vs enqueueing an asynchronous job to perform the same action later (e.g., send transactional email). Describe a decision framework covering user experience, correctness, failure modes, SLO impact, and operational complexity. Give two concrete examples (one that should be sync, one async) and justify.
Sample Answer
Decision framework (stepwise):- User experience (latency/feedback): Is the user expecting immediate confirmation? If yes, prefer sync. If user can tolerate delay and gets progress/ack, async is fine.- Correctness & consistency: Does downstream correctness require the write to be persisted before returning (strong consistency, read-after-write)? If yes → sync. If eventual consistency is acceptable → async.- Failure modes & recoverability: Can you retry/compensate easily? Async jobs allow retries/backoff and dead-letter queues; sync failures require immediate rollback and affect user flow.- SLO impact: Sync work increases tail latency and directly counts toward request SLOs; async shifts burden off request path but increases background-job SLOs and operational surface.- Operational complexity: Async needs job queues, workers, visibility, DLQs, monitoring; sync is simpler but makes scaling/DB load directly coupled to request rate.- Observability & alerting: Ensure metrics for both request path and worker pipeline; plan alerts for queue depth, retry rates, processing latency.Concrete example — Sync (should be synchronous):- Updating a user’s password reset confirmation (persisting new hashed password). Justification: Security/correctness demands the write be durable before informing the user. Read-after-write must succeed (immediate login/invalidating old tokens). Failure must be surfaced immediately so user can retry; SLOs tolerate slightly higher latency for correctness. Operationally simple and safer for security-critical actions.Concrete example — Async (should be asynchronous):- Sending a transactional email (welcome or receipt) after order completion. Justification: UX tolerates small delay; the order persistence is the critical correctness point and should be sync. Email sending can fail transiently and benefit from retries, rate limiting, and batching in a worker. Moving email to async reduces request tail latency and keeps request SLOs healthy. Operational cost: implement queue, workers, DLQ, and instrumentation, which is justified given scalability and resilience benefits.Guiding heuristics:- If user is blocked without immediate result or security/integrity at stake → sync.- If action is side-effecty, retryable, or can be reconciled later → async.Always instrument both paths, set SLOs for request latency and background processing, and provide clear user feedback (progress, queued confirmation, or error) to maintain UX.
MediumTechnical
55 practiced
Compare Kafka, RabbitMQ, and Amazon SQS for building an event-driven system. Cover core semantics (streams vs queues), throughput/latency characteristics, ordering guarantees, persistence and retention, consumer scaling, exactly-once capabilities, and operational trade-offs for SREs.
Sample Answer
High-level summary: Kafka is a distributed log (stream) optimized for high-throughput, long-retention streaming and replay; RabbitMQ is a traditional AMQP broker (queue-first) good for complex routing and low-latency RPC-style patterns; Amazon SQS is a managed at-least-once queue service focused on simplicity and operational ease.Core semantics- Kafka: append-only partitioned log (streams); consumers read offsets; decouples storage and consumption.- RabbitMQ: queues with push delivery and complex exchange-routing (fanout/topic/direct).- SQS: message queue (pull via long-polling); simple FIFO or standard queue semantics.Throughput / latency- Kafka: very high throughput (MBs–GBs/s cluster), low producer/consumer latency (ms), scales by partitions.- RabbitMQ: low latency (single-digit ms) but throughput limited by broker CPU/IO and routing complexity.- SQS: lower throughput/variable latency (tens–hundreds ms); managed scaling but higher tail latency.Ordering guarantees- Kafka: ordering within a partition; need partitioning strategy to preserve order.- RabbitMQ: ordering per queue but not guaranteed under requeues or clustering without single consumer.- SQS: standard queues: best-effort ordering; FIFO queues: strict ordering but lower throughput and dedup window.Persistence and retention- Kafka: durable on-disk retention configurable by time/size; supports long-term retention and replay.- RabbitMQ: persistent messages to disk (slower), but queues are ephemeral by design—long retention taxes broker.- SQS: durable for configured visibility/retention window (max 14 days); server-managed durability.Consumer scaling- Kafka: scale horizontally by adding consumers up to partition count (consumer groups).- RabbitMQ: supports multiple consumers per queue with competing-consumer model; scaling requires careful queue topology.- SQS: effectively unlimited consumers; each poll returns messages; scale is managed.Exactly-once capabilities- Kafka: supports EOS for producers/consumer side transactions with idempotent producers and transactional writes (complex).- RabbitMQ: at-least-once by default; can implement idempotency or use ack/nack patterns; no native EOS.- SQS: at-least-once delivery; use deduplication (FIFO queues) or idempotent consumers for effective once semantics.Operational trade-offs for SREs- Kafka: operationally heavy—Zookeeper/KRaft, careful capacity planning (disk, partitions), monitoring (ISR lag, GC), complex upgrades; excellent observability opportunities (offsets, consumer lag). Good for retention/replay, but needs strong automation (backups, rebalancing).- RabbitMQ: moderate ops—single broker or cluster with mirrored queues; care for flow control, queue explosion, and memory alarm; simpler than Kafka but routing complexity increases operational burden.- SQS: minimal ops—fully managed, auto-scaling, SLA-backed, but limited control over internals, metrics are coarse, and debugging distributed message flow harder.When to choose- Kafka: stream processing, event sourcing, high-throughput pipelines requiring replay.- RabbitMQ: complex routing, request/response patterns, low-latency messaging within a controlled cluster.- SQS: simple decoupling with minimal ops overhead, variable latency acceptable, or when using AWS ecosystem.As an SRE, factor in operational cost, required SLAs, need for replay/retention, and team expertise before selecting.
HardTechnical
45 practiced
Design a deduplication subsystem for a high-throughput message ingestion pipeline that uses a Bloom filter to reduce duplicate checks against persistent storage. Provide pseudo-code for the ingestion flow, explain how you combine probabilistic and durable structures, and analyze false-positive consequences and mitigation strategies.
Sample Answer
Approach: Use an in-memory Bloom filter as a fast front-line deduper to avoid hitting durable store for most duplicates. On positive Bloom hits (possible duplicate) consult durable store (e.g., RocksDB, Cassandra) to confirm. On Bloom negatives, accept and insert into both Bloom filter and durable store. Periodically persist Bloom (or rebuild) and use a write-ahead-log (WAL) for durability/repair.
python
# Pseudo-code (Python-like)
class Deduper:
def __init__(self, bloom, store, wal):
self.bloom = bloom # in-memory probabilistic filter
self.store = store # durable key-value store with TTL/meta
self.wal = wal # append-only log for recovery
def ingest(msg_id, payload):
# fast path: bloom negative -> accept
if not self.bloom.might_contain(msg_id):
self.wal.append(("ingest", msg_id)) # durable intent
self.store.put(msg_id, payload) # idempotent insert
self.bloom.add(msg_id)
return "accepted"
# possible duplicate -> confirm against durable store
record = self.store.get(msg_id)
if record is None:
# false positive in bloom: accept and repair
self.wal.append(("ingest", msg_id))
self.store.put(msg_id, payload)
self.bloom.add(msg_id)
return "accepted (bloom false positive)"
else:
return "duplicate"
Key concepts:- Bloom provides constant-time low-latency negative checks; durable store is source-of-truth.- WAL ensures recovery and eventual consistency (rebuild Bloom on restart).- Use sharded Bloom filters (per partition) to limit size and reduce false-positive rate.Complexity:- Time: O(1) Bloom operations; occasional O(1) store lookup on positives.- Space: Bloom size tradeoff: m = - (n ln p) / (ln2)^2.False-positive consequences & mitigations:- Consequence: legitimate messages may be misclassified as duplicates only if Bloom incorrectly indicates presence and durable store also lost the key — but our flow confirms against store, so false positives only cause extra store lookups (performance cost) not data loss.- Mitigations: - Tune Bloom parameters (m, k) for target false-positive p given throughput and memory. - Use TTLs and partitioned filters to bound memory and rebuild frequency. - Persist/periodically snapshot Bloom from WAL-store to rebuild quickly after failures. - For safety-critical paths, use a monotonic idempotency token persisted before acknowledgement (two-phase write) to avoid any chance of dropping unique messages.Operational considerations:- Monitor bloom false-positive rate, store hit rate, latency tail; autoscale shards and tune m/k.- During restarts: rebuild Bloom from store or WAL in background; accept higher false positives until rebuilt.
HardTechnical
49 practiced
Architect an observability stack specifically for caches and asynchronous pipelines. List the essential metrics, logs, traces, dashboards, and alerts you would implement (e.g., cache hit/miss, eviction rate, consumer lag, DLQ counts, processing latency p50/p95/p99). Describe sample alert rules and runbook actions for common failures.
Sample Answer
Requirements & goals:- Detect cache degradation (latency, correctness), pipeline slowdowns, data loss, backpressure.- Provide signal correlation (metrics ↔ traces ↔ logs) and alerting that separates symptomatic vs. actionable alerts.- Meet SLOs: e.g., cache availability 99.9%, pipeline end-to-end latency p95 < X.High-level observability architecture:- Metrics: Prometheus (or managed equivalent) scraping app and infra exporters.- Tracing: OpenTelemetry → Jaeger/Tempo.- Logs: Structured JSON to ELK/Opensearch or Loki.- Dashboards/Alerts: Grafana with alertmanager.- Alert enrichment: runbook links, playbook automation (PagerDuty).- Correlation: include trace_id, message_id, topic/partition, cache_key in logs/metrics.Essential metrics (per cache / per pipeline component):- Cache: hit_rate (hits / lookups), miss_rate, eviction_rate, TTL_expirations, load_latency_p50/p95/p99, stale_reads, size_bytes, memory_utilization, keys_count, error_count (e.g., connection errors).- Pipeline (per consumer/topic/partition): consumer_lag, consumption_rate, throughput_msgs/s, processing_latency_p50/p95/p99, error_rate, retries_count, DLQ_count, checkpoint_commit_latency, inflight_messages, backpressure_signals (queue_depth).- Infra: CPU, memory, disk_io, network_io, GC_pause_ms, file_descriptors.Logs:- Structured events: ingestion_time, processing_start/end, trace_id, message_id, cache_key, error_type, stacktrace, retry_count.- DLQ entries with full payload and original metadata.- Cache client warnings (timeouts, protocol errors).Traces:- End-to-end trace from producer → broker → consumer → downstream; spans for cache get/set, DB fallback, external calls. Capture timings and error tags.Dashboards:- Cache overview: hit/miss, latency p50/p95/p99, evictions, mem usage, top keys.- Pipeline health: consumer lag heatmap, throughput, latency percentiles, DLQ trend, per-partition lag.- End-to-end flow: trace sampling, bottleneck heatmap (component latencies).- Alert status & SLO burn rate.Alerting: tiered (P0,P1,P2) with rate/absolute thresholds and burn rate rules.Sample alert rules:1) P0: Cache outage- Condition: cache_error_rate > 5% AND successful_requests < 50% of baseline for 5m- Action: page on-call, mute downstream low-priority alerts- Runbook: verify cache process & network, check connection pool exhaustion, restart cache node if unresponsive, fail open to DB if safe, increase replicas, roll back recent deploys.2) P0: Consumer stuck / high consumer lag- Condition: consumer_lag > 10k msgs OR lag growth rate > 1000 msgs/min sustained 5m- Action: page on-call- Runbook: check consumer group status, check partition assignment, check broker health and ISR, inspect offsets via admin tools, restart consumer, scale consumer instances, investigate DLQ spikes.3) P1: Processing latency breach- Condition: processing_latency_p95 > SLO for 10m- Action: notify, create ticket- Runbook: sample traces for slow spans (DB/cache/external), check downstream saturation, roll traffic to fallback path, add diagnostics (increase trace sampling) and a temporary increase in logging.4) P1: DLQ rate increase- Condition: DLQ_count_rate > 10x baseline for 5m- Action: notify & create incident if payloads critical- Runbook: inspect sample DLQ messages, run canary reprocess locally, check schema changes, revert producer changes, mitigate by pausing producer or routing to staging reprocess.5) P2: Cache eviction surge- Condition: eviction_rate > threshold AND memory_utilization > 85% for 10m- Action: alert to owners- Runbook: identify hot keys, enable sampling of top keys, increase cache capacity or TTLs, implement cache warming, adjust eviction policy.Operational best practices:- SLOs + burn-rate alerts to avoid alert fatigue.- Alert rules include both absolute thresholds and anomalies (seasonal baselining).- Automated mitigations: circuit-breaker to bypass cache, auto-scale consumers, automated restart with backoff.- Post-incident: add dashboard panels, increase trace sample rate for impacted flows, add guardrails (schema validation, canary deploys).- Drill runbooks quarterly and keep runbook steps executable (CLI commands, playbook links).
Unlock Full Question Bank
Get access to hundreds of Caching and Asynchronous Processing interview questions and detailed answers.
