Event Driven and Asynchronous Architecture Questions
Designing and operating systems that decouple components using asynchronous messaging and event driven patterns. Covers message queues and brokered communication models (for example Kafka, RabbitMQ, Amazon SQS), publish subscribe patterns, producer consumer workflows, background job and task queue design, and when to prefer asynchronous versus synchronous request response interactions. Includes higher level architectural patterns such as event sourcing, Command Query Responsibility Segregation, sagas for distributed transactions, and patterns for decoupling services. Operational concerns include delivery semantics (at least once, at most once, exactly once), ordering guarantees and partitioning, dead letter queues, retry strategies, idempotency, error handling, monitoring and alerting (for example message lag, queue depth), scaling consumers, throughput and latency trade offs, consistency implications, and common use cases such as email sending, batch processing, file processing, notification delivery, and distributed work coordination.
HardTechnical
107 practiced
An incident: duplicated messages caused double charges for customers. As the Solutions Architect, outline the incident response steps you would take immediately to mitigate harm, the structure of a root-cause analysis you would lead, short-term customer remediation steps, and permanent architectural fixes you would recommend to prevent recurrence. Include stakeholder communication and post-mortem deliverables.
Sample Answer
Situation: Customers experienced duplicate messages that triggered double charges. As the Solutions Architect I would lead the technical response, customer remediation, and long-term fixes.Immediate mitigation (first 0–4 hours)- Stop the bleeding: coordinate with engineering to disable the offending message path (feature flag, pause retry worker, or route to dead-letter) and halt billing pipeline ingestion.- Contain data: snapshot relevant logs, message queues, DB transactions, and payment gateway requests for forensics.- Short circuit retries: if duplicate processing caused by retries, pause retry schedulers and third-party webhook retries (notify vendors).- Triage & monitoring: raise critical alert, increase logging and metrics on idempotency keys, queue depths, and payment attempts.- Communicate: notify Incident Commander, Legal, Compliance, Customer Support, Product, and Sales within 30–60 minutes with known impact and next steps.Root-cause analysis structure (lead within 24–72 hours)- Scope: define impacted customers, time window, systems involved.- Timeline: reconstruct event timeline from logs and traces.- Causal chain: identify initiating fault (e.g., missing idempotency, retry storm, race condition, message duplication in broker).- Contributing factors: config drift, insufficient testing, gaps in SLAs with payment provider, observability blind spots.- Evidence & hypothesis testing: reproduce in sandbox, correlate IDs, validate fixes.- Actionable fixes: short-term mitigations and long-term preventive measures with owners and deadlines.Short-term customer remediation- Stop further charges immediately and reverse pending duplicate transactions where possible (refunds or credits).- Auto-generate customer-facing emails and in-app notices with apology, explanation, and remediation timeline; empower CS with scripts and compensation rules.- Provide a one-click claim process for affected customers and track remediation SLAs.- Stabilize affected accounts and offer temporary safeguards (e.g., block further automated charges).Permanent architectural fixes- Enforce end-to-end idempotency: require unique transaction IDs stored and checked atomically before charging.- At-least-once → effectively-once handling: implement deduplication store (strongly consistent DB or idempotency table) and transactional outbox pattern.- Harden message handling: use exactly-once semantics where supported, or include sequence numbers and dedup windows; make consumers idempotent.- Improve retries: exponential backoff, jitter, circuit breakers between services and payment gateways; limit concurrent retries.- Observability & testing: add SLOs, alerts for duplicate-payments, distributed tracing linking payment IDs, chaos tests and integration tests simulating broker duplicates.- Operational controls: feature flags, kill switches, runbooks, and playbooks for payment incidents.Stakeholder communication- Immediate: one-page incident brief to Execs, Legal, Compliance, and CS within hour; public status page update.- Ongoing: hourly then bi-hourly updates during mitigation; daily status during RCA.- Customer communication: initial acknowledgement within 2 hours, remediation plan within 24 hours, final remediation confirmation and root-cause summary when available.- Post-incident: present RCA and remediation roadmap to Execs and affected customers.Post-mortem deliverables- Executive summary (impact, customer count, financial exposure).- Detailed timeline and causal analysis with supporting logs/traces.- Short-term actions taken and evidence they mitigated risk.- Permanent remediation plan with owners, priorities, and deadlines.- Updated runbooks, test plans, and architecture diagrams.- Lessons learned and metrics to measure prevention effectiveness (e.g., duplicate charge rate target).- Follow-up verification report after fixes are deployed and validated.This approach balances immediate customer protection, rigorous RCA, clear communications, and durable architectural changes to prevent recurrence.
HardSystem Design
78 practiced
Design a globally distributed event store for e-commerce orders that must support 10M orders/day worldwide, provide strong per-order ordering, allow querying recent events per customer, and offer cross-region read availability for low-latency reads and DR. Describe partitioning, replication strategy (async vs sync), consistency and conflict resolution, and expected latency trade-offs.
Sample Answer
Requirements (clarify): 10M orders/day (~120/s steady, higher bursts), strong ordering per-order (events for same order must be totally ordered), ability to query recent events per customer, global low-latency reads and DR.High-level approach:- Provide single-writer partitions to guarantee per-order ordering, synchronous intra-region durability, and async cross-region replication for read locality and DR. Maintain materialized recent-event views per customer for fast queries.Partitioning:- Primary partition key = order_id (or order_id + version). All events for an order go to the same shard to guarantee strict ordering and single-writer sequencing.- Secondary index / materialized view keyed by customer_id that is updated via the event stream (CDC / changefeed) to support "recent events per customer" queries. This view is sharded by customer_id for scale.- Shard sizing: assume each shard handles ~1000-5000 writes/sec; for 10M/day use ~200-500 shards (adjust for bursts & retention).Replication strategy:- Within-region: replica set (3+ replicas across AZs). Use synchronous or semi-sync replication with a write quorum (write to leader + 1 follower) to ensure durability and low-latency commit in the region.- Cross-region: async replication from region leader to read replicas in other regions via append-only log shipping (txn log / changefeed). Read replicas serve low-latency reads; promote replicas for DR.- Write routing: route writes to the order's primary region/leader. For geo-local writes from other regions, either: - Route to primary region (strongest consistency), or - Accept local write and forward to primary (introduces complexity). Prefer single-writer per-order and route writes to designated primary region (can be computed by consistent hashing or customer-region preference).Consistency & conflict resolution:- Guarantee: Strong per-order linearizability by single-writer + sequencing at the shard leader.- Cross-region async replication => eventual global consistency. To avoid conflicts, prevent multi-master writes for same order; if multi-master unavoidable, use application-level conflict resolution with version vectors + deterministic merge rules (e.g., command-sourcing where commands are re-ordered by timestamp + origin priority), or CRDTs for commutative event types (rare for orders).- Idempotency: Each event includes a monotonically increasing sequence number and UUID to detect duplicates on retries.- Materialized views updated in-order (apply sequence numbers) to ensure read-model consistency per-order.Query patterns:- Recent events per customer served from the customer-event view in local region (async but typically up-to-date within 100s ms–seconds).- For canonical per-order state, read from primary shard or strongly-consistent read API that contacts leader.Latency trade-offs (expected):- Local write commit (leader + AZ follower quorum): ~10–50ms depending on infra and sync config.- Cross-region durability (async replication): replication lag typically tens to hundreds ms; worst-case seconds under load or network partition.- Local read from read-replica: single-digit to 50ms depending on region proximity.- Strongly-consistent read that queries leader in another region: added cross-region RTT (50–200ms).- Synchronous cross-region replication to get strong global durability would add 100–300ms per write — too costly; prefer async for cross-region.DR and failover:- Maintain region leaders and ability to promote read-replicas to leader for affected partitions. Use safe promotion: ensure all pending logs are applied or use committed sequence numbers to avoid losing events.- Run periodic log-trimming and retention policies; archive cold events to object storage.Operational notes / trade-offs:- Single-writer per-order simplifies correctness and removes conflict resolution complexity at cost of routing complexity and possible increased latency when primary is remote.- Materialized read views trade strict freshness for low-latency queries; design SLAs per API (stale-by-100ms vs strongly-consistent).- Monitor replication lag, apply backpressure on producers, and allow per-order hot-shard rebalancing.This design gives strict per-order ordering and global read availability with predictable latency trade-offs: low-latency local writes/reads within primary region, low-latency reads from local async replicas, and controlled eventual global convergence.
MediumSystem Design
144 practiced
A Kafka topic has 100 partitions. Your system normally handles 10k events/sec and must scale to 1M events/sec during spikes. Describe how you would scale consumers to handle load spikes: consumer group sizing, partition assignment strategies, stateful vs stateless consumers, rebalancing impact, autoscaling thresholds, and trade-offs in latency vs throughput. Include ways to reduce rebalance disruption.
Sample Answer
Requirements clarification:- Topic: 100 partitions, normal load 10k events/s, spike to 1M events/s.- Need design for scaling consumers (low disruption, acceptable latency/throughput trade-offs).High-level approach:- Partition count drives max parallelism: 100 partitions ⇒ at most 100 active consumer instances in one group. For 1M eps we need far more parallelism, so first decision: increase partition count (e.g., 500–2000) and/or enable multi-threaded processing per consumer. Also review partition key strategy so load is balanced.Consumer group sizing and partition assignment:- Aim for 1 consumer task per partition for simplicity (1:1 assignment gives max parallelism and simple semantics).- If partition count cannot be increased, run fewer consumers and have each consumer process partitions concurrently with an internal worker pool (bounded thread pool), ensuring ordering per partition if required.- Use the sticky assignor or Range/CooperativeSticky assignor to minimize movement on rebalance.- Consider Kafka Streams or ksqlDB if you need stream abstraction and state management.Stateful vs stateless consumers:- Prefer stateless consumers where possible (idempotent processing, external durable sinks) — easiest to autoscale.- For stateful needs (aggregations/windowing), externalize state to RocksDB + changelog topics (Kafka Streams) or to an external store (Redis/Cassandra). This enables faster failover, fewer rebalance impacts, and scaling by adding instances with state restore.- If using local state, enable standby replicas (Kafka Streams) or use static membership to avoid full rebalancing.Rebalancing impact and reduction strategies:- Use cooperative rebalancing (incremental cooperative assignor) to avoid stop-the-world revocations.- Enable static membership (consumer.instance.id) so short restarts don’t trigger full group rebalance.- Tune session.timeout.ms and heartbeat.interval.ms (longer time reduces accidental rebalances but increases failure detection latency).- Use sticky assignor to preserve assignment stability.- Warm standby pool: keep warm idle consumers (joined to group but with zero assignment?) — instead, use static members or maintain a pool of pre-warmed instances that can take over without long startup.- Reduce max.poll.interval.ms by tuning batch sizes; for long processing per record use async commit patterns.Autoscaling thresholds and policies:- Primary signals: consumer lag, per-consumer throughput, CPU/memory, and end-to-end latency.- Example scaling rules: - Scale out when average consumer lag > 10k msgs for >1 minute OR broker-end throughput per partition > target (e.g., partitions delivering >5k eps) OR CPU > 70%. - Scale in when lag < 500 msgs and CPU < 30% for 5 minutes.- Use horizontal scaling at orchestration layer (Kubernetes HPA/VPA or custom controller) and coordinate with partition reassignments if changing number of consumers vs partitions.- When increasing partitions dynamically, perform controlled partition reassignment (rate-limited) and rebalance windows during low-traffic periods.Latency vs throughput trade-offs:- To maximize throughput: increase batch sizes (fetch.max.bytes, max.partition.fetch.bytes), increase max.poll.records, use asynchronous I/O, allow bigger in-flight batches — this increases end-to-end latency and memory usage.- To minimize latency: smaller fetch sizes, smaller max.poll.records, more consumers (more partitions), prefer per-record processing — reduces throughput efficiency.- Choose per-use-case: for spikes where throughput matters (backfill/ingestion), favor batching; for low-latency needs (real-time alerts), favor smaller batches and more parallelism.Operational notes and trade-offs:- Increasing partition count increases broker leader metadata and controller load and can cause more network chatter — balance partition count vs broker capacity.- If 100 partitions are fixed and 1:1 consumer-per-partition still insufficient, consider sharding downstream processing within a consumer, or use multiple topics/prefixes and fan-out producers.- Ensure idempotency and exactly-once where needed: enable idempotent producers and transactions if consumer side writes back to Kafka.Concrete numbers example:- 1M eps across 100 partitions ⇒ 10k eps/partition. If a single consumer thread can safely process 2k eps (including downstream IO), you need ~5 threads per partition or increase partitions to 1000 to allow 1000 consumers at 1:1. Prefer increasing partitions (e.g., to 1000) + consumer autoscaling to handle spikes smoothly.Summary actions:1. Increase partitions to enable parallelism; ensure even key distribution.2. Prefer stateless consumers or externalized state; use Kafka Streams with changelog for stateful workloads.3. Use cooperative sticky assignor, static membership, and tune timeouts to minimize rebalance disruption.4. Autoscale based on lag/throughput/CPU with conservative thresholds and warm-up strategies.5. Balance batching parameters to trade latency vs throughput depending on SLA.
EasyTechnical
84 practiced
What is a Dead Letter Queue (DLQ)? Describe a DLQ-based architecture for handling poison messages in a task processing system, including how you would instrument metrics and alerts, automate quarantining, and implement a manual review and replay workflow. Mention any retention and security considerations.
Sample Answer
A Dead Letter Queue (DLQ) is a secondary queue where messages that cannot be processed (poison messages) are routed after repeated failures, preserving them for inspection without blocking the main pipeline.DLQ-based architecture (high level):- Producer → Primary Task Queue → Worker(s)- On worker failure: increment attempt count; if attempts < threshold retry/backoff; if >= threshold, move message to DLQ with metadata (error, stack, attempts, timestamp, worker id).- DLQ backed by durable storage (e.g., Kafka topic with separate retention, SQS dead-letter queue, or an object store reference).Instrumentation & alerts:- Metrics: DLQ size, DLQ arrival rate, poison ratio (DLQ arrivals / total processed), avg time in DLQ, replay success rate.- Logs: per-message failure reasons and correlation IDs.- Alerts: high DLQ growth rate, sustained elevated poison ratio, sudden spike from a single producer/service.- Dashboards: time series + top failing error types and producers.Automated quarantining:- When moving to DLQ attach metadata and optionally quarantine by tagging source service and autocreate issue/ticket in incident system.- Auto-suppress retries for messages matching known non-retriable error patterns.- Optionally run automated enrichment/triage job that classifies messages (transient vs permanent) and auto-archive or mark for manual review.Manual review & replay workflow:- Provide a secure UI showing DLQ entries, filters (error type, source, time), and message preview with schema validation.- Review actions: mark as discard, fix & requeue (edit payload), replay (single or batch), or escalate to owner.- Replay path: validate schema, run through sandboxed test worker, then push back to primary queue with metadata to avoid loops; record audit trail.Retention & security:- Retention: set retention based on compliance (e.g., 30–90 days), with long-term archive for audit compliance. Rotate/expire automatically.- Security: encrypt messages at rest and in transit, role-based access to DLQ UI, redact PII in previews, keep audit logs for access and replay actions, and apply least privilege to service accounts that can move/replay messages.Trade-offs: durable DLQ storage increases cost but preserves forensic ability; automation reduces toil but requires careful classification to avoid data loss. This approach balances reliability, observability, and operational control.
MediumTechnical
77 practiced
For a travel booking platform coordinating flights, hotels, and car rentals, compare choreography and orchestration approaches for sagas. Discuss coupling, observability, fault isolation, retry and compensation complexity, and scenarios where central orchestration might be justified despite introducing a single control plane.
Sample Answer
Situation: Designing a saga for a travel booking platform (flight + hotel + car) requires choosing between choreography (event-driven, peer-to-peer) and orchestration (central coordinator).Comparison:- Coupling: - Choreography: Low central coupling; services subscribe to events and act independently. Good for independent teams and incremental rollout. - Orchestration: Higher logical coupling to the orchestrator; services implement well-known commands/compensations. Easier to enforce workflow policies.- Observability: - Choreography: Harder to trace end-to-end—requires distributed tracing and consistent event schemas. More work to build a holistic view. - Orchestration: Easier centralized logging/metrics; orchestrator can emit complete saga state and timelines.- Fault isolation: - Choreography: Failures can be isolated to individual services, but cascading retries/events may produce inconsistent states if not carefully designed. - Orchestration: Orchestrator mediates retries and timeouts, reducing accidental cascades; single point of control but not necessarily single point of failure if highly available.- Retry & compensation complexity: - Choreography: Each service must own its retry/compensation behavior; coordinating timing and idempotency is complex. Hard to implement global backoff and ordered compensations. - Orchestration: Centralized retry policies, ordered compensations, and coordinated timeouts simplify logic. Compensations are easier to sequence deterministically.When central orchestration is justified:- Business requires strict transactional semantics or predictable rollback ordering (e.g., charge authorization must be reversed before cancelling hotel).- Regulatory/audit needs demand central audit trails and guaranteed workflow history.- Complex conditional flows where multiple services’ states determine next steps—simpler to encode in one orchestrator.- Teams lack mature event infrastructure or consistent tracing; faster delivery via orchestrator reduces integration risk.Recommendation: For travel booking start with choreography if services are small, independent, and scale is high; prefer orchestration when auditability, complex compensations, or predictable sequencing are priorities. Consider hybrid: lightweight orchestrator for high-risk flows and events for simpler interactions. Ensure idempotency, standardized events/commands, distributed tracing, and HA orchestrator if chosen.
Unlock Full Question Bank
Get access to hundreds of Event Driven and Asynchronous Architecture interview questions and detailed answers.
