Event Driven and Asynchronous Architecture Questions

Designing and operating systems that decouple components using asynchronous messaging and event driven patterns. Covers message queues and brokered communication models (for example Kafka, RabbitMQ, Amazon SQS), publish subscribe patterns, producer consumer workflows, background job and task queue design, and when to prefer asynchronous versus synchronous request response interactions. Includes higher level architectural patterns such as event sourcing, Command Query Responsibility Segregation, sagas for distributed transactions, and patterns for decoupling services. Operational concerns include delivery semantics (at least once, at most once, exactly once), ordering guarantees and partitioning, dead letter queues, retry strategies, idempotency, error handling, monitoring and alerting (for example message lag, queue depth), scaling consumers, throughput and latency trade offs, consistency implications, and common use cases such as email sending, batch processing, file processing, notification delivery, and distributed work coordination.

HardSystem Design

94 practiced

Design a distributed rate limiter enforcing per-customer fairness across many stateless consumer instances. Describe a scalable design using Redis with token-bucket semantics, Lua scripts for atomicity, and consistent hashing for sharding customer state. Discuss correctness, race conditions, failover, and how to handle bursts and global quotas.

Sample Answer

Requirements & constraints:- Per-customer token-bucket semantics (rate r tokens/sec, burst capacity b).- Stateless consumers (many) must enforce fairness across all instances.- Scales to millions of customers, low latency, strong correctness (no over-issuance).- Use Redis, Lua for atomic updates, consistent hashing to shard state.

High-level design:- Shard customers by customer_id using consistent hashing to N Redis shards (clustered nodes or separate logical DBs). Each shard holds token-bucket state per customer: {tokens, last_ts, rate, burst, optional global_quota_window_counters}.- Consumers are stateless. On request, consumer computes shard for customer_id, sends a single Redis EVALSHA (Lua) script to that shard to tryConsume(k) tokens.

Lua script (atomic):- Load bucket (tokens, last_ts).- Recompute tokens = min(burst, tokens + (now - last_ts) * rate).- If tokens >= k: tokens -= k; persist tokens + last_ts; return allowed.- Else return denied and remaining tokens estimate.- Also increment per-customer global quota counters (sliding window or leaky-bucket) within the same script if global quota applies—ensuring atomicity of token decrement + quota increment.

Correctness & race conditions:- Single Redis shard + Lua script ensures atomic check-and-decrement; avoids races across consumers hitting same customer state.- Consistent hashing ensures all operations for a customer go to same shard; moving shards requires migrating state (see failover).- Use EVALSHA to reduce latency; include optimistic re-try on MOVED/ASK for cluster.

Failover & availability:- Redis cluster with replication and majority writes. Use Redis primary for writes; replicas for reads (but writes must go to primary).- If primary fails, Redis failover promotes replica; consumers must handle transient errors and retry with exponential backoff. To reduce lost tokens during failover, persist state frequently (each operation writes).- For shard resizes (adding nodes), use consistent-hashing ring with minimal key movement; perform background state transfer; briefly allow dual-writing during re-shard window to avoid token-loss—coordinate with a migration coordinator.

Handling bursts & global quotas:- Token-bucket inherently allows bursts up to b. Set per-customer burst tuned to business policy.- For cross-customer global quotas (e.g., total concurrent tokens across all customers), maintain a separate global quota shard or a hierarchical allocator: - Consumers first check a global Redis key (also via Lua) for remaining global capacity; if global capacity low, throttle even if per-customer tokens exist. - Alternatively implement probabilistic admission (reservoir) to limit tail behavior.- Use sliding-window counters (time-bucketed) for monthly/daily quotas—update them atomically in same script to prevent exceeding global quotas.

Monitoring, metrics & tuning:- Export metrics: allowed/denied per-customer, latency, Redis QPS, hot-key heatmaps.- Mitigate hot customers: use local leaky token cache in consumer for very high QPS customers to reduce Redis load, but always validate periodically against Redis to ensure fairness.- Backpressure: when Redis latency/high errors, degrade to conservative local rate limits to protect correctness.

Trade-offs:- Centralizing per-customer state in Redis ensures correctness and atomicity at cost of Redis throughput; sharding + consistent hashing mitigates.- Using Redis streams or CRDTs could improve availability but complicates strong-consistency guarantees.

This design gives atomic, fair, and scalable per-customer token buckets using Redis+Lua and consistent hashing while addressing races, failover, bursts, and global quotas.

HardTechnical

75 practiced

You must guarantee atomic, exactly-once state updates across a relational database and a Kafka topic when a service writes an order and emits an event. Explain how the outbox pattern combined with CDC (Change Data Capture) and Kafka Connect/ Debezium provides an atomic, reliable delivery to Kafka, and discuss operational considerations such as transactional boundaries and ordering guarantees.

Sample Answer

Approach summary:Use a transactional outbox in the relational DB combined with CDC (Debezium) + Kafka Connect to atomically publish events: the service writes order + outbox row in a single DB transaction; Debezium captures the outbox insert and Kafka Connect publishes to Kafka. This decouples event emission from business logic while preserving atomicity.

How it provides atomic, reliable delivery:- Single DB transaction: order state and outbox insert are committed together, so either both persist or neither do — eliminating the dual-write problem.- CDC reads the committed binlog/transaction stream (Debezium) and emits change events in the same transactional order the DB committed them.- Kafka Connect with the Debezium connector can use exactly-once delivery semantics to Kafka (idempotence and producer transactions) or at-least-once with idempotent consumers; combined with careful keys and deduplication, you achieve end-to-end exactly-once semantics from DB to topic.

Transactional boundaries and guarantees:- Transaction boundary is the DB commit. Ensure outbox insert is in same DB transaction as business write.- Debezium reads DB change stream: when configured to include transaction metadata (transaction IDs, commit timestamps), it can preserve ordering and group events by transaction.- To push to Kafka exactly-once, run Kafka brokers and Connect with idempotent producers and transactional writes (Kafka transactions). Debezium + Kafka Connect supports transactional producers so a connector task can atomically write all records from a DB transaction into Kafka or abort.- Use a consistent partitioning key (e.g., order_id) to preserve ordering per key; global ordering across partitions isn’t guaranteed.

Operational considerations:- Ordering: Debezium preserves commit order; Kafka preserves per-partition order. Choose partition key to match required ordering scope (per-order vs global).- Exactly-once vs at-least-once: connector + Kafka transactions give end-to-end exactly-once to topic; downstream consumers still need idempotence or dedup logic if they side-effect external systems.- Schema evolution: use Avro/Schema Registry to manage schema changes and compatibility.- Latency and throughput: CDC adds small tail-latency; tune connector tasks, binlog retention, and snapshot behavior.- Failure modes: handle connector restarts (offset storage), DB failover (Debezium HA), and Kafka outages (backpressure); monitor connector lag, DB replication lag, and Kafka producer errors.- Idempotency/dedup: include unique event id and consumer-side dedup table if downstream side-effects must be exactly-once.- Operational hygiene: secure connectors, monitor offsets/heartbeats, ensure binlog retention long enough for connector recovery, test schema changes and failovers.

Summary:Implement outbox writes inside the DB transaction, let Debezium capture committed outbox rows, and configure Kafka Connect with transactional producers and proper partitioning. This yields atomic order+event commits and reliable delivery to Kafka; ensure schema management, ordering keys, connector/Kafka transactional configs, monitoring, and consumer idempotency to handle production edge cases.

MediumSystem Design

77 practiced

Design a Kafka-based data ingestion pipeline that must handle 100,000 events per second, retain raw events for 7 days, and provide at-least-once delivery to downstream analytic sinks such as S3 and a real-time index. Describe producers, partitioning strategy, broker sizing, replication, compression, schema registry use, consumer groups, backpressure handling and monitoring.

Sample Answer

Requirements (clarify): ingest 100k events/sec, retain raw events 7 days, provide at-least-once delivery to S3 (cold) and a real-time index (hot). Assume average event size ~1 KB (adjust if you have real numbers).

High-level architecture:Producers -> Kafka Topic (raw-events) -> Consumer groups:- S3 sink (Kafka Connect or custom consumer)- Real-time index sink (Elasticsearch/OpenSearch consumer)- Optional stream processors (Spark/KSQL)

Producers- Use async batching with linger.ms (~20ms) and large batch.size to increase throughput.- Enable compression (lz4 or snappy) to reduce network & storage.- Configure acks=all and enable idempotence (enable.idempotence=true) to reduce duplicates; keep max.in.flight.requests.per.connection <= 5 to balance throughput and ordering.- Use a schema registry (Avro/Protobuf/JSON Schema) and validate on producer side; embed schema ID in message to ensure compatibility.

Partitioning strategy- Choose topic partitions for parallelism and throughput. For 100k eps @1KB ≈ 100 MB/s raw. Conservative per-partition throughput ~50 MB/s -> start with 4–8 partitions for raw throughput. For consumer parallelism and operational flexibility, prefer 50–200 partitions if you expect many downstream consumers or frequent rebalances. Partition key: choose event key with natural sharding (e.g., customerId) where ordering per key matters; otherwise use round-robin or hash to evenly distribute.

Broker sizing & storage- Raw storage: 100 MB/s * 86400 = ~8.64 TB/day → ~60.5 TB for 7 days (uncompressed).- With lz4/snappy expect 2–4x reduction; assume 3x -> ~20 TB.- Replication factor 3: cluster stores ~60 TB usable.- Use SSDs. If one broker has 4 TB usable, need ~15 brokers. Add room for overhead and spikes -> 18–20 brokers.- If not compressing or if event size larger, increase brokers. Alternatively use Kafka Tiered Storage (Confluent/Cloud-native) to offload older segments to S3 and reduce broker footprint.

Replication & durability- replication.factor=3, min.insync.replicas=2, default.replication.factor at topic creation.- acks=all at producers for durability.- Monitor under-replicated partitions (URP) and prefer rack-aware replica placement for AZ-failure tolerance.

Compression & retention- Topic-level compression will reduce network & disk.- Set retention.ms for 7 days (or retention.bytes).- Use segment.bytes to control flushing and retention, and retention policy delete (raw events). Consider log compaction for derived topics only.

Schema Registry- Use centralized Schema Registry (Avro/Protobuf/JSON Schema) to enforce backward/forward compatibility.- Producers register schemas and include schema id in the payload; consumers validate and evolve safely.- Enforce SR policy for compatibility to avoid downstream breakage.

Consumer groups & sinks- S3 sink: use Kafka Connect S3 Sink or custom consumer. For at-least-once: - Consumer reads, writes to S3 using atomic pattern (write to temp/UUID path, then rename). - Commit offsets after successful write. Handle retries with exponential backoff. - Use partitioned paths (date/hour/partition) to speed queries and recovery.- Real-time index (Elasticsearch): consumer batches and bulk indexes; on success commit offsets. Implement idempotent writes (document IDs based on event key + offset) or implement upserts to handle duplicate deliveries.- Use separate consumer groups per sink for isolation and independent scaling.

Backpressure & flow control- Producers should implement retry/backoff and monitor send failures. Throttle producers by controlling max.in.flight.requests and client-side rate limits if brokers are overloaded.- At broker/consumer side, tune replica fetchers and socket buffers. Use Kafka quotas (bytes per second) to protect cluster from noisy producers/consumers.- For sinks, use bounded queues and backpressure signals: if S3 sink lags, either buffer to durable queue (another topic) or slow producers via client throttle/traffic shaping. For critical spikes, rely on retention to absorb backlog and scale consumers.

Monitoring & alerting- Collect JMX metrics via Prometheus/Grafana. Key metrics: - Broker: UnderReplicatedPartitions, OfflinePartitionsCount, CPU, disk usage, network throughput, JVM GC, log segment sizes - Topic: bytes_in_per_sec, bytes_out_per_sec, partition count, leader distribution - Consumers: consumer_lag (per partition), commit rate, processing errors - Connectors: task failures, connector lag- Alerts: - URP > 0, Offline partitions, consumer_lag increasing over threshold, disk usage > 75%, broker CPU > 80%, GC pauses > threshold.- Use Kafka Cruise Control for rebalancing/autoscaling recommendations, and integrate with cluster autoscaling if on cloud.

Operational considerations & trade-offs- If raw storage cost on brokers is high, enable tiered storage to push older segments to S3 while keeping hot data local.- Compression reduces I/O but adds CPU; lz4 is a good balance.- Increasing partitions improves parallelism but increases controller overhead and recovery time.- Use replication and acks=all for durability, but that increases write latency—acceptable for at-least-once.

Example sizing summary (with ~1 KB events, lz4 compression ~3x):- Raw 7-day compressed: ~20 TB → with RF=3 -> ~60 TB cluster storage- Broker disk 4 TB usable -> ~15–20 brokers- Partitions: 100 partitions (balance parallelism and manageability)- Replication: rf=3, min.insync.replicas=2

This design ensures 100k eps ingestion, 7-day raw retention with capacity planning, at-least-once delivery to S3 and real-time index using consumer-side commit-after-success and idempotent or atomic sink writes, plus monitoring and backpressure controls to keep the pipeline reliable and observable. Adjust numbers when you have precise event size, compression ratio, and SLA latency requirements.

EasyTechnical

82 practiced

Explain idempotency in asynchronous processing and list practical patterns a data engineer can use to make downstream consumers idempotent. Include approaches such as idempotency keys, deduplication stores, conditional writes, and idempotent sinks, and discuss ttl and storage choices.

Sample Answer

Idempotency in asynchronous processing means applying the same message or operation multiple times has the same effect as applying it once. In distributed pipelines where retries, duplicates, and out-of-order delivery happen, making downstream consumers idempotent prevents data corruption, double-charges, or duplicated records.

Practical patterns:

- Idempotency keys - Generate a stable key per logical event (e.g., order_id + event_version). Consumer stores seen keys and ignores repeats. - Good for deduplicating at application layer.

- Deduplication stores - Use a fast key-value store (Redis, DynamoDB, Cassandra) to record processed idempotency keys. - Choose storage durability based on SLA: Redis (low-latency, optional persistence), DynamoDB (durable, scalable). - Include a TTL to expire keys when you no longer need to dedupe (controls storage cost).

- Conditional writes (atomic compare-and-set) - Use DB features: INSERT ... ON CONFLICT DO NOTHING (Postgres), conditional Put in DynamoDB, or transactions to ensure only first write succeeds. - Ensures single-writer semantics without separate read-then-write races.

- Idempotent sinks - Design sink operations to be naturally idempotent, e.g. upserts by primary key, write-ahead logs, Kafka compacted topics keyed by entity. - Use monotonic fields (e.g., last_updated timestamp or sequence number) and only apply newer updates.

Additional considerations:- TTL: set TTLs for dedupe entries based on business window for duplicates (minutes for payment retries, days for billing). Too short -> duplicates; too long -> storage blowup.- Storage choices: in-memory caches for low-latency short windows, durable NoSQL for longer windows and crash recovery, relational DBs when strong consistency/transactions needed.- Trade-offs: latency vs durability, cost vs retention, and complexity of maintaining dedupe stores. Combine approaches: conditional writes where possible, idempotency keys for business logic, and TTL to bound storage.

MediumTechnical

107 practiced

Design a governance and lineage solution for event-driven pipelines that provides schema enforcement, lineage tracing from source to sink, PII masking policies, and audit logs for regulatory compliance. Describe components, metadata stores, integration with schema registry, and how to implement enforcement at publish time and at processing time.

Sample Answer

Requirements & constraints (clarify): event-driven, low latency, support Avro/JSON/Protobuf, enforce schemas, record end-to-end lineage, apply PII masking policies, produce tamper-evident audit logs for compliance.

High-level components- Producers & Broker: Kafka (or cloud pub/sub) with producer-side serializers.- Schema Registry: Confluent/Apicurio storing Avro/JSON/Protobuf schemas and compatibility rules.- Policy & Metadata Service: central metadata catalog (Apache Atlas, Marquez, or cloud Data Catalog) extended with governance API storing dataset identities, field-level classifications (PII, sensitivity), owners, contracts.- Policy Engine: policy-as-code (OPA or custom) that evaluates masking/retention/access policies.- Stream Processing: Spark Structured Streaming / Flink / Kafka Streams for processing with schema enforcement and runtime masking.- Lineage Collector: agents integrated at broker, registry, and processing layers that emit lineage events to the catalog.- Audit Log Store: append-only, immutable storage (WORM S3 + signed ledger or cloud audit logs) for publish, schema changes, policy decisions, and masking actions.- Observability: dashboards + alerts, and access-control logging.

Metadata stores & integration- Schema Registry holds schemas, versioning, and compatibility rules. Each schema has metadata tags linking to catalog dataset IDs and PII tags.- Metadata Catalog stores dataset <-> schema mapping, owners, sensitivity labels, downstream/upstream links. Lineage events push into the catalog (e.g., Marquez API).- Policy store (OPA/Rego) holds masking rules keyed by datasetId.field with actions (redact, hash, tokenize, pseudonymize) and allowed roles.- Audit store records: event publishes (producer id, topic, schema version, timestamp), schema registry changes (who/when/version diff), policy evaluations (decision, rule id), and processing transforms (job id, input schema version, output schema version, masking applied).

Enforcement: publish time (producer-side)- Mandatory serializer wrapper: producers must use a company SDK that: - Validates payload against schema from registry (reject/raise on compatibility or missing required fields). - Attaches metadata headers: schema-id, dataset-id, producer-id, lineage-token. - Calls policy engine in fast-path (local cache) to check whether incoming message contains disallowed PII for that principal; block or redact before publish. - Emits an audit event to the audit store (async, buffered) with decision and schema id.- Broker-level validation (defence-in-depth): Kafka broker interceptors or Kafka Connect Single Message Transform (SMT) that verifies schema header matches payload and enforces topic-level policies (reject or quarantine to a dead-letter topic).

Enforcement: processing time (consumer/stream)- Immutable contract: processing jobs must declare input datasetId + expected schema-version; pipeline orchestration validates declared schemas before run.- At job start, fetch schema from registry and field-classification from catalog. Apply schema enforcement (strict parsing) and implement transformations using typed DataFrames or generated classes (Avro/Protobuf) to avoid schema drift.- PII masking: use policy engine to fetch masking rules; implement deterministic hashing, tokenization via a secure token service (HSM or cloud KMS-backed), or format-preserving masking depending on rule. Record mask operation and key id in audit.- Lineage capture: processing framework emits a lineage event that records: input datasetId:versions, transformation id (job and code bundle hash), output datasetId:versions, timestamps, and policy decisions. Push to metadata catalog.- Quarantine and retry: malformed records or policy violations routed to quarantined topics/buckets, with audit reasons.

Lineage tracing- Capture edges at three points: produce (source -> topic), process (topic -> transformed dataset/topic), sink (topic -> external sink/warehouse).- Each event includes datasetId + schemaVersion and jobId; catalog constructs DAG for traceability. Support interactive trace: given a sink dataset+partition/time range, query upstream lineage to identify source topics and schema versions.- Store fine-grained column-level lineage for transformations that map/derive fields (use expression parsing or use Spark’s Column-level lineage library).

PII masking policies- Define policy model: datasetId.field -> rule (mask_type, key, scope, allowed_roles, justification, retention).- Policy enforcement modes: PRE-PUBLISH (producer SDK), PROCESS-TIME (processor), and ACCESS-TIME (for analytical query engines).- Tokenization: use secure token service with reversible tokens for authorized roles; store token mapping in a secured vault or deterministic tokenization for joinability.- Redaction/hashing: for irreversible masking.- Policy lifecycle: approvals, audit trail, policy change events pushed to audit log and trigger reprocessing or reclassification if required.

Audit & Compliance- Audit events: immutable, append-only, cryptographically signed or storing ledger metadata (who, what, when, why, schemaVersion, action).- Retention & export: retention policies and ability to export audit trail for regulators.- Regular attestations: run automated checks (daily) to verify no data leaked unmasked into approved topics; produce compliance reports.

Operational considerations & trade-offs- Latency vs strictness: producer-side enforcement adds latency but prevents leaked data. Use local cached policies to keep latency low; fall back to async auditing for nonblocking operations where appropriate.- Schema evolution: enforce compatibility rules in registry; for breaking changes require approval/workflow and downstream impact analysis using lineage.- Performance: masking/tokenization can be expensive; prefer deterministic hashing for high-throughput streams and tokenization selectively.- Security: keys for tokenization/masking stored in KMS/HSM, access-logged, separate privileges for policy changes.

Example workflow (publish -> processed sink)1. Producer SDK validates message against schema v3, applies producer-side PII redaction for fields flagged as restricted for that principal, sets headers, sends to Kafka topic.2. Broker interceptor validates header schema-id and appends a publish-audit event to audit-store (WORM S3).3. Stream job pulls message, asserts input schema v3, fetches masking rules, applies tokenization for joinable ID and hashes SSN, writes output to delta-lake table with schema v4.4. Job emits lineage event: source topic(schema v3) -> job:hash -> delta-table(schema v4) into catalog. Job writes a processing-audit event summarizing records processed, masked counts, and policy decisions.

Testing & verification- Contract tests for producers/consumers using schema registry stubs.- Integration tests that assert masking behavior and that lineage events are emitted.- Periodic data scans to detect PII in cleartext and generate alerts.

This design provides defense-in-depth (producer, broker, processor), centralized metadata and policies, full lineage for impact analysis, PII masking with auditable decisions, and immutable audit logging for compliance.

Unlock Full Question Bank

Get access to hundreds of Event Driven and Asynchronous Architecture interview questions and detailed answers.

Join thousands of developers preparing for their dream job.