Designing and operating systems that ingest, process, and serve continuous event streams with low latency and high throughput. Core areas include architecture patterns for stream native and event driven systems, trade offs between batch and streaming models, and event sourcing concepts. Candidates should demonstrate knowledge of messaging and ingestion layers, message brokers and commit log systems, partitioning and consumer group patterns, partition key selection, ordering guarantees, retention and compaction strategies, and deduplication techniques. Processing concerns include stream processing engines, state stores, stateful processing, checkpointing and fault recovery, processing guarantees such as at least once and exactly once semantics, idempotence, and time semantics including event time versus processing time, watermarks, windowing strategies, late and out of order event handling, and stream to stream and stream to table joins and aggregations over windows. Performance and operational topics cover partitioning and scaling strategies, backpressure and flow control, latency versus throughput trade offs, resource isolation, monitoring and alerting, testing strategies for streaming pipelines, schema evolution and compatibility, idempotent sinks, persistent storage choices for state and checkpoints, and operational metrics such as stream lag. Familiarity with concrete technologies and frameworks is expected when discussing designs and trade offs, for example Apache Kafka, Kafka Streams, Apache Flink, Spark Structured Streaming, Amazon Kinesis, and common serialization formats such as Avro, Protocol Buffers, and JSON.
EasyTechnical
72 practiced
You are evaluating messaging systems for a client building an event-driven microservices platform. Compare message brokers (e.g., RabbitMQ), commit-log systems (e.g., Apache Kafka), and cloud streaming services (e.g., Amazon Kinesis). For each, describe typical use cases, ordering and delivery semantics, scaling characteristics, latency and retention trade-offs, and operational cost/complexity. As a Solutions Architect, when would you recommend each option and why?
Sample Answer
High-level summary- Message brokers (RabbitMQ): broker-centric, push-based queues for point-to-point and pub/sub with flexible routing.- Commit-log systems (Kafka): durable append-only log, partitioned, consumer-driven offsets — high throughput and replayability.- Cloud streaming (Kinesis): managed commit-log style stream with AWS integrations, tradeoffs in throughput/retention vs. operational simplicity.Typical use cases- RabbitMQ: request/reply, RPC, work queues, transactions, complex routing (topic/fanout) with moderate throughput.- Kafka: event sourcing, durable event bus, stream processing, audit logs, high-throughput telemetry.- Kinesis: similar to Kafka for AWS-centric customers needing managed scaling and easy integration with Lambda, Redshift, Glue.Ordering & delivery semantics- RabbitMQ: per-queue ordering; supports at-least-once by default (ack/nack), exactly-once achievable with idempotence and transactions but complex.- Kafka: ordering guaranteed per-partition; at-least-once by default; exactly-once semantics (EOS) available with producer/consumer transactions across Kafka Streams.- Kinesis: ordering within a shard; at-least-once delivery; de-duplication/ordering managed by application.Scaling characteristics- RabbitMQ: scale vertically or via clustering/federation; sharding queues is manual and can be complex.- Kafka: designed to scale horizontally via partitions across brokers; rebalancing needed but mature tooling.- Kinesis: autoscaling (shards) but shard limits and scaling operations have constraints and cost implications.Latency & retention trade-offs- RabbitMQ: low latency (ms) for small workloads, short retention (queues cleared on consume) unless persisting messages.- Kafka: optimized for throughput with low end-to-end latency (tens of ms); retention tunable (hours→years) for replay and analytics.- Kinesis: similar to Kafka but with slightly higher per-record latency variability; retention default 24 hrs (extendable to 7-365 days with enhanced features).Operational cost & complexity- RabbitMQ: moderate ops (broker management, HA setups); simpler for small teams.- Kafka: higher ops complexity (Zookeeper/Controller, brokers, monitoring); Confluent or managed services reduce burden.- Kinesis: lowest ops overhead (managed) but higher per-GB cost and vendor lock-in; limited protocol flexibility.Recommendations as a Solutions Architect- Choose RabbitMQ when you need simple, low-latency work queues, complex routing, or RPC patterns and want fast time-to-value with modest scale.- Choose Kafka when you require high throughput, durable event sourcing, replayability, and strict partitioned ordering for analytics and stream processing.- Choose Kinesis when you are AWS-centric, prefer managed service trade-offs, want tight Lambda/CloudWatch/Glue integration, and accept vendor constraints for lower ops burden.Trade-offs summary: prefer brokers for routing/RPC, commit-logs for durability/replay/scale, and cloud streaming for managed operational simplicity in cloud-first architectures.
EasyTechnical
33 practiced
Describe how consumer groups and partition assignment work in Kafka. Explain common partition assignment strategies, what happens during a rebalance, and the implications for in-flight processing and stateful stream processors. What patterns would you recommend to minimize the impact of rebalances on processing correctness and latency?
Sample Answer
Consumer groups: a group of consumers coordinate to read a topic’s partitions so each partition is consumed by exactly one consumer in the group. Kafka tracks group membership and assigns partitions; each consumer commits offsets to record progress.Partition assignment strategies:- Range: contiguous partition ranges per consumer (simple, can cause imbalance).- RoundRobin: partitions distributed evenly across consumers (better balance).- Sticky: tries to keep previous assignments when members change, minimizing partition movement.- Cooperative-sticky (incremental cooperative): moves partitions incrementally to avoid full group stop-the-world rebalances.What happens during a rebalance:1. Group coordinator detects membership change (join/leave/failure).2. All consumers enter a rebalance phase: current assignments revoked, new assignment computed and sent.3. Consumers receive new assignments and resume consumption.During eager rebalances consumers stop consuming until the rebalance finishes — causing pauses and duplicate processing risks if offsets and processing aren’t coordinated.Implications for in-flight processing and stateful processors:- In-flight records can be processed twice (if offsets not committed) or lost (if committed before processing completes).- Stateful stream processors (local state stores) can lose locality: partitions move to another instance, requiring state restoration (RocksDB restore from changelog) — costly in latency and IO.- Full rebalances cause longer pauses and heavy state migrations.Recommended patterns to minimize impact:- Use cooperative rebalancing or sticky assignor to reduce movement and avoid full stop-the-world pauses.- Use static membership (group.instance.id) so planned restarts don’t trigger rebalances.- Graceful shutdown: call consumer.wakeup()/close() after finishing work and committing offsets.- Commit offsets only after processing (or use transactional producers + Kafka consumer offsets commit in transaction) to achieve exactly-once semantics.- Design idempotent processing to tolerate duplicates.- For stateful apps: use standby replicas (Kafka Streams) and partition-aware routing to reduce cold starts; increase changelog retention and tune restoration concurrency.- Reduce churn: avoid frequent scaling events, tune session/heartbeat timeouts reasonably, and monitor consumer liveness.These choices trade off complexity vs. latency/throughput—favor cooperative/sticky assignment, static membership, graceful shutdown, and transactional/idempotent processing for strongest correctness with low rebalance impact.
HardTechnical
36 practiced
Design a deduplication and ordering strategy for e-commerce order events where retries can create duplicates and events may arrive out of order. Requirements: ensure a single fulfillment action per logical order, maintain auditability, and support reconciliation. Explain use of event IDs, stateful dedup stores, idempotent sinks, and downstream reconciliation jobs, including recovery scenarios.
Sample Answer
Situation & goals: We must ensure exactly-one fulfillment action per logical order despite retries/duplicates and out-of-order arrival, keep a complete audit trail, and allow periodic reconciliation to detect drift.Design overview:- Event contract: every order event carries an immutable global event_id (UUID v4 or ULID), order_id, event_type (Created/Updated/Confirmed/Canceled), event_ts (producer time), and sequence_no (optional per-order monotonic counter if available).- Ingress: a durable message bus (Kafka) with topic partitioning by order_id to help ordering locality but not rely on it for correctness.- Stateful dedup store: stream processor (e.g., Flink/Beam/Kafka Streams) with a keyed state by order_id that keeps: - last_applied_event_id(s) (sliding window of recent IDs) to detect duplicates by event_id - last_event_ts/sequence_no for ordering decisions - canonical order_state (PENDING, CONFIRMED, FULFILLED, CANCELLED) and a tombstone/soft-delete policy for retention - full audit log append (write-ahead) to an immutable event store (S3/CDC DB) for replay/reconciliation- Dedup & ordering logic: - On event arrival, first check event_id against recent IDs → drop if duplicate (record metric/audit). - If sequence_no present, accept only if seq > last_seq or idempotent (same event_id). If out-of-order but newer event_ts, use business rules (e.g., CONFIRM supersedes earlier UPDATES). - Persist state transactionally: update dedup set, state, and append audit record in a single atomic step (use transactional sink or two-phase commit with Kafka + DB).- Idempotent sinks/actions: - Fulfillment service exposes idempotent API keyed by order_id + idempotency_key (use event_id as idempotency_key). - Process emits a "fulfillment_command" only when state transitions to FULFILL and only if not already fulfilled per state store. - Downstream fulfillment must store the idempotency_key and acknowledge; retries safe.- Auditability & immutable log: - Every input event and every state transition is appended to an immutable, queryable audit store (object store or append-only DB) with causal links to event_ids.- Reconciliation: - Periodic batch/stream job compares canonical state (state store’s snapshot or materialized view) against external systems (warehouse, fulfillment DBs) using order_id and last_event_id. - Reconciliation reports mismatches, attempts automated repair using tombstone/audit trail, and generates tickets for manual ops when > threshold.- Recovery scenarios: - Processor crash: on restart, restore keyed state from checkpoint (Flink) and replay audit log from Kafka offset to ensure deterministic reapplication. Because event_id dedup prevents double-apply, replay is safe. - Lost state: rebuild state by replaying immutable audit log ordered by event_ts/sequence_no to reconstruct per-order state; reconciliation job can flag gaps. - Downstream inconsistencies: reconciliation invokes idempotent compensating actions (cancel/fulfill) using event_id as idempotency_key.Trade-offs & notes:- Requiring sequence_no simplifies ordering but may be unavailable; fallback to event_ts + business rules.- Dedup window size balances memory vs risk of reprocessing very old duplicates—immutable audit log protects against data loss.- Use strong monitoring: duplicate rate, out-of-order rate, reconciliation drift, processing lag.This design yields single fulfillment per logical order, full auditability, safe recovery/replay, and automated reconciliation.
HardTechnical
39 practiced
Discuss how to handle network partitions and split-brain scenarios in a streaming ecosystem. Explain how brokers, controllers, and stream processing frameworks behave during partitions, how to detect divergence, and strategies to reconcile diverging state after partitions heal while avoiding data loss and duplicate side effects. Use Kafka and Flink examples.
Sample Answer
Situation: In a streaming ecosystem for a mission-critical application, network partitions can split brokers, controllers, and stream processors leading to split‑brain (two active partitions accepting writes). As a Solutions Architect I design controls to prevent, detect, and reconcile divergence while minimizing data loss and duplicate side effects.How components behave:- Kafka brokers/controllers: Leader election requires quorum. If the controller or ISR loses quorum, partitions may become read-only or—if unclean leader election is enabled—an out-of-sync replica can be promoted and accept writes, causing divergence. KRaft/raft or Zookeeper roles decide controller availability.- Stream processors (Flink): TaskManagers may be partitioned from JobManager; Flink will attempt failover, reschedule tasks, or pause processing depending on checkpointing and high-availability setup. If two JobManagers become active (misconfigured HA), parallel processing can cause duplicate side effects.Detecting divergence:- Compare committed log offsets across replicas and clusters (e.g., cross‑region MirrorMaker metrics).- Monitor ISR size, leader changes, unclean.leader.election.rate, controller elections, and producer errors.- For Flink, monitor checkpoint completion rates, job restarts, and state backend metrics. Periodically compute state checksums (e.g., keyed-state hash) and compare across standby/DR clusters.Prevention and configuration best practices:- Enforce quorum-based leader election: disable unclean.leader.election.enable, use min.insync.replicas, and properly size replication factor and controller quorum.- Use idempotent and transactional producers in Kafka (enable.idempotence=true; transactions with transactional.id) to ensure exactly-once to Kafka.- Configure Flink with durable checkpointing to a replicated filesystem, externalized checkpoints/savepoints, and HA mode (ZooKeeper/KV store) so only one JobManager is active.Reconciliation strategies after healing:1. Avoid divergence by design: prefer blocking writes when quorum is lost rather than allowing split writes.2. If divergence occurred, detect by comparing offsets/state hashes and application-level sequence numbers or vector clocks embedded in events.3. Reconcile logs: - For Kafka intra-cluster divergence due to unclean election, prefer replay of authoritative leader's log; use consumer application logic to deduplicate (idempotency keys) and compensate. - For cross-cluster replication (DR), use MirrorMaker 2 with active-passive or quorum-based reconciler; pause replication on conflicting leaders, compute offset gaps, and reconcile by committing tombstones or applying compensating events.4. Application-level reconciliation: - Emit idempotency keys and causal metadata so downstream sinks can dedupe or apply idempotent writes. - Use transactional sinks or two‑phase-commit connectors (Flink's TwoPhaseCommitSinkFunction) so side effects are committed atomically with checkpoints. - For non-transactional sinks, implement outbox pattern: write side-effect requests to a Kafka topic (transactional) and have a single dispatcher consume and apply with at-least-once semantics plus idempotency/deduplication at the target.Flink+Kafka concrete example:- Prevent: Kafka producers set enable.idempotence=true and use transactions; topic replication factor 3, min.insync.replicas=2, unclean leader election disabled. Flink configured with checkpointing to S3 and exactly-once Kafka sinks using FlinkKafkaProducer with semantic=EXACTLY_ONCE (two-phase commit).- Detect: Alert on controller elections, leader changes, and failed checkpoints. Periodically write state checksums to a dedicated topic.- Reconcile: If a partition accepted conflicting writes during a network split, restore Flink from the last common externalized checkpoint, replay authoritative Kafka partitions from the checkpoint offsets, and let transactional producers ensure duplicate writes to external sinks are avoided. If external systems can't be rolled back, run compensating transactions derived from event metadata (sequence numbers, op-type) and apply idempotent fixes.Trade-offs:- Strict quorum and blocking writes minimize divergence but increase availability during partitions (CAP tradeoff). For high-availability use cases, accept eventual reconciliation and invest in strong dedup/compensation logic.- Transactions and externalized checkpoints increase latency and operational complexity.Key principles to communicate to clients:- Design for prevention (quorum + config), detect divergence early (metrics, checksums, offsets), and reconcile deterministically (transactional flows, idempotency, outbox/compensate patterns).- Document recovery runbooks: how to identify authoritative cluster, how to restore Flink jobs from checkpoints, and how to replay/reconcile Kafka topics to avoid data loss and duplicate side effects.
MediumTechnical
39 practiced
As a Solutions Architect, define a schema evolution policy for Avro messages stored in Kafka with Schema Registry. Specify compatibility mode (backward, forward, full), deprecation practices, required default values, test processes, and the governance steps to coordinate producers and consumers and prevent production breakage.
Sample Answer
Requirements & objective: prevent producer/consumer breakage while allowing safe, iterative schema change for Avro in Kafka using Confluent Schema Registry.Compatibility mode- Default: BACKWARD for topic-level schemas so new schema can read data produced with older writers. Use BACKWARD_TRANSITIVE for multi-version robustness.- Use FORWARD only for specific cases where consumers must read newer writers; FULL used rarely and with strong coordination.Deprecation practices- Mark fields as deprecated in schema docs & add a boolean "deprecated" field in schema metadata (not payload).- Phase deprecation: announce → stop producing to field → remove after 2 release cycles (or N days) and successful consumer validation.Required default values & nullability- Required fields must be non-nullable with no defaults.- Optional additions must be nullable or have a sensible default in schema to preserve backward compatibility (provide default in Avro for non-nullable additions).- Avoid changing type; use union types for evolution.Test processes- Automated CI that runs: - Compatibility checks against Registry (conftest or registry API) on PRs - Contract tests: producer contract (can write) and consumer contract (can read old/new messages) - Schema migration sandbox replay tests with representative data- Add fuzz/interop tests to catch edge cases.Governance & coordination- Schema ownership: topic-level schema owners (team + contact) registered in a governance catalog.- Change workflow: RFC → automated compatibility check → review board (schema review) → staged rollout (dev → qa → canary prod) → full rollout.- Versioning policy: use semantic version in subject naming (topic-value-vN) and keep subject-per-topic-value convention.- Emergency rollback: producers can switch to previous compatible schema; registry allows soft-rollback with approval.- Documentation: changelog, deprecation timeline, migration guides and consumer adapter examples.- Communication: automated notifications on schema change, mandatory sign-off from impacted consumer owners for non-backward-compatible changes.Best practices- Prefer additive changes only; avoid renaming (use aliases in consumer mapping); central registry enforcement and CI gates to block incompatible PRs.
Unlock Full Question Bank
Get access to hundreds of Stream Processing and Event Streaming interview questions and detailed answers.
