Designing and operating systems that ingest, process, and serve continuous event streams with low latency and high throughput. Core areas include architecture patterns for stream native and event driven systems, trade offs between batch and streaming models, and event sourcing concepts. Candidates should demonstrate knowledge of messaging and ingestion layers, message brokers and commit log systems, partitioning and consumer group patterns, partition key selection, ordering guarantees, retention and compaction strategies, and deduplication techniques. Processing concerns include stream processing engines, state stores, stateful processing, checkpointing and fault recovery, processing guarantees such as at least once and exactly once semantics, idempotence, and time semantics including event time versus processing time, watermarks, windowing strategies, late and out of order event handling, and stream to stream and stream to table joins and aggregations over windows. Performance and operational topics cover partitioning and scaling strategies, backpressure and flow control, latency versus throughput trade offs, resource isolation, monitoring and alerting, testing strategies for streaming pipelines, schema evolution and compatibility, idempotent sinks, persistent storage choices for state and checkpoints, and operational metrics such as stream lag. Familiarity with concrete technologies and frameworks is expected when discussing designs and trade offs, for example Apache Kafka, Kafka Streams, Apache Flink, Spark Structured Streaming, Amazon Kinesis, and common serialization formats such as Avro, Protocol Buffers, and JSON.
HardTechnical
46 practiced
You notice that increasing Kafka producer batch.size reduces throughput variability but increases tail latency. Explain tuning knobs across Kafka producer (batch.size, linger.ms, acks), network and OS layers, and processing engine (parallelism, checkpointing) to optimize for low 99th percentile end-to-end latency while preserving required throughput. Provide a prioritized tuning plan.
Sample Answer
Situation: We're seeing that larger Kafka producer batch.size lowers throughput variance but pushes up 99th-percentile end-to-end latency. Goal: reduce tail latency while keeping required throughput. Below is a prioritized, practical tuning plan covering producer, network/OS, and processing engine (Flink-like), with rationale and trade-offs.Key knobs and how they affect tail latency vs throughput- Producer: - batch.size: larger batches increase throughput and amortize IO but add queueing delay (increases p99). Reduce to limit per-batch queuing; pick smallest size that preserves throughput. - linger.ms: controls wait-for-batch time. Lower linger.ms to reduce added latency; increase only if throughput drop is unacceptable. - acks: acks=all improves durability but increases tail latency due to leader+ISR sync; acks=1 lowers latency but increases data loss risk. Use acks=1 if acceptable, otherwise keep all and mitigate elsewhere. - max.in.flight.requests.per.connection: >1 allows pipelining but can re-order on retries; reducing can reduce retries' impact on p99. - compression.type: compress reduces network bytes and CPU; CPU cost can add latency—use fast compressors (lz4/snappy). - retries/delivery.timeout.ms/request.timeout.ms: tune to avoid long retry stalls that spike p99; prefer bounded retries and short timeouts.- Network & OS: - socket.send.buffer.bytes / TCP_NODELAY: TCP_NODELAY reduces Nagle-induced delays; set socket buffers to match bandwidth*RTT. - net.core.rmem_max / wmem_max: increase to prevent kernel drops under bursts. - NIC settings: enable interrupt moderation tuning, offloads, adjust RSS to spread interrupts across cores. - MTU and path MTU: use jumbo frames if supported to reduce per-packet overhead. - Prioritize low jitter: isolate producer hosts from noisy neighbor CPU/network workloads, use QoS to prioritize Kafka traffic.- Processing engine (Flink): - Parallelism: increase parallelism to reduce per-record processing queueing; more consumers/producers smooth processing and reduce queuing latency. - Checkpointing: synchronous, frequent checkpoints increase tail latency. Use asynchronous checkpoints, increase checkpoint interval, enable incremental/state backend tuned (RocksDB), and set minPauseBetweenCheckpoints to avoid checkpoint overload. - Backpressure: monitor and eliminate hot operators. Break heavy operators, add buffering, use operator chaining judiciously. - Task slot / thread pools: ensure enough threads for network and IO; separate network IO threads from compute if possible. - Exactly-once vs at-least-once: exactly-once (two-phase commit) adds latency; consider at-least-once if business allows.Prioritized tuning plan (stepwise)1. Measure baseline p50/p95/p99 and where latency accumulates (producer send, network, broker ack, consumer/process). Instrument producer metrics, broker, and Flink metrics.2. Reduce producer linger.ms first (e.g., to 0-5ms) and lower batch.size incrementally until p99 improves without unacceptable throughput loss. Rationale: removes artificial batching delay.3. Tune compression to lz4/snappy to keep throughput while lowering CPU latency spikes.4. Adjust acks: if business allows some risk, try acks=1 and measure p99. If not allowed, keep acks=all and proceed.5. Constrain max.in.flight and retries/delivery timeout to avoid long stalls on transient failures.6. Network/OS: enable TCP_NODELAY, set appropriate socket buffers, tune NIC interrupt handling, and ensure sufficient bandwidth/MTU. Re-measure.7. Processing engine: increase consumer parallelism and task slots to absorb bursts; switch to async checkpoints, increase interval and minPauseBetweenCheckpoints; enable incremental checkpoints or RocksDB to reduce snapshot time.8. If p99 still high, consider architectural changes: add a small low-latency fast-path topic (smaller batch) for latency-sensitive messages; separate workloads by SLA.Trade-offs and monitoring- Reducing batch.size/linger.ms lowers latency but can raise CPU/network load and reduce throughput. Compensate with parallelism and compression.- Changing acks trades durability for latency.- Checkpoint relaxation trades recovery time for lower runtime tail latency.Success criteria and metrics- Target: p99 below SLA while throughput >= required value.- Monitor: producer batch wait time, request latency, broker ISR lag, network packet drops, Flink operator latency/backpressure, checkpoint duration and alignment.This stepwise approach lets you trade off batching vs queuing delay, isolate network/OS bottlenecks, and scale processing to preserve throughput while driving down 99th-percentile latency.
EasyTechnical
36 practiced
Explain at-least-once, at-most-once, and exactly-once processing semantics in streaming systems. For each, give a concrete example of a pipeline (producer, broker, processing engine, sink) and describe how duplicates or data loss can occur or be prevented.
Sample Answer
At-least-once, at-most-once, and exactly-once describe delivery/processing guarantees in streaming systems.At-least-once: Each event is processed one or more times (no loss, possible duplicates).Example pipeline: Producer → Kafka → Spark Streaming (micro-batches) → Data Warehouse.How duplicates occur: On failure before the consumer commits offsets, the stream job restarts and re-reads the same messages, causing duplicates in the DW.Prevention/mitigation: Make sinks idempotent (upsert with unique key), deduplicate downstream using event IDs/timestamps, or use transactional sinks if available.At-most-once: Each event is processed zero or one time (no duplicates, possible loss).Example pipeline: Producer → Kafka → simple consumer that commits offsets immediately → File sink.How loss occurs: If the consumer commits offset before processing and then crashes, the message is lost.When used: Low-latency, best-effort scenarios where occasional loss is acceptable.Exactly-once: Each event is processed exactly once end-to-end (no loss, no duplicates).Example pipeline: Producer → Kafka → Apache Flink (with checkpointing & 2PC) → transactional sink (Kafka sink with EOS, or database with transactional commits).How it works: Flink checkpoints operator state and coordinates two-phase commits to sinks; on restart it restores state and replays only what wasn’t committed, ensuring idempotent commit boundaries.Trade-offs: Higher complexity and latency; requires sink support (transactions or idempotent writes) and careful state management.Summary: choose based on correctness needs vs. complexity — at-least-once + idempotent sinks is common pragmatic choice for data engineering.
EasyTechnical
46 practiced
Explain how Kafka consumer groups and partition assignment work. Describe rebalance triggers, the difference between cooperative and eager rebalancing, how offsets are committed (auto vs manual), and practical techniques to minimize disruption from frequent rebalances in a high-throughput streaming application.
Sample Answer
Kafka consumer groups: consumers with the same group.id coordinate to consume a topic's partitions so each partition is processed by exactly one consumer in the group. The group coordinator (a broker) tracks membership and assigns partitions to consumers; offsets are per (topic, partition, group.id).Partition assignment & rebalances:- When membership or subscription changes the coordinator triggers a rebalance to reassign partitions.- Common rebalance triggers: consumer join/leave (start/stop/crash), subscription changes, consumer heartbeat/session timeout, group coordinator leader failover, or client calling subscribe/unsubscribe.Cooperative vs eager rebalancing:- Eager (stop-the-world): assignor revokes all partitions from consumers, waits for all to be revoked, then assigns new partitions. Causes full pause in processing during rebalance.- Cooperative (incremental cooperative): consumers can give up/receive partitions incrementally, enabling partial ownership transfers and much lower disruption for large groups. Requires assignor & client support (e.g., cooperative-sticky assignor + Kafka client >= supported versions).Offsets: auto vs manual- auto.commit.enable (auto.offset.reset for behavior on missing offsets): auto commits offsets periodically (enable.auto.commit=true). Pros: simple; Cons: at-least-once vs at-most-once tradeoffs and less control.- Manual commits: application calls commitSync() or commitAsync() after processing records. commitSync is blocking and confirms commit; commitAsync is non-blocking (handle failures). For exactly-once processing build idempotency or use transactions with Kafka Streams/Producer transactions.Techniques to minimize disruption in high-throughput apps:- Use cooperative assignor (sticky/cooperative-sticky) and Kafka clients that support it.- Use static group membership (group.instance.id) so restarts don’t trigger full rebalances for recoverable instances.- Tune session.timeout.ms, heartbeat.interval.ms and max.poll.interval.ms to match processing time and network characteristics (but avoid too-large timeouts that delay failure detection).- Keep poll loops fast: do small batches, process quickly or hand off heavy work to worker threads; call poll frequently to avoid coordinator thinking consumer is dead.- Use pause()/resume() around blocking work and commit offsets after processing batches.- Use async commits and idempotent downstream sinks; commit offsets after durable persistence to avoid duplicates.- Monitor rebalance frequency and consumer lag; reduce churn by making deployments rolling and avoiding frequent consumer restarts.- Reduce number of partitions consumers contend for (balance partition count vs parallelism), or scale consumers to partition ratio 1:1 to avoid repeated ownership churn.These practices reduce stop-the-world time, lower duplicate processing, and keep throughput stable in production.
MediumTechnical
49 practiced
Outline a CI/CD pipeline for continuous delivery of streaming jobs. Include unit tests, integration tests with embedded brokers or test clusters, performance tests to detect regressions, and safe deployment strategies (canary, blue-green, rolling). Mention practical tools and gating criteria for production deployment.
Sample Answer
Situation: Build a CI/CD pipeline to safely deliver streaming jobs (Spark/Flink/Beam) with full test coverage, performance regression detection, and safe rollout.Pipeline stages and practical tools:1. Code & Build (GitHub/GitLab + Docker + Maven/Gradle) - Lint, compile, build container image, sign artifact.2. Unit tests - Fast, JVM/unit tests mocking sources/sinks (Mockito, ScalaTest, JUnit). - Gate: 100% critical-path tests pass; no new flaky tests.3. Integration tests (embedded brokers/test clusters) - Use Testcontainers (Kafka, Schema Registry, Postgres) for deterministic integration tests. - Run streaming job against embedded/local Flink MiniCluster or LocalSparkRunner. - Validate end-to-end logic, schemas, exactly-once semantics where applicable. - Gate: all integration tests pass; contract/schema compatibility checks (Confluent Schema Registry).4. Contract & Data Quality checks - Schemas, Cloudevents, reject-breaking changes; run Great Expectations or Deequ for sample outputs.5. Performance and regression tests - Synthetic workload generator (k6/benchmarks, custom producers) feeding a staging cluster sized like prod. - Measure throughput, end-to-end latency, backpressure, GC/CPU, memory. - Baseline comparison + automated anomaly detection (Prometheus alert rules). - Gate: latency/throughput within X% of baseline, no increase in error-rate.6. Staging deployment & smoke tests - Deploy via Kubernetes (Helm/ArgoCD) to staging namespace; run smoke workflows and canary consumers. - Health checks, consumer lag, and metric assertions.7. Production safe rollout strategies - Canary: route small % of traffic (kafka partitioning or feature-flagged producers) to new job; monitor error-rate, processing correctness, lag for N minutes before increasing. - Blue-Green: spin up new job version in parallel, mirror input (kafka mirror or tee) and compare outputs; swap consumers when green is validated. - Rolling: for stateless processors; update pods incrementally with state migration strategies for stateful jobs (savepoints for Flink).Observability & automated gating:- Metrics: processing success/error count, e2e latency percentiles (p50/p95/p99), consumer lag, commit offsets, JVM GC, throughput.- Tools: Prometheus + Grafana, Loki, OpenTelemetry/Jaeger for traces, Alertmanager for thresholds.- Gating criteria before full prod: all smoke/canary metrics within thresholds for configurable window (e.g., 30m), no schema incompatibility, no regression in p99 latency > Y% or throughput drop > Y%, automated tests green, manual approval for major changes.Rollback & safety:- Maintain automated rollback (Argo Rollouts or Helm), use savepoints/checkpoints, idempotent sinks, and dead-letter queues.- Post-deploy: run data-consistency checks (row counts, checksums) and retain pre-deploy snapshots for troubleshooting.This pipeline ensures fast feedback (unit/integration), realistic validation (Testcontainers and mini-clusters), performance regression detection, and safe progressive rollout (canary/blue-green) with clear automated gates for production.
HardSystem Design
42 practiced
Design a real-time pricing engine using event sourcing and CQRS. Price changes are events that must produce materialized views for fast reads, support auditability, and allow replay. Choose an event store, explain compaction and snapshot strategies, how to build projections, and how to handle schema changes and projection rebuilds.
Sample Answer
Requirements & constraints:- Functional: record every price-change as an immutable event; support low-latency reads (materialized views), audit/history, replayability.- Non-functional: high throughput (thousands/sec), low read latency (<50ms), strong ordering per aggregate (e.g., product or SKU), retention & compliance.High-level architecture:- Producers (pricing service, external feeds) -> API/gateway -> Event Store -> Stream processor(s) -> Materialized view stores (read DBs / caches) + Audit store + Backfill pipelines- Use CQRS: writes only to event store; reads from projections/DBs.Event store choice:- Primary: Apache Kafka (with compacted topics) or EventStoreDB for first-class event sourcing. For large scale and ecosystem, choose Kafka+Kafka Streams/Flink. - Benefits: ordering per partition (use aggregate id -> partition), retention & compaction, replay by offset, ecosystem connectors (Debezium, Kafka Connect).Compaction & snapshots:- Use Kafka log compaction to keep latest event per key where appropriate (but careful: compaction loses history — so keep a separate immutable audit topic with time-based retention or archive to cold storage like S3).- Implement periodic snapshots per aggregate stored in a snapshot store (Redis/Mongo/Cassandra) to accelerate rebuilds: store snapshot version = last event sequence.- Snapshot strategy: hybrid — event snapshots every N events or T minutes for hot aggregates; full compaction only for snapshots not for audit.Building projections:- Use stream processors (Kafka Streams or Flink) to consume event topics, apply deterministic logic, and write to materialized views: - OLTP reads -> denormalized row store (Postgres/CockroachDB) or key-value cache (Redis) for ultra-low latency. - OLAP views -> warehouse (BigQuery/Snowflake) via connectors for analytics.- Ensure idempotency: use event sequence numbers and dedupe logic; operate transactional writes (Kafka exactly-once semantics or write with upsert keys).Schema changes & projection rebuilds:- Use schema registry (Avro/Protobuf/JSON Schema) with versioning and compatibility rules (backward/forward).- For breaking changes: introduce new event type version (price_change_v2). Keep processors supporting both versions; perform data migration gradually.- Projection rebuilds: maintain ability to replay from a chosen offset or from archived event logs. Steps: 1. Pause projection writes (or write to staging). 2. Spin up new projection instance reading from earliest or snapshot offset. 3. Use snapshots to skip long histories; if snapshot schema differs, transform snapshot during load. 4. Validate new projection against sample queries, then swap readers (canary then cutover).- Automate rebuilds with orchestration (Kubernetes jobs, Flink savepoints) and monitor correctness.Auditability & compliance:- Archive raw events to immutable object store (S3 with WORM or Glacier) and keep event metadata (who/when) for audit trails.- Provide tooling to replay events to reproduce state at any timestamp.Trade-offs:- Kafka gives scale and replay but needs careful partitioning and retention planning.- Snapshots speed rebuilds but increase complexity and must be consistent with events (store snapshot with last processed offset).- Compaction reduces storage but cannot replace audit store.Operational concerns:- Monitor lags, offsets, reprocessing time, consumer group health.- Chaos testing: simulate replays and schema migrations in staging.- Backups of materialized views and snapshots; RBAC for event mutation.This design gives deterministic, auditable pricing state, low-latency reads via CQRS projections, and safe replay/upgrade paths for schema evolution.
Unlock Full Question Bank
Get access to hundreds of Stream Processing and Event Streaming interview questions and detailed answers.
