Designing and operating systems that ingest, process, and serve continuous event streams with low latency and high throughput. Core areas include architecture patterns for stream native and event driven systems, trade offs between batch and streaming models, and event sourcing concepts. Candidates should demonstrate knowledge of messaging and ingestion layers, message brokers and commit log systems, partitioning and consumer group patterns, partition key selection, ordering guarantees, retention and compaction strategies, and deduplication techniques. Processing concerns include stream processing engines, state stores, stateful processing, checkpointing and fault recovery, processing guarantees such as at least once and exactly once semantics, idempotence, and time semantics including event time versus processing time, watermarks, windowing strategies, late and out of order event handling, and stream to stream and stream to table joins and aggregations over windows. Performance and operational topics cover partitioning and scaling strategies, backpressure and flow control, latency versus throughput trade offs, resource isolation, monitoring and alerting, testing strategies for streaming pipelines, schema evolution and compatibility, idempotent sinks, persistent storage choices for state and checkpoints, and operational metrics such as stream lag. Familiarity with concrete technologies and frameworks is expected when discussing designs and trade offs, for example Apache Kafka, Kafka Streams, Apache Flink, Spark Structured Streaming, Amazon Kinesis, and common serialization formats such as Avro, Protocol Buffers, and JSON.
HardTechnical
45 practiced
You observe that 10% of partitions receive 90% of traffic due to a few heavy users, causing hot partitions and outage-like behavior. Outline detection methods and a remediation plan: key-salting, explicit sharding, custom partitioners, dynamic rekeying, or routing heavy users to dedicated topics. Explain operational steps to migrate keys while preserving ordering when needed.
Sample Answer
Detection:- Metrics: monitor per-partition throughput/latency/error rates (bytes/sec, msgs/sec, consumer lag, request latency). Alert when top X% partitions > threshold.- Telemetry: emit and aggregate keys->partition mapping and top-k heavy keys daily/hourly (Prometheus + histogram + top-k job).- Sampling: on-produce sampling that logs heavy key IDs and origin client IDs; run offline analysis to confirm hotspot persistence vs bursts.- Tracing: distributed traces to link heavy users to APIs/clients.Remediation plan (ordered, with trade-offs):1. Short term — rate limit/throttle heavy users: quickest to relieve pressure, but may impact SLAs.2. Route heavy users to dedicated topics/partitions: create a per-tenant topic or dedicated partition range. Preserves ordering within that user's stream. Good for predictable heavy tenants.3. Explicit sharding (application-level): change producers to shard a heavy key into N sub-keys (userID#shard). Preserves ordering per sub-shard only; requires consumer-side reassembly if global order per user needed.4. Key-salting (simple rehash): append small salt to key before hashing to spread writes. Low-effort, but breaks per-key ordering unless salt is deterministic per logical bucket.5. Custom partitioner: implement a partitioner that maps heavy keys to multiple partitions based on time/window (e.g., round-robin per message for heavy keys) while mapping other keys normally. Allows partial ordering guarantees (per-window).6. Dynamic rekeying with metadata service: maintain a mapping service of logical-key → physical-shard set. Producers consult it; consumers consult it to reassemble or to read from dedicated topics.Operational migration steps preserving ordering:- Decide ordering requirement: per-key total order vs causal ordering vs per-user session ordering.- For approaches that split keys (salting/sharding), preserve ordering by: a) Grace period with dual-write: for a migration window, producers write both to old single-key topic and to new sharded/dedicated topic(s). Consumers read old topic until cutover, then switch to new topic and use an overlap window to reconcile offsets. b) Use stream-sequencing tokens: tag each message with strictly increasing sequence number per logical key (producer-side counter) so consumers can merge streams and detect gaps/duplicates. c) Repartition consumer logic: implement a rekeying consumer that reads old topic, assigns messages to new partitioned topics preserving sequence numbers, then switch downstream consumers to the new topics after verify.- Steps: 1. Identify heavy keys and plan target shards/topics. 2. Deploy mapping service + feature-flagged producer changes to support dual-write and sequence numbers. 3. Start dual-write; monitor consumers for correctness and lag. 4. Run a reprocessor job to copy historical messages (if needed) from old partitions to new target with sequence numbers. 5. Validate end-to-end ordering and completeness via checksums and sequence-gap detection. 6. Flip consumer reads to new topics/partitions and disable old writes after a safe retention/verification period. 7. Clean up old topics/partitions after retention.Key operational considerations:- Coordination across producers, consumers, and platform teams; use feature flags and small phased rollouts.- Automate verification (counts, checksums, sequence continuity) and have rollback plans.- For heavy multi-tenant environments, prefer dedicated topics or SLAs per tenant; for transient hotspots, prefer throttling + dynamic sharding.This approach balances immediacy, ordering guarantees, and engineering effort.
EasyTechnical
37 practiced
Compare Avro, Protocol Buffers (Protobuf), and JSON as serialization formats for event streams. Explain trade-offs in size, parsing speed, schema evolution capabilities, and runtime compatibility. When and why would you use a schema registry?
Sample Answer
High-level summary:- JSON: text-based, human-readable, flexible.- Avro: compact binary with explicit schema that’s separate or embedded; designed for big data/event streams.- Protobuf: compact binary with generated code and strong typing; efficient and widely used.Size:- Binary formats (Avro, Protobuf) are much smaller than JSON — often 3–10x smaller depending on data and field names.- Protobuf tends to be slightly smaller than Avro for many schemas due to varint encoding and field-number compactness; Avro can be competitive for dense records.Parsing speed:- Binary formats parse faster and use less CPU than JSON because they avoid text parsing and repeated key strings.- Protobuf and Avro offer fast deserialization; Protobuf often edges out in benchmarks, but real difference is small in most apps.Schema evolution:- Avro: excellent evolution support — schemas are separate; reader and writer schemas are resolved at read time with explicit compatibility rules (default values, field removal/addition).- Protobuf: supports evolution via optional/required (avoid required), field numbers; removing or reusing numbers is risky but well-supported if done carefully.- JSON: no built-in schema; evolution must be handled in application logic or via JSON Schema (ad hoc), which is error-prone at scale.Runtime compatibility:- Avro + Schema Registry: consumers can fetch writer schemas at runtime and resolve to reader schema — great for heterogeneous consumers.- Protobuf: typically uses generated classes; runtime compatibility requires careful management of .proto files and distribution of descriptors or using dynamic parsing APIs.- JSON: universally readable without generated code, easiest for ad-hoc consumers, but more brittle and less performant.When/how to use a schema registry:- Use a schema registry (e.g., Confluent Schema Registry) when you need centralized schema management, automated compatibility checks, and the ability for consumers to fetch schemas at read time. This is especially valuable for Avro (common pattern), and useful for Protobuf/gRPC when multiple services or languages consume events.- Benefits: prevents incompatible changes, stores schema versions, enables canonical schema IDs in messages (small binary reference instead of full schema), and simplifies cross-team coordination.- Don’t bother for small projects or purely internal JSON logs where human-readability outweighs strict schema control.Recommendation:- For high-throughput event streams with strong schema evolution needs: Avro + Schema Registry.- For low-latency RPC or tightly-coupled services with generated stubs: Protobuf.- For simple, human-readable, low-volume integration or browser-facing payloads: JSON.
EasyTechnical
62 practiced
Explain event time, processing time, and ingestion time in stream processing. How do watermarks help with out-of-order events and late arrivals? Give an example where choosing processing time semantics would produce wrong analytics compared to event time.
Sample Answer
Event time, processing time, and ingestion time are three clock semantics used in stream processing to timestamp and reason about events:- Event time: the timestamp when the event actually occurred (embedded in the event). Best reflects real-world ordering and is used for correct analytics across distributed sources.- Processing time: the time when the stream processing system observes/handles the event (machine/system clock). Cheap and low-latency but sensitive to variability (network delays, reordering).- Ingestion time: the time when the event enters the ingestion layer (e.g., the broker). It’s between event and processing time — useful when event timestamps are missing but you want a consistent ingestion point.Out-of-order events and late arrivals- Watermarks are a heuristic low-water mark that the system advances to indicate “we expect no events with event time ≤ watermark.” They let windowing operators emit results before all possible late events arrive.- Typical behavior: when watermark passes a window’s end, the window is fired. Late events arriving after the watermark can be dropped, routed to a side output, or used to update results if allowed (late-data handling).Example where processing-time semantics is wrongImagine counting unique users per minute for an international app. A user in Tokyo performs actions at 00:00 JST (which is 15:00 UTC previous day). Network delays mean the event reaches the processing cluster at 15:02 UTC. Using processing time you’d attribute the action to 15:02 window, skewing counts and sessionization across minutes. Using event time (with watermarks) you correctly assign the event to the 00:00 JST minute window, producing accurate analytics.Key tradeoffs- Processing time: simpler, lower latency, but can produce incorrect results when events are delayed or reordered.- Event time + watermarks: accurate and robust to delays, but requires event timestamps, watermark tuning, and accepts increased complexity/latency for correctness.
MediumTechnical
39 practiced
For windowed aggregations, discuss how window size, checkpoint interval, batching, and parallelism influence latency and throughput. Provide concrete knobs you would adjust to optimize for low latency vs for high throughput and explain trade-offs for state size and recoverability.
Sample Answer
Windowed aggregations trade off latency, throughput, state size and recoverability. Briefly: larger windows and larger checkpoints/batches improve throughput but increase latency and state; smaller windows/checkpoints reduce latency but increase overhead and recovery time.How each knob affects behavior:- Window size (duration & slide): larger windows accumulate more events → higher throughput (amortized work) but higher per-event latency and bigger state. Smaller windows give lower end-to-end latency but more frequent computation and higher per-second overhead.- Checkpoint interval: longer intervals reduce checkpoint I/O → higher throughput, lower CPU/IO contention, but longer recovery time and more state to replay on failure. Shorter intervals improve recoverability/less lost work but add I/O latency and CPU overhead.- Batching (micro-batch size or record accumulation before processing): larger batches increase throughput (amortize overhead) but add queuing latency. Smaller batches lower latency at cost of throughput.- Parallelism (number of task slots/partitions): more parallelism increases throughput and reduces per-task state size (if keyed evenly), but increases coordination/communication overhead and can increase checkpoint size if each task has state. Under-parallelized jobs may hit CPU or backpressure.Concrete tuning strategies:- For low latency: - Use smaller windows/slides or even session windows tuned to event patterns. - Reduce batch sizes / push to record-at-a-time processing (e.g., Flink streaming, small maxRecordsPerPoll). - Shorten checkpoint interval (but keep asynchronous checkpoints and incremental/state backend like RocksDB to reduce pause). - Increase parallelism moderately to avoid hotspots. - Use local in-memory state where safe; enable low-latency serializers. - Trade-offs: higher CPU usage, more frequent checkpoints, larger recovery work relative to last checkpoint.- For high throughput: - Increase window size/aggregate further upstream (coarser granularity). - Increase batch sizes, increase checkpoint interval, enable larger checkpoint compression and async checkpointing. - Increase parallelism to scale out; use RocksDB for large state with incremental checkpoints. - Use operator fusion to reduce network hops. - Trade-offs: higher end-to-end latency, larger state on disk, longer recovery windows, potentially more data to replay.State size & recoverability:- Larger windows and larger batching increase retained state and checkpoint size. Use incremental/differential checkpointing, compact state (TTL, incremental aggregations), and state backends that support incremental checkpoints (RocksDB) to balance throughput and recoverability.- Recovery time is roughly proportional to checkpoint size + amount of events to replay. Shorter checkpoints + smaller state → fast recovery but higher runtime overhead.Rule of thumb: decide SLO (latency vs throughput), then tune window/batch/checkpoint in that direction while using incremental checkpoints, state TTL/compaction, and sufficient parallelism to avoid hotspots. Monitor end-to-end latency, checkpoint durations, and recovery time; iterate.
HardSystem Design
33 practiced
Design an architecture to propagate backpressure from a slow downstream sink back through a distributed streaming pipeline to producers. Include mechanisms such as reactive-streams backpressure, bounded queues with rejection, rate-limiting tokens, and circuit-breakers. Explain how to avoid cascading failures and ensure graceful degradation.
Sample Answer
Requirements:- End-to-end backpressure: slow downstream sinks must signal upstream producers.- Resilience: avoid cascading failures, provide graceful degradation.- Bounded memory and predictable latency.- Support distributed producers/consumers, multi-tenant.High-level architecture:Producer -> Ingress Service (reactive stream publisher) -> Broker/Coordinator -> Worker/Processor nodes -> Sink (slow consumer)Core mechanisms:1. Reactive-streams backpressure end-to-end- Use Reactive Streams (Publisher/Subscriber/Subscription). The Sink subscribes and requests N items. If it requests 0 or fewer, upstream stops sending.- Implement at each hop using non-blocking pull: each component only pulls when it has capacity.2. Bounded queues with rejection- Each node keeps a bounded in-memory queue. When full, apply rejection policy: - Prefer backpressure propagation: signal Subscription.request(0) or reduce request rate upstream. - If upstream cannot slow (eg. external producers), reject with specific error codes (429) and optionally enqueue to DLQ.3. Rate-limiting tokens (leaky-bucket)- Use distributed token bucket per producer-group enforced by Broker. Producers must acquire tokens before producing. When sink slow, coordinator reduces token issuance frequency.- Tokens help shape ingress and prevent bursts overwhelming pipeline.4. Circuit-breakers & health scoring- Per-sink circuit breaker (closed/half-open/open). On repeated timeouts or queue saturation, open CB to stop forwarding work to that sink; instead: - Route to degraded path (batching, sampling, approximate processing) - Persist to durable queue (Kafka) as backpressure buffer- Health score per node used for routing decisions.Data flow & control:- Sink reports consumer demand and health (request N, latency metrics) to local processor which adjusts its Subscription to upstream. Processor sends aggregated signals to Broker/Coordinator. Broker adjusts token rates and can push back pressure directives to Ingress and producers.Avoid cascading failures / graceful degradation:- Multi-tier buffering: short in-memory bounded buffers + durable buffer (Kafka). Short buffers enforce quick backpressure; durable buffers absorb longer slowness without OOM.- Admission control: reject or sample low-priority traffic when capacity exhausted rather than letting system degrade globally.- Priority & QoS: tag messages with priority; high priority gets tokens preferentially.- Fast failure signaling: use health metrics and circuit-breakers to isolate failing sinks; re-route or drop non-critical work.- Observability & autoscaling: metrics (queue depth, token rate, latencies) feed autoscaling rules; scale processors or increase sink capacity where possible.Trade-offs:- Immediate rejections preserve stability but lose some data unless persisted.- Durable buffering increases latency but prevents data loss.- Token coordination (consistent store) adds complexity; can approximate with local buckets and periodic sync.Example technologies:- Reactive frameworks: Project Reactor / Akka Streams- Durable buffer: Kafka- Coordinator: Redis or etcd for token counters + circuit-breaker state- Metrics: Prometheus + alertingThis design ensures upstream producers are either slowed via reactive pulls and token limits, or asked to gracefully fall back (retry, persist, sample), preventing cascading failures while allowing controlled degradation and eventual recovery.
Unlock Full Question Bank
Get access to hundreds of Stream Processing and Event Streaming interview questions and detailed answers.
