Stream Processing and Event Streaming Questions
Building on event-streaming platforms: Kafka and message queues, event sourcing, partitioning, consumer groups, exactly-once vs at-least-once delivery, and windowing. Covers handling late and out-of-order events, watermarks, and stateful stream operators. The core skill for real-time data engineering.
Design a streaming pipeline that computes a rolling metric (for example daily or weekly active users, or a per-minute revenue total) over a high-volume event stream, where a meaningful share of events arrive late. Cover ingestion, windowing, watermark strategy, exactly-once handling, and how you'd reconcile a late-arriving correction into an already-served result.
Sample Answer
Direct answer
Designing a streaming pipeline to compute a rolling metric with meaningful late arrivals means combining event-time windowing with an explicitly chosen watermark and allowed-lateness strategy, an exactly-once (or safely idempotent) write path to the serving store, and a defined policy for how a late correction gets reflected once it's already been served.
Structured elaboration
The pipeline shape is: ingest from a durable, replayable source (a commit-log platform); window the aggregation by event time (tumbling for a clean per-period number, sliding if the metric needs to update more often than its own window length); choose a watermark and allowed-lateness setting sized against your real observed lateness distribution, not guessed; write results via an idempotent upsert keyed by the window's identity so a late-triggered recomputation of an already-served window overwrites cleanly rather than double-counting; and decide, deliberately, whether a correction after the fact re-fires and overwrites the previously served value (accepting that consumers see a number change) or is captured separately for reconciliation without disturbing what was already shown.
Worked example
Computing daily active users where up to 10% of events can arrive as much as 24 hours late: ingest from Kafka, window by calendar day (event time), set the watermark's allowed lateness wide enough to cover the bulk of that 24-hour tail (accepting that the very last, rarest late arrivals beyond that are handled by a separate side-output-and-reconciliation path rather than blocking the main pipeline indefinitely), and write the daily count to the serving store as an idempotent upsert keyed by (date), so a late-triggered recompute of yesterday's count safely overwrites the earlier, less-complete number rather than creating a duplicate entry. At 1 million events per minute, the state backend and checkpoint strategy need to be sized to hold roughly a day's worth of in-flight, not-yet-closed windows, which is the concrete capacity-planning number this design has to account for.
Trade-offs and pitfalls
A wider allowed-lateness setting sized to catch "most" late data delays every window's first result and grows in-flight state accordingly; the residual small fraction of even-later data that misses even a generous allowed-lateness window still needs an explicit, deliberate policy (silently drop it, or reconcile it later), not an accidental gap nobody decided on. The most common mistake in this exact design pattern is building the happy-path windowing and forgetting to design the idempotent-write and late-correction behavior with the same rigor, leaving a pipeline that looks correct in a demo but silently double-counts or drops data the first time a real late-arrival burst occurs in production.
Design the state-management strategy for a stateful stream-processing job holding very large per-key state (hundreds of millions of keys, terabytes total): state backend and storage choice, point-in-time restore for compliance, how you'd detect and alert on runaway state growth, and how cross-partition global aggregation (for example a global top-100) fits given each partition only sees its own keys.
Sample Answer
Direct answer
Managing state at massive scale (hundreds of millions of keys, terabytes total) means choosing a disk-backed state backend with incremental checkpointing as a baseline, adding point-in-time restore for compliance needs, actively monitoring for and alerting on runaway state growth, and, for a truly global aggregate, accepting that no single partition can compute it alone.
Structured elaboration
At this scale, an in-memory-only backend is not viable; a disk-backed backend (RocksDB or similar) with incremental checkpointing (only changed data persisted per checkpoint, not the full state) is what keeps checkpoint time and storage growth proportional to the rate of change rather than total size. Point-in-time restore for compliance (being able to reconstruct exactly what a piece of state looked like at a specific past moment) generally requires retaining a chain of checkpoints or a changelog long enough to replay back to any required point, not just the latest snapshot. Detecting runaway growth means monitoring per-key and total state size trends, not just disk utilization after the fact, since a slow, silent leak in a TTL (time-to-live) or eviction policy can go unnoticed until disk pressure becomes acute. A truly global aggregate (a global top-100 across all keys) can't be computed by any single partition alone, since each partition only ever sees its own slice of the key space; the standard pattern is a two-stage aggregation, each partition computes and periodically emits its own local top-N, and a second, much smaller aggregation stage merges those partial results into the true global top-100.
Worked example
For a job holding roughly 200 million keys at about 1KB each (~200GB of state), sharded across enough parallel tasks that each individual task's local RocksDB instance stays within a manageable size (say, a few tens of gigabytes per task), incremental checkpointing keeps the per-checkpoint cost proportional to the fraction of keys that actually changed since the last checkpoint, not the full 200GB every time. A global top-100 over this same key space is computed by having each task emit its own local top-100 candidates on each window close, then a lightweight downstream aggregator merges those (at most, task-count times 100) candidates into the true global top-100, which is a vastly cheaper operation than trying to centralize all 200 million keys' counts in one place.
Trade-offs and pitfalls
Point-in-time restore for compliance directly conflicts with an aggressive storage-cost-minimization instinct to prune old checkpoints quickly; the retention window for checkpoints has to be driven by the actual compliance requirement, not by disk-cost pressure alone. The two-stage global-aggregation pattern trades perfect accuracy for feasibility: a true global top-100 computed this way can, in principle, miss an item that ranks just outside each individual partition's local top-N but would have ranked in the true global top-100 when combined, a bounded and generally acceptable approximation error worth being explicit about.
What factors should drive partition-key selection for a topic (cardinality, hot-key avoidance, the ordering a downstream aggregation actually needs)? Walk through the trade-offs for a concrete topic with a natural user or entity ID.
Sample Answer
Direct answer
Partition-key choice is a balance between cardinality (enough distinct values to spread load evenly across partitions), hot-key avoidance (no single value dominating traffic), and preserving whatever ordering guarantee your downstream processing actually needs, which usually means keying on the entity whose events must stay in relative order.
Structured elaboration
Too low a key cardinality (or one value that's disproportionately common) concentrates traffic onto a small number of partitions regardless of how many partitions the topic has, defeating the purpose of partitioning for parallelism. The key also directly determines what ordering guarantee a consumer sees: events sharing a key are guaranteed to arrive in production order relative to each other, while events with different keys have no ordering relationship, so the key should be chosen to match whatever entity actually needs that guarantee (commonly a user ID, an account ID, or an order ID), not chosen purely for even load distribution if that conflicts with a real ordering requirement.
Worked example
For an orders topic needing per-order ordering (an order's create, update, and cancel events must be processed in that exact sequence), keying by order_id guarantees this at the cost of no guarantee across different orders, which is fine since nothing needs that. If a small number of very large customers generate a disproportionate share of all orders (a common real-world skew), keying purely by order_id still spreads load reasonably evenly since orders themselves are numerous and roughly uniform, whereas keying instead by customer_id for the same use case would concentrate a large customer's high order volume onto whichever partition their customer ID hashes to, a genuine hot-partition risk.
Trade-offs and pitfalls
A key chosen purely for even distribution, ignoring the actual ordering requirement, will pass every load test cleanly while silently violating a correctness requirement nobody tested for (out-of-order processing for a given entity) until it causes a real, hard-to-reproduce bug. Conversely, a key chosen purely for correctness without checking its cardinality and skew can pass correctness review and then fail under load the first time one key's traffic dominates; both properties need to be checked, not just one.
Explain how to achieve end-to-end exactly-once semantics for a pipeline that reads from Kafka, processes in a stream processor such as Flink, and writes to an external sink that does not natively support Kafka's transactions (for example a relational database or object store). Cover Kafka producer idempotence and transactions, the processor's checkpointing and two-phase-commit sink support, and idempotent writes at the sink.
Sample Answer
Direct answer
Achieving end-to-end exactly-once from Kafka through a stream processor to an external sink combines three layers working together: an idempotent, transactional producer so retries never duplicate a write; the stream processor's checkpointing coordinated with a two-phase-commit (or equivalent) sink so output is only visible once the corresponding input has been durably checkpointed; and, where the sink itself can't participate in that coordination, idempotent writes at the sink as the fallback that makes reprocessing harmless.
Structured elaboration
Kafka's own producer idempotence and transactions solve the producer-to-broker leg: a producer can write to multiple partitions atomically and recover from a crash without duplicating output, because a transaction is either fully committed or fully aborted. The stream processor's checkpointing (Flink's distributed snapshots, for instance) captures the exact input offset the processor has consumed up to, alongside any internal state, so recovery resumes from precisely that point rather than reprocessing or skipping. The hard part is the sink: if it's Kafka-native (another topic), the processor's transactional producer covers it directly. If it's an external system that doesn't support the same transactional protocol (a relational database, an object store), the standard techniques are a two-phase-commit sink (the processor coordinates a pre-commit and commit phase with the sink, tied to its own checkpoint barrier) or, more commonly in practice, idempotent writes at the sink (an upsert keyed by a deterministic ID derived from the source offset, so reprocessing the same input after a recovery produces the same final row rather than a duplicate one). Kafka Streams exposes this producer-side transactional support directly through its client API (initTransactions, then beginTransaction/commitTransaction around each batch of output), which is what lets a Kafka Streams application get exactly-once between its input and a Kafka-native output topic essentially for free; the hard sink-boundary problem above is specifically what remains once the destination is NOT another Kafka topic.
Worked example
A pipeline reading from Kafka, aggregating in a stream processor, and writing to a relational database: the processor's checkpoint interval determines the window of "already-processed-but-not-yet-checkpointed" work that would be replayed after a crash. If the sink write is an idempotent upsert keyed by (partition, offset_range) rather than a plain insert, replaying that window after recovery overwrites the same row with the same value instead of inserting a second row, achieving the same end result as true exactly-once without needing the sink to support two-phase commit at all. The outbox pattern (writing the intended change to a local, transactional outbox table alongside the primary write, then a separate process relaying that outbox to Kafka) is another way to get atomicity between a database write and a downstream event, useful when the direction is reversed (application to Kafka rather than Kafka to database). The same idempotent-upsert technique applies regardless of the concrete sink: a cloud data warehouse table, a Redis-backed online feature store for an ML pipeline, or a relational OLTP database all accept the identical deterministic-key upsert approach, since the technique depends only on the sink supporting an upsert, not on which specific sink product it is.
Trade-offs and pitfalls
Two-phase-commit sinks add real latency (every checkpoint now waits on the sink's commit protocol) and operational complexity (the sink needs to support being coordinated this way at all, which many managed services don't); idempotent upserts are usually the more practical default when available, since they need no special sink-side protocol support, only a stable, deterministic key to upsert against. The mistake to avoid is believing the label "exactly-once" describes the whole pipeline once the Kafka-to-processor leg is solid; if the sink write itself isn't made idempotent or transactionally coordinated, duplicates can still slip in at exactly that boundary, silently undermining every upstream guarantee. It is also worth being honest that true end-to-end exactly-once is not always worth its cost: for a sink and use case where an occasional duplicate can be tolerated or cheaply corrected downstream, at-least-once delivery plus a simpler compensating check is often the more practical target than chasing full exactly-once everywhere.
What does producer idempotence mean in a streaming platform, and how does an idempotent producer (using a producer ID and sequence numbers) prevent duplicate writes from retries at the broker level, without requiring transactions?
Sample Answer
Direct answer
Producer idempotence means the broker can recognize and silently discard a duplicate write caused by the producer's own retry, so a producer can safely retry on any doubt about whether a write succeeded, without that retry ever becoming a second copy of the message.
Structured elaboration
Without idempotence, a producer that sends a record, experiences a network timeout waiting for the acknowledgment, and retries has no way to know whether the original write actually succeeded before the timeout; if it did, the retry creates a genuine duplicate. An idempotent producer solves this by tagging every message with a producer ID (assigned once per producer session) and a strictly increasing sequence number per partition; the broker tracks the last sequence number it accepted from each producer ID per partition, and if it receives a message with a sequence number it's already seen (or below what it expects), it recognizes the retry and discards it without creating a duplicate, while still acknowledging success back to the producer.
Worked example
A producer sends message sequence number 42 to a partition; the write succeeds and is durably replicated, but the acknowledgment is lost in transit back to the producer due to a network blip. The producer, seeing no acknowledgment, retries and resends the same logical message, now tagged with the same sequence number 42. The broker recognizes sequence number 42 has already been accepted for this producer ID on this partition and discards the retry as a duplicate, while still returning a success acknowledgment to the producer, so from the producer's point of view the retry was safe and the log only has one copy of that message.
Trade-offs and pitfalls
Idempotence only protects against duplicates from the SAME producer session retrying; it does not by itself provide atomicity across multiple partitions or multiple topics in one logical operation, which is what Kafka's transactions layer on top of idempotence exists to solve. A producer restarting entirely gets a new producer ID, so idempotence guarantees reset across restarts; anything that must survive a full producer restart without duplication needs an additional mechanism, typically an application-level idempotency key carried in the message itself.
Unlock Full Question Bank
Get access to all 31 Stream Processing and Event Streaming interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.