Stream Processing and Event Streaming Questions
Building on event-streaming platforms: Kafka and message queues, event sourcing, partitioning, consumer groups, exactly-once vs at-least-once delivery, and windowing. Covers handling late and out-of-order events, watermarks, and stateful stream operators. The core skill for real-time data engineering.
Design a streaming pipeline that computes a rolling metric (for example daily or weekly active users, or a per-minute revenue total) over a high-volume event stream, where a meaningful share of events arrive late. Cover ingestion, windowing, watermark strategy, exactly-once handling, and how you'd reconcile a late-arriving correction into an already-served result.
Sample Answer
Direct answer
Designing a streaming pipeline to compute a rolling metric with meaningful late arrivals means combining event-time windowing with an explicitly chosen watermark and allowed-lateness strategy, an exactly-once (or safely idempotent) write path to the serving store, and a defined policy for how a late correction gets reflected once it's already been served.
Structured elaboration
The pipeline shape is: ingest from a durable, replayable source (a commit-log platform); window the aggregation by event time (tumbling for a clean per-period number, sliding if the metric needs to update more often than its own window length); choose a watermark and allowed-lateness setting sized against your real observed lateness distribution, not guessed; write results via an idempotent upsert keyed by the window's identity so a late-triggered recomputation of an already-served window overwrites cleanly rather than double-counting; and decide, deliberately, whether a correction after the fact re-fires and overwrites the previously served value (accepting that consumers see a number change) or is captured separately for reconciliation without disturbing what was already shown.
Worked example
Computing daily active users where up to 10% of events can arrive as much as 24 hours late: ingest from Kafka, window by calendar day (event time), set the watermark's allowed lateness wide enough to cover the bulk of that 24-hour tail (accepting that the very last, rarest late arrivals beyond that are handled by a separate side-output-and-reconciliation path rather than blocking the main pipeline indefinitely), and write the daily count to the serving store as an idempotent upsert keyed by (date), so a late-triggered recompute of yesterday's count safely overwrites the earlier, less-complete number rather than creating a duplicate entry. At 1 million events per minute, the state backend and checkpoint strategy need to be sized to hold roughly a day's worth of in-flight, not-yet-closed windows, which is the concrete capacity-planning number this design has to account for.
Trade-offs and pitfalls
A wider allowed-lateness setting sized to catch "most" late data delays every window's first result and grows in-flight state accordingly; the residual small fraction of even-later data that misses even a generous allowed-lateness window still needs an explicit, deliberate policy (silently drop it, or reconcile it later), not an accidental gap nobody decided on. The most common mistake in this exact design pattern is building the happy-path windowing and forgetting to design the idempotent-write and late-correction behavior with the same rigor, leaving a pipeline that looks correct in a demo but silently double-counts or drops data the first time a real late-arrival burst occurs in production.
Compare how Kafka producers/consumers, Flink, and Spark handle backpressure internally (buffer limits, rate limiting, disk-spilling, adaptive batching), and which of these controls you as an operator can actually tune.
Sample Answer
Direct answer
Kafka producers signal backpressure by blocking or throttling sends once local buffers fill; Flink propagates it through its own back-pressured network stack, visibly slowing upstream operators, and can spill state to disk under memory pressure; Spark's micro-batch model absorbs it differently, by adaptively sizing batches and simply taking longer per batch rather than an explicit signal, and the operator-tunable controls differ correspondingly across the three.
Structured elaboration
A Kafka producer with a full local buffer (its buffer.memory exhausted) will block the calling application thread (or throw, depending on configuration) rather than silently dropping data, which is itself the backpressure signal reaching the application code that's producing; the operator-tunable knobs here are buffer.memory and max.block.ms, trading memory footprint against how long a producer will block before failing. Flink's network stack propagates backpressure operator-by-operator: a slow downstream operator causes its input buffers to fill, which causes the upstream operator feeding it to slow its own output rate correspondingly, visible directly in Flink's own backpressure-monitoring metrics per operator; the operator-tunable controls are the network buffer sizes/counts, and for a RocksDB-backed state backend under memory pressure the backend spills state to local disk rather than failing outright, trading latency for stability, with the spill/memory threshold itself tunable. Spark Structured Streaming's micro-batch model doesn't have quite the same continuous backpressure signal; instead, it uses adaptive batching, dynamically sizing how much input a batch takes based on how long the previous batch took to process (rate-limiting the input rate to keep batch duration under the trigger interval), which an operator tunes via the rate-limiting/backpressure configuration rather than a per-record signal, and a slow batch simply takes longer to complete, delaying the next batch's trigger accordingly.
Worked example
An operator diagnosing a slowdown would check different things per engine: for a Kafka producer, buffer-memory utilization and send-blocking time, tunable via buffer.memory; for Flink, the built-in per-operator backpressure metric (which directly flags which specific operator in the topology is the actual bottleneck) plus RocksDB spill-to-disk metrics if state no longer fits in memory; for Spark Structured Streaming, the batch-duration-versus-trigger-interval metric and the adaptive input rate the backpressure mechanism computed for the next batch, watching for batches taking longer than the configured trigger interval, a sign the pipeline is falling behind its own schedule.
Trade-offs and pitfalls
An operator used to Flink's granular, per-operator backpressure visibility can be caught off guard by Spark's coarser, batch-duration-based signal, which doesn't pinpoint WHICH stage within a micro-batch is actually slow, only that the batch as a whole took too long; diagnosing the specific bottleneck within a slow Spark micro-batch generally needs the Spark UI's own stage-level breakdown rather than a single headline metric. Relying on Flink's disk-spilling as a safety net rather than tuning memory and buffer sizes properly trades a hard failure for a quieter, harder-to-notice latency regression, since a state backend spilling to disk under sustained pressure will keep the job technically running while its actual throughput degrades.
What is a watermark in stream processing, and how does allowed lateness (grace period) let a system decide when a window is 'done' while still tolerating some late-arriving events?
Sample Answer
Direct answer
A watermark is a stream processor's running estimate of "I don't expect to see any more events with a timestamp older than this," and allowed lateness is an explicit grace period past that estimate during which a window stays open anyway, so a genuinely late event can still be folded in before the window is finally closed.
Structured elaboration
Without a watermark, a processor has no principled way to decide a window is "done," since out-of-order arrival means a new, older-timestamped event could always show up later. The watermark advances based on the timestamps of events actually observed (commonly, the maximum event time seen so far, minus a bound on expected out-of-orderness), giving the processor a defensible point to say "close this window now." Allowed lateness pushes that closing point back further: a window whose natural close time has passed can still accept updates until the watermark, adjusted for the allowed-lateness grace period, finally passes it, after which any further arrivals for that window are genuinely too late and are handled separately (dropped, or routed to a side output for manual reconciliation).
Worked example
With 5 seconds of allowed lateness on a 10-second tumbling window: a window covering [00:00, 00:10) would, without allowed lateness, close the instant the watermark passes 00:10. With 5 seconds of allowed lateness, it stays open until the watermark passes 00:15, giving any event that's up to 5 seconds late (by the watermark's own estimate) a chance to still be counted in that window's result, at the cost of delaying that window's final output by those same 5 seconds.
Trade-offs and pitfalls
A looser allowed-lateness setting is more complete (catches more genuinely late data) but delays every window's final result by that same amount, which compounds if you then need to further wait on downstream consumers. A tighter setting closes windows faster but drops more late data as "too late," a trade-off that should be tuned against how late your real upstream systems actually deliver data, not set arbitrarily; setting it far looser than your real lateness distribution just adds latency for no correctness benefit.
Explain how Kafka consumer groups and partition assignment work: what happens when a consumer joins or leaves the group, and how does the choice of partition key affect the ordering guarantees a consumer sees?
Sample Answer
Direct answer
A consumer group splits a topic's partitions among its members so each partition is consumed by exactly one member at a time; when a consumer joins or leaves, the group rebalances, reassigning partitions among the remaining members, and the partition key you choose determines which records land together and are therefore ordered relative to each other.
Structured elaboration
Every partition within a consumer group is owned by exactly one consumer instance at any moment, which is what lets a group scale out: adding more consumer instances (up to the partition count) increases parallel throughput. When membership changes (a consumer crashes, is added, or a rolling deploy restarts one), the group coordinator triggers a rebalance to redistribute ownership. Because ordering is only guaranteed within a partition, the partition key is what actually decides the ordering a consumer experiences: records sharing a key always land in the same partition and are processed in the order they were produced, while records with different keys have no ordering relationship at all, even if consumed by the same consumer instance.
Worked example
A consumer group with 6 partitions and 3 consumer instances typically gets 2 partitions each. If a 4th instance joins, a rebalance reassigns ownership so each of the 4 instances gets 1 or 2 partitions. If events are keyed by user_id, all of a given user's events land on the same partition and are guaranteed to be processed in production order relative to each other by whichever consumer instance currently owns that partition, but there's no ordering guarantee between different users' events, even within the same rebalance epoch.
Trade-offs and pitfalls
More consumer instances than partitions means some instances sit idle with no partitions assigned, which is a common source of confusion when scaling out doesn't actually increase throughput. Choosing a low-cardinality or skewed partition key (for example, a key with only a handful of distinct values, or one value that dominates traffic) defeats the purpose of having many partitions, concentrating load on just a few of them regardless of how many consumer instances you add.
Explain backward, forward, and full schema compatibility modes as enforced by a schema registry. For each mode, give an example schema change (adding a field, removing a field) and say whether it's allowed.
Sample Answer
Direct answer
Backward compatibility means a new schema can read data written with the old schema; forward compatibility means an old schema can read data written with the new schema; full compatibility requires both directions to hold at once, and each mode allows a different, specific set of schema changes.
Structured elaboration
Backward compatibility (the most common mode enforced) allows adding a new field with a default value (an old-schema record simply doesn't have it, and a new-schema reader fills in the default) and removing a field that had a default, but disallows adding a field with no default (an old writer's record would be missing a value the new reader has no way to fill in) or removing a field with no default. Forward compatibility is the mirror image: it allows removing a field or adding one with a default from the reader's perspective, ensuring an OLD reader can still make sense of NEW data, even data written with fields the old reader has never heard of, which it simply ignores. Full compatibility is the intersection of both sets of allowed changes, meaning effectively only additive changes with defaults (or field removals of fields that had defaults) are safe, since anything else breaks one direction or the other.
Worked example
Adding a new optional field referrer (type nullable string, default null) to an event schema is backward compatible (a new-schema consumer reading old data just gets null for the field that wasn't there) and also forward compatible (an old-schema consumer reading new data simply never sees the extra field, ignoring it), so it satisfies full compatibility too. Removing a required field with no default, or changing an existing field's type from string to integer, breaks backward compatibility (an old record won't have a valid value under the new schema's expectations) and generally breaks forward compatibility as well, which is why compatibility-mode checks in a schema registry reject that kind of change outright before it can be registered.
Trade-offs and pitfalls
Teams sometimes register schemas under NONE compatibility mode to unblock a change the stricter mode would reject, without recognizing that this removes the safety net entirely for every future change on that subject, not just the one they were trying to push through. The right response to a rejected, genuinely breaking change is almost always a new topic (or a new schema subject) with a defined migration plan for consumers, not weakening the compatibility mode on the existing one.
Unlock Full Question Bank
Get access to all Stream Processing and Event Streaming interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.