Stream Processing and Event Streaming Questions
Building on event-streaming platforms: Kafka and message queues, event sourcing, partitioning, consumer groups, exactly-once vs at-least-once delivery, and windowing. Covers handling late and out-of-order events, watermarks, and stateful stream operators. The core skill for real-time data engineering.
You must decide whether a customer-facing real-time dashboard should wait for late data before closing a window, or close early and risk revising numbers later. Walk through the freshness-versus-correctness trade-off and how you'd set the watermark and allowed-lateness values.
Sample Answer
Direct answer
Waiting for late data before closing a window gives a more correct, final number at the cost of delaying every viewer's first look at it; closing early and revising gives an immediately available, if provisional, number that can change, and the right choice depends on whether your audience needs a stable number now or a correct one eventually.
Structured elaboration
The two watermark-tuning knobs that actually implement this decision are the watermark's own bounded-out-of-orderness assumption (how long the window waits before considering itself "done") and the allowed-lateness grace period (how much longer it stays open after that for genuinely late corrections). Setting both tight closes the window fast, showing a number quickly, but risks that number changing later if late data qualifies for a correction pass. Setting both loose delays the first number shown but reduces how often it later changes. A customer-facing dashboard's actual requirement, does the audience expect and tolerate a number that revises itself, or does it need to be stable the moment it's shown, is what should drive this choice, not a generic preference for either speed or correctness in the abstract.
Worked example
For an internal, real-time operations dashboard where operators expect and understand that early numbers are provisional (they're used to seeing a metric "settle" over a few minutes), a tight watermark showing an immediate, occasionally-revised number is the right choice: speed of the first signal matters more than the analyst having to double-check a stable answer. For a customer-facing revenue total shown once per day with no expectation of later revision, a loose watermark that waits substantially longer before closing is the right choice, since a customer noticing the number changed after they already saw it erodes trust far more than a delay in first showing it.
Trade-offs and pitfalls
The common mistake is picking one watermark configuration for the whole system and applying it everywhere, when different consumers of the same underlying data genuinely have different tolerance for a number that later revises itself; the right answer is often to expose both a fast, explicitly-labeled-provisional number and a slower, final one, rather than forcing a single trade-off on every audience.
Compare Kafka's time/size-based retention with log compaction. For a topic tracking the latest state per key (such as a user profile) versus a topic used as a raw audit log, which retention policy fits each, and what role do tombstone records play in compaction?
Sample Answer
Direct answer
Time or size-based retention deletes whole log segments once they age out or the log exceeds a size cap, while log compaction instead keeps only the latest record for each key forever (subject to a tombstone grace period), making compaction the right choice whenever you only care about a key's current state rather than its full history.
Structured elaboration
A raw audit log (every event that ever happened, for compliance or replay) wants time or size retention: keep everything for, say, 90 days, then drop old segments regardless of content. A topic modeling current state per key, like the latest known profile for each user, wants compaction: even if a user's profile was written a year ago and never touched since, compaction guarantees that record survives, because it's still the latest value for that key, while an intermediate revision from between two writes to the same key is safely discarded once compaction runs. A tombstone (a record with a null value for a key) tells the compaction process "this key has been deleted"; the tombstone itself is retained for a configurable grace period (delete.retention.ms) so that any consumer lagging behind still sees the deletion before it disappears, then it too is compacted away.
Worked example
Concretely, for two topic types: order-events-audit (raw audit log) would use cleanup.policy=delete with retention.ms set to your compliance window, e.g. 90 days, so segments older than that are deleted outright, including duplicate old revisions. user-profile-current (event-sourced current-user-state, where only the latest value per key matters) would use cleanup.policy=compact, with min.compaction.lag.ms controlling how long a record must sit before it's eligible to be compacted away by a newer write to the same key, and delete.retention.ms controlling how long a tombstone survives before the key disappears entirely. Some topics reasonably combine both (cleanup.policy=compact,delete) to compact per-key history while still expiring keys that haven't been touched in a very long time.
Trade-offs and pitfalls
Compaction is not instantaneous: it runs periodically over closed log segments, so a consumer reading the "head" of a compacted topic can still briefly see superseded records for a key until the next compaction pass catches up. A common mistake is applying compaction to a topic that's really an event log (where every historical event matters, like an audit trail), which silently discards history you actually needed; compaction only makes sense when the topic's semantics are genuinely "latest value wins."
Design a comprehensive testing strategy for a stateful stream-processing pipeline: unit tests for individual operators, integration tests against an embedded or containerized broker, and production-like end-to-end tests. What's genuinely hard to test in a streaming pipeline that isn't hard in a batch job?
Sample Answer
Direct answer
Testing a stateful stream-processing pipeline needs unit tests for individual operators' logic in isolation, integration tests running the actual topology against a real (embedded or containerized) broker to catch issues unit tests can't see, and production-like end-to-end tests validating timing-sensitive behavior; what's genuinely hard here that a batch job doesn't face is testing behavior that depends on the passage of time and out-of-order arrival, which a batch test's single, static input dataset can't naturally exercise.
Structured elaboration
Unit tests for an individual operator (a windowed aggregation function, a deduplication check) can run in-process against a small, hand-constructed sequence of inputs, verifying the operator's pure logic without needing a broker at all, exactly like testing a batch transformation function. Integration tests need an actual broker (embedded in-process for speed, or a lightweight containerized instance) to catch issues that only show up with real partitioning, real consumer-group behavior, and real serialization, none of which a pure unit test exercises. What's specifically hard and streaming-unique is testing time- and ordering-dependent behavior: a batch job's test input is a fixed, static dataset processed all at once, with no concept of "time passing" during the test; a streaming job's correctness often depends on exactly how and when events arrive relative to a watermark, which means tests need to explicitly control simulated time (advancing a test clock, injecting events out of order, holding back a watermark) to exercise late-data handling, window-close timing, and rebalance-recovery behavior at all.
Worked example
An integration test for a windowed deduplication operator would spin up an embedded broker, publish a deliberately out-of-order sequence of events (including one arriving just inside the allowed-lateness boundary and one arriving just outside it), advance the test's simulated watermark explicitly rather than relying on wall-clock time to pass, and assert that the in-boundary late event was correctly folded into its window while the out-of-boundary one was correctly dropped and counted, exactly the kind of test a purely batch-style "run the transformation on this dataset" test has no natural way to express.
Trade-offs and pitfalls
A common testing gap is validating only the happy path (events arriving in order, on time) and never exercising the late-arrival and rebalance-recovery paths at all, which are exactly the paths most likely to hide real production bugs, since they're the paths a casual manual test naturally avoids exercising. Production-like end-to-end tests (running against realistic data volume and timing) are the most expensive to build and run, so most of a test suite's coverage should come from cheaper unit and integration tests, reserving the expensive end-to-end tests for a smaller number of scenarios that specifically need production-realistic scale or timing to validate.
Define at-most-once, at-least-once, and exactly-once processing guarantees in a streaming system. For each, give a concrete example of how a producer, broker, and consumer would need to behave to provide it, and why exactly-once is the hardest to guarantee end to end.
Sample Answer
Direct answer
At-most-once means a record might be lost but never processed twice; at-least-once means a record might be processed twice but never lost; exactly-once means each record's effect is applied precisely once, which is the hardest to guarantee because it requires coordinating the producer, the broker, and the consumer's side effects all together rather than any one of them acting alone.
Structured elaboration
At-most-once falls out naturally from a consumer that commits its offset before processing (or a producer that doesn't retry on failure): a crash at the wrong moment simply drops the record. At-least-once falls out from a consumer that commits its offset only after processing completes, combined with a producer that retries on any doubt: a crash at the wrong moment causes reprocessing rather than loss, which is why at-least-once plus idempotent processing is the most common practical target. Exactly-once additionally requires that the ENTIRE pipeline, producer write, broker replication, and the consumer's downstream effect, behaves as a single atomic unit, which is why it typically needs either an idempotent producer plus transactional writes (Kafka's approach) or an idempotent sink that makes reprocessing harmless even under at-least-once delivery.
Worked example
For a payment-processing consumer: at-most-once would risk silently never charging a customer if a crash happens between reading the event and processing it (the offset was already committed, so no retry). At-least-once with retry would guarantee the charge attempt is never silently dropped, but risks charging twice if the consumer crashes after charging but before committing its offset, unless the charge operation itself is idempotent (say, keyed by a unique payment-request ID so a retried charge is recognized and ignored). True exactly-once removes even that residual risk by making the whole read-process-commit sequence atomic.
Trade-offs and pitfalls
Most systems in practice target at-least-once delivery plus an idempotent downstream operation, rather than reaching for full exactly-once semantics everywhere, because true exactly-once has real throughput and complexity costs and is often unnecessary when the operation itself can simply be made safe to repeat. The common mistake is treating "exactly-once" as a label you can slap on a system rather than a property you have to actually construct end to end; a pipeline that's exactly-once from producer to broker but writes to a non-transactional external sink without any idempotency mechanism is not exactly-once overall, no matter how the middle of the pipeline is configured.
A consumer group is repeatedly rebalancing, causing task restarts, garbage-collection pauses, and duplicate processing. Walk through your troubleshooting approach and how you'd stabilize the group.
Sample Answer
Direct answer
Repeated rebalances causing task restarts and GC spikes usually trace back to one slow or overloaded consumer instance being falsely suspected dead (missing its heartbeat or session timeout) and getting kicked out, which itself then causes more load on the remaining consumers, creating a self-reinforcing cycle.
Structured elaboration
The troubleshooting sequence should separate cause from symptom. First, check whether any consumer instance is exceeding its max.poll.interval.ms (taking too long between polls, often because its per-record processing logic is doing too much synchronous work) or missing session-timeout heartbeats due to long GC pauses; either causes the group coordinator to consider that instance dead and trigger a rebalance, kicking it out and redistributing its partitions to already-busy peers. Those peers now have more work, which can push them into the same GC-pause or slow-poll trouble, causing a cascading, self-sustaining rebalance storm. The GC pauses themselves are frequently a symptom of memory pressure from accumulating local state or an undersized heap for the actual working set.
Worked example
Concretely: check max.poll.interval.ms against actual observed time-between-polls for the affected consumers (a mismatch here is the direct trigger); check GC logs for pause durations approaching or exceeding the session timeout around the time of each rebalance; check whether the set of partitions reassigned during each rebalance keeps growing (a sign the cascade is spreading to more consumers each round). The fix usually combines increasing max.poll.interval.ms and session.timeout.ms to tolerate realistic processing time, reducing per-poll batch size so a single poll cycle does less synchronous work, and addressing the underlying memory pressure causing the GC pauses (heap sizing, or moving large per-key state off-heap).
Trade-offs and pitfalls
Simply raising the timeouts without addressing the underlying slowness just delays detection of a genuinely stuck consumer, trading faster failure detection for tolerance of temporary slowness; the two need to be balanced against how quickly you actually need to detect a truly dead consumer versus how much legitimate processing-time variance you need to tolerate. A common mistake is treating each rebalance as an isolated incident rather than checking whether it's part of a growing cascade, which delays recognizing the self-reinforcing nature of the problem.
Unlock Full Question Bank
Get access to all 31 Stream Processing and Event Streaming interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.