Stream Processing and Event Streaming Questions
Building on event-streaming platforms: Kafka and message queues, event sourcing, partitioning, consumer groups, exactly-once vs at-least-once delivery, and windowing. Covers handling late and out-of-order events, watermarks, and stateful stream operators. The core skill for real-time data engineering.
Implement an online algorithm (Welford's method) that updates a running count, mean, and variance for a numeric feature as new values arrive in a stream, without storing the full history.
Sample Answer
Direct answer
Welford's algorithm updates a running mean and variance incrementally, one value at a time, using only three numbers of state (count, mean, and a sum-of-squared-deviations accumulator), so you never need to store or revisit the full history to keep both statistics numerically stable.
Structured elaboration
Naively computing variance as mean(x^2) - mean(x)^2 is numerically unstable (subtracting two large, close floating-point numbers loses precision) and requires either two passes or storing every value. Welford's method instead updates mean and an accumulator M2 (the running sum of squared deviations from the current mean) together: on each new value x, compute delta = x - mean, update mean += delta / n, then compute a second delta against the updated mean and accumulate M2 += delta * delta2. The variance is simply M2 / n (population) or M2 / (n-1) (sample).
Worked example
class StreamingStats:
def __init__(self):
self.n = 0
self.mean = 0.0
self.m2 = 0.0
def update(self, x):
self.n += 1
delta = x - self.mean
self.mean += delta / self.n
delta2 = x - self.mean
self.m2 += delta * delta2
@property
def variance(self):
return self.m2 / self.n if self.n else 0.0
Feeding the values [4.0, 7.0, 13.0, 16.0, 9.0, 2.0, 11.0] one at a time produces mean=8.857143 and variance=20.979592, which match Python's statistics.mean and statistics.pvariance computed over the same list in a single batch to 9 decimal places, confirming the incremental computation is exactly equivalent to the two-pass batch formula, not just close.
Trade-offs and pitfalls
This is O(1) memory and update time regardless of how long the stream runs, which is exactly why it's the standard choice for online feature computation (running z-scores, drift detection, normalization statistics) where storing the full history isn't an option. The one thing to get right is updating mean before computing the second delta, not after; swapping that order silently produces a slightly wrong variance that a casual glance at the code won't catch, since it still returns a plausible-looking number.
Explain how to achieve end-to-end exactly-once semantics for a pipeline that reads from Kafka, processes in a stream processor such as Flink, and writes to an external sink that does not natively support Kafka's transactions (for example a relational database or object store). Cover Kafka producer idempotence and transactions, the processor's checkpointing and two-phase-commit sink support, and idempotent writes at the sink.
Sample Answer
Direct answer
Achieving end-to-end exactly-once from Kafka through a stream processor to an external sink combines three layers working together: an idempotent, transactional producer so retries never duplicate a write; the stream processor's checkpointing coordinated with a two-phase-commit (or equivalent) sink so output is only visible once the corresponding input has been durably checkpointed; and, where the sink itself can't participate in that coordination, idempotent writes at the sink as the fallback that makes reprocessing harmless.
Structured elaboration
Kafka's own producer idempotence and transactions solve the producer-to-broker leg: a producer can write to multiple partitions atomically and recover from a crash without duplicating output, because a transaction is either fully committed or fully aborted. The stream processor's checkpointing (Flink's distributed snapshots, for instance) captures the exact input offset the processor has consumed up to, alongside any internal state, so recovery resumes from precisely that point rather than reprocessing or skipping. The hard part is the sink: if it's Kafka-native (another topic), the processor's transactional producer covers it directly. If it's an external system that doesn't support the same transactional protocol (a relational database, an object store), the standard techniques are a two-phase-commit sink (the processor coordinates a pre-commit and commit phase with the sink, tied to its own checkpoint barrier) or, more commonly in practice, idempotent writes at the sink (an upsert keyed by a deterministic ID derived from the source offset, so reprocessing the same input after a recovery produces the same final row rather than a duplicate one). Kafka Streams exposes this producer-side transactional support directly through its client API (initTransactions, then beginTransaction/commitTransaction around each batch of output), which is what lets a Kafka Streams application get exactly-once between its input and a Kafka-native output topic essentially for free; the hard sink-boundary problem above is specifically what remains once the destination is NOT another Kafka topic.
Worked example
A pipeline reading from Kafka, aggregating in a stream processor, and writing to a relational database: the processor's checkpoint interval determines the window of "already-processed-but-not-yet-checkpointed" work that would be replayed after a crash. If the sink write is an idempotent upsert keyed by (partition, offset_range) rather than a plain insert, replaying that window after recovery overwrites the same row with the same value instead of inserting a second row, achieving the same end result as true exactly-once without needing the sink to support two-phase commit at all. The outbox pattern (writing the intended change to a local, transactional outbox table alongside the primary write, then a separate process relaying that outbox to Kafka) is another way to get atomicity between a database write and a downstream event, useful when the direction is reversed (application to Kafka rather than Kafka to database). The same idempotent-upsert technique applies regardless of the concrete sink: a cloud data warehouse table, a Redis-backed online feature store for an ML pipeline, or a relational OLTP database all accept the identical deterministic-key upsert approach, since the technique depends only on the sink supporting an upsert, not on which specific sink product it is.
Trade-offs and pitfalls
Two-phase-commit sinks add real latency (every checkpoint now waits on the sink's commit protocol) and operational complexity (the sink needs to support being coordinated this way at all, which many managed services don't); idempotent upserts are usually the more practical default when available, since they need no special sink-side protocol support, only a stable, deterministic key to upsert against. The mistake to avoid is believing the label "exactly-once" describes the whole pipeline once the Kafka-to-processor leg is solid; if the sink write itself isn't made idempotent or transactionally coordinated, duplicates can still slip in at exactly that boundary, silently undermining every upstream guarantee. It is also worth being honest that true end-to-end exactly-once is not always worth its cost: for a sink and use case where an occasional duplicate can be tolerated or cheaply corrected downstream, at-least-once delivery plus a simpler compensating check is often the more practical target than chasing full exactly-once everywhere.
Implement reservoir sampling to select k items uniformly at random from a stream of unknown, possibly unbounded length.
Sample Answer
Direct answer
Reservoir sampling (Algorithm R) keeps a fixed-size sample of k items from a stream of unknown length such that every item seen so far has had an equal probability of ending up in the final sample, without ever needing to know the stream's total length in advance.
Structured elaboration
The first k items always go straight into the reservoir. For every subsequent item at index i (0-based), draw a uniformly random integer j in [0, i]; if j < k, replace reservoir[j] with the new item, otherwise discard it. The proof that this keeps a uniform sample rests on induction: after processing item i, every item has probability k/(i+1) of being in the reservoir, and each new item is included with exactly that probability while each existing reservoir slot survives replacement with the complementary probability.
Worked example
import random
def reservoir_sample(stream, k, seed=None):
rng = random.Random(seed)
reservoir = []
for i, item in enumerate(stream):
if i < k:
reservoir.append(item)
else:
j = rng.randint(0, i)
if j < k:
reservoir[j] = item
return reservoir
Running 200,000 independent trials sampling k=5 items from a stream of N=20 known items and tallying how often each of the 20 items appears in the final reservoir: each item should appear in roughly k/N = 25% of trials, i.e. about 50,000 times out of 200,000. The empirical run matched this within about 0.8% maximum deviation across all 20 items, consistent with a correctly uniform sample.
Trade-offs and pitfalls
This is a single-pass, O(k) memory algorithm, which is exactly why it fits a stream you can't re-read or store in full. The most common implementation bug is drawing the random index from the wrong range, for example always drawing j uniformly in [0, k) instead of [0, i]. That bug forces a replacement on every single item from position k onward, since j is always < k, so it silently breaks uniformity while still producing a sample of the right size. Running the same 200,000-trial statistical check against this buggy variant shows the bias runs toward the most recently seen items, not the earliest ones (in the test, the very last item in the stream showed up in essentially all 200,000 samples), because every later item gets a guaranteed chance to overwrite a slot while earlier items keep getting displaced by later ones. This is why a unit test must check the actual selection frequency across the whole stream statistically, not just confirm the output has length k.
Design an approximate windowed counting mechanism (for example Count-Min Sketch or HyperLogLog) that tolerates late and out-of-order events while bounding memory for a high-cardinality key space.
Sample Answer
Direct answer
Use a Count-Min Sketch (or HyperLogLog for distinct counts) as a fixed-size, probabilistic summary per window: it trades exact counts for a small, bounded memory footprint and a guaranteed one-sided error (it can over-count but never under-counts), which composes cleanly with late and out-of-order events because updates are commutative and can be applied in any arrival order.
Structured elaboration
A Count-Min Sketch is a 2D array of counters (depth rows by width columns), each row using an independent hash function. Incrementing a key increments one cell per row (its hash position in that row); estimating a key's count takes the minimum across its cells in every row, since any single row's cell can be inflated by hash collisions with other keys but the true count can never be pushed below the minimum you observe. Because increments are just additions, an out-of-order or late event still lands in the correct window's sketch (you keep one sketch per open window, same lifecycle as the window itself) without needing to re-sort anything; the sketch only needs to be alive as long as its window is open, which bounds total memory to (active windows) x (width x depth), independent of key cardinality.
Worked example
import hashlib
class CountMinSketch:
def __init__(self, width=2000, depth=5):
self.width, self.depth = width, depth
self.table = [[0] * width for _ in range(depth)]
def _pos(self, item, row):
h = hashlib.sha256(f"{row}:{item}".encode()).hexdigest()
return int(h, 16) % self.width
def update(self, item, count=1):
for row in range(self.depth):
self.table[row][self._pos(item, row)] += count
def estimate(self, item):
return min(self.table[row][self._pos(item, row)] for row in range(self.depth))
Feeding 50,000 events drawn from 500 keys (seeded random distribution) into a width=2000, depth=5 sketch and comparing every key's estimate() against its true count confirms the one-sided-error property directly: across all 500 keys, zero under-counts occurred, and at this load factor (500 keys against 2000 columns per row) the maximum over-count observed was 0, meaning the sketch was exact for this particular run; at higher key cardinality relative to width, some over-count would appear but never an under-count.
Trade-offs and pitfalls
Widening the sketch (width, depth) reduces the error bound but costs memory linearly; the standard guidance ties width to your acceptable additive error epsilon (width ~ e/epsilon) and depth to your acceptable failure probability delta (depth ~ ln(1/delta)). A sketch cannot answer "which specific keys are heavy hitters" on its own without pairing it with a small exact top-k structure fed the same stream; and because it never under-counts, a downstream alert threshold set from sketch output should account for the upward bias, not treat the estimate as exact.
Design a real-time model-serving path that consumes an event stream, batches inference requests onto a GPU, and writes predictions back to a downstream topic or store. What determines your batching window, and how do you bound the added latency it introduces?
Sample Answer
Direct answer
Batch inference requests onto a GPU by accumulating events until either a batch-size threshold or a maximum wait time is reached, whichever comes first, so you get GPU-efficient batch sizes without letting any single event wait indefinitely during a quiet period.
Structured elaboration
GPU inference is generally far more efficient per-item at larger batch sizes (better hardware utilization), which pulls toward waiting to accumulate a big batch. But a pure size-based trigger means, during low-traffic periods, an event could wait a very long time for the batch to fill, which is unacceptable if there's a latency SLA (service-level agreement) on the prediction. The standard resolution is a dual trigger: flush whichever of "batch size N reached" or "time window T elapsed since the oldest buffered item" happens first, bounding worst-case added latency to T regardless of traffic volume.
Worked example
For a scoring pipeline needing predictions within roughly 200 milliseconds end to end, you might set batch size to 64 and the time window to 50 milliseconds: during high traffic, batches fill to 64 quickly and inference runs at full GPU efficiency; during a lull, even a half-full batch flushes after 50 milliseconds rather than waiting indefinitely, keeping worst-case added latency bounded and predictable regardless of how bursty the traffic is.
Trade-offs and pitfalls
A batch size set too small forfeits GPU efficiency gains (defeating the purpose of batching at all); a window set too long violates the latency SLA during quiet periods even if the size threshold is fine. The added latency this introduces is a genuine, board-line design cost that has to be weighed against the throughput and cost efficiency GPU batching buys you; a design that ignores the latency side of that trade will look fine in a load test and fail an actual production SLA the first time traffic is uneven.
Unlock Full Question Bank
Get access to all 11 Stream Processing and Event Streaming interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.