Stream Processing and Event Streaming Questions
Building on event-streaming platforms: Kafka and message queues, event sourcing, partitioning, consumer groups, exactly-once vs at-least-once delivery, and windowing. Covers handling late and out-of-order events, watermarks, and stateful stream operators. The core skill for real-time data engineering.
What triggers a consumer-group rebalance, and how does cooperative-sticky rebalancing reduce disruption to a stateful stream processor compared with the eager rebalance protocol?
Sample Answer
Direct answer
A rebalance triggers whenever group membership changes (a consumer joins, leaves, crashes, or is considered dead by a missed heartbeat) or the subscribed topic's partition count changes; cooperative-sticky rebalancing reduces disruption by only reassigning the specific partitions that actually need to move, instead of revoking every partition from every consumer and reassigning from scratch.
Structured elaboration
The older, eager rebalance protocol works by having every consumer in the group give up all of its partitions before any reassignment happens, meaning even consumers whose ownership isn't actually changing briefly stop processing. This is especially costly for a stateful stream processor, where losing a partition means dropping and later rebuilding local state for that partition's keys. Cooperative-sticky rebalancing instead computes the minimal set of partition moves needed and only revokes those specific partitions, letting consumers keep processing their unaffected partitions the whole time, which is a large win for a job carrying meaningful local state.
Worked example
With 3 consumers each owning 2 of 6 partitions, and a 4th consumer joining, the eager protocol revokes all 6 partitions from all 3 existing consumers, temporarily stops processing on every partition, then reassigns fresh (likely 1-2 partitions per consumer, 4 consumers now). Cooperative-sticky instead computes that only 1 or 2 specific partitions need to move to the new consumer, and only those are revoked; the other 4-5 partitions are never interrupted, so 2 of the 3 original consumers keep processing uninterrupted through the whole rebalance.
Trade-offs and pitfalls
Cooperative-sticky rebalancing takes potentially two rounds of the rebalance protocol to converge (since a partition being revoked from one consumer isn't immediately assigned to another in the same round, to avoid a brief window where two consumers might both think they own it), which is a small added latency cost in exchange for the much larger win of not disrupting unaffected partitions. Mixing consumers on the older eager protocol with ones configured for cooperative-sticky in the same group is not safe and generally requires a coordinated rollout across the whole consumer group.
Design an approach to enrich a high-rate event stream with a slowly-changing dimension table (such as a product catalog or user profile) that updates infrequently. Compare caching with a time-to-live, asynchronous lookups with a fallback, and maintaining the dimension as local processor state.
Sample Answer
Direct answer
Enriching a high-rate stream with a slowly-changing dimension means choosing between caching the dimension locally with a time-to-live, doing an asynchronous lookup with a fallback for cache misses, or maintaining the dimension as local processor state kept fresh by its own change stream, and the right choice depends mainly on how stale a value you can tolerate and how large the dimension is.
Structured elaboration
A TTL-based cache (an in-memory map refreshed periodically, or on a miss) is the simplest option and works well when the dimension is small enough to fit comfortably in memory and moderate staleness (minutes) is acceptable. Asynchronous lookups (calling out to the dimension's source system on a cache miss, with a fallback value or a brief buffering delay while waiting) trade added latency and an external dependency for always-fresh data, appropriate when staleness truly can't be tolerated and the dimension is too large or too volatile to cache wholesale. Maintaining the dimension as local processor state, fed by its own change-data-capture or event stream (essentially a stream-table join where the "table" side is itself streamed as changes happen), avoids both staleness and per-event external calls entirely, at the cost of needing to consume and maintain that second stream as part of the job's own state.
Worked example
For a product catalog that changes a few times per hour and is small enough (a few hundred thousand SKUs) to fit comfortably in memory, maintaining it as local state fed by a change stream from the catalog service gives always-fresh enrichment with no per-event external call and no cache-staleness trade-off at all, the strongest option when it's actually feasible size-wise. For a much larger or more volatile dimension where maintaining full local state isn't practical, for example a 10-million-entry key-value store of user attributes too large to comfortably hold in every processing task's memory, a TTL cache accepting a few minutes of staleness is usually good enough for most product-analytics use cases (such as computing click-through rate per product), and is far simpler to operate than either alternative. A related variant of the local-state approach is maintaining a slowly-changing dimension as a full Type 2 history (keeping every prior value per key with its effective date range, not just the latest), which the same change-stream-fed local state can support if the join needs to know what a dimension's value was AT THE TIME of the fact event, not just its current value.
Trade-offs and pitfalls
Asynchronous lookups with a fallback value need a clear, deliberate answer for what that fallback actually means downstream (a stale-but-known value, or an explicit "unknown" marker), since silently substituting a default that looks like real data can quietly corrupt an aggregate without any visible error. Maintaining a dimension as local state is the most operationally robust option when feasible, but it means the job now depends on and must correctly consume a second stream, doubling the sources of failure that need monitoring.
Explain the difference between event time, processing time, and ingestion time in stream processing, and give a concrete example where using the wrong one produces an incorrect result.
Sample Answer
Direct answer
Event time is when something actually happened in the real world; processing time is when your system happens to handle it; using processing time when event time is what actually matters produces a result that reflects your pipeline's own timing quirks (delays, retries, backpressure) rather than the reality you're trying to measure.
Structured elaboration
The two only coincide when data arrives instantly and in order, which almost never holds in a real distributed system: network delays, retries, mobile devices batching and later flushing offline events, and backpressure all mean a batch of events can arrive minutes or hours after they actually occurred, and can arrive out of the order they happened in. Any metric computed by grouping on processing time ("count of events I happened to receive this hour") answers a different, less useful question than the one grouped by event time ("count of events that actually happened this hour"), and the two answers diverge exactly when your pipeline experiences any delay or reordering, which is precisely when correctness matters most.
Worked example
A mobile app logs a purchase event at 11:58 PM while the phone is offline; the event only reaches the server and gets processed at 12:15 AM the next calendar day, once connectivity returns. A metric grouped by processing time would count that purchase in the wrong day's revenue entirely, inflating the next day and understating the true day it happened on. Grouped by event time (the 11:58 PM timestamp recorded on the device), the purchase correctly lands in the original day's total, which is the number a finance team actually needs.
Trade-offs and pitfalls
Event time requires trusting the timestamp the event carries, which is only as reliable as its source (a device with a wrong clock, or a system that stamps arrival time rather than occurrence time, undermines the whole approach); processing time needs no such trust and is simpler to reason about, which is why it's still the right choice for genuinely processing-time-relevant metrics (like measuring your own pipeline's throughput or latency, where you actually do want to know when things were handled, not when they happened). The mistake is applying one uniformly everywhere rather than choosing per metric based on what question is actually being asked.
For a real-time scoring use case, compare embedding a machine learning model's inference directly inside the stream-processing job versus calling out to a separate model-serving service. What drives the choice?
Sample Answer
Direct answer
Embedding a model's inference directly inside the stream job avoids network round-trip latency and an external dependency, favoring the lowest-latency path; calling out to a separate model-serving service centralizes model versioning, scaling, and monitoring for the model independent of the streaming job, favoring operational cleanliness at the cost of added latency and a new failure mode.
Structured elaboration
Embedding the model means the model's weights and inference code ship as part of the stream job's own deployment artifact, so scoring an event never leaves the process, the fastest possible path but meaning every model update requires redeploying the streaming job itself, and the job's own resource footprint (memory, potentially GPU) now includes the model's requirements too. A separate serving call decouples model lifecycle from the streaming job's lifecycle entirely (the model team can deploy a new model version independently, with its own canary and rollback process), and lets the model-serving layer scale and be monitored on its own, at the cost of a network call on the hot path (latency, and a new dependency that can fail or slow down independent of the streaming job's own health).
Worked example
A use case needing sub-millisecond scoring latency at extremely high event volume, where the model changes infrequently, favors embedding: no external call, and infrequent model updates mean the redeploy-to-update cost is acceptable. A use case where the model is updated frequently (multiple times a day, by a separate ML team iterating quickly) and can tolerate a few milliseconds of added latency favors a separate serving call, decoupling the model team's release cadence entirely from the streaming job's own deployment schedule.
Trade-offs and pitfalls
Embedding ties the streaming job's own release cycle to the model's release cycle, which can become a real organizational bottleneck if the model changes far more often than the streaming logic itself does. Calling out to a serving layer adds a genuine new failure mode (the serving layer being slow or down) that the streaming job needs an explicit fallback behavior for (skip scoring, use a cached prior score, or apply backpressure), not silently blocking indefinitely on a stalled call.
Design a capacity-planning approach for stateful stream-processing jobs using a persistent state backend: forecast disk, memory, CPU, network, and checkpoint-storage growth as a function of business volume, how you'd detect and alert on state blowups, and how you'd choose checkpoint storage (local disk, HDFS, object storage) at multi-terabyte scale while keeping cloud cost under control.
Sample Answer
Direct answer
Capacity planning for a persistent state backend means forecasting disk, memory, CPU, network, and checkpoint-storage growth from your actual per-user or per-key state footprint times expected volume growth, choosing a checkpoint-storage backend that fits your scale and cost target, and alerting on state-size trend directly, not just on disk utilization after the fact.
Structured elaboration
The forecast starts from a concrete unit economics number: bytes of state per key, multiplied by expected key-count growth, gives a disk-growth projection; checkpoint-storage growth follows a similar multiplication but also depends on checkpoint frequency and whether checkpointing is incremental. Memory sizing needs to account for the backend's hot-key cache, sized to keep the actively-touched fraction of keys served from memory rather than disk. CPU forecasting tracks separately: serialization/deserialization cost during checkpointing and restore, plus (for a RocksDB-style backend) background compaction work, both scale with state size and checkpoint frequency, so CPU headroom has to grow alongside disk, not be assumed constant. Network forecasting matters most for checkpoint uploads to remote storage and for state redistribution during rescaling, since both move state-sized payloads across the network on a schedule (checkpoint interval) or an event (a parallelism change), and undersized network capacity shows up as checkpoint duration creeping up long before disk becomes the bottleneck. Choosing the checkpoint-storage backend is its own explicit trade-off: local disk is fastest for checkpoint/restore I/O but doesn't survive node loss and doesn't scale past a single machine's capacity; HDFS gives durable, cluster-local storage with good throughput but adds the operational overhead of running and capacity-planning a second cluster; object storage (S3-compatible) scales effectively without capacity planning and is usually the cheapest per-GB at multi-terabyte scale, at the cost of higher and less predictable latency per checkpoint operation, which matters most when checkpoint or restore time is itself latency-sensitive. Alerting on the state-size trend itself (not just resulting disk utilization, which lags behind and gives less lead time) catches a runaway growth problem while there's still time to act, whether that's an eviction-policy bug or genuine unplanned business growth.
Worked example
For a job forecasted to grow from 50 million to 200 million keys over the next year at roughly 1KB per key, disk capacity planning targets roughly 200GB of state plus headroom for checkpoint storage overhead, and CPU/network capacity are sized to keep checkpoint duration flat as that state grows, not just disk. At this multi-terabyte scale across many such jobs, checkpoint storage moves to object storage specifically for its cost-per-GB and lack of a capacity ceiling, accepting its higher per-operation latency as the trade-off, with cost-reduction levers (moving colder checkpoint history to cheaper storage tiers, tuning TTL (time-to-live) to expire genuinely stale keys sooner) applied specifically to keep the checkpoint-storage cost from growing linearly with raw state size.
Trade-offs and pitfalls
Aggressively minimizing storage cost by trimming TTLs or checkpoint retention can conflict with a genuine business need to hold state longer (a compliance requirement, a feature needing longer-lived history); capacity planning should surface this trade-off explicitly to stakeholders rather than making the cost-minimizing choice unilaterally. Choosing object storage purely for cost without accounting for its latency profile can also silently blow a tight checkpoint-duration or restore-time budget, so the storage-backend choice needs to be validated against your actual latency requirements, not picked on cost alone.
Unlock Full Question Bank
Get access to all 29 Stream Processing and Event Streaming interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.