Stream Processing and Event Streaming Questions
Building on event-streaming platforms: Kafka and message queues, event sourcing, partitioning, consumer groups, exactly-once vs at-least-once delivery, and windowing. Covers handling late and out-of-order events, watermarks, and stateful stream operators. The core skill for real-time data engineering.
Explain the core building blocks of Apache Kafka: topics, partitions, brokers, replication, and leader/follower roles for a partition. How does a producer's message end up durably stored and available to consumers?
Sample Answer
Direct answer
A producer's message becomes durable when it's written to the leader replica of a partition and then copied to enough in-sync replicas to satisfy the acknowledgment level the producer requested; consumers then read it in the exact order it was appended within that partition.
Structured elaboration
A topic is a named, logical stream of records, split into one or more partitions for parallelism. Each partition is an append-only, ordered log; every record gets a monotonically increasing offset within its partition. Each partition has one leader broker (which serves all reads and writes for it) and zero or more follower replicas that continuously copy the leader's log. Only followers that are caught up within a bounded lag are considered in-sync replicas (ISR); a write is durable once it's replicated across the ISR set, not merely written to the leader's local disk.
Worked example
Concretely: a producer sends a record with key user-42 to topic clicks. The partitioner hashes the key to pick, say, partition 3. The leader broker for partition 3 appends the record at the next offset (say offset 8842), then two followers replicate it. Once both are confirmed in-sync, the broker acknowledges the write back to the producer (assuming acks=all). A consumer subscribed to partition 3 will always see offset 8842 after offset 8841 and before offset 8843, because ordering is only ever guaranteed within a single partition, not across the whole topic.
Trade-offs and pitfalls
Because ordering is per-partition, spreading a topic across more partitions increases throughput and parallel consumption but only preserves ordering for records sharing the same partition (typically enforced via a consistent partition key). A common misconception is expecting global, topic-wide ordering by default; that only holds if the topic has exactly one partition, which caps throughput to a single broker's capacity for that topic.
How would you diagnose and respond to network partitions and split-brain-like symptoms in a streaming ecosystem where brokers, a controller/coordination layer, and downstream stream processors all disagree about cluster state?
Sample Answer
Direct answer
Network partitions and split-brain-like symptoms in a streaming ecosystem show up as brokers, their controller/coordination layer, and downstream processors disagreeing about who's the leader for a partition or which consumer owns which work; the fix is always to trust the coordination layer's quorum decision and force everything else (leader assignment, consumer offsets) to converge to it, never to let two sides keep operating independently.
Structured elaboration
A network partition can split a broker cluster into two groups that can each see a majority (or, worse, neither sees a majority) of the coordination service. If the coordination layer correctly demotes a minority-side broker (a broker that can no longer reach quorum steps down as leader for its partitions), the system self-heals once connectivity is restored: the minority side simply catches up as a follower. The dangerous case is a controller or coordination-layer bug (or misconfiguration like unclean leader election enabled) that lets a minority-side broker keep acting as leader, accepting writes that the majority side never sees, which produces genuinely divergent histories for the same partition, a real split-brain.
Worked example
Concretely, your triage sequence: first check the coordination layer's own health and quorum status (is it itself partitioned, or reporting a clean picture); then check for any partition whose reported leader differs between what different brokers believe versus what the coordination layer says, which is the direct signature of a stale or diverged leader; then check consumer groups for members that appear registered from two different broker views simultaneously, an equivalent split-brain at the consumer-group level. Once found, the resolution is to force the ecosystem to converge on the coordination layer's authoritative view: fence off (or restart) any broker or processor still operating on a stale view, and, if unclean leader election was involved, treat the affected partitions' recent history as suspect and reconcile against a known-good replica.
Trade-offs and pitfalls
Disabling unclean leader election trades availability for consistency: if the only in-sync replica is unreachable, the partition simply goes unavailable rather than electing a stale replica that could silently lose recently acknowledged writes. Teams under availability pressure sometimes enable unclean leader election broadly to avoid outages, without recognizing they've traded a temporary availability problem for a permanent, silent data-consistency problem that's much harder to detect after the fact.
What is tiered storage for a commit-log platform, and how does offloading older log segments to object storage change broker disk usage, read/write latency, and the economics of long-retention or replay-heavy topics?
Sample Answer
Direct answer
Tiered storage moves older log segments from local broker disk to cheaper object storage, which shrinks the broker's local storage footprint and cost dramatically at the price of higher latency for reads that reach back into that older, offloaded data.
Structured elaboration
Without tiered storage, a broker's local disk has to hold every byte of retained data for every partition it leads, which caps how long you can afford to retain data (or requires expensive, large local disks). Tiered storage splits the log into a "hot" tier still on fast local disk (recent segments, actively being written and read) and a "cold" tier offloaded to object storage (older, closed segments). Reads for recent data are unaffected; reads reaching into the cold tier incur the latency of fetching from object storage, generally acceptable for the kind of use case tiered storage targets: long-retention or replay-heavy topics, not latency-critical hot-path consumption.
Worked example
A topic needing 2 years of retention for regulatory replay, but where 99% of real read traffic only ever touches the last 24 hours, is the textbook fit: local disks only need to hold roughly a day's worth of data (dramatically reducing the broker fleet's total disk footprint and cost) while object storage cheaply holds the other 729 days, accessed only on the rare occasion someone needs to replay old history.
Trade-offs and pitfalls
A workload that frequently reads far back into history (contradicting the "cold data is rarely read" assumption tiered storage is built around) will see a real latency and cost hit from constant object-storage fetches, potentially worse than just provisioning enough local disk in the first place. Tiered storage also adds an operational dependency on the object-storage layer's availability for any read that reaches the cold tier, which is a new failure mode a purely local-disk deployment didn't have.
Explain how Kafka consumer groups and partition assignment work: what happens when a consumer joins or leaves the group, and how does the choice of partition key affect the ordering guarantees a consumer sees?
Sample Answer
Direct answer
A consumer group splits a topic's partitions among its members so each partition is consumed by exactly one member at a time; when a consumer joins or leaves, the group rebalances, reassigning partitions among the remaining members, and the partition key you choose determines which records land together and are therefore ordered relative to each other.
Structured elaboration
Every partition within a consumer group is owned by exactly one consumer instance at any moment, which is what lets a group scale out: adding more consumer instances (up to the partition count) increases parallel throughput. When membership changes (a consumer crashes, is added, or a rolling deploy restarts one), the group coordinator triggers a rebalance to redistribute ownership. Because ordering is only guaranteed within a partition, the partition key is what actually decides the ordering a consumer experiences: records sharing a key always land in the same partition and are processed in the order they were produced, while records with different keys have no ordering relationship at all, even if consumed by the same consumer instance.
Worked example
A consumer group with 6 partitions and 3 consumer instances typically gets 2 partitions each. If a 4th instance joins, a rebalance reassigns ownership so each of the 4 instances gets 1 or 2 partitions. If events are keyed by user_id, all of a given user's events land on the same partition and are guaranteed to be processed in production order relative to each other by whichever consumer instance currently owns that partition, but there's no ordering guarantee between different users' events, even within the same rebalance epoch.
Trade-offs and pitfalls
More consumer instances than partitions means some instances sit idle with no partitions assigned, which is a common source of confusion when scaling out doesn't actually increase throughput. Choosing a low-cardinality or skewed partition key (for example, a key with only a handful of distinct values, or one value that dominates traffic) defeats the purpose of having many partitions, concentrating load on just a few of them regardless of how many consumer instances you add.
A production broker cluster is showing frequent leader elections, shrinking in-sync-replica sets, and rising tail latency (or has just lost a large fraction of brokers with unclean leader election enabled). Walk through your triage checklist and how you'd stabilize the cluster and validate no data was silently lost.
Sample Answer
Direct answer
Frequent leader elections and a shrinking in-sync-replica set point to brokers falling behind or becoming unreachable; the triage priority is to stabilize replication first (find and fix whatever's causing followers to fall behind), and only then worry about performance tuning, because an unstable ISR is an active data-safety risk, not just a latency problem.
Structured elaboration
A broker showing repeated leader elections is either crashing/restarting, suffering long garbage-collection pauses that make it look unresponsive to the coordination layer, or is genuinely network-partitioned from the rest of the cluster. A shrinking ISR set means followers can't keep up with the leader's write rate, most commonly from disk I/O saturation, network bandwidth exhaustion, or the same GC-pause problem affecting the replication path. If a large fraction of brokers is lost and unclean leader election is enabled, any partition whose remaining replicas are all out-of-sync will still elect a leader from among them, silently accepting a replica that may be missing recently-acknowledged writes.
Worked example
A concrete triage checklist: (1) confirm the coordination layer's own health first, since a lagging controller can itself cause spurious leader-election churn; (2) check broker-level GC logs and disk/network I/O metrics for the affected brokers, correlated against the timestamps of the leader elections; (3) identify every partition whose ISR shrank and whether unclean leader election activated for any of them; (4) once the root cause (say, a disk nearing capacity causing write stalls) is fixed and brokers rejoin the ISR, run a targeted comparison (checksums or record counts) on any partition that underwent unclean leader election against the last known-good replica, to find out whether any acknowledged writes were actually lost.
Trade-offs and pitfalls
Restarting a struggling broker without first understanding why it's struggling can make things worse, briefly forcing every partition it led to fail over at once and adding load to the very brokers that must now serve as new leaders. Unclean leader election should generally be treated as an emergency-only escape hatch, not a standing configuration on any topic where losing recently-acknowledged data is unacceptable; the moment it fires, you should treat the affected data as needing a verification pass, not assume everything is fine because the cluster is technically serving traffic again.
Unlock Full Question Bank
Get access to all 9 Stream Processing and Event Streaming interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.