Stream Processing and Event Streaming Questions
Building on event-streaming platforms: Kafka and message queues, event sourcing, partitioning, consumer groups, exactly-once vs at-least-once delivery, and windowing. Covers handling late and out-of-order events, watermarks, and stateful stream operators. The core skill for real-time data engineering.
A client needs effectively-exactly-once writes into a system of record (for example a ledger or data warehouse) whose sink does not support distributed transactions. Given the available techniques (Kafka transactions with a two-phase-commit sink, idempotent upserts with dedup keys, outbox-plus-CDC, an external coordinator), which would you actually implement for a high-throughput case, and why?
Sample Answer
Direct answer
Given a sink that doesn't support distributed transactions, the most broadly practical choice is idempotent upserts keyed by a deterministic ID derived from the source offset, because it needs no special protocol support from the sink and degrades gracefully; reach for Kafka transactions with a two-phase-commit-capable sink, outbox-plus-CDC, or a fully external coordinator only when the sink genuinely can't express an idempotent write (a pure append-only log with no upsert semantics) or the write itself has side effects that can't be made idempotent.
Structured elaboration
Each option has a distinct failure mode to weigh. Kafka transactions (the producer's transactional.id, initTransactions/beginTransaction/commitTransaction API, paired with consumers reading at isolation.level=read_committed) give exactly-once atomicity across multiple Kafka topics and partitions, and can be extended end-to-end only if the sink itself participates in a two-phase-commit protocol (pre-commit, then a durable commit marker written once the transaction coordinator confirms); this works well when the sink is Kafka itself or a connector explicitly built for it, but most warehouses and ledgers don't expose that kind of commit protocol, and even when they do, the extra coordinator round-trip per transaction adds real latency that hurts at high throughput. Idempotent upserts (keyed by, say, partition + offset) are simple and need no sink cooperation beyond supporting an upsert, but require the write itself to be naturally expressible as "set this row to this value" rather than an unconditional append or an external side effect (charging a card, calling a third-party API) that can't be replayed harmlessly. Outbox-plus-CDC (writing the intended change to a local transactional table, then relaying it via change-data-capture) buys atomicity between a primary write and a downstream event at the cost of an extra moving part (the CDC relay) and added latency. A fully external coordinator (building your own two-phase-commit-like protocol across the source and sink, distinct from Kafka's own built-in transactions) gives a similarly strong guarantee to the Kafka-transactions option but without even Kafka's native support, making it the most operationally complex to build and maintain correctly; it's rarely justified unless neither of the simpler options fits.
Worked example
For a high-throughput case writing aggregated metrics into a data warehouse table: an idempotent upsert keyed by (window_start, dimension_key) is almost always sufficient, since re-computing and re-writing the same aggregate for the same window and key after a recovery produces an identical row, with no need for the warehouse to understand Kafka transactions or a two-phase-commit protocol at all. This is the implementation you'd actually ship for that case; Kafka transactions with a 2PC-capable sink would add a coordinator round-trip to every commit for a guarantee the idempotent upsert already provides more cheaply here, and building a fully external coordinator would add even more operational surface for no additional benefit in this scenario.
Trade-offs and pitfalls
The idempotent-upsert approach breaks down the moment the operation has an external side effect that can't be expressed as a deterministic upsert (sending an email, calling a payment gateway); those cases genuinely need either Kafka transactions paired with a sink that can honor a two-phase commit, a fully external coordinator, or an idempotency-key mechanism enforced at the external system's own API boundary, which the streaming pipeline alone can't manufacture. Choosing the heaviest option (Kafka transactions with a 2PC sink, or a hand-built external coordinator) by default, out of caution, adds real operational and latency cost that's usually unjustified when a simpler idempotent write would have been sufficient for the actual side effect involved.
Design a multi-region event-streaming topology where each region accepts local writes and reads with low local latency, while still maintaining a globally consistent materialized view. What replication approach and conflict-handling would you use?
Sample Answer
Direct answer
A multi-region streaming topology that keeps local writes and reads fast while still maintaining a globally consistent view means accepting eventual (not immediate) global consistency, replicating each region's local events asynchronously to every other region, and giving the application an explicit way to resolve or accept conflicts when the same entity is updated from two regions close together in time.
Structured elaboration
Requiring every write to synchronously reach every region before acknowledging it would defeat the entire purpose of local, low-latency writes, since every write would then pay full cross-region round-trip latency. The practical design instead has each region accept writes locally (low latency, immediately acknowledged), then asynchronously replicate those events to every other region's copy of the topic, so each region eventually has a complete picture of global activity, just not instantaneously. Conflict handling (two regions modifying the same entity within the replication lag window) needs an explicit, chosen strategy: last-write-wins by timestamp is simplest but can silently discard a legitimate concurrent update; a CRDT-based structure sidesteps the conflict entirely for data types that support it; or, for anything higher-stakes, routing all writes for a given entity to one designated "home" region (sacrificing true multi-region write locality for that entity specifically) removes the conflict possibility altogether.
Worked example
For a global product catalog where different regions rarely update the exact same SKU at the exact same moment, asynchronous cross-region replication with last-write-wins conflict resolution is usually acceptable, since actual conflicts are rare and low-stakes if they occur. For something higher-stakes, like an account balance that must never be double-spent across regions, routing all writes for a given account to a single home region (accepting non-local write latency for THAT account specifically, from other regions) avoids the conflict problem for the data that actually can't tolerate it, while less sensitive data elsewhere in the same platform still enjoys full multi-region write locality.
Trade-offs and pitfalls
Asynchronous replication means a region can, briefly, serve a locally-fast but globally-stale read (an entity that was just updated in another region hasn't replicated here yet); if the application can't tolerate that staleness for a specific read path, that path needs to explicitly route to the entity's home region rather than reading local replicas. The choice of conflict-resolution strategy has to be made per data type based on its actual business tolerance for a silently-dropped concurrent update, not applied blanket across the whole platform.
Design a plan to re-partition a Kafka topic to support significantly higher throughput while preserving per-key ordering for existing consumers, including how you'd cut over without downtime or duplicate processing.
Sample Answer
Direct answer
A safe re-partitioning plan increases the topic's partition count for headroom, lets the consumer group rebalance onto the new partitions, and separately addresses any stateful consumer's need to reconcile its per-key state, since the key-to-partition mapping changes for existing keys the moment the partition count changes.
Structured elaboration
The throughput problem itself is solved simply: more partitions means more consumer instances can process in parallel. The subtlety is that most partitioning schemes compute hash(key) % partition_count, so changing partition_count changes where an existing key lands, which breaks per-key ordering continuity for any consumer maintaining state keyed to "whichever partition this key used to be on." A safe cutover sequences this deliberately: increase the partition count during a low-traffic window if possible; let the consumer group's next rebalance pick up the new partitions naturally; and, for any stateful consumer, either accept a one-time state-rebuild (flush and recompute from a bounded recent window) or use a stable virtual-partitioning scheme (a large, fixed number of virtual keys pre-hashed to a smaller number of physical partitions) designed from the start to make future partition-count changes non-disruptive to the key mapping.
Worked example
Doubling a topic's partition count from 20 to 40 to relieve consumer saturation: any consumer maintaining a simple per-key running total needs a plan for the moment of cutover, since a given key's events, previously landing consistently on partition 7, may now land on a different partition (say, 27) post-expansion; without a stable virtual-partitioning scheme designed in advance, the practical plan is to flush and rebuild affected per-key state from a recent, bounded replay window immediately after the partition-count change takes effect, accepting a brief window of reduced completeness during that rebuild.
Trade-offs and pitfalls
A topic's partition count can be increased but essentially never decreased without recreating the topic entirely, so this decision should be made with real headroom in mind rather than incrementally chasing each new scaling wall; a virtual-partitioning scheme designed in from the start avoids the whole key-remapping problem for future expansions, but has to be planned before the topic starts accumulating meaningful stateful consumers, since retrofitting it onto an already-stateful pipeline is exactly the disruptive migration this question is describing.
Design the monitoring and alerting strategy for a production streaming pipeline that feeds business dashboards: which metrics would you track across brokers, producers, consumers, and the processing job, and what would trigger a page versus a ticket?
Sample Answer
Direct answer
Monitoring and alerting for a production streaming pipeline needs to cover broker health (replication and disk), producer health (send-error and retry rates, request latency), consumer health (lag, error rate), and processing-job health (checkpoint duration, processing latency), with paging reserved for signals that mean users are seeing wrong or stale data right now, and tickets for degradation trends that haven't yet crossed that line.
Structured elaboration
At the broker layer: under-replicated partitions, ISR (in-sync-replica) shrinkage, and disk-usage trend are the leading indicators of a durability or availability risk. At the producer layer: send-error rate and retry rate reveal whether producers are actually succeeding at getting data into the pipeline at all, a rising retry rate is an early warning of a broker or network problem well before it shows up as consumer lag, and request/produce latency together with how full the producer's internal send buffer is reveal whether producers are being throttled or are buffering faster than they can flush, which, left unnoticed, becomes silent data loss (a failed send after retries are exhausted) rather than merely a visible delay. At the consumer layer: consumer lag (both raw and converted to estimated time-behind-real-time) is the single most important signal, since it directly reflects freshness, alongside error and retry rates. At the processing-job layer: checkpoint duration and success rate reveal whether the job can actually recover within an acceptable time if it crashed right now, and end-to-end processing latency reveals whether the pipeline is meeting its actual freshness commitment to consumers of its output. Paging should be reserved for signals meaning something is ALREADY visibly wrong (lag past a threshold that violates a committed SLA (service-level agreement), checkpoints failing outright, or a producer's send-error rate showing actual data loss after exhausted retries); a ticket, reviewed during business hours, fits a slower degradation trend (disk usage climbing toward a threshold weeks out, or producer retry rate creeping up without failures yet) that doesn't need someone woken up at 3am.
Worked example
A concrete alerting policy: page when consumer lag, converted to estimated time-behind-real-time, exceeds the pipeline's documented freshness SLA for more than 5 consecutive minutes (filtering out a brief rebalance-induced blip); page when checkpoints fail 3 times consecutively (a single transient failure retries automatically and doesn't need a human yet); page when a producer's send-error rate (sends that failed after exhausting retries, meaning the record never reached the broker) exceeds even a small threshold, since that's actual, unrecoverable data loss, not just delay; file a ticket when producer retry rate climbs above its normal baseline without yet crossing the failure threshold, as an early warning worth investigating during business hours; file a ticket when disk usage on any broker crosses 70% of capacity, giving time to add capacity before it becomes a page-worthy emergency at 90%.
Trade-offs and pitfalls
Alerting on every metric individually, rather than composing a smaller number of business-meaningful thresholds (like "the SLA is at risk") from the underlying metrics, produces alert fatigue that trains the on-call team to ignore pages, which is more dangerous than having too few alerts. Producer-side alerting is also the layer most often skipped in practice, since producers are frequently owned by a different team than the pipeline itself, and it's tempting to assume someone else is watching them; that assumption leaves an actual data-loss failure mode (a send failing after exhausted retries) invisible to the team that most needs to know. The common mistake is copying a generic monitoring template wholesale rather than tying each alert back to an actual, specific consequence for users if it fires.
Explain how to achieve end-to-end exactly-once semantics for a pipeline that reads from Kafka, processes in a stream processor such as Flink, and writes to an external sink that does not natively support Kafka's transactions (for example a relational database or object store). Cover Kafka producer idempotence and transactions, the processor's checkpointing and two-phase-commit sink support, and idempotent writes at the sink.
Sample Answer
Direct answer
Achieving end-to-end exactly-once from Kafka through a stream processor to an external sink combines three layers working together: an idempotent, transactional producer so retries never duplicate a write; the stream processor's checkpointing coordinated with a two-phase-commit (or equivalent) sink so output is only visible once the corresponding input has been durably checkpointed; and, where the sink itself can't participate in that coordination, idempotent writes at the sink as the fallback that makes reprocessing harmless.
Structured elaboration
Kafka's own producer idempotence and transactions solve the producer-to-broker leg: a producer can write to multiple partitions atomically and recover from a crash without duplicating output, because a transaction is either fully committed or fully aborted. The stream processor's checkpointing (Flink's distributed snapshots, for instance) captures the exact input offset the processor has consumed up to, alongside any internal state, so recovery resumes from precisely that point rather than reprocessing or skipping. The hard part is the sink: if it's Kafka-native (another topic), the processor's transactional producer covers it directly. If it's an external system that doesn't support the same transactional protocol (a relational database, an object store), the standard techniques are a two-phase-commit sink (the processor coordinates a pre-commit and commit phase with the sink, tied to its own checkpoint barrier) or, more commonly in practice, idempotent writes at the sink (an upsert keyed by a deterministic ID derived from the source offset, so reprocessing the same input after a recovery produces the same final row rather than a duplicate one). Kafka Streams exposes this producer-side transactional support directly through its client API (initTransactions, then beginTransaction/commitTransaction around each batch of output), which is what lets a Kafka Streams application get exactly-once between its input and a Kafka-native output topic essentially for free; the hard sink-boundary problem above is specifically what remains once the destination is NOT another Kafka topic.
Worked example
A pipeline reading from Kafka, aggregating in a stream processor, and writing to a relational database: the processor's checkpoint interval determines the window of "already-processed-but-not-yet-checkpointed" work that would be replayed after a crash. If the sink write is an idempotent upsert keyed by (partition, offset_range) rather than a plain insert, replaying that window after recovery overwrites the same row with the same value instead of inserting a second row, achieving the same end result as true exactly-once without needing the sink to support two-phase commit at all. The outbox pattern (writing the intended change to a local, transactional outbox table alongside the primary write, then a separate process relaying that outbox to Kafka) is another way to get atomicity between a database write and a downstream event, useful when the direction is reversed (application to Kafka rather than Kafka to database). The same idempotent-upsert technique applies regardless of the concrete sink: a cloud data warehouse table, a Redis-backed online feature store for an ML pipeline, or a relational OLTP database all accept the identical deterministic-key upsert approach, since the technique depends only on the sink supporting an upsert, not on which specific sink product it is.
Trade-offs and pitfalls
Two-phase-commit sinks add real latency (every checkpoint now waits on the sink's commit protocol) and operational complexity (the sink needs to support being coordinated this way at all, which many managed services don't); idempotent upserts are usually the more practical default when available, since they need no special sink-side protocol support, only a stable, deterministic key to upsert against. The mistake to avoid is believing the label "exactly-once" describes the whole pipeline once the Kafka-to-processor leg is solid; if the sink write itself isn't made idempotent or transactionally coordinated, duplicates can still slip in at exactly that boundary, silently undermining every upstream guarantee. It is also worth being honest that true end-to-end exactly-once is not always worth its cost: for a sink and use case where an occasional duplicate can be tolerated or cheaply corrected downstream, at-least-once delivery plus a simpler compensating check is often the more practical target than chasing full exactly-once everywhere.
That is every published Stream Processing and Event Streaming question for Full-Stack Developer so far. Browse the other topics in this category, or practice this one interactively.