Stream Processing and Event Streaming Questions
Building on event-streaming platforms: Kafka and message queues, event sourcing, partitioning, consumer groups, exactly-once vs at-least-once delivery, and windowing. Covers handling late and out-of-order events, watermarks, and stateful stream operators. The core skill for real-time data engineering.
Design an idempotent sink that writes streaming results into an external database that does not support distributed transactions, ensuring no duplicate rows even when the streaming job restarts and reprocesses.
Sample Answer
Direct answer
Design the sink so writes are naturally idempotent, typically an upsert keyed by a deterministic identifier derived from the source event (its partition and offset, or a stable business key), so a replayed write after a restart produces the identical row rather than a second one.
Structured elaboration
Without distributed-transaction support at the sink, you can't get the streaming job's checkpoint and the sink's write to commit as a single atomic unit; the workaround is to make the write itself safe to repeat. Concretely: derive a deterministic key for every write (source offset, or a natural key already unique to the business event), and express the write as "set this row's value" (an upsert) rather than "append a new row." On a restart-and-replay after a crash, the same source records get reprocessed and re-written with the identical key and value, so the external database simply overwrites the same row with the same content, producing no visible duplicate.
Worked example
A sink writing aggregated hourly revenue per product to a relational table keyed by (product_id, hour): an upsert (INSERT ... ON CONFLICT (product_id, hour) DO UPDATE SET revenue = excluded.revenue) means that if the job crashes after writing hour 14's revenue but before its checkpoint commits, and recovery reprocesses and rewrites hour 14's revenue again, the second write simply overwrites the row with (in this case) the same recomputed value, leaving no duplicate row and no incorrect double-counted total.
Trade-offs and pitfalls
This only works cleanly when the write is naturally expressible as an idempotent upsert; an operation with an external side effect (sending a notification, charging a payment) can't be made idempotent this way without an additional mechanism, typically an idempotency key checked and recorded at the point of the side effect itself, not just at the database write. A common mistake is deriving the upsert key from something that isn't actually stable across a replay (a randomly generated ID rather than the source offset or a genuine business key), which silently reintroduces duplicates despite the upsert pattern looking correct.
Compare Kafka's time/size-based retention with log compaction. For a topic tracking the latest state per key (such as a user profile) versus a topic used as a raw audit log, which retention policy fits each, and what role do tombstone records play in compaction?
Sample Answer
Direct answer
Time or size-based retention deletes whole log segments once they age out or the log exceeds a size cap, while log compaction instead keeps only the latest record for each key forever (subject to a tombstone grace period), making compaction the right choice whenever you only care about a key's current state rather than its full history.
Structured elaboration
A raw audit log (every event that ever happened, for compliance or replay) wants time or size retention: keep everything for, say, 90 days, then drop old segments regardless of content. A topic modeling current state per key, like the latest known profile for each user, wants compaction: even if a user's profile was written a year ago and never touched since, compaction guarantees that record survives, because it's still the latest value for that key, while an intermediate revision from between two writes to the same key is safely discarded once compaction runs. A tombstone (a record with a null value for a key) tells the compaction process "this key has been deleted"; the tombstone itself is retained for a configurable grace period (delete.retention.ms) so that any consumer lagging behind still sees the deletion before it disappears, then it too is compacted away.
Worked example
Concretely, for two topic types: order-events-audit (raw audit log) would use cleanup.policy=delete with retention.ms set to your compliance window, e.g. 90 days, so segments older than that are deleted outright, including duplicate old revisions. user-profile-current (event-sourced current-user-state, where only the latest value per key matters) would use cleanup.policy=compact, with min.compaction.lag.ms controlling how long a record must sit before it's eligible to be compacted away by a newer write to the same key, and delete.retention.ms controlling how long a tombstone survives before the key disappears entirely. Some topics reasonably combine both (cleanup.policy=compact,delete) to compact per-key history while still expiring keys that haven't been touched in a very long time.
Trade-offs and pitfalls
Compaction is not instantaneous: it runs periodically over closed log segments, so a consumer reading the "head" of a compacted topic can still briefly see superseded records for a key until the next compaction pass catches up. A common mistake is applying compaction to a topic that's really an event log (where every historical event matters, like an audit trail), which silently discards history you actually needed; compaction only makes sense when the topic's semantics are genuinely "latest value wins."
Write a short producer configuration snippet (any mainstream Kafka client) for a use case demanding durability over raw throughput: no message loss even under broker failure, and no duplicate writes from producer retries. Explain what each setting does.
Sample Answer
Direct answer
For durability over throughput, you want the producer to wait for full replication acknowledgment, retry indefinitely on transient failures, and deduplicate its own retries at the broker so a retried send never becomes a duplicate message.
Structured elaboration
Three settings do the real work: acks=all makes the broker only acknowledge a write once it's replicated to every in-sync replica, not just the leader, so a leader failure right after the write doesn't lose it. enable.idempotence=true turns on the idempotent producer, which tags each message with a producer ID and a per-partition sequence number so the broker can detect and silently drop a duplicate caused by the producer's own retry (this is what makes retries safe to set high without risking duplicate writes). retries set high (or effectively infinite) combined with idempotence means a transient broker hiccup gets retried rather than dropped, without the retry itself becoming a second copy of the message.
Worked example
from kafka import KafkaProducer
producer = KafkaProducer(
bootstrap_servers=["broker1:9092", "broker2:9092"],
acks="all", # wait for all in-sync replicas, not just the leader
enable_idempotence=True, # broker dedupes retries via producer-id + sequence number
retries=2**31 - 1, # retry (near-)indefinitely on retriable errors
max_in_flight_requests_per_connection=5, # safe upper bound while idempotence is on
)
producer.send("orders", key=b"order-123", value=b'{"status":"created"}')
producer.flush()
max_in_flight_requests_per_connection matters because idempotence only preserves ordering guarantees up to 5 in-flight requests per connection; going above that with idempotence enabled is rejected by the client.
Trade-offs and pitfalls
This configuration trades throughput and latency for durability: acks=all waits for the slowest in-sync replica on every write, and high retries can mask a genuinely broken downstream if you don't also alert on retry counts. The common mistake is enabling high retries without idempotence, which silently reintroduces duplicate writes on exactly the failure this configuration is meant to prevent.
What window trigger types are available in stream processing (count-based, time-based, continuous, custom), and for a live analytics use case needing low latency, how would you choose between early triggers and waiting for the window to close?
Sample Answer
Direct answer
Trigger types decide WHEN a window emits a result, independent of the window's size or type: count-based fires after N elements, time-based fires on a clock interval, continuous fires on every new element, and custom triggers let you combine or override these, most commonly to emit an early, provisional result before the window formally closes.
Structured elaboration
A window's type (tumbling, sliding, session) defines what data belongs together; the trigger defines when you actually get a result for that grouping. The default trigger for an event-time window is typically "fire once, when the watermark passes the window's end," but a low-latency use case often can't wait that long for the final, complete answer. A custom trigger lets you emit early, provisional results (say, every 10 seconds while the window is still open) alongside the final, watermark-triggered result, giving users a live-updating number that gets corrected once the authoritative, complete value is available.
Worked example
For a live dashboard showing "revenue in the current hour," waiting for the full hour's watermark to pass before showing anything would mean the number is blank for up to an hour. A custom trigger firing every 30 seconds on the still-open current-hour window, in addition to the final watermark-triggered close, gives viewers a continuously updating (if provisional) number, which then snaps to its final, authoritative value once the hour's watermark passes and any late data has been accounted for.
Trade-offs and pitfalls
Early, provisional results change over the life of the window, which downstream consumers (dashboards, alerts) need to be built to handle gracefully (as an updating value, not a one-time final answer), or they'll misleadingly present a partial number as though it were complete. Firing too frequently on a large window adds real processing and downstream-update overhead for marginal freshness benefit past a certain point, so the trigger interval itself is a trade-off between how live the number needs to feel and how much churn the system can absorb.
Design a multi-region event-streaming topology where each region accepts local writes and reads with low local latency, while still maintaining a globally consistent materialized view. What replication approach and conflict-handling would you use?
Sample Answer
Direct answer
A multi-region streaming topology that keeps local writes and reads fast while still maintaining a globally consistent view means accepting eventual (not immediate) global consistency, replicating each region's local events asynchronously to every other region, and giving the application an explicit way to resolve or accept conflicts when the same entity is updated from two regions close together in time.
Structured elaboration
Requiring every write to synchronously reach every region before acknowledging it would defeat the entire purpose of local, low-latency writes, since every write would then pay full cross-region round-trip latency. The practical design instead has each region accept writes locally (low latency, immediately acknowledged), then asynchronously replicate those events to every other region's copy of the topic, so each region eventually has a complete picture of global activity, just not instantaneously. Conflict handling (two regions modifying the same entity within the replication lag window) needs an explicit, chosen strategy: last-write-wins by timestamp is simplest but can silently discard a legitimate concurrent update; a CRDT-based structure sidesteps the conflict entirely for data types that support it; or, for anything higher-stakes, routing all writes for a given entity to one designated "home" region (sacrificing true multi-region write locality for that entity specifically) removes the conflict possibility altogether.
Worked example
For a global product catalog where different regions rarely update the exact same SKU at the exact same moment, asynchronous cross-region replication with last-write-wins conflict resolution is usually acceptable, since actual conflicts are rare and low-stakes if they occur. For something higher-stakes, like an account balance that must never be double-spent across regions, routing all writes for a given account to a single home region (accepting non-local write latency for THAT account specifically, from other regions) avoids the conflict problem for the data that actually can't tolerate it, while less sensitive data elsewhere in the same platform still enjoys full multi-region write locality.
Trade-offs and pitfalls
Asynchronous replication means a region can, briefly, serve a locally-fast but globally-stale read (an entity that was just updated in another region hasn't replicated here yet); if the application can't tolerate that staleness for a specific read path, that path needs to explicitly route to the entity's home region rather than reading local replicas. The choice of conflict-resolution strategy has to be made per data type based on its actual business tolerance for a silently-dropped concurrent update, not applied blanket across the whole platform.
Unlock Full Question Bank
Get access to all Stream Processing and Event Streaming interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.