Batch, Streaming, and Real-Time Serving Trade-offs Questions
Reasoning about when to use batch, micro-batch, or continuous streaming and how to serve low-latency analytics: latency, cost, complexity, and correctness trade-offs; lambda vs kappa architectures; and reprocessing semantics. Covers real-time aggregation, freshness vs consistency trade-offs, and reconciling streaming results with batch ground truth, including geospatial and high-throughput real-time workloads under eventual consistency. The data-systems judgment topic for choosing and reconciling batch versus real-time approaches, distinct from the hands-on streaming transport itself.
Explain the real differences between batch processing and stream processing for a production data platform: latency, throughput, cost, operational complexity, and correctness. Give one concrete workload that clearly favors each approach, and describe a scenario where a hybrid of the two is the right call.
Sample Answer
Direct answer
Batch and streaming are two ways of answering the same question (what does the data say right now) at different points on the latency/cost/complexity curve. Batch reads a large accumulated chunk on a schedule; streaming processes each event as it arrives. Pick batch by default and reach for streaming only when a specific business decision genuinely needs data fresher than the next scheduled run can provide.
Structured elaboration
Latency. Batch is minutes to a day (whatever the schedule is); streaming is sub-second to a few seconds, bounded mainly by how long you wait for late data (the watermark).
Throughput. Batch amortizes overhead across a huge chunk, so it is usually the cheaper way to move the same total volume of data. Streaming pays a small per-event tax (serialization, network hops, state lookups) on every single record, so the same total throughput costs more compute.
Cost. A batch job runs, finishes, and releases its compute. A streaming job holds compute (and often memory-resident state) 24/7 whether or not there's a burst of traffic, so streaming infrastructure has a real always-on cost floor that batch does not.
Operational complexity. Batch failure recovery is simple: rerun the job. Streaming failure recovery has to reason about partial state, checkpoints, exactly-once vs at-least-once delivery, and consumer lag, which means more moving parts, more monitoring surface, and a team that has to understand event-time semantics, not just SQL.
Correctness. Batch naturally sees all the data for a period before computing anything, so late-arriving records are simply part of the input. Streaming has to make an explicit decision about how long to wait for late data (allowed lateness) before it emits a result, which means a streaming aggregate can legitimately differ from the eventual batch recompute of the same period.
| Axis | Batch | Streaming |
|---|---|---|
| Latency | minutes to a day | sub-second to seconds |
| Throughput cost per unit volume | lower (amortized) | higher (per-event overhead) |
| Infra cost floor | zero between runs | always-on |
| Ops complexity | low (rerun on failure) | higher (state, checkpoints, lag) |
| Correctness model | sees everything before computing | must decide how long to wait for late data |
Worked example
A nightly-refreshed revenue dashboard is a clean batch case: the business consumes it once a day, a few hours of latency is invisible to the user, and a failed run just reruns. A fraud-detection system that has to block a card swipe before it completes is a clean streaming case: by the time a batch job would even start, the transaction has already succeeded or failed. A hybrid shows up constantly in practice: a company might run streaming only for the handful of metrics that trigger pages or block a transaction, and leave everything else (the other 90% of reporting) on batch, because paying the always-on cost and operational overhead of streaming for a dashboard nobody checks more than once a day has no payoff.
Trade-offs and pitfalls
The most common mistake is treating this as a technology choice instead of a latency-requirement choice: teams reach for Kafka and Flink because streaming sounds more modern, then discover they've taken on 24/7 operational burden for data nobody looks at more than once a day. The second most common mistake is the opposite: assuming batch is always simpler, when a batch job that has grown large enough to blow its nightly window is itself an operational risk. The right first question is always "what decision does this data drive, and how fresh does it actually need to be to drive that decision correctly," not "which technology is state of the art."
Design a hybrid pipeline where nightly batch jobs compute historical values and a streaming job updates recent values on top of them. How do you reconcile the overlap between the two (streamed recent updates vs. batch historical values), prevent double-counting once the batch job catches up, and ensure the combined result is monotonic and reproducible?
Sample Answer
Direct answer
The key design decision is a clean, well-defined handoff boundary: the streaming path owns everything newer than a fixed cutoff (say, the last 24-48 hours), the batch path owns everything older, and every value in the combined output is tagged with which path produced it, so "double counting" becomes structurally impossible rather than something you have to detect after the fact.
Structured elaboration
The handoff boundary: pick a cutoff time window (based on how long your data typically takes to fully settle, including late arrivals and any correction processes) before which data is considered "batch-owned." The streaming path only ever writes to the window newer than that cutoff; the nightly batch job only ever (re)writes to the window at or older than the cutoff. Neither path writes into the other's territory, which is what actually prevents double-counting, rather than trying to deduplicate after merging two overlapping outputs.
Reconciling overlap: as the cutoff advances each day (yesterday's "streaming-owned" data becomes old enough to be "batch-owned" today), the batch job recomputes that newly-crossed-over window from the full, settled data and overwrites whatever the streaming path had written for it, rather than appending to it. This overwrite, not merge, is what keeps the combined table free of duplicates.
Monotonicity and reproducibility: because the batch job always fully recomputes (and overwrites) its owned windows from the immutable source log, rerunning it for the same historical window always produces the same result, which is what "reproducible" means here. Monotonicity (values only get more correct over time, never inconsistently jump around) follows from the batch job being the sole writer once a window crosses the cutoff; the streaming path never touches that window again.
Implementation detail that matters: this requires the storage layer to support an efficient overwrite-by-partition operation (most warehouse and lakehouse formats do, via a partition-level replace or a MERGE keyed on the time window), not an append-only log, otherwise "overwrite" becomes an expensive full-table rewrite.
Worked example
With a 24-hour cutoff: today's data (0-24 hours old) is served entirely from the streaming path's continuously-updated aggregate. Tomorrow, once today's data is fully settled, the nightly batch job recomputes today's aggregate from the raw log and overwrites the partition the streaming path had been writing, at which point that partition is permanently batch-owned and the streaming path stops writing to it. A user querying "yesterday's total" always gets the batch-recomputed, final value; a user querying "today's total" always gets the streaming path's continuously-updating value, and the two never both contribute to the same partition.
Trade-offs and pitfalls
The common mistake is designing this as an append-and-deduplicate system (both paths write, then a dedup step tries to reconcile duplicates by, say, keeping the most recent write per key), which is fragile: dedup logic itself can have bugs, and it obscures which value is authoritative at any given moment. The partition-ownership-by-time-window design avoids that whole class of bug by construction. The remaining risk is choosing a cutoff that's too short for how long your data actually takes to settle, in which case the batch job will be overwriting a partition that hasn't fully arrived yet, reintroducing exactly the streaming-vs-batch discrepancy this design is meant to eliminate; the cutoff should be set from measured late-arrival data, not a round number picked by convention.
A team needs both retrainable ML models (which want reproducible, accurate historical data) and low-latency online scoring. Which would you recommend, Lambda or Kappa, and why? Sketch how you would migrate from the other architecture with minimal risk, and describe how you'd keep the online features and the periodic batch snapshots used for training reconciled with each other.
Sample Answer
Direct answer
For a team that needs both retrainable models on reproducible historical data and low-latency online scoring, I'd lean toward a Kappa-style single streaming codebase with an explicit batch snapshot layer bolted on for training, rather than a full Lambda split, because the two consumers (the online scorer and the training pipeline) actually want the same underlying feature-computation logic, and duplicating that logic in a full Lambda setup is where ML teams most often introduce training-serving skew.
Structured elaboration
The key insight is that "low-latency scoring" and "reproducible training data" don't need two different processing engines, they need one feature-computation codebase and two different consumption patterns of its output: a live path for scoring, and periodic durable snapshots of the same computed features for training.
If choosing Kappa-leaning: run the feature computation as a single streaming job (Flink or Kafka Streams) that both serves online reads and periodically materializes a snapshot of feature values to durable storage (a feature store's offline layer or a warehouse table) at a fixed cadence. Training reads the snapshots; serving reads the live streaming state. Because both paths run the same code, a definition change (say, a new bucketing rule for a categorical feature) automatically applies to both, which is exactly the alignment you want.
Migration steps from a Lambda-style setup: (1) identify where the batch and speed layer logic has already diverged, and reconcile them into one shared definition first, before touching infrastructure; (2) stand up the streaming feature-computation job so it can both serve online reads and write periodic snapshots, running it in shadow (not yet serving) against production traffic; (3) validate that the streaming job's snapshots match the batch layer's historical output for a trailing window, within an agreed tolerance; (4) cut serving traffic over, keep the old batch layer running in parallel for a defined bake period as a fallback; (5) once validated, deprecate the separate batch feature pipeline.
Keeping online features and batch snapshots reconciled: version every feature definition (not just the values), so an offline snapshot can be tagged with exactly which code version produced it; run a scheduled comparison job that samples recent online feature values against what the batch snapshot computed for the same entities and time window, and alert if they diverge past a tolerance (some drift is expected from timing, not necessarily a bug, so the tolerance matters).
Worked example
A personalization model retrains nightly on nightly feature snapshots but scores online with sub-second freshness. Before migration, the team has two implementations of "user's 7-day click rate": a Spark job for training data and a Flink job for serving, and they've already seen the two disagree by a few percent because of a rounding difference in a time-window boundary. After migrating to the shared streaming codebase, the same Flink job computes the click rate for both serving and, once a day, snapshots it to the training table, so the discrepancy is structurally impossible rather than something the team has to keep re-verifying.
Trade-offs and pitfalls
The risk in this approach is treating the snapshot cadence as free: if training needs feature values reproducible for a specific historical instant (for point-in-time correctness, avoiding label leakage), the snapshot job has to be built with that discipline from day one, not bolted on later, since a naive "just query current state" snapshot will leak future information into training data. The other pitfall is skipping the shadow-validation step under time pressure; training-serving skew introduced silently during a rushed migration is far more expensive to debug after the fact than the extra week the validation step costs upfront.
Design a near-real-time feature store that supports both offline batch features and online features with a 1-minute freshness target. How do you reconcile the two, so a value computed by the nightly batch job and the same value computed by the online path don't silently drift apart, and how do you version features as the reconciliation logic changes?
Sample Answer
Direct answer
The way to keep an offline batch-computed value and an online streaming-computed value from silently drifting apart is to make them provably the same computation, not two independently-tuned approximations of the same idea, by sharing the feature definition's code (or at minimum its exact specification) between both paths and continuously measuring the gap between them rather than assuming it's zero.
Structured elaboration
Reconciliation mechanism: implement the feature's transformation logic once, in a form that both the offline batch job and the online serving path can execute (a shared library, or a framework that compiles one definition to both a batch and a streaming runner). This removes the most common cause of drift, which is two engineers implementing "the same" feature slightly differently (different time-window boundary conventions, different null-handling, different rounding).
Feature versioning: attach a version identifier to every feature definition, and store that version alongside every computed value, both online and in the offline snapshot table. This lets you answer "which exact logic produced this number" months later, and lets you roll out a definition change safely (both paths read the new version's code, and old data stays tagged with the old version rather than being silently reinterpreted).
Measuring drift, not assuming it away: run a scheduled job that samples recent entities, pulls both the online-computed value and what the offline batch snapshot computed for the same entity and time, and reports the distribution of differences. Some non-zero drift is expected and fine (timing differences mean the online value reflects slightly more recent data than the batch snapshot captured), but the monitoring job should alert if the drift distribution shifts meaningfully, which usually signals a real bug (a definition mismatch, a bad deploy) rather than expected timing noise.
Handling the reconciliation logic changing over time: when the feature definition itself changes (a new bucketing rule, say), version the change explicitly, backfill historical offline snapshots under the new version if training needs consistent history, and don't silently apply the new definition to old data without re-versioning it, since that would make the drift-monitoring job (and any model trained on the old data) unable to tell what actually changed.
Worked example
A "user's 7-day purchase count" feature is defined once, in a shared library, and compiled to run both inside the online Flink job (serving reads) and the nightly batch snapshot job (training reads). The drift-monitoring job samples 10,000 users daily, compares online vs. snapshot values for the same user and day, and typically finds a small, explainable gap (the online value sometimes includes a purchase from the last few minutes that the prior night's snapshot didn't yet have). When a deploy accidentally changes the online path's window boundary from inclusive to exclusive, the drift-monitoring job's next run shows a sudden, systematic shift in the difference distribution (not just noise), which is exactly the signal that catches the bug before it silently degrades whatever model consumes the feature.
Trade-offs and pitfalls
The most common mistake is building the online and offline computations as two separate implementations "because they're different systems anyway" and trusting code review to keep them aligned; code review reliably misses subtle semantic differences (an inclusive versus exclusive window boundary looks identical in two different codebases written by two different people). The second mistake is treating any measured drift as automatically a bug and chasing it to zero; some drift is structurally expected from timing alone, and the useful signal is a change in the drift distribution, not its mere existence.
Your team processes 100M events/day through a batch pipeline and retrains models every few hours. Someone proposes adding real-time features for personalization. Before committing to that, how would you evaluate whether the need is real: what minimal experiment or prototype would you run, and what objective success and cost criteria would decide whether it's worth building?
Sample Answer
Direct answer
Before building anything, I'd run a cheap, time-boxed experiment (a data-only prototype, not a real streaming pipeline) that estimates the upper bound of value real-time features could add, using data the team already has, and only commit to the real build if that upper bound clears a cost threshold worth the engineering investment.
Structured elaboration
The minimal experiment: take a sample of historical events, and instead of building a live streaming pipeline, simulate what a real-time feature would have looked like by re-deriving it from the existing batch data at fine time granularity (for example, reconstructing "user's activity in the last 5 minutes" from timestamped logs offline). Feed that simulated feature into an offline evaluation of whatever downstream system would use it (a recommendation model, a personalization rule), and measure the lift over the current nightly-batch-derived feature. This gives a real signal about value without paying for any streaming infrastructure.
Why this bounds the value from above: a production real-time pipeline will always be somewhat noisier and lossier than a perfectly-reconstructed offline simulation (late data, partial state on cold start, occasional gaps), so if the simulated version doesn't show a meaningful lift, a real implementation won't either, and you've avoided the build entirely.
Objective success criteria: define the lift threshold before running the experiment, not after seeing the result, ideally in terms the business cares about (a lift in a downstream metric like conversion rate or session length, not just a proxy metric like feature correlation). Also define a cost threshold: estimate the ongoing infrastructure and operational cost of a real streaming build (compute, on-call burden, added system complexity), and require the measured lift to translate into more business value than that ongoing cost, not just be statistically distinguishable from zero.
Worked example
Suppose the offline simulation shows that using 5-minute-fresh features instead of the current several-hours-old batch features would lift a downstream conversion metric by an estimated 0.3 percentage points on a baseline of 4%, a real but modest improvement. If that lift, extrapolated across current volume, is worth less per month than the estimated cost of running the streaming infrastructure to get 5-minute freshness in production, that's a clear "not yet" answer, and the team should revisit only if volume grows enough to change that math, or if the freshness need becomes strategic rather than incremental.
Trade-offs and pitfalls
The main pitfall is running the offline simulation but skipping the cost side of the comparison, which tends to happen when the team is technically excited about streaming and only wants to prove the value case, not weigh it. The other pitfall is picking a lift threshold that's too easy to clear (any statistically significant lift, however tiny) rather than one tied to whether the lift is large enough to justify the ongoing operational cost; a real-time feature that's measurably better but not meaningfully better is a bad trade once you account for what running it in production actually costs.
Unlock Full Question Bank
Get access to all 9 Batch, Streaming, and Real-Time Serving Trade-offs interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.