Batch, Streaming, and Real-Time Serving Trade-offs Questions

Direct answer

Batch and streaming are two ways of answering the same question (what does the data say right now) at different points on the latency/cost/complexity curve. Batch reads a large accumulated chunk on a schedule; streaming processes each event as it arrives. Pick batch by default and reach for streaming only when a specific business decision genuinely needs data fresher than the next scheduled run can provide.

Structured elaboration

Latency. Batch is minutes to a day (whatever the schedule is); streaming is sub-second to a few seconds, bounded mainly by how long you wait for late data (the watermark).

Throughput. Batch amortizes overhead across a huge chunk, so it is usually the cheaper way to move the same total volume of data. Streaming pays a small per-event tax (serialization, network hops, state lookups) on every single record, so the same total throughput costs more compute.

Cost. A batch job runs, finishes, and releases its compute. A streaming job holds compute (and often memory-resident state) 24/7 whether or not there's a burst of traffic, so streaming infrastructure has a real always-on cost floor that batch does not.

Operational complexity. Batch failure recovery is simple: rerun the job. Streaming failure recovery has to reason about partial state, checkpoints, exactly-once vs at-least-once delivery, and consumer lag, which means more moving parts, more monitoring surface, and a team that has to understand event-time semantics, not just SQL.

Correctness. Batch naturally sees all the data for a period before computing anything, so late-arriving records are simply part of the input. Streaming has to make an explicit decision about how long to wait for late data (allowed lateness) before it emits a result, which means a streaming aggregate can legitimately differ from the eventual batch recompute of the same period.

Worked example

A nightly-refreshed revenue dashboard is a clean batch case: the business consumes it once a day, a few hours of latency is invisible to the user, and a failed run just reruns. A fraud-detection system that has to block a card swipe before it completes is a clean streaming case: by the time a batch job would even start, the transaction has already succeeded or failed. A hybrid shows up constantly in practice: a company might run streaming only for the handful of metrics that trigger pages or block a transaction, and leave everything else (the other 90% of reporting) on batch, because paying the always-on cost and operational overhead of streaming for a dashboard nobody checks more than once a day has no payoff.

Trade-offs and pitfalls

The most common mistake is treating this as a technology choice instead of a latency-requirement choice: teams reach for Kafka and Flink because streaming sounds more modern, then discover they've taken on 24/7 operational burden for data nobody looks at more than once a day. The second most common mistake is the opposite: assuming batch is always simpler, when a batch job that has grown large enough to blow its nightly window is itself an operational risk. The right first question is always "what decision does this data drive, and how fresh does it actually need to be to drive that decision correctly," not "which technology is state of the art."

Direct answer

Lambda architecture runs two separate codepaths, a batch layer for accuracy and a speed layer for freshness, and merges their outputs at serving time. Kappa architecture collapses that into a single streaming codepath with full replay capability, using reprocessing instead of a second batch system to get accuracy. Lambda buys you a battle-tested batch layer as a correctness backstop at the cost of maintaining two implementations of your business logic; Kappa buys you one codebase at the cost of your streaming engine and storage having to support cheap, full-history replay.

Structured elaboration

Lambda's components: a batch layer (recomputes accurate views from the full historical log, usually on a schedule), a speed layer (a stream processor providing low-latency, possibly-approximate views of very recent data), and a serving layer that merges the two, typically overwriting the speed layer's view for a time window once the batch layer's more-accurate result for that window lands.

Kappa's components: a single durable, replayable event log (the source of truth) and one stream-processing codebase that both handles live traffic and, when you need to fix a bug or backfill, gets pointed at an earlier offset and replayed through the same code.

The real cost of Lambda is that your batch and speed layers are two independent implementations of the same business logic, in whatever language/framework each layer uses, and they can silently drift apart (a bug fixed in one and not the other, a metric defined slightly differently). That duplication is the actual argument against it, not raw complexity for its own sake.

The real cost of Kappa is that it pushes all correctness work onto the streaming engine: state management, exactly-once guarantees, and cheap full-history replay for potentially years of data, all inside one system, which is a harder engineering problem than "run a batch job over the same log."

Worked example

A team building fraud scoring that needs both a fast online signal and periodic full retraining on corrected historical labels is a natural Lambda fit: the speed layer scores in real time with best-effort features, the batch layer recomputes ground truth nightly once labels settle, and a small serving-layer reconciliation step swaps in the corrected value. A team building a single, well-defined metrics pipeline (say, per-minute active users) where the exact same logic should always apply, live or replayed, is a better Kappa fit: one Flink job computes it, and a bug fix just means replaying the log from the point of the bug, not fixing and re-deploying two systems.

Trade-offs and pitfalls

A common mistake is choosing Kappa because "one codebase" sounds obviously simpler, without checking whether the team's streaming engine and storage can actually afford cheap, full-history replay at their retention window and data volume; if replay is slow or expensive, Kappa quietly turns into "one codebase, but reprocessing takes three days," which defeats the purpose. The opposite mistake is defaulting to Lambda out of caution and then never actually keeping the two layers' logic in sync, which produces the exact dual-codepath drift Lambda is often criticized for.

Direct answer

For any given analytics workflow, the choice isn't a single spectrum from batch to streaming; it's two separate decisions layered on top of each other: how fresh does this need to be (which points you toward batch or streaming), and, if you need streaming, do you also need a batch backstop for correctness (which is the Lambda-vs-Kappa question).

Structured elaboration

Start with the freshness requirement. If the consumer can tolerate minutes-to-a-day of latency, pure batch wins on cost and operational simplicity; there's nothing to gain from adding streaming machinery. If the consumer needs sub-second-to-seconds latency, you need some form of streaming, full stop, because no amount of clever batch scheduling gets you there.

Once you've established you need streaming, the second decision is whether a single streaming codepath (Kappa) is trustworthy enough on its own, or whether you want a separate, deterministic batch recomputation as a correctness backstop (Lambda). That decision usually turns on: how expensive is being wrong (financial reporting wants a backstop; a live view-count doesn't), and how cheap is full-history replay in your streaming stack (if replay is fast and reliable, Kappa's single codepath is enough; if it isn't, or if a regulator wants a simple, deterministic recompute story, add the Lambda-style batch layer).

Worked example

Take a concrete workflow: computing daily active users for a product dashboard, with a secondary need for a live "users online right now" counter.

• Pure batch fits the daily-active-users number: it's consumed once a day, needs to be exactly right, and a nightly job scanning the day's log is both cheap and simple.
• Pure streaming fits the live counter: it needs to update within seconds and an occasional off-by-a-few-users error during a brief network blip is not a business problem.
• Kappa would fit if this company later wants daily active users computed from the same streaming pipeline as the live counter, and their streaming engine can cheaply replay a day's worth of events to recompute that day's number deterministically, avoiding a second batch codebase.
• Lambda would fit if daily active users feeds into a revenue-attribution or billing calculation where the company genuinely wants a batch recompute as an audit trail, independent of whatever the streaming engine's live counter reported.

Trade-offs and pitfalls

The pitfall in this kind of question is answering "it depends" without giving the two-step decision structure that makes it not ambiguous: freshness requirement first, correctness-backstop need second. Answering with just "streaming is for real-time and batch is for reports" skips the actual judgment call, which is when a single streaming codebase is trustworthy enough to be the only source of truth, and when it isn't.

Direct answer

These two needs sit at opposite ends of the latency spectrum and should not be forced onto one pipeline: daily revenue reports are a batch problem, real-time fraud alerts are a streaming problem, and trying to serve both from the same infrastructure choice usually means over-paying for one or under-serving the other.

Structured elaboration

For (A), daily aggregated revenue reports: analysts consume this once a day, the numbers need to be complete and auditable (finance will reconcile them against source systems), and a few hours of processing latency is invisible to the consumer. A nightly batch job that waits for the full day's data, recomputes cleanly, and can be rerun if something goes wrong is the right shape: simple failure recovery, lower compute cost per unit of data processed, and no need to reason about partial/late data during the run because by the time it runs, the day is over.

For (B), fraud alerts within 10 seconds: by definition, this decision has to be made before a batch job would even have started. This is not a latency optimization on top of batch, it is a different processing model: a streaming (or at minimum sub-minute micro-batch) pipeline that scores each transaction as it arrives, with a state store holding whatever recent history the model needs (velocity checks, recent device/IP behavior) and an alerting path that can act within the window.

Worked example

Concretely: revenue reporting reads from the same append-only event log as the fraud pipeline, but on a completely separate cadence and infrastructure. The fraud path needs a stateful stream processor (state per card/account, sub-second decisions) and pays 24/7 compute for that state and the always-on stream. The reporting path needs a scheduled job (Airflow-triggered Spark or a warehouse-native transform) that reads yesterday's partition once, and pays only for the run itself, typically a fraction of the fraud path's monthly compute for a similar data volume, because it isn't holding anything resident between runs.

Trade-offs and pitfalls

The trap is assuming one architecture has to serve both because they draw from the same source events. That's a data-modeling question (do both consume the same log, yes), not a processing-model question (do both need the same latency, no). Building the fraud pipeline on top of the revenue pipeline's batch cadence would make fraud detection useless (alerts arrive after the damage is done); building revenue reporting on top of the fraud pipeline's streaming infrastructure would mean paying always-on compute and operational overhead for a report nobody looks at outside business hours, with no benefit, since "more real-time" doesn't make a once-a-day report more correct or more useful.

Direct answer

The key design decision is a clean, well-defined handoff boundary: the streaming path owns everything newer than a fixed cutoff (say, the last 24-48 hours), the batch path owns everything older, and every value in the combined output is tagged with which path produced it, so "double counting" becomes structurally impossible rather than something you have to detect after the fact.

Structured elaboration

The handoff boundary: pick a cutoff time window (based on how long your data typically takes to fully settle, including late arrivals and any correction processes) before which data is considered "batch-owned." The streaming path only ever writes to the window newer than that cutoff; the nightly batch job only ever (re)writes to the window at or older than the cutoff. Neither path writes into the other's territory, which is what actually prevents double-counting, rather than trying to deduplicate after merging two overlapping outputs.

Reconciling overlap: as the cutoff advances each day (yesterday's "streaming-owned" data becomes old enough to be "batch-owned" today), the batch job recomputes that newly-crossed-over window from the full, settled data and overwrites whatever the streaming path had written for it, rather than appending to it. This overwrite, not merge, is what keeps the combined table free of duplicates.

Monotonicity and reproducibility: because the batch job always fully recomputes (and overwrites) its owned windows from the immutable source log, rerunning it for the same historical window always produces the same result, which is what "reproducible" means here. Monotonicity (values only get more correct over time, never inconsistently jump around) follows from the batch job being the sole writer once a window crosses the cutoff; the streaming path never touches that window again.

Implementation detail that matters: this requires the storage layer to support an efficient overwrite-by-partition operation (most warehouse and lakehouse formats do, via a partition-level replace or a MERGE keyed on the time window), not an append-only log, otherwise "overwrite" becomes an expensive full-table rewrite.

Worked example

With a 24-hour cutoff: today's data (0-24 hours old) is served entirely from the streaming path's continuously-updated aggregate. Tomorrow, once today's data is fully settled, the nightly batch job recomputes today's aggregate from the raw log and overwrites the partition the streaming path had been writing, at which point that partition is permanently batch-owned and the streaming path stops writing to it. A user querying "yesterday's total" always gets the batch-recomputed, final value; a user querying "today's total" always gets the streaming path's continuously-updating value, and the two never both contribute to the same partition.

Trade-offs and pitfalls

The common mistake is designing this as an append-and-deduplicate system (both paths write, then a dedup step tries to reconcile duplicates by, say, keeping the most recent write per key), which is fragile: dedup logic itself can have bugs, and it obscures which value is authoritative at any given moment. The partition-ownership-by-time-window design avoids that whole class of bug by construction. The remaining risk is choosing a cutoff that's too short for how long your data actually takes to settle, in which case the batch job will be overwriting a partition that hasn't fully arrived yet, reintroducing exactly the streaming-vs-batch discrepancy this design is meant to eliminate; the cutoff should be set from measured late-arrival data, not a round number picked by convention.