Batch, Streaming, and Real-Time Serving Trade-offs Questions

Direct answer

For 5 TB/day with an hourly freshness requirement, hourly micro-batch is the right default over always-on streaming: you get the freshness the business actually asked for without paying to keep compute resident between hours, and the cost gap is real, not marginal.

Structured elaboration

The freshness bar here is hourly, not sub-second, so the entire benefit streaming exists to provide (low end-to-end latency) is unused: an always-on stream processor would finish computing an hour's aggregate the moment the last relevant event lands, but nobody reads that number until the top of the next hour anyway. What streaming does cost, always, is a resident cluster: compute (and often state) held 24/7 regardless of whether data is flowing.

Micro-batch instead spins up compute, processes the hour's slice, writes the result, and releases the compute, so you only pay for the minutes you're actually crunching data, not the 50+ minutes per hour you're idle. On operational complexity specifically: micro-batch keeps the team on familiar, simple failure recovery (a failed hourly run just reruns from the last good partition), whereas an always-on streaming job adds real operational surface, checkpoints, consumer lag, state backends, that this workload's relaxed freshness bar doesn't buy back any benefit for.

Concrete optimizations that matter more than the batch-vs-stream choice itself:

• File format: land raw as Parquet (columnar, compressed) rather than raw JSON; this cuts both storage footprint and the bytes the hourly job has to scan.
• Partitioning: partition by hour (and a natural key like region or event type if queries filter on it), so the hourly job and any downstream ad-hoc query only touch the slice it needs.
• Compaction: many small files from continuous ingestion kill batch-read performance; compact into fewer, larger files (target roughly 128MB-1GB per file) either at write time or as a periodic housekeeping job.
• Compression: Snappy or Zstd on top of Parquet trades a small CPU cost for a meaningfully smaller scan volume, which usually pays for itself at this data size.

Worked example

Assume the streaming option is a small always-on cluster (8 vCPUs) running 24/7, and the micro-batch option is a bigger ephemeral cluster (32 vCPUs) that only runs for about 10 minutes each hour, at an illustrative $0.05/vCPU-hour:

DAILY_TB = 5.0
HOURS_PER_DAY, DAYS_PER_MONTH = 24, 30
STREAMING_CLUSTER_CORES, CORE_HOUR_COST = 8, 0.05
BATCH_CLUSTER_CORES, BATCH_MINUTES_PER_RUN = 32, 10

streaming_hours_per_month = HOURS_PER_DAY * DAYS_PER_MONTH        # 720
streaming_core_hours = STREAMING_CLUSTER_CORES * streaming_hours_per_month  # 5,760
streaming_cost = streaming_core_hours * CORE_HOUR_COST             # $288.00

batch_runs_per_month = HOURS_PER_DAY * DAYS_PER_MONTH              # 720
batch_core_hours = BATCH_CLUSTER_CORES * (BATCH_MINUTES_PER_RUN/60) * batch_runs_per_month  # 3,840
batch_cost = batch_core_hours * CORE_HOUR_COST                     # $192.00

print(f"streaming_core_hours={streaming_core_hours}")
print(f"streaming_cost={streaming_cost:.2f}")
print(f"batch_core_hours={batch_core_hours:.0f}")
print(f"batch_cost={batch_cost:.2f}")
print(f"cost_ratio={streaming_cost/batch_cost:.2f}")
print(f"savings={streaming_cost-batch_cost:.2f}")
time_fraction_pct = (BATCH_MINUTES_PER_RUN/60) * 100
cost_fraction_pct = (batch_core_hours/streaming_core_hours) * 100
idle_fraction_pct = 100 - time_fraction_pct
print(f"time_fraction_pct={time_fraction_pct:.1f}")
print(f"cost_fraction_pct={cost_fraction_pct:.1f}")
print(f"idle_fraction_pct={idle_fraction_pct:.1f}")

Executed output: streaming = 5,760 core-hours -> $288.00/mo; micro-batch = 3,840 core-hours -> $192.00/mo. That's a 1.5x cost ratio in streaming's favor for micro-batch, a $96/month saving in this illustrative sizing. The 16.7% figure (10 of the 60 minutes in each hour) is the batch cluster's time fraction, not its cost fraction: the batch cluster is 4x the size of the streaming one (32 vCPUs vs. 8), so its actual core-hour draw is 4 times 16.7%, that is 66.7%, of what the always-on cluster consumes, which is exactly the ratio of 3,840 to 5,760 core-hours that produces the 1.5x cost gap. A cluster that is 4x bigger but idle 83.3% of the time still costs less than one running continuously at a quarter of the size, but it is the combination of the bigger core count and the shorter run time, not the 16.7% time fraction by itself, that produces the $96 saving. At a real 5 TB/day scale the cluster sizes and $/core-hour will differ from this illustration, but the shape of the trade (pay only for active minutes vs. pay for 24/7 residency) holds regardless of the specific numbers.

Trade-offs and pitfalls

The common mistake is picking streaming because 5 TB/day sounds like it needs "big data" infrastructure. Data volume and latency requirement are independent axes: high volume with a relaxed latency bar is exactly the case micro-batch is built for, and paying an always-on tax for freshness nobody asked for is pure waste. The flip side pitfall: if the business later needs the freshness bar tightened to minutes, don't just shrink the batch interval indefinitely, since per-run overhead (cluster startup, small-file proliferation) stops being a rounding error somewhere around 5-15 minute intervals, and that's the point to seriously evaluate a real streaming engine instead.

Direct answer

The decision framework has five parts (latency, cost, operational complexity, data correctness, and business value), and the answer to "should we migrate" only becomes yes once a genuine business need for latency below what batch can deliver is established; the migration itself should be executed as a parallel-run, not a cutover, so correctness is proven before the old system is retired.

Structured elaboration

Decision framework:

• Latency: quantify the gap between what batch currently delivers and what the business need actually requires; if the gap is small (batch already delivers close to what's needed), the case for migrating weakens regardless of the other axes.
• Cost: streaming's always-on compute and operational overhead versus batch's pay-per-run model; this is a real, ongoing cost increase that should be weighed against the latency gain, not treated as a rounding error.
• Operational complexity: honestly assess whether the team has (or can build) the skills to run a production streaming system: state management, monitoring, on-call for a system that, unlike batch, can't just be "rerun tomorrow" if it breaks.
• Data correctness: streaming introduces a genuinely different correctness model (event-time semantics, watermarks, late data) that batch never had to reason about; this isn't free even with a well-executed migration.
• Business value: translate the latency improvement into an actual business outcome (faster fraud detection catching more fraud, fresher dashboards changing a decision sooner), not just "more real-time is better."

Phased migration plan:

1. Parallel run: stand up the streaming pipeline alongside the existing batch pipeline, both reading from the same source, without the streaming pipeline serving any production traffic yet.
2. Parity validation: compare the streaming pipeline's output against the batch pipeline's output over a meaningful historical window (including known edge cases like month-end, late data, and any past incidents), at a fine enough granularity that compensating errors can't hide behind a matching aggregate total.
3. Cutover: once parity holds for an agreed bake period, switch downstream consumers to the streaming pipeline's output, ideally behind a flag that can be reversed quickly.
4. Deprecation: only after the cutover has run cleanly through a full business cycle (including whatever periodic events, like month-end close, are most likely to expose a gap) does the old batch pipeline get formally retired.

Rollback strategy: keep the batch pipeline's code and infrastructure intact (not deleted, just not the primary path) until deprecation, and make the cutover a config/flag change, not a one-way infrastructure migration, so reverting doesn't require rebuilding anything under incident pressure.

Worked example

A 200 TB/day pipeline is approaching its nightly batch window's limit (the job is starting to run past when downstream consumers need it ready). Rather than treating this purely as "the batch job needs to be faster," the team quantifies the actual business latency need (in this case, dashboards need to be ready by 7am, and the batch job is now finishing at 6:45am with shrinking margin) and concludes a streaming migration is justified not by a desire for lower latency in the abstract, but by a concrete, worsening operational risk in the current batch approach. The migration proceeds through the four phases above, with parity validated against three months of historical data including the last two month-end closes, which is where a subtle late-data handling bug in the new streaming pipeline is actually caught, before it ever touched production.

Trade-offs and pitfalls

The most common failure is skipping straight to "we need streaming" once a batch job is straining, without first asking whether the underlying problem (a batch job that's grown too large or inefficient) has a batch-side fix (better partitioning, incremental processing, a bigger cluster) that's cheaper and lower-risk than a full paradigm migration. The second failure is compressing or skipping the parallel-run/parity-validation phase under deadline pressure, which is exactly how subtle correctness bugs (event-time edge cases, a different late-data policy than the old system implicitly had) end up discovered in production instead of in validation.

Direct answer

Migrating from Lambda to Kappa is fundamentally a data-parity project before it's an infrastructure project: the risk isn't standing up a streaming job, it's proving the new single codepath reproduces what the old batch layer was producing, for both live and replayed data, before you retire the batch layer that's been your correctness backstop.

Structured elaboration

Code refactoring: consolidate the batch and speed layer's business logic into one streaming-native implementation. In practice this usually means porting whatever the batch layer does (often SQL or a Spark job) into the stream processor's semantics (windowing, watermarks, state), which is where subtle behavior changes creep in, since "the same aggregation" can mean something different once you have to define how long to wait for late data.

State migration: any state the batch layer implicitly held (via full recompute each run) has to become explicit, durable, checkpointed state in the streaming job. Size this carefully; a batch job that scanned petabytes of history each run was never a good model for what needs to live in a stream processor's state store going forward, so decide what recent window actually needs to be live state versus what can be queried from cold storage.

Reprocessing/backfill plan: this is Kappa's whole value proposition, so prove it works before cutover. Pick a historical window, replay it through the new streaming job from the durable log, and confirm the output matches the old Lambda system's recorded output for that same window, not just that it runs without errors.

Parity tests: run the new Kappa pipeline in shadow, consuming the same live traffic as the existing Lambda system, for long enough to see your full range of edge cases (month boundaries, late data, any known-tricky periods). Diff outputs field by field, not just at an aggregate level, since a matching daily total can hide compensating errors underneath.

Rollback strategy: keep the old Lambda pipeline (or at minimum its batch layer) running and untouched until the new system has passed parity for an agreed bake period, and keep the cutover reversible, meaning downstream consumers should be able to be pointed back at the old output source without a schema change.

Worked example

Suppose the Lambda system's batch layer computes 30-day rolling revenue by user cohort, and the new Kappa job needs to replicate that with event-time windowing and a watermark policy. The parity test replays the last 90 days through the new job and compares its 30-day rolling revenue against the historical batch output, cohort by cohort. If the new job matches on 995 of 1,000 sampled cohort-days and the 5 mismatches all trace to one edge case (users who churned and re-signed up within the window), that's a concrete, fixable code bug (not a shrug-and-ship situation) found before cutover instead of in production.

Trade-offs and pitfalls

The main risk across all of this is declaring victory on aggregate-level parity ("the daily total matches") without checking finer granularity, since two wrong numbers can cancel out. The second risk is under-provisioning replay capacity: if replaying a full backfill window takes days instead of hours because the new streaming engine's state store or checkpointing wasn't sized for a large backfill, you've quietly recreated a batch job's latency inside a system that was supposed to make backfills fast. The third, org-level risk is retiring the old batch layer too early under pressure to "finish the migration"; keep it as a fallback until the new system has survived at least one real edge case (a late-arriving correction, a schema change) in production, not just in the parity test.

Direct answer

For any given analytics workflow, the choice isn't a single spectrum from batch to streaming; it's two separate decisions layered on top of each other: how fresh does this need to be (which points you toward batch or streaming), and, if you need streaming, do you also need a batch backstop for correctness (which is the Lambda-vs-Kappa question).

Structured elaboration

Start with the freshness requirement. If the consumer can tolerate minutes-to-a-day of latency, pure batch wins on cost and operational simplicity; there's nothing to gain from adding streaming machinery. If the consumer needs sub-second-to-seconds latency, you need some form of streaming, full stop, because no amount of clever batch scheduling gets you there.

Once you've established you need streaming, the second decision is whether a single streaming codepath (Kappa) is trustworthy enough on its own, or whether you want a separate, deterministic batch recomputation as a correctness backstop (Lambda). That decision usually turns on: how expensive is being wrong (financial reporting wants a backstop; a live view-count doesn't), and how cheap is full-history replay in your streaming stack (if replay is fast and reliable, Kappa's single codepath is enough; if it isn't, or if a regulator wants a simple, deterministic recompute story, add the Lambda-style batch layer).

Worked example

Take a concrete workflow: computing daily active users for a product dashboard, with a secondary need for a live "users online right now" counter.

• Pure batch fits the daily-active-users number: it's consumed once a day, needs to be exactly right, and a nightly job scanning the day's log is both cheap and simple.
• Pure streaming fits the live counter: it needs to update within seconds and an occasional off-by-a-few-users error during a brief network blip is not a business problem.
• Kappa would fit if this company later wants daily active users computed from the same streaming pipeline as the live counter, and their streaming engine can cheaply replay a day's worth of events to recompute that day's number deterministically, avoiding a second batch codebase.
• Lambda would fit if daily active users feeds into a revenue-attribution or billing calculation where the company genuinely wants a batch recompute as an audit trail, independent of whatever the streaming engine's live counter reported.

Trade-offs and pitfalls

The pitfall in this kind of question is answering "it depends" without giving the two-step decision structure that makes it not ambiguous: freshness requirement first, correctness-backstop need second. Answering with just "streaming is for real-time and batch is for reports" skips the actual judgment call, which is when a single streaming codebase is trustworthy enough to be the only source of truth, and when it isn't.

Direct answer

Batch and streaming are two ways of answering the same question (what does the data say right now) at different points on the latency/cost/complexity curve. Batch reads a large accumulated chunk on a schedule; streaming processes each event as it arrives. Pick batch by default and reach for streaming only when a specific business decision genuinely needs data fresher than the next scheduled run can provide.

Structured elaboration

Latency. Batch is minutes to a day (whatever the schedule is); streaming is sub-second to a few seconds, bounded mainly by how long you wait for late data (the watermark).

Throughput. Batch amortizes overhead across a huge chunk, so it is usually the cheaper way to move the same total volume of data. Streaming pays a small per-event tax (serialization, network hops, state lookups) on every single record, so the same total throughput costs more compute.

Cost. A batch job runs, finishes, and releases its compute. A streaming job holds compute (and often memory-resident state) 24/7 whether or not there's a burst of traffic, so streaming infrastructure has a real always-on cost floor that batch does not.

Operational complexity. Batch failure recovery is simple: rerun the job. Streaming failure recovery has to reason about partial state, checkpoints, exactly-once vs at-least-once delivery, and consumer lag, which means more moving parts, more monitoring surface, and a team that has to understand event-time semantics, not just SQL.

Correctness. Batch naturally sees all the data for a period before computing anything, so late-arriving records are simply part of the input. Streaming has to make an explicit decision about how long to wait for late data (allowed lateness) before it emits a result, which means a streaming aggregate can legitimately differ from the eventual batch recompute of the same period.

Worked example

A nightly-refreshed revenue dashboard is a clean batch case: the business consumes it once a day, a few hours of latency is invisible to the user, and a failed run just reruns. A fraud-detection system that has to block a card swipe before it completes is a clean streaming case: by the time a batch job would even start, the transaction has already succeeded or failed. A hybrid shows up constantly in practice: a company might run streaming only for the handful of metrics that trigger pages or block a transaction, and leave everything else (the other 90% of reporting) on batch, because paying the always-on cost and operational overhead of streaming for a dashboard nobody checks more than once a day has no payoff.

Trade-offs and pitfalls

The most common mistake is treating this as a technology choice instead of a latency-requirement choice: teams reach for Kafka and Flink because streaming sounds more modern, then discover they've taken on 24/7 operational burden for data nobody looks at more than once a day. The second most common mistake is the opposite: assuming batch is always simpler, when a batch job that has grown large enough to blow its nightly window is itself an operational risk. The right first question is always "what decision does this data drive, and how fresh does it actually need to be to drive that decision correctly," not "which technology is state of the art."