Batch, Streaming, and Real-Time Serving Trade-offs Questions

Direct answer

At 20,000 events/sec with a sub-5-second ingestion latency target, the ingestion layer needs only a small number of partitions to handle the raw throughput comfortably, and the harder engineering problem is actually the read side: getting thousands of dashboard queries per second down to sub-second latency, which a caching layer in front of the serving store solves cheaply if the read pattern is repetitive (as dashboard traffic usually is).

Structured elaboration

Sizing the ingestion layer: with an average event size and a per-partition throughput ceiling (both stated assumptions here, since real limits depend on your specific broker and hardware), compute the number of partitions from both the message-rate cap and the raw-bytes-per-second cap, and provision for the larger of the two plus headroom for skew and growth.

Processing layer: a stream processor consuming those partitions, doing whatever light transformation/aggregation the metrics require, with parallelism matched to the partition count so no single consumer instance becomes a bottleneck. At this volume (about 19 MB/s), this is a modest cluster, not a large one; the sub-5-second latency bar is comfortably achievable without exotic tuning.

Storage layer for serving: choose a real-time-oriented store (a fast OLAP engine like ClickHouse/Druid, or a low-latency key-value/columnar store) over a traditional data warehouse for the serving path, since warehouses are optimized for large scans, not thousands of small, low-latency point/aggregate queries per second.

Caching strategy for sub-second reads: dashboard read traffic is highly repetitive (many viewers polling the same small set of metric keys), which makes it an excellent caching target. Put a cache (in-memory, short TTL matched to your freshness bar) in front of the serving store, so only a small fraction of reads (cache misses) actually hit the store, and even those hit a store designed for low-latency point reads.

Worked example

Partition sizing (assumptions: 1 KB average event size; a single partition sustains 10,000 msgs/sec or 10 MB/sec, whichever binds first):

EVENTS_PER_SEC, AVG_EVENT_BYTES = 20_000, 1_000
PER_PARTITION_MSG_CAP, PER_PARTITION_MB_CAP = 10_000, 10

throughput_mb_s = EVENTS_PER_SEC * AVG_EVENT_BYTES / (1024*1024)   # 19.07 MB/s
partitions_by_msg_rate = -(-EVENTS_PER_SEC // PER_PARTITION_MSG_CAP)  # ceil -> 2
partitions_by_mb_rate = -(-throughput_mb_s // PER_PARTITION_MB_CAP)   # ceil -> 2

print(f"throughput_mb_s={throughput_mb_s:.2f}")
print(f"partitions_by_msg_rate={partitions_by_msg_rate}")
print(f"partitions_by_mb_rate={partitions_by_mb_rate}")

Executed output: raw throughput is 19.07 MB/s; both the message-rate and byte-rate caps independently require 2 partitions; provisioning 3 (50% headroom) comfortably absorbs skew and near-term growth without needing to repartition soon.

Cache sizing (assumptions: 500 distinct dashboard metric keys, 50 viewers per key, polling every 5 seconds, 98% cache hit rate, 150ms store read latency vs. 3ms cache read latency):

read_qps = 500 * 50 / 5                                    # 5,000 reads/sec, all viewers
store_qps = read_qps * (1 - 0.98)                          # 100 reads/sec actually hit the store
blended_latency_ms = 0.98*3 + 0.02*150                      # 5.94 ms

print(f"read_qps={read_qps:.0f}")
print(f"store_qps={store_qps:.0f}")
print(f"blended_latency_ms={blended_latency_ms:.2f}")

Executed output: 5,000 total read QPS collapses to only 100 QPS actually reaching the serving store once the cache absorbs 98% of reads, and the blended expected latency across all reads is 5.94 ms, comfortably under a one-second target even though a single uncached store read (150ms) is far from instant on its own.

Trade-offs and pitfalls

The mistake to avoid is over-provisioning the ingestion layer out of an instinct that "20,000 events/sec sounds like a lot," when the arithmetic shows it's a genuinely modest throughput for a modern streaming platform; the real engineering effort here is on the read side, where naively hitting the serving store for every dashboard poll (5,000 QPS with no cache) would either require a much more expensive store or blow the sub-second latency target under load. The other pitfall is setting the cache TTL longer than the stated freshness requirement to squeeze out a higher hit rate; the cache's TTL has to be bounded by the same 5-second latency promise the whole system is built to keep, not chosen purely to optimize the cache-hit number.

Direct answer

Migrating from Lambda to Kappa is fundamentally a data-parity project before it's an infrastructure project: the risk isn't standing up a streaming job, it's proving the new single codepath reproduces what the old batch layer was producing, for both live and replayed data, before you retire the batch layer that's been your correctness backstop.

Structured elaboration

Code refactoring: consolidate the batch and speed layer's business logic into one streaming-native implementation. In practice this usually means porting whatever the batch layer does (often SQL or a Spark job) into the stream processor's semantics (windowing, watermarks, state), which is where subtle behavior changes creep in, since "the same aggregation" can mean something different once you have to define how long to wait for late data.

State migration: any state the batch layer implicitly held (via full recompute each run) has to become explicit, durable, checkpointed state in the streaming job. Size this carefully; a batch job that scanned petabytes of history each run was never a good model for what needs to live in a stream processor's state store going forward, so decide what recent window actually needs to be live state versus what can be queried from cold storage.

Reprocessing/backfill plan: this is Kappa's whole value proposition, so prove it works before cutover. Pick a historical window, replay it through the new streaming job from the durable log, and confirm the output matches the old Lambda system's recorded output for that same window, not just that it runs without errors.

Parity tests: run the new Kappa pipeline in shadow, consuming the same live traffic as the existing Lambda system, for long enough to see your full range of edge cases (month boundaries, late data, any known-tricky periods). Diff outputs field by field, not just at an aggregate level, since a matching daily total can hide compensating errors underneath.

Rollback strategy: keep the old Lambda pipeline (or at minimum its batch layer) running and untouched until the new system has passed parity for an agreed bake period, and keep the cutover reversible, meaning downstream consumers should be able to be pointed back at the old output source without a schema change.

Worked example

Suppose the Lambda system's batch layer computes 30-day rolling revenue by user cohort, and the new Kappa job needs to replicate that with event-time windowing and a watermark policy. The parity test replays the last 90 days through the new job and compares its 30-day rolling revenue against the historical batch output, cohort by cohort. If the new job matches on 995 of 1,000 sampled cohort-days and the 5 mismatches all trace to one edge case (users who churned and re-signed up within the window), that's a concrete, fixable code bug (not a shrug-and-ship situation) found before cutover instead of in production.

Trade-offs and pitfalls

The main risk across all of this is declaring victory on aggregate-level parity ("the daily total matches") without checking finer granularity, since two wrong numbers can cancel out. The second risk is under-provisioning replay capacity: if replaying a full backfill window takes days instead of hours because the new streaming engine's state store or checkpointing wasn't sized for a large backfill, you've quietly recreated a batch job's latency inside a system that was supposed to make backfills fast. The third, org-level risk is retiring the old batch layer too early under pressure to "finish the migration"; keep it as a fallback until the new system has survived at least one real edge case (a late-arriving correction, a schema change) in production, not just in the parity test.

Direct answer

In a past disagreement like this, the resolution came from replacing the abstract argument ("real-time" versus "batch is cheaper") with a concrete number: what specific decision does the real-time data drive, and what does it cost the business, in dollars or lost opportunity, if that decision is made an hour later instead of instantly. Once that number existed, the debate stopped being about technology preference and became a straightforward cost-benefit call.

Structured elaboration

Situation: sales had promised a customer "real-time analytics" as part of a deal, without a specific latency number attached; engineering pushed back that a full streaming build-out was expensive and risky to ship on the deal's timeline, and wanted to ship a 15-minute batch refresh instead.

Approach: rather than debating in the abstract, I asked sales what the customer actually did with the data and, specifically, what would go wrong if it were 15 minutes stale instead of instant. It turned out the customer's use case was a daily operations review meeting, not a moment-to-moment trading-style decision, so "real-time" in the sales conversation had really meant "noticeably fresher than the competitor's daily-batch product," not sub-second.

Evaluation: I put together a short comparison for both sides: a true streaming build would take roughly six additional weeks of engineering time and add ongoing operational cost neither team had budgeted for, versus a 15-minute micro-batch refresh that could ship within the existing deadline and would still be dramatically fresher than the customer's status quo.

Worked example

We brought both options to the customer directly, with sales in the room, and let the customer confirm what they actually needed: they cared about seeing yesterday's promotional campaign's performance before the next morning's stand-up, not second-by-second updates. A 15-minute refresh comfortably cleared that bar. That reframed the internal conversation entirely: sales stopped pushing for "real-time" as a literal spec and started selling "fast enough to change your morning decisions," which the 15-minute batch pipeline delivered.

Trade-offs and pitfalls

The outcome was that we shipped on time with the simpler system, and it's still running years later without the operational overhead a real streaming build would have added. The lesson I took from it: "real-time" as stated by a non-technical stakeholder is almost always a proxy for some other requirement (fresher than a competitor, fast enough for a specific workflow), and the fastest way to resolve the disagreement is to find out what that underlying requirement actually is, rather than litigating the word "real-time" itself.

Direct answer

A consistent 2% streaming-vs-batch shortfall with a fixed 10-minute allowed-lateness window is the classic fingerprint of the watermark dropping late events the batch layer still counts, so the investigation should start there before assuming a bug, and the fix is either widening the lateness window or explicitly reconciling the dropped tail, not rewriting the aggregation logic.

Structured elaboration

Step 1, quantify the late-event distribution: pull the true arrival-time-versus-event-time delta for a sample of production events from the raw log (not from the streaming job's output, which has already discarded what it dropped). If a meaningful fraction of events arrive more than 10 minutes after their event time, that's a sufficient explanation for a shortfall, and the size of that fraction should roughly match the 2% gap.

Step 2, confirm the mechanism, not just correlate it: rather than assuming, build the arithmetic. If X% of daily events arrive later than the 10-minute allowed-lateness window, the streaming aggregate will structurally be short by approximately X% (modulo any events that arrive within the window and are still counted). This step (below) shows a concrete, reproducible version of that check.

Step 3, remediate: two independent levers, not mutually exclusive: (a) widen the allowed-lateness window if the operational cost (more held-open state, slightly higher latency before a window finalizes) is acceptable, which recovers some of the gap directly; (b) accept that some tail will always arrive later than any reasonable window and build an explicit reconciliation step that corrects the streaming-derived historical numbers against the batch recompute on a schedule, rather than treating the streaming number as ever being the final one for past periods.

Step 4, correct history: for the past month's data, don't try to "fix" the streaming aggregate retroactively; simply replace the affected historical daily totals with the batch-recomputed values (which, since batch sees the full late tail, are the accurate ones), and communicate clearly to downstream consumers that historical numbers for that period were corrected and why.

Worked example

A synthetic reproduction of the mechanism, with all inputs pinned and seeded:

import random
random.seed(7)

N_EVENTS = 200_000
ON_TIME_FRACTION = 0.965
n_on_time = int(N_EVENTS * ON_TIME_FRACTION)
n_late = N_EVENTS - n_on_time

# late events: minutes-late drawn from an exponential with mean 25 minutes
late_minutes = [random.expovariate(1 / 25) for _ in range(n_late)]
allowed_lateness_minutes = 10

late_but_within_window = sum(1 for x in late_minutes if x <= allowed_lateness_minutes)
streaming_count = n_on_time + late_but_within_window
shortfall_pct = (N_EVENTS - streaming_count) / N_EVENTS * 100

print(f"n_on_time={n_on_time}")
print(f"n_late={n_late}")
print(f"late_but_within_window={late_but_within_window}")
print(f"streaming_count={streaming_count}")
print(f"shortfall_pct={shortfall_pct:.2f}")

Executed output: 200,000 total events (batch ground truth); 193,000 on-time; 7,000 late (of which 2,380 arrive within the 10-minute window and are still counted, 4,620 arrive later and are dropped); streaming count = 195,380; shortfall = 4,620 events = 2.31% of the batch total. This is a synthetic reproduction (not the actual production event-time histogram), but it demonstrates that a late-arrival distribution with roughly this shape, about 3.5% of events arriving late with a mean lateness of 25 minutes against a 10-minute window, is fully sufficient to produce a ~2% streaming shortfall on its own, with no bug required. A real investigation would replace this synthetic distribution with the actual measured one from step 1 and confirm the numbers line up.

Trade-offs and pitfalls

The biggest mistake here is assuming a discrepancy this size must be a bug and going straight to a code review of the aggregation logic, when the far more common cause (a mismatch between the allowed-lateness policy and the real-world late-arrival distribution) is cheap to check first and, in this case, fully explains the gap. The second mistake is widening the lateness window as a blanket fix without checking the cost: every extra minute of allowed lateness means every window stays open longer, holding more state and delaying when downstream consumers can treat a number as final, so the right lever depends on how much that delay actually costs versus how much of the 2% gap it recovers.

Direct answer

Lambda architecture runs two separate codepaths, a batch layer for accuracy and a speed layer for freshness, and merges their outputs at serving time. Kappa architecture collapses that into a single streaming codepath with full replay capability, using reprocessing instead of a second batch system to get accuracy. Lambda buys you a battle-tested batch layer as a correctness backstop at the cost of maintaining two implementations of your business logic; Kappa buys you one codebase at the cost of your streaming engine and storage having to support cheap, full-history replay.

Structured elaboration

Lambda's components: a batch layer (recomputes accurate views from the full historical log, usually on a schedule), a speed layer (a stream processor providing low-latency, possibly-approximate views of very recent data), and a serving layer that merges the two, typically overwriting the speed layer's view for a time window once the batch layer's more-accurate result for that window lands.

Kappa's components: a single durable, replayable event log (the source of truth) and one stream-processing codebase that both handles live traffic and, when you need to fix a bug or backfill, gets pointed at an earlier offset and replayed through the same code.

The real cost of Lambda is that your batch and speed layers are two independent implementations of the same business logic, in whatever language/framework each layer uses, and they can silently drift apart (a bug fixed in one and not the other, a metric defined slightly differently). That duplication is the actual argument against it, not raw complexity for its own sake.

The real cost of Kappa is that it pushes all correctness work onto the streaming engine: state management, exactly-once guarantees, and cheap full-history replay for potentially years of data, all inside one system, which is a harder engineering problem than "run a batch job over the same log."

Worked example

A team building fraud scoring that needs both a fast online signal and periodic full retraining on corrected historical labels is a natural Lambda fit: the speed layer scores in real time with best-effort features, the batch layer recomputes ground truth nightly once labels settle, and a small serving-layer reconciliation step swaps in the corrected value. A team building a single, well-defined metrics pipeline (say, per-minute active users) where the exact same logic should always apply, live or replayed, is a better Kappa fit: one Flink job computes it, and a bug fix just means replaying the log from the point of the bug, not fixing and re-deploying two systems.

Trade-offs and pitfalls

A common mistake is choosing Kappa because "one codebase" sounds obviously simpler, without checking whether the team's streaming engine and storage can actually afford cheap, full-history replay at their retention window and data volume; if replay is slow or expensive, Kappa quietly turns into "one codebase, but reprocessing takes three days," which defeats the purpose. The opposite mistake is defaulting to Lambda out of caution and then never actually keeping the two layers' logic in sync, which produces the exact dual-codepath drift Lambda is often criticized for.