Data Pipeline Architecture and Design Questions

Direct answer

Batch processes data in scheduled, bounded chunks, trading latency for simplicity and easier correctness reasoning over large windows; streaming processes each event (or small micro-batch) continuously as it arrives, trading operational complexity for low end-to-end latency. Which one fits a given pipeline stage comes down to how quickly the downstream consumer of that specific stage needs the data, and whether the transformation genuinely needs to see many records together to be correct.

Structured elaboration

What's actually different, beyond speed

• Latency floor: batch's floor is the schedule interval; streaming's floor is close to network plus per-event processing time.
• Processing model: batch works over a bounded, known dataset (you can look at "all of yesterday" at once); streaming works over an unbounded sequence, which needs explicit mechanisms (windows, watermarks) to decide when a group of events is "done," something batch gets for free by waiting for the whole file or table partition to land.
• Resource pattern: batch spins compute up for a burst and releases it; streaming holds resources continuously, so idle-time cost matters more.
• Failure and replay model: batch failures resume naturally by re-running the job over the same input range; streaming failures need checkpointing and offset tracking to resume from where processing actually stopped.

Decision criteria for a given stage

Applying this per stage, not to the whole pipeline

The choice is not binary across an entire pipeline. An ingestion stage might run as streaming so no event is ever lost on the way in, while a downstream aggregation stage stays batch because it genuinely needs a full day's data to compute a stable metric. Decide stage by stage based on what that stage's own consumer needs, not what the source technology happens to support end to end.

Worked example

Say a stakeholder's service-level agreement (SLA) requires a metric to be no more than 5 minutes stale, but the relevant stage currently runs on a 24-hour batch schedule:

$\frac{24 \times 60 \text{ min}}{5 \text{ min}} = 288$

The batch cadence is 288 times looser than the freshness requirement, which rules batch out for that stage regardless of how fast the job itself runs once triggered; something closer to streaming or frequent micro-batch is needed there. Contrast a weekly executive report with the same 24-hour batch cadence: nightly batch already clears that bar with days to spare, so adding a streaming layer there would add operational cost without changing the outcome anyone sees.

Trade-offs & pitfalls

• Choosing streaming because it feels more modern, when the downstream consumer only checks the data once a day, means paying continuous infrastructure cost for freshness nobody uses.
• Trying to shrink a batch stage's schedule down to minutes eventually rebuilds a fragile streaming system, without the tooling (checkpointing, backpressure, idempotent writes) that makes real streaming safe.
• Streaming generally raises the bar on ordering and duplicate handling, since there is no "look at the whole day at once" moment to reconcile against; batch defers that complexity to reprocessing time instead.
• Treating the choice as pipeline-wide, rather than per stage, is the most common wrong turn; most real pipelines end up as a mix once examined stage by stage.

Direct answer

Schema evolution is a deliberate, managed change to a data contract over time, such as adding a field or widening a type; schema drift is the same kind of change happening unmanaged, whenever a producer changes its output without coordinating with consumers. It is risky with many downstream consumers because each one decoded the old contract into its own assumptions (column order, types, required fields), and a change that looks harmless to the producer, such as renaming a field or tightening a type, can silently break, or worse silently mis-parse, every consumer nobody told.

Structured elaboration

Evolution, drift, and enforcement

Enforcement and evolution are complementary controls, not opposites: enforcement decides what is allowed to change at all, rejecting anything not covered by a known, versioned schema; evolution defines the compatible ways a schema is allowed to change without being rejected. A pipeline that only enforces, rejecting anything not byte-identical to the original schema, turns every legitimate producer change into an outage; a pipeline that only tolerates drift, accepting anything and adapting silently, makes every accidental change invisible until a consumer breaks downstream.

Why more consumers means more risk

Every additional consumer is another independent assumption about the contract, a required field it always expects, a type it parses a specific way, a column it reads by position. A change compatible with nine consumers can still break the tenth. Consumers rarely deploy in lockstep with a producer's change, so even a compatible change has a window where old and new consumers coexist and must both keep functioning against whichever schema version they actually receive. The failure mode is often silent rather than a crash: a numeric field that starts arriving as a string might get parsed as zero or null instead of raising an error, corrupting downstream aggregates without anyone noticing until the numbers look wrong.

Making evolution safe instead of just possible

Classify every proposed change as backward-compatible (old consumers can still read new data, such as adding an optional field with a default), forward-compatible (new consumers can still read old data), or breaking (neither), and require the first category for routine changes. Version the schema explicitly and require producers to register a change before shipping, so an incompatible change is rejected before it ever reaches a consumer, enforcement and evolution working together. Give consumers a tolerant-reading discipline, ignoring unknown fields and applying explicit defaults for newly added ones, so the registration rules are not the only thing standing between a schema change and a broken consumer.

Worked example

A shared events contract has a field that has always held a small, fixed set of string values. A producer team widens it to include a new value without registering the change. Consumers that validate against the original fixed set now reject or mis-bucket every event carrying the new value; consumers that do not validate at all just silently mis-categorize those records in downstream reports. Had the change gone through a compatibility check first, adding a value while the field stays the same type is backward-compatible for consumers that do not hard-validate the value set, but a known breaking change for the ones that do, the validating consumers could have been notified and updated before the producer shipped, instead of the team finding out from a support ticket.

Trade-offs & pitfalls

• Enforcing so strictly that every legitimate, backward-compatible change requires a coordinated multi-team release pushes teams to bypass the pipeline entirely for "small" changes, which just relocates the drift somewhere else.
• Relying purely on tolerant reading everywhere and skipping registration entirely lets drift stay invisible until an aggregate looks wrong, by which point the root cause could be weeks old.
• Backward-only compatibility is cheaper to enforce than full, both-direction compatibility, but it constrains rolling a change back, since old consumers reading new data is covered while new consumers reading not-yet-upgraded old data is not.
• Treating schema evolution as purely a technical versioning problem is a common wrong turn; the real risk is coordinating independently deployed consumers, which a version number alone does not solve.

Direct answer

The speed layer owns the window of data the batch layer has not yet finished reprocessing, typically the last partial day or the last few hours, while everything older than the batch layer's most recently completed run is served exclusively from batch output, because batch has already recomputed it correctly. The handoff is a watermark: once batch advances that watermark forward, the speed layer's corresponding window is dropped and superseded, not merged value-by-value, because the entire reason the speed layer existed for that window was that it had not yet been through the authoritative computation.

Structured elaboration

The speed layer's job. Give an approximate, low-latency view of data that has not yet gone through the accurate batch recompute. It tolerates being wrong in small ways, duplicate counts, out-of-order updates, because it will be superseded once batch catches up.

The batch layer's job. Perform a full, authoritative recompute over a complete time window on a schedule, correcting for late-arriving and out-of-order events and enforcing deduplication. Its output for a given period becomes authoritative the moment the run completes.

The boundary, concretely. Define a watermark $W$ equal to the timestamp through which the most recent batch run is complete. Any query for time before $W$ reads only from batch output. Any query for time at or after $W$ reads from the speed layer. When the next batch run finishes and advances $W$ forward, the speed layer's now-covered range is simply discarded and replaced by batch's output for that range, not reconciled row by row, because batch is strictly more correct for anything it has already covered.

Why no per-row reconciliation is needed. The speed layer's numbers for a given window are never "corrected" against batch; they are superseded wholesale once batch's watermark passes that window. This is what keeps the two layers from needing complex agreement logic between them.

Delivery guarantees differ by layer. The speed layer can tolerate at-least-once or approximate counting because its output is provisional by design. The batch layer needs deduplication and exactly-once-equivalent accounting because its output is what actually persists.

flowchart LR
  E[Event stream] --> Sp[Speed layer: stream processor]
  Sp --> Fv[Fast view: low-latency store]
  E --> Rs[Raw storage: append-only]
  Rs --> Bl[Batch layer: full recompute]
  Bl --> Av[Authoritative view]
  Fv --> Serve[Serving layer]
  Av --> Serve
  Bl -->|advances watermark W| Sp

Worked example

Batch runs nightly starting at 02:00, covering the prior 24 hours, and takes 3 hours to complete due to full reprocessing and joins, so its watermark $W$ only advances to "yesterday 00:00" once the run finishes around 05:00:

If a dashboard is queried at 15:00 the same day, the speed layer is the only source for everything from $W$ to now:

Everything before that 39-hour window comes from last night's batch output. Once tonight's 02:00 to 05:00 run completes, $W$ advances to "today 00:00," and the speed layer's live window shrinks back down to just the hours elapsed so far today.

Trade-offs & pitfalls

If a batch run fails or runs long, the speed layer's provisional window grows and stays uncorrected for longer, since nothing has caught up to supersede it; this is why a maximum-staleness service-level agreement (SLA) on the batch run, with alerting if it's missed, matters as much as the batch logic itself. A common wrong turn is trying to reconcile or merge speed and batch numbers cell by cell instead of a clean cutover at the watermark; that doubles the reconciliation logic for no benefit, since batch is always strictly more correct for anything it has already covered. The other well-known cost of this architecture is maintaining two independent codepaths, a streaming job and a batch job, that must implement the same business logic and never drift apart; this exact duplication is what kappa architecture removes by reprocessing history through the same streaming code instead of a separate batch pipeline.

Direct answer

A cron job only knows a clock time; it has no idea whether the upstream stage it depends on actually finished, failed, or is still half-written, so a multi-stage pipeline built on cron entries is really a set of independent guesses about how long each stage takes. A workflow orchestrator (Airflow, Dagster, Prefect and similar) models the pipeline as a directed acyclic graph (DAG) of tasks with explicit dependency edges, so a downstream task only starts when its upstream task actually reports success, and the system gives you retries, backfills, and one place to see the whole pipeline's health for free.

Structured elaboration

Dependency-awareness versus time-awareness. Cron triggers on wall-clock time alone. If stage two is scheduled 30 minutes after stage one, that gap is a guess about stage one's runtime, not a guarantee. An orchestrator's DAG edge says "task B runs after task A succeeds," which is true regardless of how long A actually takes that day.

Failure propagation. In a DAG, if a task fails, its downstream dependents simply do not run (or run a defined failure branch), automatically. With cron, each stage fires on its own schedule no matter what happened upstream, so a failed stage one silently lets stage two start against stale or partial data.

Retries with policy. Orchestrators let you attach a retry policy per task (bounded attempts, backoff), so a transient failure recovers without a human noticing. A cron job that fails just waits for its next scheduled fire, however far away that is.

Backfill and replay. Because tasks are parameterized by the DAG's logical run (a date or partition), you can rerun a single historical slice through the whole dependency chain. Cron has no concept of "this stage, for that date," so backfilling means manually re-invoking scripts in the right order.

Observability. A DAG-based orchestrator gives one UI or API surface for task duration, success/failure history, and where in the chain something is stuck. Cron gives you whatever each script happens to log, scattered across machines and cron logs.

When cron is still the right tool. A single, independent job with no downstream dependents and no need for backfill or retry policy does not need a DAG orchestrator; adding one there is pure overhead.

Worked example

Take a 3-stage pipeline: extract, transform, load. With cron, each stage gets a fixed offset:

This assumes extract always finishes within 30 minutes. If the source is slow once and extract takes 45 minutes, transform still fires at 01:30 and reads a half-written file; cron has no signal that would stop it. Generalizing to a chain of $N$ stages, cron needs $N$ independently tuned fixed offsets, each one a manual guess that has to be re-tuned whenever runtime varies:

For $N = 5$: cron requires 5 separately maintained time guesses; the DAG requires 4 edges, and those edges stay correct no matter how long any one stage takes, because the trigger is "the previous task instance succeeded," not a clock offset.

Retries make the gap sharper. Suppose a task fails transiently (a brief network blip to the source). An orchestrator retries with exponential backoff, say attempts at 1, 2, and 4 minutes after failure:

A cron-based stage that fails has no retry concept built in: the next opportunity to reprocess is the next scheduled fire, often the next day:

That gap, 7 minutes versus 1440 minutes, is the practical cost of "no retry semantics," not an abstract one.

Trade-offs & pitfalls

An orchestrator is another system to run and keep healthy (unless you use a managed offering), so standing one up for a single independent job with no dependents is pure overhead with no payoff. Sensor or polling patterns (a task that waits by repeatedly checking for a file or a signal) can add their own latency or cost if the poll interval is misconfigured, so "dependency-aware" isn't automatically "instant." A common wrong turn is putting heavy transformation logic directly inside orchestrator tasks instead of having the orchestrator call out to dedicated compute; the orchestrator's job is to sequence and observe work, not to do the work itself.

Direct answer

At-most-once delivers each event zero or one times, never retrying on an uncertain outcome, so failures can silently drop data; at-least-once retries until delivery is confirmed, so events arrive one or more times and consumers must tolerate duplicates; exactly-once means every event is reflected in the result precisely once, with neither loss nor duplication. True end-to-end exactly-once is hard because it requires every hop between producer and final sink to agree, atomically, on whether a given unit of work has already taken effect, and real pipelines are built from independent components that do not share one transaction.

Structured elaboration

The three guarantees

Why exactly-once is hard to actually get end to end

In practice it is "effectively-once," built from at-least-once delivery (never silently drop, always retry on doubt) plus idempotent processing at the consumer, so a duplicate delivery has no additional effect. A literal single-delivery guarantee, with no retries and no duplicates ever occurring anywhere, is not achievable across an unreliable network; the achievable guarantee is that duplicates never MATTER, not that they never happen. Each hop, producer to broker, broker to processor, processor to sink, has its own failure and acknowledgment model, so getting a coordinated guarantee across all of them means either every hop supports the same commit protocol, which is rare outside a single vendor's tightly integrated stack, or idempotency has to be built explicitly at whichever hop lacks it. The hardest case is the last hop, to a sink outside the pipeline's control, such as a third-party system with no concept of the pipeline's delivery semantics; a shared transaction cannot be imposed on a system that does not participate in one, so exactly-once there always reduces to at-least-once delivery plus an idempotent write at the sink.

The practical takeaway

Most production pipelines that describe themselves as exactly-once mean this composed guarantee, at-least-once delivery deduplicated by a stable key at whichever hop needs it, not a literal single-delivery promise. Designing for it means finding the hop with the weakest native guarantee and adding idempotency there, rather than trying to make the whole chain transactional.

Worked example

An order event travels from a producer through a message broker to a stream processor and then to an external payment application. The broker can guarantee at-least-once delivery to the processor through offset commits after processing. The processor can deduplicate internally using a stable order identifier. But the payment application is a third party; if the processor's call to it times out, the processor cannot tell whether the payment was already charged or not. The only real exactly-once guarantee left available at that last hop is the payment application itself supporting an idempotency key, so it deduplicates repeat calls carrying the same key, which is a property of the external system, not something the pipeline can impose.

Trade-offs & pitfalls

• Advertising "exactly-once" for the whole system when it is only true up to a boundary the pipeline does not control, such as a downstream application without idempotency support, overstates the guarantee.
• Choosing at-most-once for cost or simplicity where a dropped event is actually expensive, such as a billing event, when at-least-once plus a dedupe key would have been just as simple to build, is a costly shortcut.
• At-least-once plus idempotent consumers is usually cheaper and more robust to build than coordinating true distributed transactions across every hop, but it pushes real design work onto every consumer, which is easy to underestimate.
• Treating exactly-once as a checkbox a messaging system provides, rather than an end-to-end property that depends on the weakest hop, including hops outside the pipeline's control, is the most common wrong turn.