Data Pipeline Architecture and Design Questions

Direct answer

Records that fail parsing or validation go to a dead-letter queue (DLQ), a separate durable holding area, tagged with the original payload, the specific error, and a stable identifier tying them back to their place in the source. Correction happens outside the main flow (automated repair for known-fixable patterns, manual remediation for the rest), and replay pushes the corrected record back through the exact same validation gate it failed under a new attempt identifier, so it either passes cleanly or fails again visibly. Nothing is ever silently patched and slipped back in.

Structured elaboration

Put the gate as early as possible. Validation belongs at the ingestion boundary, before a record reaches any join, transform, or aggregation stage. Catching a bad record before it's mixed into shared state is cheap; unwinding an aggregate it already corrupted is not.

Parsing failures and validation failures are different problems. A parsing failure (malformed structure, broken encoding) usually points at a producer bug and is rarely safe to auto-fix. A validation failure (a missing required field, a value out of range, a reference that doesn't resolve) is often fixable given a bit more context, which is why it's worth routing these two differently rather than treating every failure the same way.

What has to travel with a record into the DLQ. The raw original payload, the specific rule or field that failed, a timestamp, a stable identifier that preserves its position in the source stream, and a retry or attempt counter. Without this, remediation is guesswork.

Two correction paths, not one. A known, safe failure pattern (for example, a schema version the pipeline hasn't rolled out support for yet) can go through an automated repair-and-resubmit step with no human involved. Everything else goes to a remediation surface where a person can see the failure reason and the payload, fix or annotate it, and mark it ready to go back through.

Replay through the same gate, under a new identity. A corrected record is resubmitted through the identical validation logic it originally failed, not around it, so a bad fix still gets caught. It carries a new attempt identifier rather than overwriting the original entry, preserving an audit trail of what failed, what changed, and when it was retried.

Watch the queue itself, not just individual failures. A DLQ that only ever grows is itself a signal. The right response to a small, steady trickle (normal background noise) is different from the right response to a sudden spike (usually a systemic break, like an upstream schema change), and the pipeline needs a way to tell those apart rather than routing an ever-growing stream to a queue nobody is watching.

Worked example

Say the pipeline ingests 2,400,000 records a day, and historically 0.25 percent fail validation:

Spread evenly across 24 hours, that's a baseline arrival rate of:

Now suppose an upstream schema change causes 8 percent of one hour's arriving records (100,000 that hour) to fail validation:

Compared to the 250-per-hour baseline, that's:

If the alerting rule fires whenever the DLQ arrival rate exceeds 5 times the trailing baseline, this spike (32 times baseline) triggers it clearly, which is exactly the signal that should prompt someone to check for a systemic cause, like a schema change, instead of assuming it's more of the same background trickle.

flowchart LR
    A[Incoming record] --> B[Validation gate]
    B -->|passes| C[Continue through pipeline]
    B -->|fails| D[Dead-letter queue]
    D --> E[Automated repair, known patterns]
    D --> F[Manual remediation]
    E --> G[Replay, new attempt id]
    F --> G
    G --> B

Trade-offs & pitfalls

• A DLQ with no remediation and replay loop is a write-only graveyard; it's no better than silently dropping the records, just more expensive to store.
• Reprocessing a corrected record under its original identity, instead of a new attempt identifier, erases the audit trail of "this failed once, here was the original data."
• Auto-retrying validation failures that are genuinely permanent (a business-rule violation, not a transient glitch) wastes effort on records that will fail identically every time until the rule or the data changes.
• Placing the validation gate after joins or aggregation means a bad record has already corrupted shared downstream state before anyone catches it; moving the gate earlier is nearly free, unwinding a corrupted aggregate is not.
• Treating every spike in DLQ volume as "more of the same" instead of a signal worth investigating is how a schema break goes unnoticed for hours.

Direct answer

Keeping raw ingestion separate means the raw event log or landing files stay retained independently of whatever transform code currently runs on top of them, so when a bug is found in the transform logic, the fix is just code plus a replay of the same untouched history. If ingestion and transformation are tightly coupled, typically because the ingestion step only keeps output the transform has already touched, or discards source events once processed, fixing that same bug means the input that would prove the fix, and let you reprocess history, may no longer exist in the form it originally had.

Structured elaboration

Walk through what actually breaks when the two are coupled, step by step, using a concrete bug: a join-key normalization error that undercounts one category.

With decoupled raw storage. Raw events for the affected period still exist untouched, byte-for-byte as originally ingested. Fix the transform code, rerun it over the same raw window, get corrected output. No data was lost, only recomputed.

With ingestion and transformation coupled in one pass. The job extracts, transforms, and loads in a single step and only ever writes the transformed result. The raw form the bug depended on is gone. Reprocessing means re-extracting from the source system itself, which may no longer have that historical data at all if its own retention window has passed, or, worse, may return something different because source rows have since been updated or deleted.

Non-determinism creeps in over time. Even if the source's retention happens to still cover the period, its current state reflects everything that has happened to those rows SINCE, updates and deletes included. A "replay" built from a live re-extract silently reflects today's state of the source, not the historical state the original run actually saw, producing a different, non-reproducible answer than a true raw-log replay would give.

Blast radius of testing the fix. In the coupled design, testing the fix means running it against the live production extract path, since there is no stored raw form to safely target instead, which risks re-hitting the source system's load and re-triggering any side effects the ingestion step has, if that step isn't a pure read.

Worked example

A category-normalization bug is found to affect 45 days of history. With raw storage retained for 90 days (or archived further to cold storage), the fix reprocesses exactly those 45 days from the untouched raw copy, a bounded, fully deterministic amount of recompute. With ingestion and transformation coupled, and the source system itself retaining only 30 days of live queryable history before purging or archiving older rows elsewhere:

Those 15 days can no longer be correctly reproduced from the source at all; that portion of history is permanently uncorrectable without a raw copy that was never kept.

Trade-offs & pitfalls

Decoupling has a real cost: storing raw data for a long window means paying to keep data that's mostly redundant with its own transformed output. The mitigation is a defined raw-retention policy, keep raw only as long as the realistic "we might need to reprocess" window actually is, matching how far back schema or logic bugs are typically caught, rather than retaining it forever by default with no justification. Raw retention only helps if raw is genuinely immutable and untouched; if the ingestion step itself silently applies "a little cleaning for convenience" on the way in, the coupling has just moved one layer down and reintroduced the same problem under a different name.

Direct answer

Schema evolution is a deliberate, managed change to a data contract over time, such as adding a field or widening a type; schema drift is the same kind of change happening unmanaged, whenever a producer changes its output without coordinating with consumers. It is risky with many downstream consumers because each one decoded the old contract into its own assumptions (column order, types, required fields), and a change that looks harmless to the producer, such as renaming a field or tightening a type, can silently break, or worse silently mis-parse, every consumer nobody told.

Structured elaboration

Evolution, drift, and enforcement

Enforcement and evolution are complementary controls, not opposites: enforcement decides what is allowed to change at all, rejecting anything not covered by a known, versioned schema; evolution defines the compatible ways a schema is allowed to change without being rejected. A pipeline that only enforces, rejecting anything not byte-identical to the original schema, turns every legitimate producer change into an outage; a pipeline that only tolerates drift, accepting anything and adapting silently, makes every accidental change invisible until a consumer breaks downstream.

Why more consumers means more risk

Every additional consumer is another independent assumption about the contract, a required field it always expects, a type it parses a specific way, a column it reads by position. A change compatible with nine consumers can still break the tenth. Consumers rarely deploy in lockstep with a producer's change, so even a compatible change has a window where old and new consumers coexist and must both keep functioning against whichever schema version they actually receive. The failure mode is often silent rather than a crash: a numeric field that starts arriving as a string might get parsed as zero or null instead of raising an error, corrupting downstream aggregates without anyone noticing until the numbers look wrong.

Making evolution safe instead of just possible

Classify every proposed change as backward-compatible (old consumers can still read new data, such as adding an optional field with a default), forward-compatible (new consumers can still read old data), or breaking (neither), and require the first category for routine changes. Version the schema explicitly and require producers to register a change before shipping, so an incompatible change is rejected before it ever reaches a consumer, enforcement and evolution working together. Give consumers a tolerant-reading discipline, ignoring unknown fields and applying explicit defaults for newly added ones, so the registration rules are not the only thing standing between a schema change and a broken consumer.

Worked example

A shared events contract has a field that has always held a small, fixed set of string values. A producer team widens it to include a new value without registering the change. Consumers that validate against the original fixed set now reject or mis-bucket every event carrying the new value; consumers that do not validate at all just silently mis-categorize those records in downstream reports. Had the change gone through a compatibility check first, adding a value while the field stays the same type is backward-compatible for consumers that do not hard-validate the value set, but a known breaking change for the ones that do, the validating consumers could have been notified and updated before the producer shipped, instead of the team finding out from a support ticket.

Trade-offs & pitfalls

• Enforcing so strictly that every legitimate, backward-compatible change requires a coordinated multi-team release pushes teams to bypass the pipeline entirely for "small" changes, which just relocates the drift somewhere else.
• Relying purely on tolerant reading everywhere and skipping registration entirely lets drift stay invisible until an aggregate looks wrong, by which point the root cause could be weeks old.
• Backward-only compatibility is cheaper to enforce than full, both-direction compatibility, but it constrains rolling a change back, since old consumers reading new data is covered while new consumers reading not-yet-upgraded old data is not.
• Treating schema evolution as purely a technical versioning problem is a common wrong turn; the real risk is coordinating independently deployed consumers, which a version number alone does not solve.

Direct answer

A cron job only knows a clock time; it has no idea whether the upstream stage it depends on actually finished, failed, or is still half-written, so a multi-stage pipeline built on cron entries is really a set of independent guesses about how long each stage takes. A workflow orchestrator (Airflow, Dagster, Prefect and similar) models the pipeline as a directed acyclic graph (DAG) of tasks with explicit dependency edges, so a downstream task only starts when its upstream task actually reports success, and the system gives you retries, backfills, and one place to see the whole pipeline's health for free.

Structured elaboration

Dependency-awareness versus time-awareness. Cron triggers on wall-clock time alone. If stage two is scheduled 30 minutes after stage one, that gap is a guess about stage one's runtime, not a guarantee. An orchestrator's DAG edge says "task B runs after task A succeeds," which is true regardless of how long A actually takes that day.

Failure propagation. In a DAG, if a task fails, its downstream dependents simply do not run (or run a defined failure branch), automatically. With cron, each stage fires on its own schedule no matter what happened upstream, so a failed stage one silently lets stage two start against stale or partial data.

Retries with policy. Orchestrators let you attach a retry policy per task (bounded attempts, backoff), so a transient failure recovers without a human noticing. A cron job that fails just waits for its next scheduled fire, however far away that is.

Backfill and replay. Because tasks are parameterized by the DAG's logical run (a date or partition), you can rerun a single historical slice through the whole dependency chain. Cron has no concept of "this stage, for that date," so backfilling means manually re-invoking scripts in the right order.

Observability. A DAG-based orchestrator gives one UI or API surface for task duration, success/failure history, and where in the chain something is stuck. Cron gives you whatever each script happens to log, scattered across machines and cron logs.

When cron is still the right tool. A single, independent job with no downstream dependents and no need for backfill or retry policy does not need a DAG orchestrator; adding one there is pure overhead.

Worked example

Take a 3-stage pipeline: extract, transform, load. With cron, each stage gets a fixed offset:

This assumes extract always finishes within 30 minutes. If the source is slow once and extract takes 45 minutes, transform still fires at 01:30 and reads a half-written file; cron has no signal that would stop it. Generalizing to a chain of $N$ stages, cron needs $N$ independently tuned fixed offsets, each one a manual guess that has to be re-tuned whenever runtime varies:

For $N = 5$: cron requires 5 separately maintained time guesses; the DAG requires 4 edges, and those edges stay correct no matter how long any one stage takes, because the trigger is "the previous task instance succeeded," not a clock offset.

Retries make the gap sharper. Suppose a task fails transiently (a brief network blip to the source). An orchestrator retries with exponential backoff, say attempts at 1, 2, and 4 minutes after failure:

A cron-based stage that fails has no retry concept built in: the next opportunity to reprocess is the next scheduled fire, often the next day:

That gap, 7 minutes versus 1440 minutes, is the practical cost of "no retry semantics," not an abstract one.

Trade-offs & pitfalls

An orchestrator is another system to run and keep healthy (unless you use a managed offering), so standing one up for a single independent job with no dependents is pure overhead with no payoff. Sensor or polling patterns (a task that waits by repeatedly checking for a file or a signal) can add their own latency or cost if the poll interval is misconfigured, so "dependency-aware" isn't automatically "instant." A common wrong turn is putting heavy transformation logic directly inside orchestrator tasks instead of having the orchestrator call out to dedicated compute; the orchestrator's job is to sequence and observe work, not to do the work itself.

Direct answer

The speed layer owns the window of data the batch layer has not yet finished reprocessing, typically the last partial day or the last few hours, while everything older than the batch layer's most recently completed run is served exclusively from batch output, because batch has already recomputed it correctly. The handoff is a watermark: once batch advances that watermark forward, the speed layer's corresponding window is dropped and superseded, not merged value-by-value, because the entire reason the speed layer existed for that window was that it had not yet been through the authoritative computation.

Structured elaboration

The speed layer's job. Give an approximate, low-latency view of data that has not yet gone through the accurate batch recompute. It tolerates being wrong in small ways, duplicate counts, out-of-order updates, because it will be superseded once batch catches up.

The batch layer's job. Perform a full, authoritative recompute over a complete time window on a schedule, correcting for late-arriving and out-of-order events and enforcing deduplication. Its output for a given period becomes authoritative the moment the run completes.

The boundary, concretely. Define a watermark $W$ equal to the timestamp through which the most recent batch run is complete. Any query for time before $W$ reads only from batch output. Any query for time at or after $W$ reads from the speed layer. When the next batch run finishes and advances $W$ forward, the speed layer's now-covered range is simply discarded and replaced by batch's output for that range, not reconciled row by row, because batch is strictly more correct for anything it has already covered.

Why no per-row reconciliation is needed. The speed layer's numbers for a given window are never "corrected" against batch; they are superseded wholesale once batch's watermark passes that window. This is what keeps the two layers from needing complex agreement logic between them.

Delivery guarantees differ by layer. The speed layer can tolerate at-least-once or approximate counting because its output is provisional by design. The batch layer needs deduplication and exactly-once-equivalent accounting because its output is what actually persists.

flowchart LR
  E[Event stream] --> Sp[Speed layer: stream processor]
  Sp --> Fv[Fast view: low-latency store]
  E --> Rs[Raw storage: append-only]
  Rs --> Bl[Batch layer: full recompute]
  Bl --> Av[Authoritative view]
  Fv --> Serve[Serving layer]
  Av --> Serve
  Bl -->|advances watermark W| Sp

Worked example

Batch runs nightly starting at 02:00, covering the prior 24 hours, and takes 3 hours to complete due to full reprocessing and joins, so its watermark $W$ only advances to "yesterday 00:00" once the run finishes around 05:00:

If a dashboard is queried at 15:00 the same day, the speed layer is the only source for everything from $W$ to now:

Everything before that 39-hour window comes from last night's batch output. Once tonight's 02:00 to 05:00 run completes, $W$ advances to "today 00:00," and the speed layer's live window shrinks back down to just the hours elapsed so far today.

Trade-offs & pitfalls

If a batch run fails or runs long, the speed layer's provisional window grows and stays uncorrected for longer, since nothing has caught up to supersede it; this is why a maximum-staleness service-level agreement (SLA) on the batch run, with alerting if it's missed, matters as much as the batch logic itself. A common wrong turn is trying to reconcile or merge speed and batch numbers cell by cell instead of a clean cutover at the watermark; that doubles the reconciliation logic for no benefit, since batch is always strictly more correct for anything it has already covered. The other well-known cost of this architecture is maintaining two independent codepaths, a streaming job and a batch job, that must implement the same business logic and never drift apart; this exact duplication is what kappa architecture removes by reprocessing history through the same streaming code instead of a separate batch pipeline.