ETL and ELT Design Patterns Questions

A full load re-reads and rewrites the entire target table on every run. An incremental load reads and applies only what changed since the last run. Incremental is the default because it is almost always cheaper on both source load and warehouse compute, but a full refresh is still the right call in a few specific situations.

When a full refresh beats incremental

• The table is small enough that reading it entirely costs nothing meaningful (a few thousand rows), and the extra engineering to track a cursor isn't worth it.
• The source can't reliably tell you what changed: no reliable updated_at, no CDC (Change Data Capture) feed, and rows can be deleted without leaving any trace an incremental scan would catch.
• Historical corrections are common and hard to bound: if upstream data gets silently rewritten in ways you can't detect with a watermark, a full refresh is the only way to guarantee correctness.
• The table is genuinely small relative to the operational complexity incremental logic adds (deletion handling, backfill logic, drift between the target and source over time).

Signals that confirm an incremental load actually worked

1. Row-count delta sanity check. Compare the number of rows the incremental load touched against a rough expectation (source's own change rate, a moving average of daily deltas). A load that touched zero rows on a day you know had activity is a red flag, not a quiet success.
2. Watermark/cursor advancement. Confirm the stored high-water-mark actually moved forward after the run. A load that silently fails to update its cursor will look "successful" (job exit code 0) while re-processing the same window forever, or worse, never picking up new data if the failure is downstream of the cursor update.
3. Reconciliation against source counts. Periodically (daily or weekly), compare a full count (or checksum) from the source against the target for a bounded window, to catch drift that accumulates silently across many "successful" incremental runs, e.g. missed deletes or an off-by-one in the cursor comparison.

Pitfall worth naming: teams often treat "full load" and "incremental load" as a permanent architectural choice, when the honest answer is usually "incremental with a scheduled full-refresh reconciliation" (nightly incremental, weekly full rebuild) precisely because incremental logic silently accumulates drift that a full comparison catches and a purely incremental pipeline never will on its own.

Architecting this pipeline means deliberately splitting transformation work by WHERE it needs to run, not defaulting to one pattern everywhere. The two workloads (sub-5-second personalized recommendations, and daily batch analytics) have genuinely different constraints, and the split should be visible in the design, not incidental to it.

The transform-placement decision, explicit

• Clickstream, feeding real-time recommendations (pre-load ETL, by necessity): light validation and enrichment (session stitching, basic feature computation) has to happen in the streaming path itself, before the recommendation service can use it, because ELT's "load raw, transform later in the warehouse" pattern is fundamentally too slow for a 5-second SLA (service-level agreement). This is ETL because it MUST be, not by preference.
• CDC (Change Data Capture) from transactional databases, feeding daily analytics (ELT): these changes get captured with minimal transformation (log-based CDC via Debezium, landed close to raw) and the heavy transformation, joins, aggregation, dimensional modeling, happens after landing in the warehouse, where it benefits from batch compute, is trivially reprocessable if a transform has a bug, and doesn't need to keep pace with a sub-5-second SLA.
• The clickstream ALSO feeds daily analytics, and for that path, the same raw (or lightly-enriched) events land in a data lake and get transformed in batch alongside everything else, meaning the same source data takes two different paths depending on which downstream consumer it's serving.

Architecture

flowchart LR
  CS[Clickstream 1B events/day] --> K[Kafka]
  DB[(Transactional DBs)] --> CDC[Debezium CDC] --> K
  K --> SP[Stream processor: light ETL - validate, enrich, feature-compute]
  SP --> RT[(Real-time feature store)]
  RT --> REC[Recommendation service <5s SLA]
  K --> LAKE[(Raw landing: data lake)]
  LAKE --> ELT[Batch ELT: dbt models in the warehouse]
  ELT --> DW[(Analytics warehouse)]
  DW --> BI[Daily analytics and dashboards]

Storage choices: the data lake (object storage, Parquet) is the durable raw landing zone for everything, cheap and replayable. The real-time feature store (a low-latency key-value store, not the warehouse) serves the recommendation path because a warehouse query, even a fast one, is the wrong latency class for a 5-second end-to-end budget that also has to include model inference. The analytics warehouse holds the ELT-transformed, dimensionally modeled data for BI.

Streaming vs. batch split: the recommendation path is pure streaming, no batch step in its critical path at all. The daily-analytics path is batch (scheduled dbt runs), consuming from the same raw landing zone the streaming path also reads from, so there is exactly one ingestion point (Kafka/Debezium into the lake) feeding two independently-paced downstream consumers.

Failure and recovery: the streaming path's failure mode is "recommendations get staler or fall back to a non-personalized default," recoverable by replaying from the last committed Kafka offset once the stream processor is healthy again. The batch ELT path's failure mode is "the daily analytics refresh is late," recoverable by simply rerunning the failed dbt run, since ELT models built on landed, durable raw data are naturally re-runnable without any data loss risk, which is exactly the property that justifies deferring most of the transformation work to that side of the split.

The honest answer is "it depends on where the transform has to live to do its job well," not a blanket rule. Four factors actually drive the decision in practice.

The factors

1. Compute cost and billing model. Modern cloud warehouses bill compute separately from storage and scale near-instantly, which is the single biggest reason ELT has become the default: pushing transformation into the warehouse means you're not paying for a separate always-on compute layer, and you scale transformation compute the same way you scale query compute.
2. Governance and access control. If a transformation needs to touch PII (personally identifiable information) before it's ever visible in the warehouse (masking, tokenization, filtering out fields a downstream consumer legally can't see), it has to run upstream, before load, because "load raw then mask" means the unmasked data existed in the warehouse even briefly.
3. Iteration speed and ownership. SQL-in-the-warehouse (ELT via dbt) is something analysts and analytics engineers can write and iterate on themselves. Upstream ETL usually lives in a codebase only data engineering touches, which is slower to change but keeps transformation logic close to the source system's quirks, which sometimes only the team that owns that source truly understands.
4. Coupling to the source schema. Transformations that depend heavily on source-system internals (a microservice's internal event shape, a legacy system's undocumented status codes) are easier to keep close to that source, so the person who understands the source owns the translation, rather than propagating source-specific knowledge into a shared warehouse layer everyone else has to learn.

How the answer changes by source type
For a small relational CRM table, ELT is almost always right: the data is small, the warehouse can trivially absorb the compute, and keeping the raw table around costs nothing meaningful. For a high-volume event stream (billions of rows/day), some transformation often still has to happen upstream regardless: basic validation, PII stripping, and possibly enrichment that would be prohibitively expensive to redo on every downstream query if left entirely raw. The realistic answer for a high-volume stream is usually a hybrid: light, cheap-to-skip transforms upstream (schema validation, PII masking) and everything else pushed down into ELT, where it can be reprocessed and iterated on freely.

The trap: treating this as a one-time architectural decision instead of a per-transformation one. A real pipeline typically runs BOTH patterns simultaneously, some transforms pre-load out of necessity (governance, cost at extreme volume), most transforms post-load because that's cheaper and easier to iterate on, and conflating "our pipeline uses ELT" with "every transform lives in the warehouse" is where teams get surprised by a governance or cost problem they didn't design for.

The MERGE, loading orders_stg(order_id, amount, last_modified, deleted) into orders(order_id PK, amount, last_modified, is_deleted):

MERGE INTO orders AS t
USING (
  SELECT order_id, amount, last_modified, deleted
  FROM (
    SELECT *, ROW_NUMBER() OVER (
      PARTITION BY order_id ORDER BY last_modified DESC
    ) AS rn
    FROM orders_stg
  )
  WHERE rn = 1
) AS s
ON t.order_id = s.order_id
WHEN MATCHED AND s.last_modified > t.last_modified THEN
  UPDATE SET amount = s.amount, last_modified = s.last_modified, is_deleted = s.deleted
WHEN NOT MATCHED THEN
  INSERT (order_id, amount, last_modified, is_deleted)
  VALUES (s.order_id, s.amount, s.last_modified, s.deleted);

Verified end to end with a target row for order 1, and a staging batch containing an update for order 1, a stale duplicate row for order 1 with an older last_modified, a new order 2, and a new order 3 that's inserted and soft-deleted in the same batch:

before: orders = [(1, 100.00, 09:00, false)]

staging batch:
  (1, 150.00, 11:00, false)   -- real update
  (1,  90.00, 08:00, false)   -- STALE duplicate for the same key, older timestamp
  (2,  40.00, 11:05, false)   -- new
  (3,  10.00, 11:10, true)    -- new, arrives already deleted

after MERGE:
  order_id | amount | last_modified | is_deleted
  1        | 150.0  | 11:00         | False
  2        | 40.0   | 11:05         | False
  3        | 10.0   | 11:10         | True

re-running the SAME merge again: identical output (confirmed idempotent)

The inner ROW_NUMBER() window is what makes this correct rather than merely lucky: without it, the MERGE spec would try to match target row 1 against BOTH staging rows for order 1, and most engines either error on a duplicate match in the USING clause or apply them in an undefined order, which can silently leave the stale (90.00, 08:00) value as the final state instead of the correct 150.00. Deduplicating the staging batch down to one row per key, keeping the latest by last_modified, before it ever reaches the MERGE's ON clause is what guarantees a deterministic, correct result regardless of what order duplicate source rows happen to arrive in.

What makes this safe under retries: the WHEN MATCHED AND s.last_modified > t.last_modified guard means re-applying an already-applied batch is a no-op, since the incoming last_modified will no longer be greater than what's already in the target. Combined with the per-batch dedup, the MERGE produces the same end state whether it runs once or is retried after a partial failure and re-run from scratch.

What makes this safe under concurrent runs: a single MERGE statement is atomic in every warehouse that supports it (Snowflake, BigQuery, Postgres), so two concurrent MERGEs against the same target either serialize (one waits for the other's lock) or one fails with a concurrent-modification error that the caller retries, rather than interleaving in a way that corrupts a row. The dangerous version of "concurrent" is two SEPARATE INSERT/UPDATE statements instead of one MERGE, where a race between them can produce a duplicate insert; using one atomic MERGE statement is what removes that class of bug entirely.

Indexes/constraints: a unique constraint (or primary key) on order_id in the target table is what turns "the MERGE has a bug that lets two source rows both count as NOT MATCHED" into a loud constraint violation instead of a silent duplicate row, so it's not optional, it's the backstop for exactly the kind of duplicate-match scenario this dedup step exists to prevent.

A pipeline run is idempotent if running it twice with the same input produces the same result as running it once. That matters because retries are not an edge case, they are the normal failure-recovery mechanism: a job times out and gets re-triggered, a message gets redelivered because the consumer crashed before acking it, an operator reruns a failed DAG (directed acyclic graph) task. If reprocessing the same batch can double-count rows or corrupt state, every one of those ordinary recovery paths becomes a data-corruption risk.

Two concrete techniques

1. Unique dedup key plus upsert/merge at the sink. Give every record a stable identity (a natural key, or a deterministic hash of its contents) and write with MERGE/UPSERT semantics instead of a bare INSERT. Re-applying the same record just overwrites itself with identical values; nothing accumulates.

2. A run identifier plus an idempotency check before writing. Tag each pipeline run with a unique run id (or, for append-only writes, embed a deterministic idempotency key derived from the source event), and before writing check whether that key has already been applied (a "processed batches" ledger table, or a unique constraint on the sink that makes a duplicate write a no-op or a safely-caught conflict rather than a silent double-insert).

A worked scenario: a streaming consumer reads an event, writes it to the warehouse, then crashes before it can commit its Kafka offset. On restart, the consumer re-reads the same event (at-least-once delivery is the norm for most streaming systems) and would double-write it under naive append-only INSERT logic. With an idempotency key (say, the event's own event_id) and a MERGE keyed on that id instead of INSERT, the redelivered event just re-writes the identical row, producing the exact same end state as if it had been delivered once.

Trade-offs: append-only + dedup-on-read (keep every write, filter duplicates at query time) is cheaper to write but pushes cost onto every downstream reader and doesn't bound storage growth. Merge/upsert-at-write is more expensive per write (it has to look up the existing row) but keeps the table itself clean and correct, which is usually worth it once more than one consumer reads the table.