Clarify goal: convert monolithic nightly recompute into incremental, idempotent, late-data-tolerant ETL that updates multiple analytics tables with SQL-first changes.Plan summary:1. Add change-tracking columns- Source tables: ensure (created_at, updated_at, deleted_at) or change_date and op_type if CDC available.- Target analytic tables: add last_processed_at, source_change_date, batch_id.2. Partitioning & clustering- Partition targets by logical time window (event_date or ingestion_date) AND cluster by keys used in joins/filters (user_id, product_id).- Keep source raw/landing tables partitioned by ingestion_date for efficient reads.3. Incremental selection (watermark)- Maintain a watermark table per pipeline: last_successful_max_updated_at (and optionally max_source_cdc_lsn).- Each run: SELECT rows WHERE updated_at > watermark OR updated_at BETWEEN watermark - late_threshold AND now() to capture late arrivals.4. Idempotent writes using MERGE (upsert) and soft deletesExample MERGE:sql
MERGE INTO analytics.orders tgt
USING (
SELECT id, user_id, total, event_date, updated_at
FROM staging.orders_delta
) src
ON tgt.id = src.id
WHEN MATCHED AND src.updated_at > tgt.source_change_date THEN
UPDATE SET user_id = src.user_id, total = src.total, source_change_date = src.updated_at, last_processed_at = CURRENT_TIMESTAMP
WHEN NOT MATCHED THEN
INSERT (id, user_id, total, event_date, source_change_date, last_processed_at)
VALUES (src.id, src.user_id, src.total, src.event_date, src.updated_at, CURRENT_TIMESTAMP);
5. Late-arriving data handling- Reprocess windows for N days back (late_threshold), e.g., re-run increments for event_date >= CURRENT_DATE - INTERVAL '7 days'.- Use change_date comparison to minimize rows reprocessed.6. Backfills and full-refreshs- Support explicit backfill jobs that write to a separate batch_id; after validation, swap partitions or update watermark.7. Resilience & retries- Wrap pipeline in transactional steps where supported; write intermediate deltas to durable staging table with batch_id and status (running/committed/failed) for replay.8. Monitoring & correctness queries- Watermark check:sql
SELECT pipeline, last_successful_max_updated_at, max_src_updated_at FROM pipeline_watermarks;
- Row counts & diffs per partition:sql
SELECT event_date, COUNT(*) as tgt_count, SUM(CASE WHEN src.updated_at > tgt.source_change_date THEN 1 ELSE 0 END) as pending_updates
FROM staging.src_delta src
LEFT JOIN analytics.tgt ON src.id = tgt.id
WHERE event_date BETWEEN ... GROUP BY event_date;
- Idempotency test: run same batch twice and assert no change:sql
SELECT COUNT(*) FROM analytics.tgt WHERE last_batch_id = :batch_id;
- Data quality checks: referential integrity, null thresholds, distribution changes.Trade-offs:- Using updated_at relies on source correctness; if not trustworthy use CDC or hash-based row fingerprints.- Partition granularity balances query pruning vs small-file overhead.This SQL-first approach minimizes recompute, provides idempotent MERGE semantics, reprocesses recent windows for late data, and uses watermarking + monitoring to validate correctness.