Requirements & constraints:- Daily incremental load from staging into SCD2 dimension + daily fact- Idempotent (re-runs safe), no double-counting- Support updates to dimension attributes and historical correctnessHigh-level approach:1) Dedupe staging (deterministic pick — latest event_timestamp)2) Detect new vs changed dimension rows using a stable business key + hash of attributes3) Apply SCD2 MERGE into dimension with effective_date / end_date and versioning4) Upsert facts using idempotent dedupe keys (business_key, event_date, source_id) and avoid re-insert of already-loaded rowsSQL patterns (pseudo-SQL):1) Deduplicate stagingsql
with staged_ranked as (
select *, row_number() over(partition by business_key, event_id order by event_ts desc, source_order desc) rn
from raw_staging_daily
)
, staged as (
select * from staged_ranked where rn = 1
)
2) Identify new / changed dimension rows (compute hash)sql
with candidates as (
select
s.business_key,
s.attr1, s.attr2,
md5(concat_ws('|', s.attr1::text, s.attr2::text)) as row_hash
from staged s
)
, current_dim as (
select business_key, row_hash as current_hash, dim_id
from dim_table
where end_date is null
)
, to_upsert as (
select c.*,
case
when d.dim_id is null then 'INSERT'
when d.current_hash <> c.row_hash then 'UPDATE'
else 'NOOP'
end as action,
d.dim_id
from candidates c
left join current_dim d using (business_key)
)
3) SCD2 MERGE (idempotent)sql
begin transaction;
/* CLOSE existing rows that changed */
update dim_table
set end_date = staged_load_date - interval '1 second'
where business_key in (select business_key from to_upsert where action = 'UPDATE')
and end_date is null;
/* INSERT new version for UPDATED and for INSERT */
insert into dim_table (business_key, attr1, attr2, row_hash, effective_date, end_date, version)
select business_key, attr1, attr2, row_hash, staged_load_date, null, coalesce(
(select coalesce(max(version),0)+1 from dim_table d where d.business_key = t.business_key), 1)
from to_upsert t
where action in ('INSERT','UPDATE');
commit;
Notes: use transactions, locks, or serialized jobs to avoid races. Use row_hash to make re-run idempotent: identical attributes produce same hash -> action = NOOP.4) Load fact table without double-counting- Derive foreign key by joining to dim_table as of load_date (effective_date <= event_date and (end_date is null or end_date >= event_date))- Use a dedupe key (business_key, event_id, source_id)- Upsert facts idempotently or insert only rows not already presentsql
with facts_ready as (
select s.*, d.dim_id
from staged s
join dim_table d
on s.business_key = d.business_key
and d.effective_date <= s.event_date
and (d.end_date is null or d.end_date >= s.event_date)
)
insert into fact_table (dim_id, event_date, metric, load_id, source_id)
select fr.dim_id, fr.event_date, fr.metric, :load_id, fr.source_id
from facts_ready fr
left join fact_table f
on f.source_id = fr.source_id
and f.event_id = fr.event_id
where f.event_id is null;
Idempotency & safety patterns:- Use a stable load_id (e.g., date + batch id) and store it on inserted rows- Enforce unique constraint/index on fact dedupe key (source_id,event_id) and on dim business_key with effective_date uniqueness to guard races- Use row_hash to avoid unnecessary updates and make repeated runs no-op- Maintain an ingestion watermark table to quickly skip already-seen raw batches- Wrap dimension changes in transactions and/or serialize daily jobEdge cases:- Late-arriving data: optionally backfill by reprocessing ranges; SCD2 logic handles history- Partial failures: job should be retryable; idempotent steps avoid duplicates- Concurrent runs: enforce single scheduler lock or use DB advisory locksThis pattern yields reproducible, idempotent incremental ETL with SCD2 history and safe fact joins that prevent double-counting.