Netflix Data Engineer (Staff) Interview Preparation Guide 2026

Requirements clarification:

• Country-level residency: raw PII must never leave originating country/region.
• Central analytics: only aggregated, non-identifiable metrics allowed centrally.
• Ingestion: 5 TB/day (assume global aggregate).
• Minimize cross-region egress and provide auditable controls.

High-level architecture (regional-first, central-aggregate):

1. Edge ingestion (per-country/region)

   • Regionally deployed ingestion layer (APIs / Kafka / Kinesis / PubSub) collects events and writes to a regional hot store (object store + streaming).
   • Ingest -> regional raw data lake (S3/GCS/Azure Blob in-region) partitioned by country/tenant; use Schema Registry for consistent schemas.

2. Regional processing & enforcement

   • Regional ETL (Spark/EMR/Dataflow) performs:
     • PII detection & classification (DLP)
     • Tokenization / encryption of direct identifiers using region-specific Customer-Managed Keys (CMKs)
     • Anonymization/aggregation transforms that produce GDPR-safe metrics (differential privacy, k-anonymity, or hashed buckets + noise)
   • Raw PII stays in regional cold store with strict access controls and retention lifecycle.

3. Central analytics tier

   • Only aggregated, non-identifiable datasets (pre-approved aggregates) are exported to central analytics region(s).
   • Central analytics runs OLAP/warehouse (Snowflake/Redshift/BigQuery) fed by regionally-pushed aggregated files or federated queries to regional read-only aggregate endpoints.
   • Use message queues or signed manifests to transfer aggregates; all transfers are of non-PII.

4. Governance & audit plane

   • Global catalog (read-only metadata of data locations, lineage) with per-region access controls (Lake Formation / Data Catalog).
   • Audit logs (ingest events, key usage, data access, transfers) forwarded to a secure SIEM; retention policies per regulation.
   • Policy engine (OPA or cloud policy service) enforces residency before any transfer.

Partitioning strategy

• Primary partition by country_code and ingestion_date for fast residency isolation and retention enforcement.
• Secondary partition by tenant/product for parallelism.
• Use region-local shards for streaming topics to align with country partitions.

Encryption & key management

• TLS for in-transit.
• At-rest encryption using region-scoped CMKs (KMS) per country; keys cannot be exported.
• Envelope encryption: data encrypted with data keys; data keys encrypted by regional CMKs.
• Central aggregated datasets use separate CMKs and are audited; aggregated exports are verified to exclude PII before encryption and transfer.

Replication strategy

• Raw PII: no cross-region replication. Replicate within-country zones only (AZ-level) for durability.
• Aggregated metrics: replicated to central analytics region(s); use asynchronous transfer after anonymization.
• Metadata, schema, and audit logs: replicate globally but strip PII; maintain provenance linking to region-of-origin.

Handling 5 TB/day & cost optimization

• Ingest partitioning + parallel streaming clusters sized per regional load; use autoscaling.
• Minimize egress by:
  • Doing heavy transforms regionally and only transferring compact aggregates (orders-of-magnitude smaller than raw).
  • Where possible run federated queries (BigQuery/Athena federated connectors) so central compute pulls only required aggregated slices.
  • Use compression (Parquet/ORC, snappy) and columnar formats.
  • Batch transfers to avoid per-request overheads.
• Trade-offs:
  • Centralized compute (easier analytics) vs regional compute (lower egress). Prefer regional transforms + smaller central aggregator to minimize egress.
  • Stronger anonymization (more privacy) may reduce analytic fidelity — choose DP parameters with stakeholders.
  • Cold storage vs more frequent recomputation: store raw PII regionally cheaper vs replicating; recompute aggregates when needed.

Auditable controls & compliance

• Immutable audit trail: log every ingest, access, transfer, key usage (CloudTrail-like), stored regionally and archived to WORM storage.
• Role-based access + attribute-based policies; approvals and data export workflows with recorded attestations.
• Periodic policy enforcement audits, automatic tests (canary queries) verifying exported datasets contain no PII (DLP scan + statistical checks for uniqueness).
• Retention & right-to-erasure flows: implement delete/expire pipelines that purge raw and derived datasets within retention windows and propagate to catalogs & indexes.

Why this design

• Ensures legal residency by keeping raw PII in-country and enforcing key separation.
• Minimizes egress by designing regional-first transforms and transferring only compact aggregates.
• Supports central analytics while preserving privacy via cryptographic controls and anonymization.
• Provides full auditability and policy enforcement with minimal impact on analytical capability.

Implementation notes

• Use infrastructure-as-code for reproducibility; automated data classification and unit tests for anonymization guarantees; SLA-driven monitoring for ingestion/backpressure.
• Choose cloud services that support regionally-scoped keys and fine-grained IAM; if operating in restricted countries, consider dedicated regional accounts/tenants.

Edge cases

• Countries that forbid cross-border transfer of even aggregated stats — maintain analytics within country and provide federated dashboards.
• Real-time analytics needs — implement regional streaming materialized views and send downsampled summaries centrally.

Requirements & constraints:

• Low-latency reads for EU and US customers (read locality)
• GDPR-like residency for personal data: EU personal data must remain in EU unless lawful transfer
• Minimize cross-region egress costs
• Strong encryption, auditable access, separate KMS control per region
• Operational: consistency, failure tolerance, manageable complexity

High-level architecture:

• Two primary regions: EU (e.g., eu-west) and US (e.g., us-east). Each region hosts full read-serving stacks for local customers.
• Single logical namespace with selective replication: personal (PII) vs non-personal data classification.
• Storage: region-local data lake/warehouse (S3/Blob) + regional read-optimized stores (Cassandra/Spanner/managed DB + read replicas).
• Global control plane for metadata, policy decisions (hosted in a region with strict access rules).

Selective replication & partitioning:

• Data classification pipeline tags records as EU-personal, US-personal, or non-personal/global at ingestion.
• Partition by geo attribute + customer tenancy. Records marked EU-personal only stored and replicated within EU region; US-personal only in US.
• Non-personal or consented global datasets replicated to both regions for low-latency reads.
• For shared aggregate results, replicate derived aggregates rather than raw PII to other regions to limit transfers.

Replication patterns:

• Asynchronous replication for non-personal/global data to minimize write latency and egress (batch delta transfer).
• No cross-region replication for EU-personal unless explicit legal basis — support on-demand controlled export workflow with audit.

Encryption & KMS separation:

• Region-local at-rest encryption using region-specific customer-managed KMS keys (separate key per region and per environment).
• Keys for EU-personal data owned and controlled by EU security team, never exportable.
• Envelope encryption for objects; key wraps stored regionally.
• TLS for in-transit. Use client-side encryption for especially sensitive fields where needed.

Access control:

• Fine-grained IAM and RBAC per region. Limit cross-region admin accounts.
• Attribute-based access control (ABAC) that checks data classification and request origin.
• Service principals constrained with condition keys (e.g., aws:RequestedRegion, source IP, VPC) to prevent accidental cross-region access.
• Zero-trust: mutual TLS between services; short-lived credentials (STS).

Logging, audit, and proving compliance:

• Immutable, region-local audit logs (write-once) for access, replication events, key usage, and data exports.
• Aggregate audit views made available to compliance via secure read-only channels — only metadata exported, not raw EU-personal.
• Use SIEM and automated alerts for suspicious cross-region transfers or KMS usage patterns.
• Periodic attestations: maintain provenance chain (ingest tag, policy evaluation, replication decisions) to prove where each record resides and why.

Minimizing egress costs:

• Avoid continuous cross-region replication for large raw datasets; replicate processed/aggregated results.
• Use compression, delta encoding and schedule large transfers during low-cost windows if needed.
• Push compute to the data (run analytics in-region and ship only results).
• Leverage CDN/edge caches for static, globally-consented data.

Consistency, failure modes & trade-offs:

• Local reads: strong local consistency where possible; cross-region eventual consistency for replicated global data.
• Trade-off: stricter residency reduces global analytics freshness; mitigate by running federated queries: query region-local stores and combine results in the requesting region.
• Provide an opt-in export workflow for customers who consent to cross-border storage, with explicit consent capture and audit trail.

Operational considerations:

• Automated policy engine for classification and enforcement; unit/integration tests for residency rules.
• Regular compliance drills and pen-testing.
• Documentation and playbooks for lawful transfer requests, data subject access requests, and key compromise procedures.

This design balances low-latency regional reads, legal residency, and cost by combining selective replication, partitioning, regional KMS ownership, strict access controls, and auditable logging while pushing compute to data and replicating only what’s necessary.

Direct answer

Label skew is when the distribution of labels in a training set doesn't match the real population the model will be scored against, for example a fraud model trained on data artificially balanced to 10% positive when real traffic is 0.5% positive. Feature skew is when an input feature's distribution at training time differs from what the model actually receives at serving time, for example a feature computed from batch logs during training but computed live, with a different lookback window, during serving. Both quietly bias a model because it learns decision boundaries calibrated to a distribution it will never actually see in production, and nothing about training loss or offline accuracy flags this on its own.

Structured elaboration

Sources of label skew: sampling procedures that oversample positives for training convenience; selection bias in which examples get a label at all (only labeling transactions a human analyst chose to review, which already correlates with what made them look suspicious); and label lag, where positive labels for recent examples haven't arrived yet, making recent data look artificially clean.

Sources of feature skew: two different code paths computing "the same" feature for training versus serving (this specific case is called training-serving skew); the feature's real-world distribution genuinely shifting over time between when training data was collected and when the model serves; and default or fallback values behaving differently online, where serving substitutes a placeholder for a missing value that training data never actually contained.

What to actually do about it:

1. Compare training-set label prevalence against a recent, unbiased sample of the true population prevalence, not just the labeled set itself, and correct for known sampling procedures explicitly (reweight examples inversely to their sampling probability, or recalibrate the model's output probabilities against the true base rate) rather than trusting the training set's balance at face value.
2. Compute per-feature distribution statistics on both the training set and a live production sample, using the exact same feature-computation code for both, and diff them on a recurring schedule, so a divergence reflects a real shift rather than a code-path bug.
3. Audit where labels come from before trusting them, specifically checking whether the labeling process was itself conditioned on an earlier automated or human decision. This is the surest way label skew hides, since it never shows up as a data-quality problem, only as a silently wrong modeling assumption.

Worked example

Consider a fraud model with a true production positive rate of 0.5%. Suppose the training set keeps all positives and downsamples negatives to reach a 10% positive rate for training convenience. In a random 100,000-transaction sample, the true rate implies about $100{,}000 \times 0.005 = 500$ positives and 99,500 negatives. To reach a 10% training positive rate while keeping all 500 positives, negatives must be reduced to $n$ such that:

$\frac{500}{500+n} = 0.10 \;\Rightarrow\; 500 = 0.10 \times (500+n) \;\Rightarrow\; 5000 = 500 + n \;\Rightarrow\; n = 4500$

So negatives go from 99,500 down to 4,500, a keep-rate of $4500/99500 \approx 4.52\%$. If the model's raw output is used directly against a fixed business threshold, its probabilities will be systematically too high, since they're calibrated to a 10% base-rate world, not the true 0.5% one. Since positives were fully retained (keep-rate 1.0) and negatives were kept at rate $s \approx 0.0452$, the standard correction for this kind of downsampling rescales the raw score back to the true base rate:

$p_{\text{corrected}} = \frac{s \cdot p_{\text{model}}}{s \cdot p_{\text{model}} + (1-p_{\text{model}})}$

If a transaction scores $p_{\text{model}}=0.5$ under the training-calibrated model, the corrected probability is:

$p_{\text{corrected}} = \frac{0.0452 \times 0.5}{0.0452 \times 0.5 + 0.5} = \frac{0.0226}{0.5226} \approx 0.0432$

So a transaction that looks like a coin flip under the training distribution is really only about 4.3% likely to be fraud in the true population, which is the concrete reason applying a naive 50% cutoff directly to raw model output, without this correction, over-flags a large share of legitimate transactions.

Trade-offs & pitfalls

• Validating against a holdout drawn from the same skewed training process doesn't catch label skew at all; the validation set needs to reflect the true population, or be explicitly reweighted to it, not just be a random split of the same biased sample.
• The correction above assumes negatives were downsampled uniformly at random; if the downsampling was non-random (keeping whichever negatives were easiest to log), the correction's assumption breaks and the fix silently fails.
• Feature skew caused by two independent code paths is not fixed by any statistical correction; it needs a single shared feature-computation path, not a calibration formula.
• Chasing an exact match between training and production prevalence is itself a trap in genuinely rare-event problems: forcing training data down to the true base rate can leave too few positive examples for the model to learn from at all, so downsampling combined with an explicit, documented correction is usually the better choice than pretending the imbalance doesn't exist.

Direct answer

Treat the dead-letter queue (DLQ) as a state machine, not just a holding queue: every quarantined message moves through explicit states, and nothing gets replayed until it passes two specific automated checks, has this exact message already been applied downstream (idempotency), and does its payload still match what the current consumer code expects (schema compatibility). Skipping either check on replay is the single most common way a "fixed" DLQ incident turns into a second incident.

Structured elaboration

Idempotency check before replay. A message can land in the DLQ after a later step in its processing failed, not the first one, for example a payment charge that succeeded but a subsequent confirmation-email step that did not. Replaying that message from the top without first checking whether it already partially or fully succeeded would double-apply the parts that already worked. Before replay, re-run the exact same dedup or idempotency lookup the normal delivery path would use, so a message that actually did succeed is recognized and skipped rather than blindly reprocessed.

Schema compatibility check before replay. A message can sit in the DLQ for days or weeks while the consumer code keeps evolving. Replaying it against the current code without checking whether its payload still matches the current expected schema risks a deserialization failure (landing right back in the DLQ) or, worse, being silently misinterpreted by code that has since changed its assumptions about a field's meaning. Validate the payload against the current schema before replay; a compatible message replays normally, an incompatible one gets transformed if a safe mapping exists, or archived with a note for manual handling rather than replayed blind.

Visibility into failure reasons. Every quarantined message should carry its classification (why it failed, and which state it is in) so a reviewer, or the automation itself, can act without re-diagnosing from scratch.

Automated replay or archive. Auto-classify at arrival (the same failure-reason tagging as the DLQ architecture question), auto-archive messages that pass a retention cutoff without being addressed, and auto-replay only for a narrowly pre-approved class of situations (for example, "downstream service X was down for known maintenance window Y, safe to replay everything quarantined during that window") with the idempotency and schema checks still applied even to auto-replays, not just manual ones. Anything outside that narrow, pre-approved class gates on human approval.

Monitoring DLQ health. Depth alone is not enough. Track the age of the oldest quarantined message (a slow leak looks fine on depth if replay keeps pace, age catches it), the replay success rate (a replay that lands right back in the DLQ signals the underlying fix did not actually work), and the arrival rate of new quarantines relative to a normal baseline.

stateDiagram-v2
  [*] --> Quarantined: retries exhausted
  Quarantined --> UnderReview: on-call or automation inspects
  UnderReview --> SchemaCheck: candidate for replay
  SchemaCheck --> IdempotencyCheck: schema compatible
  SchemaCheck --> Archived: schema incompatible
  IdempotencyCheck --> Replayed: safe to reprocess
  IdempotencyCheck --> Archived: unsafe or already applied
  Replayed --> [*]
  Archived --> [*]

Worked example

An illustrative incident, numbers chosen for the walkthrough, not measured: a downstream service outage sends 200 messages to the DLQ. On recovery, the on-call engineer runs the automated checks before batch-replaying. Of the 200: 12 had actually already succeeded on a delayed retry that landed just before the DLQ escalation was processed, the idempotency check filters these out and marks them replayed-as-already-done rather than reprocessing them; 3 have a payload shape that predates a schema migration made two weeks earlier, these route to archive with a note for manual follow-up; the remaining 185 pass both checks and replay cleanly. As a sanity check on the walkthrough's own numbers: 12 + 3 + 185 = 200, accounting for the full batch.

Trade-offs and pitfalls

• Skipping the idempotency re-check on replay is the most common way a DLQ "fix" causes a new incident, double-processing something that had already partially succeeded.
• An auto-replay policy that is too broad (for example, "replay everything older than a fixed age" with no per-message check) reintroduces exactly the failure mode this workflow exists to prevent.
• Monitoring only depth misses a slow, steady quarantine growth that never actually gets worked down, age and replay-success-rate catch what depth alone cannot.
• Not capturing why a message was archived (versus replayed) leaves nothing for someone auditing the incident later to reconstruct the decision.

Direct answer

NULL means "unknown," not "empty" and not "zero," and it does not behave like an ordinary value in comparisons: any direct comparison involving NULL (NULL = NULL, x = NULL) evaluates to NULL (neither true nor false), which is why WHERE x = NULL never matches anything, even rows where x actually is NULL.

Structured elaboration

1. NULL vs empty string vs zero, by type: in a string column, '' is a real, present value (zero-length text) while NULL means no value was recorded; in a numeric column, 0 is a real, present value while NULL again means unknown; in a date column there is no equivalent "empty" value at all, only NULL, which is a common source of confusion when developers try to use a sentinel date instead.
2. Pitfall 1, filtering: WHERE x = NULL silently returns zero rows; the correct form is WHERE x IS NULL.
3. Pitfall 2, aggregation: COUNT(column) skips NULLs while COUNT(*) counts every row regardless; SUM/AVG over a column with NULLs ignore the NULLs rather than treating them as zero, which changes an average's denominator in a way that is easy to miss.
4. Pitfall 3, joins: a LEFT JOIN with no match produces NULLs for every column from the right-hand table, not zeros or empty strings; code that assumes a numeric join result defaults to 0 will get NULL instead unless it explicitly wraps the column in COALESCE.
5. Handling strategies: COALESCE(x, default) to substitute a value for display or arithmetic; explicit IS NULL / IS NOT NULL checks in filters; and, where the schema allows it, deciding at write time whether "unknown" should even be representable, versus requiring a value.

Worked example

SELECT id FROM users2
WHERE email IS NULL OR TRIM(email) = '';

Against rows (1, NULL), (2, ''), (3, ' '), (4, 'a@x.com'), this correctly returns ids 1, 2, and 3 (verified): NULL, an empty string, and a whitespace-only string are three distinct representations of "no real email," and all three need to be checked explicitly, none of them is caught by checking for only one of the three.

Trade-offs and pitfalls

• COALESCE(amount, 0) is the right fix when "unknown" should be treated as zero for a sum, but the wrong fix when "unknown" should exclude the row from an average entirely (it will pull the average toward zero).
• Comparing a nullable column to another nullable column with plain = silently drops rows where either side is NULL from an equi-join; some engines offer a NULL-safe equality operator (IS NOT DISTINCT FROM in Postgres, <=> in MySQL) for the cases where NULL should be treated as matching NULL.
• Casting a NULL numeric to a string for display purposes can turn it into the literal text "None" or "null" in some client libraries if not explicitly handled, which then fails an IS NULL check silently downstream.

Direct answer
Acceptance criteria for a corrective action should be a specific, measurable, time-boxed statement of what 'fixed' looks like, defined before the work starts, not after. A verification plan then confirms that criterion is actually met using real data, not just confidence that the fix was implemented correctly.

Structured elaboration

• Define the metric and target explicitly. Not 'reduce timeouts' but 'reduce the service's timeout rate from its current baseline to a specific target percentage, measured over a specific window.' A vague criterion can't be verified; a specific one can.
• Set a monitoring window long enough to be meaningful. Too short a window risks declaring success on noise; too long delays knowing whether the fix worked. The right window depends on the incident's natural frequency, for example enough days to capture a representative mix of peak and off-peak traffic.
• Separate short, medium, and long-term verification. Immediately after deploying the fix: a targeted test or synthetic check confirms the mechanism works as intended. Over the following weeks: real production monitoring against the target metric confirms it holds under real conditions, not just in a controlled test. Longer term: a periodic audit or scheduled re-check confirms the improvement is durable and hasn't quietly regressed.
• Define what "success" and "failure" mean numerically in advance, including what would trigger reopening the item if the target isn't met, so there's no ambiguity or motivated reasoning once the data comes in.
• Name who signs off, so verification isn't just a self-assessment by whoever implemented the fix.

Worked example
A corrective action targets reducing a service's timeout rate from 0.5% to 0.05% within 30 days. Acceptance criteria: timeout rate, measured as a 7-day rolling average, must be at or below 0.05% for two consecutive weeks within the 30-day window, using the same monitoring dashboard and definition of 'timeout' used to measure the original 0.5% baseline. Verification plan: short-term, a synthetic load test immediately after deploy confirms the fix reduces timeout rate under simulated peak load; medium-term, the real 7-day rolling average is checked weekly against the target for the full 30 days; long-term, the metric is re-checked at 90 days to confirm it hasn't quietly crept back up as traffic patterns shift. If the 30-day window ends with the metric at 0.15%, that's a defined failure, not an ambiguous 'mostly worked,' and it triggers a re-investigation of whether the fix addressed the actual root cause or only a symptom.

Trade-offs and pitfalls
The most common mistake is defining acceptance criteria loosely enough that almost any outcome can be called success, which defeats the purpose of having criteria at all. A second is skipping the longer-term recheck: many fixes look successful in the first two weeks and then quietly regress as conditions change, and without a scheduled longer-term verification, that regression goes unnoticed until the incident recurs.

Proposal: Architectural changes — implement a lakehouse with curated, privacy-preserving feature stores and nearline materialized views to serve analysts quickly; add event streaming + real-time aggregation layer and self-service sandboxed query endpoints. Process changes — introduce "fast-path" templates for common analyses, standardized measurement libraries, and analyst-run model templates. Governance — strict privacy templates, automated lineage and schema checks, RBAC for sensitive datasets, and a rapid-approval path for low-risk requests. Operationally, enable CI for analytics (tests for data quality and instrumentation) and provide runbooks. Estimate risks: biggest engineering risk is building reliable nearline pipelines without increasing maintenance (risk: data drift/consistency); biggest organizational risk is change resistance and misuse of faster access leading to privacy mistakes. Mitigations: phased rollout with pilots, tight monitoring, enforcement of automated privacy checks, and training. Expected outcome: reduce time-to-insight by caching common aggregates, streamlining request-to-delivery workflows, and empowering PMs with vetted self-serve tools while keeping privacy controls intact.

Situation: I have an interview with a team building a consumer analytics platform and I need to quickly understand their mission, product, users, and metrics so I can ask informed questions and explain how I’d add value.

Task: Research thoroughly (public + private) and synthesize findings into a one-page prep doc I can reference in the interview.

Action:

• Public sources I’d consult:

  • Company website & product pages — answer: What problem does the product solve? What features are highlighted?
  • Blog / engineering blog / case studies — answer: recent launches, technical stack hints, data challenges described.
  • LinkedIn profiles of team members — answer: team size, roles, technologies listed.
  • Job postings (other data roles) — answer: recurring tech requirements, priorities.
  • Press releases / news — answer: business goals, partnerships, growth signals.
  • App store reviews / product reviews (if applicable) — answer: user pain points, feature demand.
  • Glassdoor / Reddit / Twitter — answer: culture, cross-functional pain points.

• Private/internal sources to ask for (pre-interview or during process):

  • Brief from recruiter or hiring manager — answer: key success metrics for the role, priority projects.
  • High-level architecture diagram or data glossary (if available) — answer: data sources, pipelines, storage.
  • Recent dashboards or OKRs — answer: primary KPIs the platform tracks.

• Concrete questions I aim to answer for each area:

  • Mission: What business outcomes does analytics enable? (e.g., retention uplift, personalization)
  • Product features: Which events and user properties matter? Real-time vs batch needs?
  • Users: Who consumes analytics (growth, product, marketing)? What SLAs do they expect?
  • Key metrics: Core metrics (DAU/MAU, retention, LTV, conversion funnels), freshness, accuracy thresholds.

• Synthesis into a short prep document:

  • One-page layout: Top line summary (mission + why analytics matters), bulleted product features, primary data sources/events, consumers & their needs, top 5 KPIs with definitions, inferred tech stack, 3-5 intelligent questions to ask in interview (e.g., "Which events are considered source-of-truth?" "What is acceptable data latency for activation use cases?"), and 2–3 examples of how I’d contribute (pipeline patterns, data quality tests).
  • Keep it a single PDF or note in my interview app so I can quickly reference facts and tailored questions.

Result: This structured prep ensures I can demonstrate product awareness, ask targeted technical and product questions, and show how my data engineering skills map to their immediate needs.

Direct answer

Use a compacted topic keyed by the event's composite dedup key to make the last-seen record for that key durable and queryable, and pair it with local processor state (a keyed state store with a time-to-live) so the hot dedup check happens in memory without hitting the compacted topic on every event.

Structured elaboration

The design has three layers:

1. Composite key: define the dedup key from whatever combination of fields makes an event unique (for example (user_id, event_id) or (source, natural_key, event_date) when no single ID is globally unique).
2. Local state with TTL: on each event, look up the key in local keyed state. If present and not expired, it's a duplicate: drop it. If absent, process it and write the key into state with an expiry.
3. Compacted topic as the durable backstop: local state is an in-memory (or RocksDB-backed) working set; the compacted topic is what survives a restart. On startup, the processor replays the compacted topic to rebuild its local dedup state before serving live traffic (this is exactly how a Kafka Streams state store's changelog topic works).
   Out-of-order arrivals are handled by keying the TTL to event time, not processing time: an event's TTL clock starts from its own event time, so an out-of-order but still-within-window duplicate is still caught, while genuinely ancient duplicates outside the TTL are allowed to reprocess (a bounded, accepted risk you size the TTL against).

Worked example

The dedup check itself is a straightforward TTL-bounded set membership test; the emphasis in review should be the TTL math, since that is where correctness is actually decided:

class TTLDedupStore:
    def __init__(self, ttl_s):
        self.ttl_s = ttl_s
        self.seen = {}  # key -> last_event_time

    def is_duplicate(self, key, event_time):
        last = self.seen.get(key)
        is_dup = last is not None and (event_time - last) <= self.ttl_s
        # only refresh on a fresh sighting (absent, or expired past the TTL); refreshing
        # on every duplicate would slide the clock forward on each touch and let a
        # steady trickle of duplicates outlive the TTL indefinitely
        if not is_dup:
            self.seen[key] = event_time
        return is_dup


store = TTLDedupStore(ttl_s=3600)
assert store.is_duplicate(("u1", "e1"), 0) is False       # first sighting
assert store.is_duplicate(("u1", "e1"), 1800) is True      # duplicate, within 1h TTL
assert store.is_duplicate(("u1", "e1"), 5000) is False     # 5000s later, outside the 3600s TTL -- allowed to reprocess

This matches the intended TTL semantics: a duplicate arriving within the window is caught; one arriving after the TTL is treated as a fresh event because holding every key forever isn't affordable at scale.

Trade-offs and pitfalls

The TTL is a memory-versus-correctness knob: too short and you miss real duplicates that happen to arrive slowly (a retried request an hour later); too long and per-key state grows without bound on a high-cardinality key space. Compaction alone is not sufficient for exactly-once: a consumer that crashes after processing but before its dedup-state write is committed can still reprocess, so the TTL store's write and the downstream side effect need to be committed together (or the downstream write itself needs to be idempotent) to close that gap.

Direct answer

An event-sourced fact design appends every state-change event and derives current or historical state by replaying or aggregating events; a snapshot fact design periodically materializes the current state directly. Event-sourcing preserves the fullest history and audit trail at the cost of replay complexity; snapshotting is simpler to query but only as granular in time as the snapshot interval, and derived-state correctness depends entirely on getting the replay logic right.

Structured elaboration

• Storage growth: event-sourced designs grow with every state change, however small; snapshot designs grow with the number of entities times the number of snapshot periods, which can be smaller or larger depending on how frequently entities actually change versus how often you snapshot.
• Query complexity: reconstructing "current state" or "state as of date X" from an event-sourced table requires replaying or aggregating all relevant events, which is more complex (and more compute-expensive per query, unless materialized separately) than a direct snapshot lookup.
• Correctness under replay: event-sourcing is only as correct as its replay logic; a subtly wrong aggregation rule silently produces wrong derived state for every downstream consumer, and bugs can be hard to detect since the raw events "look right" individually. A periodic snapshot's correctness is easier to spot-check directly against the source system at any given time.
• When each fits: event-sourcing fits when you need a full audit trail of every change and the ability to answer arbitrary historical "what happened" questions you didn't anticipate in advance (since you still have the raw events). Snapshotting fits when you know in advance which point-in-time states matter (daily, hourly) and want simple, fast queries without replay logic.

Worked example

Tracking an order's status: an event-sourced design stores order_status_event(order_id, status, changed_at), one row per transition (placed, paid, shipped, delivered), and "current status" is derived by taking the latest event per order. A snapshot design instead stores order_status_daily(order_id, date_key, status), one row per order per day, directly queryable without replay, but unable to tell you the exact minute a status changed within a day, only what it was as of each day's snapshot.

Trade-offs and pitfalls

A common middle ground is event-sourcing the raw truth (since it's cheap to append and preserves everything) while ALSO maintaining a derived snapshot or accumulating-snapshot table for the common query patterns, so most queries hit the cheap, simple snapshot while the full event history remains available for the rare cases needing finer-grained replay. Pure event-sourcing without any derived materialization pushes replay cost onto every single query, which rarely holds up under real dashboard load.