Data Reliability and Fault Tolerance Questions

Direct answer

Exactly-once via transactional commit adds a coordination step (a batch of records must wait for a transaction to commit before becoming visible downstream) that at-least-once processing simply does not pay. Working through concrete numbers: a worker processing at 5,000 events/sec raw, batching into transactions committed every 100ms with a 20ms commit overhead, drops to roughly 4,167 events/sec effective throughput (a 16.7% reduction) and adds roughly 70ms of average end-to-end latency (up to 120ms worst case) that at-least-once processing does not incur. For most ML pipelines, where a slightly duplicated or slightly stale feature is far cheaper than added latency or infrastructure cost, at-least-once plus compensating actions (or plain deduplication) is usually the better default; exactly-once is worth its cost specifically when a duplicate's downstream impact is expensive or irreversible.

Structured elaboration

The quantitative model, worked step by step. Assume a single worker processes events at a raw rate $R = 5{,}000$ events/sec ($0.2$ ms/event) when NOT paying any transactional overhead, this is the at-least-once baseline. For exactly-once via Kafka-style transactions, events are batched into a transaction committed every interval $I = 100$ ms, and the commit itself (a two-phase coordination round trip with the broker) takes $C = 20$ ms, during which no new events in that transaction become visible downstream.

Throughput calculation.

$$\text{events per window} = \frac{I}{1000} \times R = \frac{100}{1000} \times 5{,}000 = 500 \text{ events}$$ $$\text{effective cycle time} = I + C = 100 + 20 = 120 \text{ ms}$$ $$\text{exactly-once throughput} = \frac{500 \text{ events}}{0.120 \text{ sec}} \approx 4{,}166.7 \text{ events/sec}$$ $$\text{throughput reduction} = \frac{5{,}000 - 4{,}166.7}{5{,}000} \times 100\% \approx 16.7\%$$

Latency calculation. An event arriving right at the START of a transaction window waits the FULL cycle ($I + C = 120$ ms) before becoming visible downstream; an event arriving right before commit waits only $C = 20$ ms. For events arriving uniformly across the window, average added wait before commit begins is $I/2 = 50$ ms, plus the commit itself, $C = 20$ ms:

$$\text{average added latency} = \frac{I}{2} + C = 50 + 20 = 70 \text{ ms}$$ $$\text{worst-case added latency} = I + C = 120 \text{ ms}$$

against an at-least-once baseline visibility latency of roughly the raw per-event processing time itself ($0.2$ ms), negligible by comparison, so the 70ms average / 120ms worst-case figures above are essentially the FULL added cost of exactly-once in this model, not a marginal increment.

Cluster-scale cost. To sustain a target aggregate throughput of 50,000 events/sec: at-least-once needs $50{,}000 / 5{,}000 = 10$ workers; exactly-once needs $50{,}000 / 4{,}166.7 \approx 12$ workers, 2 additional workers (a 20% increase in worker count) purely to make up for the transactional throughput tax, a real, budgetable infrastructure cost.

Worked example: when at-least-once plus compensating actions is preferable for ML pipelines

Consider a real-time feature-ingestion pipeline writing to an online feature store, feeding a model with a 200ms P95 serving latency budget. Exactly-once's worst-case 120ms of ADDED latency alone would consume 60% of that entire budget before any model inference even begins, likely incompatible with the SLA outright. At-least-once plus an idempotent, keyed upsert (essentially free relative to a raw write) achieves the SAME final correctness (no double-counted feature updates) with none of the transactional latency tax, since idempotent-write deduplication happens at write time, not via a blocking cross-record coordination step. The "compensating action" angle applies to the rarer case where the SIDE EFFECT is not naturally idempotent (an alert already fired based on a duplicate-processed event); there, a cheap, async, after-the-fact compensating action costs far less than paying the exactly-once tax on every single event to prevent a rare, correctable duplicate.

Trade-offs and pitfalls

• Common mistake: treating "exactly-once" as strictly better and "worth it by default." As the numbers above show, it has a real, quantifiable cost (throughput AND latency AND infrastructure); the question is always whether a specific pipeline's downstream correctness requirement justifies that cost, not whether exactly-once is theoretically the strongest guarantee.
• Common mistake: assuming the commit interval I and overhead C are fixed, universal constants. They are real, TUNABLE parameters (a shorter I reduces latency at the cost of MORE frequent commits, meaning less amortization of C and thus a WORSE throughput hit; a longer I amortizes C better but adds more latency); the specific numbers used here (100ms/20ms) are illustrative, and any real system should measure its own values.
• Latency-sensitive pipelines (most ML serving) should almost always default to at-least-once plus idempotent writes, reserving true transactional exactly-once for pipelines where a latency budget genuinely tolerates the tax (typically batch or near-real-time analytics, not sub-second serving).
• The 2 extra workers in the cluster-scale example is a REAL, recurring infrastructure cost, not a one-time engineering effort; this is the kind of number worth surfacing explicitly in a design review, since "exactly-once" as a checkbox requirement can silently commit a team to a permanent ~20% larger fleet for the affected pipeline.

Direct answer

A write-ahead log (WAL) is an append-only log that records every state mutation before the mutation is applied to the actual data structure, so if the process crashes mid-update, replaying the WAL from the last known-good point reconstructs exactly the state that existed right before the crash. It trades a small amount of write latency (every change is logged first) for a strong recovery guarantee (no acknowledged write is ever lost) and for cheap durability (sequential appends are fast even on spinning disks, let alone SSDs). PostgreSQL's WAL, RocksDB's WAL, and a Kafka topic used as a commit log are all instances of the same idea: an ordered, append-only, replayable record of intent.

Structured elaboration

The core mechanism. Before any change is applied to the in-memory or on-disk data structure, a record describing that change is appended to the log and (usually) fsynced to durable storage. The system acknowledges the write only after the log record is durable. On crash, recovery replays every log record after the last confirmed checkpoint, in order, reconstructing state deterministically. This gives crash recovery without requiring every single mutation to be immediately reflected in the final data structure (which would be far slower, since random-access updates to a B-tree or an in-memory table are expensive compared to a sequential append).

Benefits.

• Durability with sequential I/O: appending to a log is cheap relative to random-access updates to the primary data structure, so the WAL buys durability without forcing every write to be a slow in-place update.
• Deterministic, replayable recovery: replaying the same log from the same starting point always produces the same end state, which is what makes crash recovery predictable and testable.
• Decouples "durable" from "applied": a write can be considered safely committed as soon as it is in the log, even before the corresponding in-memory structure or index is updated, which is what lets systems batch or defer the expensive part of a write.

Drawbacks.

• Extra write amplification: every logical write becomes at least two physical writes (the log record, then eventually the applied change to the primary structure), and often more once compaction and replication are counted.
• Unbounded growth without compaction: a log that is never truncated grows forever; something must periodically fold applied changes into a checkpoint and discard (or archive) the WAL segments before that point.
• Recovery time scales with log length: if a system crashes long after its last checkpoint, replay has to walk a long WAL, which increases the time-to-recover. This is precisely why WALs are paired with periodic checkpoints.

Corruption detection. Every WAL record typically carries a checksum (a CRC over the record's bytes) and often a monotonically increasing sequence number (Postgres calls this the LSN, log sequence number). On replay, the recovery process verifies each record's checksum before applying it and confirms sequence numbers are contiguous; a checksum mismatch or a gap in the sequence signals a torn write (a partial disk write from a crash mid-fsync) or on-disk corruption, and recovery stops there rather than applying a possibly-corrupt record, treating everything after that point as "not durably committed."

Concrete examples.

• PostgreSQL's WAL: every row-level change is logged before the corresponding heap page is modified on disk; checkpoints periodically flush all dirty pages so WAL segments before the checkpoint can be recycled or archived, and streaming replication ships WAL records to replicas so they can replay the same log independently.
• RocksDB's WAL: writes go to an in-memory memtable plus a WAL simultaneously; when the memtable is flushed to an immutable, sorted on-disk file (an SST file), the WAL segment covering those writes becomes unnecessary and is deleted, so the WAL only ever holds "not yet flushed to SST" data, bounding its size to roughly one memtable's worth of writes.
• A Kafka topic as commit log: the topic itself is the durable, ordered, replayable record of every event; a stream processor's local state (say, a RocksDB-backed aggregation store) can be fully reconstructed at any time by replaying the topic from the earliest retained offset (or from the last checkpointed offset), which is why Kafka's own retention window functions as the outer bound on how far back "replay" can reach.

Interaction with checkpoints and compaction. A checkpoint is a snapshot statement: "everything up to WAL position X is durably reflected in the primary data structure." Once a checkpoint completes, WAL segments entirely before position X are no longer needed for crash recovery (though they may still be retained for other purposes, like point-in-time recovery or replication) and can be reclaimed. Compaction is the analogous idea applied within a log-structured store itself: periodically merging and rewriting data to remove obsolete or duplicate entries, bounding the on-disk footprint. Together, checkpoint-then-truncate is what keeps a WAL's storage usage bounded instead of growing linearly with total lifetime write volume.

Worked example

Consider a stream processor doing stateful aggregation at 50,000 events/sec, checkpointing every 30 seconds, where each event produces roughly a 200-byte WAL record (event payload plus metadata: sequence number, timestamp, checksum).

$$50{,}000 \text{ events/sec} \times 30 \text{ sec} = 1{,}500{,}000 \text{ events between checkpoints}$$ $$1{,}500{,}000 \times 200 \text{ bytes} = 300{,}000{,}000 \text{ bytes} \approx 286 \text{ MiB of WAL per checkpoint interval}$$

If the process crashes 29 seconds after the last checkpoint (worst case, just before the next one completes), recovery must replay up to that ~286 MiB of WAL, deserializing and applying roughly 1.5 million records. Halving the checkpoint interval to 15 seconds halves the worst-case replay volume to ~143 MiB (750,000 records), cutting recovery time roughly in half, at the cost of checkpointing twice as often (more overhead on the steady-state write path). This is the concrete shape of the trade-off the drawbacks section names abstractly: shorter checkpoint intervals bound recovery time tighter, at a steady-state throughput cost.

Trade-offs and pitfalls

• Common mistake: treating the WAL as a permanent audit log. A WAL's job is crash recovery up to the next checkpoint; using it as a long-term event history without a separate retention/archival policy either grows storage unboundedly or gets silently truncated by the very compaction process meant to bound it, losing history nobody intended to lose.
• Common mistake: fsync-ing to the wrong barrier. If the acknowledgment to the caller happens before the WAL record is actually durable (e.g. the OS write() call returned but the data is still in a page cache with no fsync), a crash can lose "acknowledged" writes despite the WAL's presence; the durability guarantee only holds up to whatever barrier (fsync, or a replicated quorum ack) the system actually waits on.
• Group commit as the standard mitigation for fsync cost. Fsync-ing after every single write is safe but slow; most production WAL implementations batch multiple concurrent writers' log records into one fsync (group commit), trading a small amount of added latency per write for dramatically higher throughput, since the fsync cost is amortized across the batch.
• A WAL alone does not give you exactly-once, only durable, replayable at-least-once recovery of the log itself; combining it with an idempotent apply step (a keyed upsert or dedup check on write) is what turns "replay never loses data" into "replay never double-applies data" as well.

Direct answer

A circuit breaker wraps a downstream call (an API request from a data pipeline) with a state machine that tracks that dependency's recent health and stops calling it entirely once it looks broken, rather than letting every caller keep retrying into a known-failing dependency. It has three states: CLOSED (normal, calls pass through), OPEN (the dependency is considered failing, calls are rejected immediately without even attempting the network call), and HALF-OPEN (a cooldown has elapsed, a small number of probe calls are allowed through to test recovery). This protects both the caller (no more time wasted waiting on doomed calls) and the struggling dependency (no continued load from callers who cannot succeed anyway).

Structured elaboration

Failure threshold. The circuit opens once a configured fraction of recent calls fail (e.g., more than 50% of the last 20 calls, or more than N consecutive failures), not on the first single failure, since a single transient blip should not trip a breaker meant to catch SUSTAINED trouble; the exact threshold trades false-positive risk (opening on normal, isolated hiccups) against false-negative risk (staying closed too long into a genuine outage, still sending traffic that will fail).

Cooldown window. Once open, the circuit stays open for a fixed cooldown period before allowing any probe calls, giving the struggling dependency time to recover WITHOUT continued load from this caller during that window. Too short a cooldown re-opens the circuit into a still-broken dependency repeatedly (thrashing); too long delays recovery detection once the dependency IS actually healthy again.

Interaction with retry/backoff. The circuit breaker and retry-with-backoff operate at different granularities and are complementary, not redundant: backoff governs how AGGRESSIVELY a single caller retries an individual failed call, while the circuit breaker governs whether to attempt the call AT ALL, given the dependency's recent aggregate health. A well-designed system checks the circuit breaker state FIRST (fail fast if open, skip backoff entirely) and only applies backoff-and-retry logic for calls that proceed because the circuit is closed or half-open.

Interaction with backpressure. When the circuit is open, calls that would have gone to the failing dependency are rejected immediately rather than queued indefinitely; this is itself a form of backpressure, signaling upstream (the pipeline stage feeding this call) to either buffer (write to a durable queue), drop, or reroute, rather than accumulating unbounded in-flight work waiting on a dependency that will not respond in time anyway.

Worked example

A pipeline calls a downstream enrichment API at 2,000 requests/sec. The dependency begins failing at 80% of requests (a partial but severe degradation). With a threshold of "open if more than 50% of the last 20 calls failed": within roughly $20 / 2{,}000 = 0.01$ seconds of the degradation beginning (the time to accumulate 20 calls at this rate), the failure ratio crosses 50%, and the circuit opens. From that point, roughly:

$$2{,}000 \text{ requests/sec} \times \text{cooldown duration}$$

worth of requests per second of cooldown are rejected immediately (fast, cheap rejections) instead of each attempting a doomed network call and waiting for its own timeout; at a 30-second cooldown, this is 60,000 requests that would otherwise have each paid a network round-trip's worth of latency waiting to fail, now failing in microseconds instead. After the cooldown, a small number of half-open probe calls (not all 2,000/sec resuming at once) test whether the dependency has recovered; if they succeed, the circuit closes and full traffic resumes; if they still fail, the circuit reopens for another cooldown period.

Trade-offs and pitfalls

• Common mistake: opening on the first failure. This makes the breaker indistinguishable from "stop on any error," far too aggressive for a dependency that has occasional, normal transient blips; the threshold should reflect SUSTAINED degradation, not any single failure.
• Common mistake: resuming full traffic immediately after cooldown instead of a small half-open probe. Sending all 2,000 requests/sec back at once the instant cooldown ends risks immediately re-triggering the same overload/failure condition if the dependency has not FULLY recovered, undoing the cooldown's benefit in the first probe cycle.
• A circuit breaker without any coordination with backpressure just moves the problem, not solves it, per the worked example: rejected calls still need somewhere to go (buffer, drop, or reroute), a design decision separate from the circuit breaker itself.
• Per-dependency circuit state, not a single global breaker, is essential once a pipeline calls MULTIPLE downstream dependencies; a struggling dependency should not trip a breaker that also blocks calls to healthy, unrelated dependencies.

Direct answer

Checkpointing captures the state needed to resume a running job from where it left off, periodically and (in modern stream processors) asynchronously, without stopping the pipeline; snapshotting usually refers to a full, consistent copy of all operator state at one instant. In practice the terms overlap (Flink calls its periodic mechanism "checkpointing" and reserves "savepoint" for a manually triggered, more portable full snapshot), but the useful distinction for interview purposes is incremental versus full: an incremental checkpoint records only what changed since the last one, a full snapshot re-persists everything. For 50,000 events/sec, incremental checkpointing every 10 to 30 seconds is a reasonable default, and an on-call engineer restoring after a crash needs the last completed checkpoint's state, the source offsets it corresponds to, and confirmation that external sinks are consistent with that same checkpoint (not ahead of it).

Structured elaboration

What gets checkpointed and why. Three things: operator state (aggregation accumulators, windows in flight, join buffers), the position in the input (source offsets, e.g. a Kafka partition-offset map), and enough metadata to know these two are mutually consistent (they describe the exact same logical point in the stream). Without this, restoring operator state without also restoring the matching input position either replays already-processed events against already-updated state (double counting) or skips events the state does not yet reflect (data loss).

Implementation. A full snapshot pauses (or barrier-synchronizes) all operators, serializes their complete state to durable storage (S3, HDFS, a distributed filesystem), and records the corresponding source offsets, all as one atomic unit. Incremental checkpointing (Flink's RocksDB-backed incremental checkpoints are the canonical example) instead persists only the state changes (the RocksDB SST files created since the last checkpoint) via a background process, using a barrier that flows through the operator graph (Chandy-Lamport-style) to mark a consistent cut without pausing processing.

Latency and throughput impact. A full, blocking snapshot pauses processing for its duration, directly adding to end-to-end latency and briefly dropping throughput to zero; this gets worse as state size grows, since a bigger state takes longer to serialize. Incremental, asynchronous checkpointing keeps the pipeline processing while checkpointing happens in the background, at the cost of a small, steady overhead (I/O contention, CPU for serialization) rather than a periodic pause, and a more complex recovery path (you may need to replay a chain of incremental deltas, not just load one snapshot).

Recovery-time trade-off. Full snapshots are simpler and faster to restore from (load one complete state), since there is nothing to replay past that. Incremental checkpoints are cheaper to take but can be slower to restore from if many increments have accumulated since the last full base snapshot, since recovery has to apply that whole chain. Production systems bound this by periodically taking a full snapshot anyway (e.g. Flink's incremental checkpointing still relies on RocksDB's own compaction to prevent an unbounded chain of deltas).

When incremental beats full. Once state size grows large relative to the achievable checkpoint interval (state in the tens of GB or larger per operator, common for large windowed aggregations or ML feature stores), a full snapshot's pause time becomes unacceptable for a low-latency SLA. Incremental checkpointing is the standard answer: state size no longer directly gates checkpoint duration, since each interval only pays for what changed.

Worked example

For a pipeline at 50,000 events/sec with a stated low-latency requirement, the checkpoint interval is chosen by balancing two costs: checkpoint overhead (more frequent = more steady-state cost) against worst-case recovery time (less frequent = a longer replay window after a crash).

$$\text{worst-case events-to-replay} = \text{events/sec} \times \text{checkpoint interval (sec)}$$

At a 20-second interval: $50{,}000 \times 20 = 1{,}000{,}000$ events of worst-case replay. At a 5-second interval: $50{,}000 \times 5 = 250{,}000$ events, a 4x reduction in worst-case recovery replay volume, at 4x the checkpointing frequency (4x the steady-state overhead cost). A reasonable default for a genuinely low-latency pipeline is 10 to 20 seconds: frequent enough to bound recovery to roughly the same order of magnitude as the pipeline's own end-to-end latency target, infrequent enough that checkpoint overhead stays a small fraction of total processing time. The exact number is a measured trade-off specific to the operator's state size and I/O budget, not a fixed constant; teams tune it empirically against their own checkpoint-duration metrics.

What an on-call engineer needs to restore after an operator crash. Three concrete facts, in order: (1) the ID/timestamp of the last successfully completed checkpoint (not one that was in progress and aborted), (2) the exact source offsets recorded in that checkpoint's metadata, so the job resumes reading from precisely that point, not from "wherever it last was" which may be ahead of the checkpoint, and (3) whether external sinks (a database, a downstream topic) are consistent with that same checkpoint or need reconciliation, since a checkpoint only captures the job's own internal state; a sink write acknowledged after the checkpoint but not yet reflected anywhere durable on the source-offset side can be silently redone (safe only if the sink write is itself idempotent) or silently lost (if it is not), which is exactly why idempotent-sink design (a keyed upsert or dedup check at the write) and checkpoint design must be engineered together, not independently.

Trade-offs and pitfalls

• Common mistake: restoring operator state without restoring the exact matching offsets. This desynchronizes "what the job thinks it processed" from "what it will process next," producing either duplicate processing (offsets too far behind) or silent gaps (offsets too far ahead).
• Common mistake: treating "checkpoint completed" and "checkpoint durable" as the same event. A checkpoint write acknowledged by the local filesystem but not yet replicated to durable storage (S3, a replicated filesystem) can be lost in the same failure that triggered the need to recover from it; the checkpoint storage backend needs its own durability guarantee independent of the job's own process.
• Barrier alignment cost. Chandy-Lamport-style barrier checkpointing requires operators with multiple input streams to wait for the barrier to arrive on all inputs before proceeding (alignment), which under skewed or backpressured inputs can stall the fastest input waiting for the slowest; unaligned checkpointing (buffering in-flight records instead of blocking) trades a larger checkpoint size for lower tail latency under skew.
• State size, not event rate alone, usually dominates checkpoint duration. A pipeline at low event rate but with very large windowed state (e.g. a long session window holding months of history) can have checkpoint duration problems a naive "we're only at 500 events/sec, checkpointing should be cheap" assumption misses; always size the checkpoint interval against measured state size and serialization time, not event throughput alone.

Direct answer

At 1 million events/sec with a sub-200ms latency target and potentially billions of distinct keys, no single machine can hold all state in memory, so the architecture has to shard state by key across many workers, externalize each shard's state to a compaction-friendly store (RocksDB is the standard choice) rather than pure in-memory maps, and use bloom filters to cheaply reject lookups for keys that are cold (not present) without paying a full disk read. Checkpointing at this scale must be incremental and asynchronous, and recovery must be partitioned so a single worker's failure only requires restoring that worker's shard, not the entire job's state, keeping both checkpoint duration and restore time practical at this scale.

Structured elaboration

Partitioning strategy. Key-based hash partitioning (consistent hashing, so a change in worker count reshuffles a bounded fraction of keys rather than everything) distributes both the incoming event load and the state itself. At 1 million events/sec across, say, 200 workers, each worker handles roughly 5,000 events/sec, a per-worker load that is comfortably achievable with a well-tuned pipeline.

State sharding and externalized state backends. Each worker's state shard is backed by an embedded, on-disk key-value store (RocksDB is the canonical choice for exactly this use case) rather than a pure in-memory structure, so per-worker memory usage is bounded by the store's block-cache size, not by total distinct keys assigned to that worker. Compaction (RocksDB's own background merge of on-disk SST files) bounds on-disk footprint growth from repeated updates to the same keys. TTL on state entries expires genuinely stale keys automatically, bounding total stored key count for workloads where "cold" keys eventually become irrelevant (a user who stopped being active months ago).

Bloom filters for cold-key filtering. With billions of distinct keys, most individual keys are touched rarely; a bloom filter in front of each RocksDB shard (RocksDB supports this natively per SST file) lets a lookup for a key that was never written be rejected in memory, in O(1), without touching disk at all, which matters enormously at this event rate: even a small fraction of disk-bound lookups at 1 million events/sec would overwhelm typical SSD IOPS budgets.

Checkpoint frequency and storage. Incremental checkpointing is mandatory at this state size; a full snapshot of billions of keys' worth of state would make any reasonable checkpoint interval prohibitively slow. Checkpoints persist to a distributed, durable store (S3 or equivalent) so a single worker's local disk failure does not also destroy its checkpoint history. Checkpoint interval is chosen the same way it always is: balancing steady-state overhead against worst-case replay volume, but at this event rate even a short interval (5-10 seconds) still represents millions of events of worst-case replay, making the recovery PLAN (below) as important as the interval choice itself.

Recovery plan minimizing data loss and downtime. Recovery is partitioned: a failed worker's shard is restored independently from its own last checkpoint plus WAL replay since that checkpoint, while OTHER workers' shards continue processing uninterrupted, so a single-worker failure degrades throughput by roughly 1/num_workers rather than stalling the entire job. This is the key scale-specific design point: at small scale, "restart the whole job from the last global checkpoint" is an acceptable recovery story; at 1 million events/sec across hundreds of workers, a whole-job restart would mean stalling ALL traffic for the restore time of the SLOWEST shard, which does not meet a sub-200ms-serving SLA even briefly, so per-shard independent recovery is not an optimization, it is a requirement at this scale.

Worked example

200 workers, each holding roughly 10 million distinct keys' worth of state (2 billion total, consistent with "billions of distinct keys"), each shard's RocksDB instance sized with a 1% bloom-filter false-positive rate needing roughly 1.2 bytes/key for the filter alone:

$$10{,}000{,}000 \text{ keys} \times 1.2 \text{ bytes} \approx 12{,}000{,}000 \text{ bytes} \approx 11.4 \text{ MiB of bloom-filter memory per worker}$$

comfortably small relative to typical worker memory budgets, confirming bloom filters are a cheap, high-leverage addition at this scale. If one worker fails and must restore from its last checkpoint (say, 8 seconds old) plus WAL replay:

$$5{,}000 \text{ events/sec (this worker's share)} \times 8 \text{ sec} = 40{,}000 \text{ events of WAL replay for the failed shard}$$

a bounded, fast recovery (well under a second of actual replay work at typical per-event processing costs), affecting only $1/200 = 0.5\%$ of total traffic for that brief window, versus the alternative of restoring 2 billion keys' worth of aggregate state and 1 million events/sec of aggregate WAL if recovery were NOT partitioned per-shard, illustrating concretely why partitioned recovery, not just partitioned processing, is what makes this architecture's downtime story practical at this scale.

Trade-offs and pitfalls

• Common mistake: partitioning event processing by key but NOT partitioning recovery the same way. If all workers' checkpoints are coordinated into one global barrier and restored as one atomic unit, a single worker's failure forces a full-job restore, exactly the failure mode the worked example shows this architecture must avoid.
• Bloom filters trade a small, bounded false-positive rate for a large disk-IO reduction. A false positive here means an unnecessary disk read for a key that turns out not to exist (a performance cost, not a correctness one, since RocksDB still checks the actual data after the bloom filter says "maybe present"), so this is a safe optimization, not a risk to data integrity.
• TTL-based expiry for cold keys is a real product decision, not just an engineering knob. Expiring a key that a downstream consumer still expects to see (even rarely) silently changes behavior from "known value" to "cold-start default," which needs explicit product agreement, not just an engineering default chosen for memory convenience.
• Consistent hashing bounds reshuffle cost on worker-count changes, but does not eliminate it. Scaling the worker pool up or down still moves some keys' state between workers, and that migration itself needs to be a safe, checkpoint-consistent operation, not a live cutover that risks losing in-flight state for the migrating keys.