Data Pipeline Scalability and Performance Questions

Direct answer

With multiple parallel workers writing immutable objects into a data lake, the risk is not one worker crashing mid-write (that is the single-writer recovery problem), it is two workers both believing they own the same logical write and both producing an object for it, or one worker retrying and producing a second object for work it already completed. The fix combines a deterministic deduplication key (so retries are self-cancelling), an atomic commit manifest (so partial or duplicate work is never visible to readers), and a coordinator or distributed-idempotency-check layer (so two workers do not redundantly do the same logical work in the first place), plus a reconciliation approach that only checks what changed rather than rescanning the whole dataset.

Structured elaboration

Deduplication keys. Every object a worker writes is named deterministically from the logical unit of work it represents (a partition key plus a source-side version or a content hash), never from a worker-local counter or a fresh UUID per attempt. Two workers (or one worker retried) producing output for the SAME logical unit then produce the SAME object key, so the data lake's own overwrite-by-key semantics collapse them, exactly the same content-addressable pattern used for idempotent writes generally, applied at the granularity of "one worker's unit of work" rather than "one event."

Atomic commit manifests. Individual object writes to a data lake are not enough on their own, since a reader listing the target prefix mid-write sees an inconsistent, partial set of objects. A manifest (or a table format's transaction log, Delta/Iceberg being the standard examples) is written LAST, atomically, and enumerates exactly which objects constitute the current, complete, correct version. Readers consult the manifest, never the raw object listing, so partial or in-flight work from any worker is invisible until the manifest says it is done, this is the same atomic-manifest-commit mechanism a single job uses, extended here to many concurrent writers producing manifest entries.

Write-ahead logs or staging plus atomic rename. Before committing to the manifest, workers write to a staging area (a private prefix or a versioned object). This isolates in-progress work from the visible manifest entirely; a crashed worker's staged-but-uncommitted objects are simply orphaned data, cleaned up later by compaction, never something a reader could accidentally observe as "the current state."

Coordinator patterns versus distributed idempotency checks. Two structurally different ways to prevent redundant work:

• Coordinator pattern: a single component (a lock service, a leader-elected assignment table) decides which worker owns which unit of work before any writing starts, so at most one worker is ever actively producing output for a given logical unit at a time. Simple to reason about; the coordinator itself becomes a single point that must be highly available and must itself recover cleanly from a crash.
• Distributed idempotency check: no central assignment; any worker may attempt any unit of work, and correctness relies entirely on the deduplication key plus atomic commit to collapse duplicate attempts after the fact. No coordinator to keep available, but more redundant work is actually performed (multiple workers may compute the same output before one "wins" at commit time), which is wasteful of compute even though it stays correct.
  The right choice depends on how expensive redundant computation is: cheap transformations tolerate the distributed-idempotency-check approach's wasted work; expensive computations (a large join, an ML feature computation) usually justify paying for a coordinator to avoid redundant work outright.

Efficient reconciliation without reprocessing everything. Track, per logical unit, the source version it was derived from (the same version/timestamp used in the dedup key). A reconciliation pass compares the manifest's recorded source versions against the current source, and only reprocesses units whose source version has actually advanced, an incremental check bounded by what changed, not a full rescan of the entire dataset.

Worked example

10 parallel workers process a data lake write for 100,000 logical partitions (say, one partition per customer per day), using the distributed-idempotency-check approach (no coordinator). Assume a modest 2% rate of overlapping work due to dynamic task rebalancing (two workers both picking up the same partition before either commits):

$$100{,}000 \times 0.02 = 2{,}000 \text{ partitions computed redundantly by two workers}$$

Each of those 2,000 partitions is computed twice (4,000 total computations for 2,000 logical units), wasting roughly:

$$\frac{4{,}000 - 2{,}000}{100{,}000 + 2{,}000} \approx 1.96\% \text{ extra compute across the whole run}$$

but produces exactly 100,000 manifest entries at commit time, since the deterministic dedup key means both workers' output for the same partition either lands on the identical object key (content-addressed) or the manifest-commit step accepts only the first writer to successfully append that partition's entry and the second's redundant object becomes an orphan for later compaction. Correctness is unaffected by the redundant computation; only compute cost is. Switching to a coordinator pattern for this same workload would eliminate the ~2% redundant compute at the cost of the coordinator's own availability and latency overhead, a real trade worth making explicitly rather than defaulting to either choice.

Trade-offs and pitfalls

• Common mistake: relying on "last writer wins" without a deterministic tiebreak. If two workers' output differs even slightly (a subtle nondeterminism in a downstream transformation, floating-point summation order), an arbitrary "whoever commits last" rule can silently pick an inconsistent result across runs; ties should be broken by an explicit, stable rule (a fixed worker ID ordering, or the manifest's own append-only compare-and-swap): the same discipline any deterministic tiebreak on a natural key plus source timestamp needs.
• Coordinator-pattern availability is a real production dependency, not free. A coordinator outage without a fallback either halts all writers (safe but stalls the pipeline) or, worse, tempts a "just proceed without coordination during the outage" shortcut, which reintroduces exactly the duplicate-work problem the coordinator exists to prevent.
• Reconciliation that skips "what changed" tracking degrades into a full rescan under scale. Without a recorded source version per manifest entry, incremental reconciliation cannot tell what needs reprocessing and either reprocesses everything (correct but wastes the exact efficiency this pattern is meant to provide) or trusts staleness silently (efficient but wrong).
• Immutable objects mean updates are new objects, not in-place edits. A "correction" to an already-committed partition is a new object plus a new manifest entry superseding the old one, not a mutation; forgetting this and attempting to overwrite an existing immutable object in place either fails outright (many data lakes reject overwrite) or, worse, silently succeeds and breaks any concurrent reader relying on that object's immutability.