Event-Driven Architecture and Asynchronous Messaging Questions

Direct answer

Treat events as immutable and never edit history in place: back up the raw event log and periodic snapshots separately, version every schema change so old events remain interpretable forever, and do all reprocessing (replaying events into a new or corrected projection) through a governed pipeline with a dry run, a diff against known-good output, and a documented rollback before anything touches production. The core discipline is that a schema change never rewrites what already happened; it only changes how new readers interpret it, via explicit transformation logic kept alongside the schema registry.

Structured elaboration

Immutability of events

• A stored event is never edited or deleted (subject to legal retention limits) once written; every event carries its schema version so a reader always knows which interpretation rules to apply.
• Corrections to a wrong past fact are themselves new events (a compensating or correcting event), not silent edits to history. This preserves the audit trail's core value: what actually happened is recoverable exactly as it happened, including any mistake and its correction.

Event versioning

• Every event carries explicit metadata: event type, schema version, and a timestamp. A schema registry tracks compatibility rules (backward, forward, or full) per event type.
• Prefer additive changes (new optional fields with defaults). For a genuinely breaking change (a field removed, or its meaning changed), introduce a new event type or version rather than mutating the old one's meaning in place.
• Maintain a version-aware deserializer, or an explicit "upcaster" registry mapping older event versions to the current shape, so replay from the very beginning of the log always works even after several schema generations.

Enrichment vs transformation

• Enrichment happens at read time: a consumer or projector adds derived fields (a lookup join, a denormalized label) without changing the stored event bytes at all. This is the default, lower-risk choice whenever the correction or addition can live entirely downstream of the log.
• Transformation is a deliberate, governed rewrite: when correctness or performance genuinely requires a canonical change to what is stored, it happens by emitting new-version events through an idempotent migration pipeline, never by editing the original bytes. Escalate to transformation only when enrichment cannot solve the problem, since transformation carries materially more operational risk.

Backup and restore

• Back up the raw append-only event log continuously (or on a tight schedule) and take periodic snapshots of aggregate state and read-model projections separately; store both in durable, immutable, checksummed storage with clear retention policy.
• Restore in the correct order: bring back the raw events first (the source of truth), then replay them into whatever projection code and snapshot generation is current, rather than restoring a stale snapshot as if it were authoritative on its own.

Tools and practices for replaying events into new projections (folding the media/blob-payload nuance)

• A replay orchestration service accepts a manifest: which stream(s), which offset range (or the full history), which projection code version, and a dry-run flag; it should support partitioned, parallelizable replay with rate limiting so a full historical replay does not overwhelm downstream systems.
• Binary or media payloads (an image, an audio clip, a large document attached to an event) need different handling than a small JSON field: store the actual bytes in object storage and put only a reference (a pointer plus a content checksum and a payload-format version) inside the event itself. This keeps the event log itself small and fast to replay, and it makes media-format evolution (a new image encoding, a new document schema) its own explicitly versioned concern, checked via the checksum and the payload-format version rather than assumed compatible.
• Run new projection logic in shadow first: replay into a separate, non-serving projection, and diff its output against the currently-serving one (or against known-correct reconciliation totals) before cutting any production read traffic over.

Safe rollbacks if reprocessing causes issues

• Dry-run and checksum-validate before any production replay actually writes anything that serves traffic.
• Canary the replay: run it against a small time window or a small partition subset first, compare digests or reconciliation totals against the original, and only widen the replay once that canary is clean.
• Keep the previous projection's snapshot intact and untouched during a reprocessing run, so if the new projection is wrong, the rollback is: stop, discard the new projection's output, and resume serving from the untouched previous snapshot, no data was lost because the source events were never modified.

Worked example

An events log for a document-management system stores 12 million events over 3 years, including DocumentUploaded events that reference a scanned PDF stored in object storage (never inlined in the event itself: the event carries {doc_id, storage_key, checksum, payload_format_version: 1}). The team introduces payload_format_version: 2, switching to a new PDF/A archival encoding for new uploads.

• Enrichment path: a new "searchable text" field is added by a downstream OCR job that reads the referenced PDF and writes the extracted text into a separate read model. This never touches the original event or its stored bytes, so it ships with zero replay risk.
• Transformation path: 3 years later, the team decides all archived documents should be re-encoded to the new format for long-term storage compliance. This is a deliberate, governed transformation: a migration job reads each DocumentUploaded event (all 12 million, in a rate-limited, partitioned replay), re-encodes the referenced PDF, writes it to a new storage key, and emits a new DocumentReencoded event carrying the new key, checksum, and payload_format_version: 2, rather than overwriting the original event or the original stored bytes. The original event, and the original file, remain exactly as they were.

Trade-offs and pitfalls

• Common wrong turn: reaching for a transformation (a rewrite) when enrichment (an additive, read-time derivation) would have solved the same business need with far less risk and no touching of history.
• Common wrong turn: inlining large or binary payloads directly in events "for simplicity." It makes the event log itself expensive to store and slow to replay, and couples every future replay to whatever encoding was in use when the event was written.
• Common wrong turn: restoring a snapshot without also restoring and verifying the underlying event log's integrity first, which can silently paper over a corrupted or incomplete backup of the actual source of truth.
• Senior signal: keeping enrichment as the default and treating transformation as an escalation that requires its own governed pipeline (dry run, canary, diff, rollback plan), rather than treating all schema evolution as equally low-risk.

Direct answer

For a global streaming pipeline serving developer tools that need low-latency delivery, at-least-once semantics, and variable throughput, managed Kafka is the strongest fit when cross-region replication and strict per-key ordering matter enough to justify its operational profile; Google Pub/Sub (or a similar fully-managed global event bus) is the strongest fit when minimizing operational overhead and getting native global reach matter more than fine-grained partition control; RabbitMQ is generally the weakest fit for a genuinely global, high-throughput streaming pipeline, since cross-region replication and elastic scaling are not its core strengths.

Structured elaboration

Compare all three on the six named axes:

Decision criteria. Weight cross-region replication and per-key ordering strength heavily if the developer-tools audience needs strict correctness guarantees per user/session across regions; weight operational overhead and TCO heavily if the team is small relative to the platform's reach, in which case Pub/Sub's serverless model is likely to win. RabbitMQ should only be the recommendation if the deployment is effectively single-region despite the "global" framing, or if the workload's actual value is complex content-based routing rather than raw throughput and reach.

Monitoring plan. Track, per region: end-to-end publish-to-consume latency at p50/p95/p99 (50th/95th/99th percentile) to catch regional degradation early; consumer lag (for Kafka) or oldest-unacked-message age (for Pub/Sub) to detect a falling-behind consumer before it breaches downstream SLAs; dead-letter queue (DLQ) growth rate as the leading indicator of a poison-message or downstream-outage problem; and, specifically for the cross-region requirement, replication lag between regions, since a lagging replica can silently serve stale data to a region that failed over to it.

Worked example

A developer-tools company streams build-status events (build started, build passed, build failed) globally to CI (continuous integration) dashboards used by engineering teams on three continents. The requirement for low-latency delivery per developer's region, combined with variable throughput (dozens of events/sec quietly, spikes to thousands/sec during a company-wide merge window), and a need for near-real-time correctness with no strict global ordering requirement (each build's events only need to be ordered relative to that build, not globally), points toward Google Pub/Sub (or a similar managed global event bus): its native multi-region reach and serverless elastic scaling directly match the "global + variable throughput" shape of this workload, and per-build ordering can be achieved using an ordering key set to the build ID, without needing Kafka's manual partition-count planning for a workload whose peak-to-trough ratio is large and hard to predict in advance.

Trade-offs and pitfalls

The main pitfall is picking managed Kafka for its brand-name familiarity without actually needing its per-key partition control or its higher, standing operational cost, when the workload's real shape (global, bursty, no strict cross-entity ordering requirement) is better served by a serverless event bus. The opposite pitfall is picking a serverless event bus for a workload that does need Kafka-style strict ordering guarantees across a large key space, and discovering the chosen service's ordering-key feature is more constrained (e.g., scoped per region, or with lower throughput per key) than expected. A senior answer ties the recommendation explicitly to which of the six axes the specific workload stresses hardest, rather than defaulting to whichever technology is most familiar to the team.

Direct answer

Build the transformer as its own consumer/producer pair sitting between the legacy topic and the new one: consume the legacy message, apply a deterministic mapping keyed by an idempotency key, and publish the transformed result, treating "exactly-once effect" as something achieved through idempotent, deduplicated writes rather than a literal guarantee the messaging platform hands you for free. Minimal downtime comes from running the new pipeline in shadow behind a staging topic first, then cutting traffic over, never from trying to swap the running consumer in place under live load.

Structured elaboration

Architecture

flowchart LR
  Legacy[(Legacy topic v1)] --> Consumer[Transform consumer]
  Consumer --> Dedup{Idempotency key seen before?}
  Dedup -->|yes| Skip[Skip, already published]
  Dedup -->|no| Transformer[Apply v1 to v2 mapping]
  Transformer --> Publish[Publish to new topic]
  Publish --> New[(New topic v2)]
  Publish --> OffsetStore[(Durable offset and dedup store)]
  New --> Shadow[Shadow validator: compare v1 vs v2 output]

• Consumer reads from the legacy topic in batches, committing its position only after a successful, durable publish downstream, never before.
• Transformer applies a deterministic mapping from the legacy schema to the new one; deterministic means the same input message always produces the same output message, which is what makes retries safe.
• Publisher writes to the new topic and, in the same logical step, records that this source message's idempotency key has been handled, in a durable offset/dedup store.

Ensuring at-least-once or exactly-once semantics for the transformed output

• At-least-once (the practical default): if the publish succeeds but the consumer's offset commit fails afterward, the message may be reprocessed. This is safe as long as the transformer is idempotent: reprocessing the same source message produces the same output message, detected and deduplicated by its idempotency key (built from the source message's own identifier and topic/partition/offset) before a second copy is actually published.
• Exactly-once, in practice, means "effectively-once" achieved one of two ways:
  1. If the messaging platform supports transactional produce (consume-transform-produce as one atomic unit, committing the offset and the output write together), use that: either both happen or neither does, eliminating the reprocessing window entirely.
  2. If it does not, maintain an explicit dedup store (a fast key-value store) keyed by idempotency key with a bounded retention window, and check-and-set against it before every publish, exactly the same pattern as a consumer-side dedup store for redelivered messages generally.
• Whichever mechanism is used, the transformer's mapping logic itself must be deterministic; a non-deterministic transform (one that depends on wall-clock time, external random state, or an external lookup that can change between retries) breaks the "same input always produces same output" property both mechanisms depend on.

Handling schema drift

• Validate every incoming legacy message against its expected schema before transforming; on an unrecognized or malformed shape, do not guess: route it to a dead-letter queue (DLQ, dead-letter queue) for inspection rather than silently emitting a best-effort, possibly-wrong transformed record.
• Support multiple legacy schema versions in the mapping logic explicitly (the transformer selects a mapping based on the detected input version), since "legacy messages" in a real migration are rarely all on one exact schema by the time the transformer ships.
• Track parse-failure and unmapped-field rates as an ongoing metric, not just at cutover time; schema drift on a legacy system can continue to happen slowly even after the migration is declared done.

Minimal-downtime cutover and rollback

1. Run the transformer in shadow: consume the legacy topic, publish to a staging topic (not the real new topic), and compare its output against expectations (either a hand-built validator or, if one exists, the previous system's output) without any real consumer reading from the staging topic yet.
2. Canary: point a small percentage of real downstream traffic at the staging topic's output, monitor error rates and duplicate rates, before moving everyone.
3. Cutover: once the canary is clean, redirect the transformer's publish target from staging to the real new topic, and redirect downstream consumers to read from it. Because the consumer's offset was only ever committed after a durable publish, no messages are lost in this switch.
4. Rollback: if a problem appears post-cutover, stop the transformer, and either point downstream consumers back at the previous system's output (if it is still running in parallel for the deprecation window) or replay from a known-good source offset once the mapping bug is fixed, since the legacy topic itself is untouched by any of this and remains a valid replay source.

Worked example

A retailer migrates inventory-events-legacy (an older, flat schema) to inventory-events-v2 (a nested schema with explicit currency and unit fields) with 6 downstream consumers.

• The transformer runs in shadow against a staging topic for one week, publishing at the same rate as the legacy topic's actual traffic; a validator checks that quantity totals reconciled between the legacy and transformed streams match exactly for every one of the roughly 2 million messages processed that week (an exact-match reconciliation check, not a sampled one, since the mapping is meant to be lossless for these fields).
• 3 of the 6 downstream consumers are moved to the staging topic as an early canary; after their error rates stay flat for the week, all 6 are cut over to the real inventory-events-v2 topic.
• Two days after cutover, a schema-drift issue appears: a small number of legacy messages (about 40 out of that day's roughly 300,000) carry a field format the transformer's mapping did not anticipate (an unexpected currency code). Because those specific messages were routed to the dead-letter queue rather than silently mis-transformed, they are visible immediately as a DLQ growth alert, not as a silent data-quality bug discovered downstream weeks later. The mapping logic is patched to handle the new format, the 40 dead-lettered messages are replayed through the corrected transformer, and the pipeline continues, none of the other roughly 299,960 messages that day were affected.

Trade-offs and pitfalls

• Common wrong turn: committing the consumer's offset before confirming the downstream publish succeeded. That ordering is what turns an ordinary transient failure into silent data loss; committing after publish is what makes at-least-once (with idempotent handling) safe instead of lossy.
• Common wrong turn: treating "exactly-once" as something the messaging platform provides end-to-end for free without checking whether it actually supports transactional produce for this specific pipeline; where it doesn't, an application-level dedup store is doing the real work, not a marketing label.
• Common wrong turn: silently best-effort transforming a message with an unrecognized shape instead of dead-lettering it; a wrong but successfully-published transformed record is far more damaging than a visible pipeline stall.
• Senior signal: naming which of the two practical exactly-once mechanisms (platform transactions, or an application dedup store) actually applies to the specific messaging platform in play, rather than asserting "exactly-once" as an unqualified property of the system.

Direct answer

Event contract governance is a process problem wrapped around a small set of tools: every schema has one owning team registered in a schema registry, every change to it runs an automated compatibility check in continuous integration (CI) before merge, every consumer's real expectations are captured as executable consumer-driven contracts rather than assumed, and every deprecation has a stated window and a communication channel. As technical lead, the job is to make doing the right thing (checking compatibility, giving consumers notice) the path of least resistance, not a policy nobody reads until it is violated.

Structured elaboration

Ownership and the registry

• Every topic and its schema has exactly one owning team, registered centrally with a semantic changelog and a contact channel, so "who do I ask about this field" always has one answer.
• Compatibility policy defaults to backward compatibility; forward or full compatibility is opt-in per topic with an explicit reason, not a blanket setting.

Deprecation windows

• Owners declare a deprecation time-to-live (TTL), a default window (for example 90 days) during which a field or event version is marked deprecated but still functional; tooling should reject new production usage of a deprecated schema past its TTL unless a consumer team has an explicit approved extension.
• The registry surfaces deprecated fields and their TTL countdown to every consumer team, not just to the producer, so nobody discovers a deprecation the week it expires.

Automated compatibility checks in CI

• Every schema pull request runs the registry's compatibility check against the target topic automatically; an incompatible change fails the build rather than reaching a human reviewer as the first line of defense.
• Semantic versioning on schemas (major for breaking, minor for compatible additions, patch for non-behavioral changes) gives reviewers a fast signal for how much scrutiny a change needs.

Consumer-driven contracts

• Consumer teams publish machine-checkable expectations (a contract-testing tool in the Pact family, or an equivalent event-contract test harness) describing exactly which fields they read and how.
• Producer CI runs the full set of published consumer contracts against any proposed schema change; a change that breaks a consumer's stated contract fails CI even if it technically passes the registry's generic compatibility check, because a consumer's real usage can be narrower (or occasionally stricter) than the schema alone reveals.

Handling breaking changes in production while minimizing consumer disruption

1. Prefer avoiding the breaking change entirely: an additive field, a new event type, rather than a change to existing semantics.
2. If a breaking change is unavoidable, publish a new schema major version and, where the platform supports it, a new topic suffix; do not force it through the existing topic under a compatibility mode that cannot actually accommodate it.
3. Dual-write or bridge: producers write both the old and new shape for the deprecation window, or a small transform job derives the old shape from the new one, so consumers who have not migrated keep working unmodified.
4. Gate rollout with a canary: a small percentage of traffic (or a small subset of consumers) on the new schema first, monitored for consumer-side errors and increased message-rejection rates, before full rollout.
5. Communicate the deprecation TTL and migration deadline explicitly to every consumer team, and track migration status on a dashboard rather than relying on teams to self-report.

Governance and personally identifiable information (PII)

• Contract governance is not only about shape; it also has to enforce what data is allowed to travel on an event at all. Producers publishing personally identifiable information (PII) on an event contract should mask or tokenize sensitive fields at the source, and the schema registry's metadata should tag which fields are PII so both a review process and automated policy checks can catch a new field that leaks sensitive data before it ships.
• Data lineage tracking, recording which producer emitted a field and which consumers read it, is what makes it possible to answer "if we deprecate this field, who breaks" and "if this field turns out to be PII we should not have shipped, who has already consumed it" without a manual audit each time. Wire lineage capture into the same registry and CI pipeline that already enforces compatibility, rather than building it as a separate, easily-neglected system.

Worked example

A logistics platform's shipment-events topic is owned by the fulfillment team, with 12 consumer teams subscribed. The fulfillment team wants to deprecate a legacy carrier_code field in favor of a richer carrier object.

• Governance process kicks in: the registry marks carrier_code deprecated with a 90-day TTL. Lineage tracking shows exactly which of the 12 consumer teams currently read carrier_code (say, 5 of them; the other 7 never touch it).
• CI enforces the deprecation: any of those 5 teams' consumer-driven contracts still asserting on carrier_code continue to pass throughout the 90-day window (the field is still populated, just deprecated), giving them time to migrate on their own schedule.
• At day 90, the registry's tooling flags any of the 5 teams whose contract has not been updated to use carrier instead. Two teams have migrated; three have not. Rather than removing the field on schedule and breaking three teams' consumers, the extension process from the deprecation policy applies: those three teams get a documented, time-bounded extension, tracked on the migration dashboard, not a silent indefinite delay.

Trade-offs and pitfalls

• Common wrong turn: writing a deprecation policy document that nothing in CI actually enforces. A policy nobody's tooling checks is a policy nobody follows once there's a deadline of their own to hit.
• Common wrong turn: treating consumer-driven contracts as equivalent to the schema registry's compatibility check. They answer different questions: the registry asks "is this schema technically compatible with a generic reader," contracts ask "does this specific consumer's actual usage still work." Both are needed; neither substitutes for the other.
• Common wrong turn: bolting PII governance on as a separate, manual review outside the normal schema change process, which is exactly when it gets skipped under deadline pressure. It belongs in the same automated gate as compatibility.
• Senior signal: distinguishing the governance-and-process framing (who owns what, what the deprecation window is, how a breaking change gets communicated) from a purely technical schema-evolution strategy; both matter, but this question is asking specifically for the former.

Direct answer

There are two standard mechanisms for this, and the choice depends on whether the work already has a natural partition structure. If it does, let a partitioned topic's consumer-group rebalance protocol hand out ownership 1:1, no external coordinator needed. If it does not, use an external coordination service (ZooKeeper or etcd) to hand out a leased, fenced lock per task, so a worker that stops renewing its lease loses ownership automatically, and a monotonically increasing fencing token stops a worker that only appears dead (a long garbage-collection pause, not a real crash) from corrupting the result after ownership has already moved on.

Structured elaboration

Reading "exactly N out of M workers." The dominant real case is N = 1: exactly one of M available workers should own a given task at a time, with the other M - 1 idle or working other tasks. The same mechanism generalizes to N > 1 by treating a task as N independent lease slots (task/slot-0 ... task/slot-N-1) instead of one, each slot claimed and fenced exactly like the N = 1 case below.

Kafka partitioning approach. Model the tasks as a fixed number of partitions on a topic (or hash a task id onto one of a fixed set of partitions), and let Kafka's own consumer-group protocol assign partitions to consumers, exactly one consumer owns a given partition at a time, enforced by session heartbeats to the group coordinator. Extra workers beyond the partition count sit idle as hot standby. This gives "at most one active owner per partition" for free, with no separate lock service to run. Failure handling is a rebalance: a missed heartbeat triggers reassignment of that consumer's partitions to a live one; because offset commits and processing are not perfectly atomic, the new owner may reprocess the last uncommitted batch, so tasks still need to be safe to reprocess (idempotent), the transport guarantee alone is not enough.

ZooKeeper or etcd lock approach. For tasks with no natural partition structure, each task gets its own lock. ZooKeeper's standard recipe: a worker creates a sequential ephemeral node under the task's lock path; the worker holding the lowest sequence number holds the lock (this doubles as the leader-election primitive if a single coordinator process needs to be elected too). Etcd's equivalent: a worker acquires a lease with a time-to-live (TTL) and performs a compare-and-swap on a key tied to that lease. In both, if the worker's session stops renewing (crash, or the process is killed), the ephemeral node or lease expires and another worker can acquire it.

The subtlety failure handling has to cover. A worker whose process is still alive but paused (a long garbage-collection pause, a network partition that cuts it off from the coordinator but not from the downstream system it writes to) still believes it holds the lock after the coordinator has already reassigned it to someone else. This is the split-brain case: two workers, each convinced it alone owns the task. The fix is a fencing token, a number that increases every time the lock changes hands, which the downstream resource being written to checks and rejects any write carrying a token lower than the highest one it has already accepted. This makes "no task processed more than once" hold even when a worker never technically crashed, it just woke up too late.

Leader election specifically. It is only needed if a single coordinator process is the one deciding which task goes to which worker (rather than workers self-organizing via the lock primitive directly). ZooKeeper and etcd's leader-election recipes elect that coordinator the same way: an ephemeral node or leased key, re-elected automatically on failure, so the assignment logic itself does not become a single point of failure.

sequenceDiagram
  participant WA as Worker A
  participant Coord as Coordinator etcd/ZooKeeper
  participant WB as Worker B
  participant Res as Downstream resource
  WA->>Coord: acquire lease(task)
  Coord-->>WA: token=1
  WA->>Coord: renew(token=1)
  Note over WA: Worker A stalls (GC pause or crash)
  WB->>Coord: acquire lease(task) after TTL expiry
  Coord-->>WB: token=2
  WB->>Res: write(result, token=2)
  Res-->>WB: accepted
  WA->>Res: write(result, token=1)
  Res-->>WA: rejected (stale token)

Worked example

A deterministic simulation of the diagram above, using a manual step counter instead of real time so the sequence is fully reproducible.

class Coordinator:
    def __init__(self, lease_ttl_steps: int):
        self.lease_ttl_steps = lease_ttl_steps
        self.next_token = 0
        self.leases = {}  # task_id -> {owner, token, last_renew_step}

    def try_acquire(self, task_id, worker_id, step):
        lease = self.leases.get(task_id)
        if lease is not None and (step - lease["last_renew_step"]) < self.lease_ttl_steps:
            return None  # still held by someone else, not expired
        self.next_token += 1
        self.leases[task_id] = {"owner": worker_id, "token": self.next_token, "last_renew_step": step}
        return self.next_token

    def renew(self, task_id, worker_id, token, step):
        lease = self.leases.get(task_id)
        if lease and lease["owner"] == worker_id and lease["token"] == token:
            lease["last_renew_step"] = step
            return True
        return False


class DownstreamResource:
    def __init__(self):
        self.max_token_seen = 0
        self.committed_writes = []

    def write(self, task_id, worker_id, token, value):
        if token < self.max_token_seen:
            return {"accepted": False, "reason": f"stale token {token} < {self.max_token_seen}"}
        self.max_token_seen = token
        self.committed_writes.append({"task_id": task_id, "worker_id": worker_id, "token": token, "value": value})
        return {"accepted": True}

Run against the scenario in the diagram: worker A acquires the lease at step 0, renews at steps 1 and 2, then stalls (never renews again). Worker B acquires the same task at step 5, after the 3-step lease TTL has elapsed since worker A's last renewal.

step 0: worker-A acquires task, token=1
step 1: worker-A renew ok=True
step 2: worker-A renew ok=True
step 5: worker-B acquires expired task, token=2
step 6: worker-B write -> {'accepted': True}
step 7: worker-A (stale) write -> {'accepted': False, 'reason': 'stale token 1 < 2'}
committed writes: [{'task_id': 'batch-task-42', 'worker_id': 'worker-B', 'token': 2, 'value': 'result-from-B'}]
invariant holds: exactly one committed write, from the worker holding the live lease

Worker A's late write at step 7 (it never crashed, it just paused past the lease TTL) is rejected because its token (1) is lower than the highest token the resource has already accepted (2). Exactly one write commits.

Trade-offs and pitfalls

Complexity and edge cases. Lock acquire and renew are O(1) coordinator operations per task per worker. The main edge case is lease TTL sizing: too short relative to realistic garbage-collection pause or network-blip durations causes thrashing (a live worker keeps losing and reacquiring its own lease); too long delays failover after a genuine crash, letting a task sit unclaimed for that whole window. Clock skew between workers and the coordinator also matters: a lease TTL is only meaningful relative to the coordinator's own clock, workers renewing based on their own clock without accounting for skew can renew "late" from the coordinator's point of view.

• The partition-based approach couples task granularity to a fixed partition count and rebalances at whole-partition granularity, coarser than per-task locks.
• The lock-based approach gives fine, per-task granularity but adds an external coordination service as a new dependency with its own availability story.
• Relying on "the worker believes it is still alive" without fencing is the pitfall the worked example exists to demonstrate: it looks correct until a real garbage-collection pause or network blip happens in production.
• Fencing only works if the downstream resource itself enforces the token check; if the resource cannot support that (a legacy system with no such field), the fencing discipline has nowhere to be enforced and the split-brain risk returns.