Event-Driven Architecture and Asynchronous Messaging Questions

Direct answer

For an internal eventing platform at 50,000 events/sec with durable replay for analytics, at-least-once delivery for many consumers, and per-key ordering for some consumers, Kafka is the better fit: its partitioned, replayable log natively supports per-key ordering (all events for a given key land on the same partition) and long retention, which RabbitMQ does not provide as first-class capabilities. RabbitMQ remains the better choice when the workload is closer to task distribution with complex routing needs and no replay requirement, which this workload is not.

Structured elaboration

Evaluation criteria, applied to the four stated constraints:

• Sustained throughput at 50k events/sec: both platforms can reach this rate, but the design work to get there differs. Kafka scales throughput by increasing partition count (more parallelism), which is a capacity-planning exercise done up front. RabbitMQ scales by adding queues and consumers, and can also sustain very high throughput, but does not offer partition-based ordered parallelism as a native primitive.
• Durable replay for analytics: this is the decisive criterion. Kafka retains events for a configured window regardless of consumption and lets any consumer group replay from an arbitrary offset. RabbitMQ removes messages once acknowledged; replaying "already-processed" history is not a built-in capability and would require bolting on a separate durable store.
• At-least-once delivery for many consumers: both support at-least-once delivery natively. Kafka's consumer-group model lets many independent consumer groups each read the full stream at-least-once without competing with each other; RabbitMQ supports at-least-once via manual acknowledgment, but achieving "many independent full-stream readers" specifically (rather than competing consumers on one queue) requires fanning the exchange out to a separate queue per consumer group, which is more configuration to maintain as consumers are added.
• Ordered delivery per key for some consumers: Kafka provides this natively via partition-key hashing, exactly the "some consumers require ordered delivery per key" requirement. RabbitMQ can approximate per-key ordering with a single active consumer per queue and a routing key per entity, but that limits parallelism for that queue to one consumer, which is a real constraint at 50k events/sec.

Recommendation: Kafka, specifically because the combination of durable replay and per-key ordering are both native to its architecture, whereas RabbitMQ would need workarounds for both.

Mitigations for Kafka's weaknesses. Kafka's two well-known weak points relative to RabbitMQ are operational complexity (a broker cluster, plus metadata quorum management, to run and monitor) and the lack of complex, content-based routing that RabbitMQ's exchange types provide natively. Mitigate the operational-complexity weakness by using a managed offering (e.g., Amazon MSK or Confluent Cloud) rather than self-hosting the cluster. Mitigate the routing-flexibility weakness by keeping routing logic in the consuming application (a consumer subscribes to specific topics and filters or re-publishes as needed) rather than expecting the broker to route by message content, which is a reasonable trade given this workload's requirements are about ordering and replay, not complex routing.

Worked example

Sizing partitions for 50,000 events/sec with per-key ordering: assume, as a planning input for this example (not a general Kafka benchmark), that a single partition on the chosen broker instance size can sustainably sustain roughly 5,000 events/sec for this message payload size. The minimum partition count is:

$N_{\text{partitions}} \ge \left\lceil \frac{50{,}000 \text{ events/s}}{5{,}000 \text{ events/s per partition}} \right\rceil = 10$

Ten partitions is the bare minimum to sustain the target throughput with zero headroom; in practice you would provision with headroom for growth and for uneven key distribution (a small number of very active keys, or "hot keys," can overload their assigned partition even when the average across all partitions is fine), so a reasonable starting point is roughly double the bare minimum, i.e. 20-24 partitions, revisited once real key-distribution data from production traffic is available. This calculation is illustrative: the 5,000 events/sec per-partition figure is a stated planning assumption for this example, to be replaced with a measured number from a load test against your actual message size and hardware before committing to a partition count in production.

Trade-offs and pitfalls

The main pitfall is under-provisioning partitions based only on average throughput and ignoring hot-key skew; a partition count that comfortably covers the 50k events/sec average can still fall over if one key (e.g., one very active customer or entity) accounts for a disproportionate share of traffic, since that key's events are all pinned to a single partition by design. A second pitfall is choosing partition count too high without a plan to ever reduce it: Kafka does not support shrinking partition count on an existing topic without breaking the per-key-to-partition mapping (and therefore ordering) for every key, so partition sizing decisions are effectively one-directional and should be made deliberately, not just rounded up generously. A third pitfall specific to this scenario is assuming "at-least-once for many consumers" and "ordered per key for some consumers" are the same requirement: ordering is a per-partition property that some consumers need to respect (by not parallelizing within a key), while at-least-once is a delivery guarantee every consumer needs to handle via idempotency regardless of whether it cares about ordering.

Direct answer

Default every tenant to a shared cluster with per-tenant topic/queue namespacing, authenticated identity, enforced throughput quotas, and fair-share scheduling across tenants, then provide an escape hatch to move an individual noisy or high-value tenant onto a dedicated cluster without changing the platform's control plane, authentication, or billing model. At 10,000 tenants, shared multi-tenancy has to be the default for cost and operability; dedicated clusters are the exception path for tenants whose throughput, isolation, or compliance needs the shared tier can't satisfy.

Structured elaboration

Tenancy models: shared versus dedicated. Shared tenancy (many tenants' topics/queues coexist on the same broker cluster, isolated logically by namespace and access control) is far cheaper to operate at 10,000-tenant scale, since running 10,000 individual clusters is operationally and financially unworkable. Dedicated tenancy (a tenant gets its own cluster or a strongly isolated cell) gives the strongest isolation and predictable performance, at much higher per-tenant cost, and is reserved for tenants that need hard isolation guarantees (regulatory requirements, or a single tenant whose volume would otherwise dominate a shared cluster). The platform should support both under one control plane, so a tenant migration between tiers doesn't mean rebuilding integration code, only a data-plane move.

Isolation mechanisms. Namespace every tenant's topics/queues with a tenant-scoped prefix (for example tenant-42.orders) so naming collisions across tenants are structurally impossible. Enforce that isolation with access control: a tenant's credentials can only produce/consume within its own namespace, denied by default for every other tenant's namespace, so isolation is a hard access-control boundary, not just a naming convention that a misconfigured client could bypass. Resource isolation (so one tenant's burst can't starve another's throughput) is handled by the quota and fair-share mechanisms below, and encryption/network isolation per tenant (so one tenant's traffic and at-rest data are not readable by another even at the infrastructure layer) closes the loop with the security-isolation angle: encrypt each tenant's data with tenant-scoped keys and deny cross-tenant network paths by default, the same discipline a dedicated single-tenant secure event bus would provide, just implemented as policy on a shared cluster rather than via physical separation.

Authentication and authorization. Each tenant (and each service acting on that tenant's behalf) gets its own credential, never a shared platform-wide credential, issued and rotated through the platform's identity system. Authorization is namespace-scoped: a credential's access-control list only grants actions within that tenant's namespace prefix.

Quota enforcement. Each tenant gets a configured throughput ceiling (for example, a maximum produce/consume rate), enforced at the gateway/broker layer so one tenant cannot consume more than its allotted share of shared cluster capacity even during a traffic spike. Combine hard per-tenant quotas (a firm ceiling, protecting other tenants) with fair-share scheduling below the ceiling (so unused capacity from quiet tenants is available to active tenants in real time, rather than sitting idle), which is what "fair-share consumption across tenants" in the requirement is asking for: not just "nobody exceeds their cap," but "capacity that isn't being used gets redistributed fairly, not wasted."

Migrating a tenant between clusters. Because every tenant already interacts with the platform through a stable, tenant-scoped namespace and a stable client-facing endpoint (not a direct cluster address), moving a tenant from the shared cluster to a dedicated one (or between shared clusters for rebalancing) is a data-plane migration behind that stable interface: replicate the tenant's topics/queues to the destination cluster, cut over routing at the gateway once replication is caught up, and retire the source-side namespace. The tenant's producers/consumers should not need to change their connection configuration for this to work, which is what makes the "ability to migrate a tenant between clusters" requirement operationally realistic at 10,000-tenant scale rather than a disruptive, coordinated cutover with every tenant.

Billing metrics. Capture, per tenant: message produce/consume volume, bytes in/out, peak and average throughput against quota, and storage retained. These feed both the quota-enforcement decision (has this tenant exceeded its allotment) and the actual billing/usage report a customer sees, so the same metric pipeline serves both operational and commercial purposes rather than maintaining two separate counting systems that can drift apart.

Worked example

Tenant A is provisioned with a quota of 500 messages/sec on the shared cluster. During a normal day it uses roughly 80 messages/sec, well under quota, so the fair-share scheduler makes that unused headroom available to other tenants sharing the same broker resources. During a flash-sale event, tenant A's traffic climbs toward its 500/sec ceiling; the quota enforcer throttles it at 500/sec rather than letting it degrade sibling tenants' throughput, and its billing metrics record both the elevated volume and the fact that it briefly ran at its quota ceiling, which is itself a useful signal for a account team to have a capacity conversation with that tenant. If tenant A's sustained volume subsequently outgrows what's reasonable to keep absorbing on the shared cluster, the platform migrates it to a dedicated cluster via the namespace-stable migration path above, with no change to tenant A's client configuration.

Trade-offs and pitfalls

Shared clusters without hard per-tenant quotas are the most common real-world failure: a single noisy or misbehaving tenant degrades every other tenant sharing that cluster, exactly the "noisy neighbor" problem multi-tenancy exists to prevent. Conversely, quotas set too conservatively (to be "safe") waste capacity and frustrate tenants during legitimate bursts, which is why fair-share scheduling below the hard ceiling matters, not just the ceiling itself. Namespace prefixing without enforced access control is a naming convention, not isolation, since nothing stops a misconfigured or compromised client from addressing another tenant's namespace directly if the access-control layer doesn't independently deny it. Finally, a migration mechanism that requires every tenant's client to change its connection details defeats the purpose of a stable multi-tenant control plane; the routing/gateway layer, not the tenant's integration, should absorb the complexity of a cluster migration.

flowchart TD
    T1[Tenant 1] --> GW[AuthN and AuthZ gateway]
    T2[Tenant 2] --> GW
    T3[Tenant N] --> GW
    GW --> QE[Quota enforcer]
    QE --> NS1[Namespace tenant-1]
    QE --> NS2[Namespace tenant-2]
    QE --> NSN[Namespace tenant-n]
    NS1 --> SC[(Shared cluster)]
    NS2 --> SC
    NSN -->|noisy tenant migrated| DC[(Dedicated cluster)]
    SC --> BM[Billing metrics collector]
    DC --> BM

Direct answer

For a 60-second persist service-level agreement (SLA), the metric that actually matters is the projected time-to-persist for the oldest unprocessed event. Everything else, producer rate, consumer processing rate, dead-letter-queue rate, error rate, raw throughput and latency, exists to explain why that projection is moving, and paging should be tied to SLA risk crossing a threshold, not to any single metric wobbling on its own.

Structured elaboration

What to track.

• Producer rate: events published per second, the input side of the balance.
• Consumer processing rate (throughput): events actually persisted per second, the output side.
• Consumer lag: how far behind the consumer is, in event count or in the age of the oldest unprocessed event, the single number that most directly answers whether the pipeline is on track to breach the 60-second SLA.
• Dead-letter queue (DLQ) rate and depth: events that failed processing and were routed out of the normal flow. Growth here means a subset of events are silently not going to be persisted on time, or at all, until someone intervenes.
• Error rate: the fraction of processing attempts failing, whether or not they eventually land in the DLQ, often a leading indicator that rises before lag does.
• End-to-end latency: measured from event creation to successful persist, the metric the SLA is actually stated in terms of, and it should be tracked directly wherever the event carries a creation timestamp, not just inferred from lag and rate.

What's worth paging for. Not every metric wobble. Page on SLA risk specifically: when the projected additional latency for the oldest currently unprocessed event crosses a threshold meaningfully inside the 60-second budget, enough time left for someone to actually act, not an alert that fires the same moment the SLA is already breached, or when DLQ depth grows continuously rather than staying flat. A flat, near-zero DLQ is healthy; a climbing one means a systemic issue is actively producing failures faster than anyone is triaging them. A momentary lag blip that self-resolves within the next reporting interval, or a brief error-rate spike that recovers, is exactly the kind of thing that belongs on a dashboard, not something that should wake someone up.

Runbook: what to check first. With lag rising or an SLA-risk page firing, the ordered checks are: first, is the producer rate elevated, an unusual traffic spike that may just need consumer autoscaling to catch up, or is the consumer processing rate depressed, a downstream dependency slowdown, a bad deploy, resource exhaustion on the workers. This distinguishes "we need more capacity" from "something is broken." Second, check the DLQ: is it growing, and if so, sample a few entries to see whether they share a common cause, one bad event shape from one producer versus a systemic downstream failure. Third, check recent deploys and recent consumer restarts or scaling events; a consumer-group churn event, workers restarting or the assignment of work across them changing, is a common, usually self-resolving cause of a short lag spike, worth ruling out before assuming a deeper problem. Fourth, check the downstream dependency the persist step actually writes to, since a slow or unavailable datastore on the write side shows up first as processing-rate degradation, not as anything on the ingestion side.

Worked example

Suppose consumer lag is currently 12,000 unprocessed events and the consumer is processing at a steady 500 events/sec:

The oldest unprocessed event would take about 24 seconds to be reached and persisted at the current rate, comfortably inside the 60-second SLA. If lag instead grows to 27,000 events at the same 500 events/sec processing rate, the projected additional latency is 27000/500 = 54 seconds, past a page threshold set at 75% of the SLA budget:

and closing in on the 60-second breach itself. That is the SLA-risk crossing that should page, not the raw lag number in isolation, the same 12,000-event lag would be a non-event if the processing rate were higher, or would already be an active breach if the rate had dropped low enough.

Trade-offs and pitfalls

• Paging on raw lag count without dividing by current processing rate produces false urgency during high-throughput periods and false calm during low-throughput periods. The SLA is stated in time; the alert should be too.
• Alerting on every DLQ message individually instead of on DLQ growth rate and depth creates alert fatigue fast. A slow, steady trickle into the DLQ from known bad producer data is a triage-queue item, not a page; a sudden acceleration is.
• Treating a transient consumer-group reassignment as an incident before checking whether lag self-resolved wastes on-call attention on a self-healing event; conversely, dismissing every lag spike as probably just a reassignment, without checking, risks missing a real regression. The runbook should say check it, not assume it.
• Tracking throughput and latency as separate, unrelated numbers instead of connecting them to the SLA budget makes the dashboard descriptive but not actionable. The projection, lag divided by rate, compared to the SLA, is what turns raw metrics into a decision.

Direct answer

A Dead Letter Queue (DLQ) is a separate queue or topic a consumer (or the broker on its behalf) diverts a message to once it has failed processing beyond an agreed limit, so one message that will never succeed, a poison message, cannot block or endlessly retry-loop every other message queued behind it. A DLQ architecture is not just that holding area, it is the metrics and alerting that make its arrivals visible, the automated classification that quarantines messages sensibly, and the manual review and replay workflow that actually resolves them.

Structured elaboration

Why a message ends up in the DLQ. Broadly three reasons: a permanent business-logic or validation failure (the payload is malformed or violates a business rule, retrying changes nothing), a transient failure that simply exhausted its retry budget (the downstream dependency was down long enough that every retry also failed), or a poison message that crashes or hangs the consumer process itself (for example, a payload that triggers a parsing exception on every attempt). Distinguishing these at arrival is what makes the rest of the workflow tractable, they need different handling.

Metrics and alerts. Track DLQ depth (how many messages are currently quarantined), the age of the oldest DLQ message (a slow leak looks fine on depth alone if replay keeps pace, age catches it), arrival rate into the DLQ, and a breakdown by failure reason. Alert on any new arrivals to a topic that is normally near-zero, and on depth or age crossing a threshold tied to how quickly the team can realistically review them (a service-level objective, or SLO, for DLQ review turnaround).

Automated quarantining. Classify each message at the moment it arrives in the DLQ, not later: capture the failure reason, the error type or exception, the attempt count, timestamps, and the originating topic or partition as metadata alongside the payload. Tag messages that look transient (timeouts, connection resets, a known downstream outage window) as candidates for automatic replay after a cool-down; tag messages that look permanent (schema validation failures, business-rule violations) for manual review instead of blind retry.

Manual review and replay workflow. A reviewer inspects the payload, the failure reason, and the attempt history, then decides to replay (re-inject to the main queue), archive (keep for record but do not reprocess), or fix-forward (patch the producer or consumer, then replay). Replayed messages must go back through the exact same idempotency and deduplication path any normal delivery would use, not a special-cased bypass, otherwise a message that partially succeeded before its original failure can be double-processed on replay.

Retention. DLQ messages typically need a longer retention window than the main topic, since a human needs time to notice and act, but not indefinite retention; tie it to compliance and data-retention requirements rather than leaving it as an afterthought default.

Security. A DLQ commonly holds the exact same sensitive payloads as the main data path (a payment event that failed validation is still a payment event), so it needs the same encryption-at-rest and access control as the primary pipeline. Treating the DLQ as a lower-security scratch space is a common and risky oversight, precisely the hardest-to-process messages are the ones most likely to sit there.

flowchart LR
  P[Producer] --> Q[Main queue or topic]
  Q --> C[Consumer]
  C -->|success| Ack[Ack / commit offset]
  C -->|failure, retries exhausted| DLQ[Dead-letter queue]
  DLQ --> M[Metrics: DLQ depth, age]
  M --> Alert[Alert on-call]
  DLQ --> Review[Manual review / classification]
  Review -->|fixable| Replay[Replay to main queue]
  Review -->|not fixable| Archive[Archive / discard]

Worked example

A payment consumer starts failing on three messages because an upstream producer shipped a breaking field rename. Each message exhausts its retry budget and is diverted to the DLQ. The unusual arrival rate (normally zero, now three in a minute) trips an alert. Automated classification tags all three as schema-related, since the failure is a deserialization error rather than a downstream timeout, routing them to manual review instead of an automatic retry loop that would just fail the same way again. On-call fixes the producer (or adds a compatibility shim on the consumer side), confirms the fix against one message manually, then replays the remaining three, and access logs record who replayed them, satisfying the audit angle of the security requirement.

Trade-offs and pitfalls

• A DLQ with no automated classification or alerting quietly becomes a graveyard nobody looks at, the single most common real-world failure of this pattern, not a hypothetical.
• Replaying a message without re-running it through idempotency checks is the most common way a "fixed" DLQ incident turns into a second incident, double-processing.
• Indefinite DLQ retention creates both a storage cost problem and, for sensitive payloads, a compliance liability; set retention deliberately.
• Treating every DLQ arrival identically (always retry, or always require manual review) wastes either engineering attention or retry budget; the transient-versus-permanent classification is what makes the workflow scale.

Direct answer

There are two standard mechanisms for this, and the choice depends on whether the work already has a natural partition structure. If it does, let a partitioned topic's consumer-group rebalance protocol hand out ownership 1:1, no external coordinator needed. If it does not, use an external coordination service (ZooKeeper or etcd) to hand out a leased, fenced lock per task, so a worker that stops renewing its lease loses ownership automatically, and a monotonically increasing fencing token stops a worker that only appears dead (a long garbage-collection pause, not a real crash) from corrupting the result after ownership has already moved on.

Structured elaboration

Reading "exactly N out of M workers." The dominant real case is N = 1: exactly one of M available workers should own a given task at a time, with the other M - 1 idle or working other tasks. The same mechanism generalizes to N > 1 by treating a task as N independent lease slots (task/slot-0 ... task/slot-N-1) instead of one, each slot claimed and fenced exactly like the N = 1 case below.

Kafka partitioning approach. Model the tasks as a fixed number of partitions on a topic (or hash a task id onto one of a fixed set of partitions), and let Kafka's own consumer-group protocol assign partitions to consumers, exactly one consumer owns a given partition at a time, enforced by session heartbeats to the group coordinator. Extra workers beyond the partition count sit idle as hot standby. This gives "at most one active owner per partition" for free, with no separate lock service to run. Failure handling is a rebalance: a missed heartbeat triggers reassignment of that consumer's partitions to a live one; because offset commits and processing are not perfectly atomic, the new owner may reprocess the last uncommitted batch, so tasks still need to be safe to reprocess (idempotent), the transport guarantee alone is not enough.

ZooKeeper or etcd lock approach. For tasks with no natural partition structure, each task gets its own lock. ZooKeeper's standard recipe: a worker creates a sequential ephemeral node under the task's lock path; the worker holding the lowest sequence number holds the lock (this doubles as the leader-election primitive if a single coordinator process needs to be elected too). Etcd's equivalent: a worker acquires a lease with a time-to-live (TTL) and performs a compare-and-swap on a key tied to that lease. In both, if the worker's session stops renewing (crash, or the process is killed), the ephemeral node or lease expires and another worker can acquire it.

The subtlety failure handling has to cover. A worker whose process is still alive but paused (a long garbage-collection pause, a network partition that cuts it off from the coordinator but not from the downstream system it writes to) still believes it holds the lock after the coordinator has already reassigned it to someone else. This is the split-brain case: two workers, each convinced it alone owns the task. The fix is a fencing token, a number that increases every time the lock changes hands, which the downstream resource being written to checks and rejects any write carrying a token lower than the highest one it has already accepted. This makes "no task processed more than once" hold even when a worker never technically crashed, it just woke up too late.

Leader election specifically. It is only needed if a single coordinator process is the one deciding which task goes to which worker (rather than workers self-organizing via the lock primitive directly). ZooKeeper and etcd's leader-election recipes elect that coordinator the same way: an ephemeral node or leased key, re-elected automatically on failure, so the assignment logic itself does not become a single point of failure.

sequenceDiagram
  participant WA as Worker A
  participant Coord as Coordinator etcd/ZooKeeper
  participant WB as Worker B
  participant Res as Downstream resource
  WA->>Coord: acquire lease(task)
  Coord-->>WA: token=1
  WA->>Coord: renew(token=1)
  Note over WA: Worker A stalls (GC pause or crash)
  WB->>Coord: acquire lease(task) after TTL expiry
  Coord-->>WB: token=2
  WB->>Res: write(result, token=2)
  Res-->>WB: accepted
  WA->>Res: write(result, token=1)
  Res-->>WA: rejected (stale token)

Worked example

A deterministic simulation of the diagram above, using a manual step counter instead of real time so the sequence is fully reproducible.

class Coordinator:
    def __init__(self, lease_ttl_steps: int):
        self.lease_ttl_steps = lease_ttl_steps
        self.next_token = 0
        self.leases = {}  # task_id -> {owner, token, last_renew_step}

    def try_acquire(self, task_id, worker_id, step):
        lease = self.leases.get(task_id)
        if lease is not None and (step - lease["last_renew_step"]) < self.lease_ttl_steps:
            return None  # still held by someone else, not expired
        self.next_token += 1
        self.leases[task_id] = {"owner": worker_id, "token": self.next_token, "last_renew_step": step}
        return self.next_token

    def renew(self, task_id, worker_id, token, step):
        lease = self.leases.get(task_id)
        if lease and lease["owner"] == worker_id and lease["token"] == token:
            lease["last_renew_step"] = step
            return True
        return False


class DownstreamResource:
    def __init__(self):
        self.max_token_seen = 0
        self.committed_writes = []

    def write(self, task_id, worker_id, token, value):
        if token < self.max_token_seen:
            return {"accepted": False, "reason": f"stale token {token} < {self.max_token_seen}"}
        self.max_token_seen = token
        self.committed_writes.append({"task_id": task_id, "worker_id": worker_id, "token": token, "value": value})
        return {"accepted": True}

Run against the scenario in the diagram: worker A acquires the lease at step 0, renews at steps 1 and 2, then stalls (never renews again). Worker B acquires the same task at step 5, after the 3-step lease TTL has elapsed since worker A's last renewal.

step 0: worker-A acquires task, token=1
step 1: worker-A renew ok=True
step 2: worker-A renew ok=True
step 5: worker-B acquires expired task, token=2
step 6: worker-B write -> {'accepted': True}
step 7: worker-A (stale) write -> {'accepted': False, 'reason': 'stale token 1 < 2'}
committed writes: [{'task_id': 'batch-task-42', 'worker_id': 'worker-B', 'token': 2, 'value': 'result-from-B'}]
invariant holds: exactly one committed write, from the worker holding the live lease

Worker A's late write at step 7 (it never crashed, it just paused past the lease TTL) is rejected because its token (1) is lower than the highest token the resource has already accepted (2). Exactly one write commits.

Trade-offs and pitfalls

Complexity and edge cases. Lock acquire and renew are O(1) coordinator operations per task per worker. The main edge case is lease TTL sizing: too short relative to realistic garbage-collection pause or network-blip durations causes thrashing (a live worker keeps losing and reacquiring its own lease); too long delays failover after a genuine crash, letting a task sit unclaimed for that whole window. Clock skew between workers and the coordinator also matters: a lease TTL is only meaningful relative to the coordinator's own clock, workers renewing based on their own clock without accounting for skew can renew "late" from the coordinator's point of view.

• The partition-based approach couples task granularity to a fixed partition count and rebalances at whole-partition granularity, coarser than per-task locks.
• The lock-based approach gives fine, per-task granularity but adds an external coordination service as a new dependency with its own availability story.
• Relying on "the worker believes it is still alive" without fencing is the pitfall the worked example exists to demonstrate: it looks correct until a real garbage-collection pause or network blip happens in production.
• Fencing only works if the downstream resource itself enforces the token check; if the resource cannot support that (a legacy system with no such field), the fencing discipline has nowhere to be enforced and the split-brain risk returns.