Automated Incident Response and Cross-Phase Incident Scenarios Questions

Direct answer

Safe retry semantics have to be designed per connector type, because each one offers a different native primitive for idempotency or atomicity: Kafka producers get exactly-once via the idempotent producer plus transactions; Kafka consumers get it via read_committed isolation reading only committed transactional output; database writes get it via native upserts or local transactions; REST calls to a third-party get it via an idempotency-key header when the API supports one, or an outbox-plus-proxy pattern when it does not; and object storage like S3 gets it via content-addressed keys or an atomic manifest commit. There is no single mechanism that covers all four; the architecture's job is to pick the right one per connector and make sure they compose correctly end to end.

Structured elaboration

Kafka producers. Enable the idempotent producer (enable.idempotence=true), which assigns each producer a unique ID and each message a sequence number, letting the broker deduplicate retried sends from the SAME producer session automatically. For cross-partition or cross-topic atomicity (writing to multiple topics as one unit), wrap the writes in a Kafka transaction (initTransactions, beginTransaction, commitTransaction), which the broker either fully commits or fully aborts.

Kafka consumers. Reading a transactional producer's output requires setting the consumer's isolation level to read_committed, so aborted or in-flight transactions are invisible; a consumer left at the default read_uncommitted would see uncommitted, possibly-aborted data, silently breaking the exactly-once guarantee the producer side worked to provide. Consumer offset commits should be tied to downstream processing completion (commit the offset only after the corresponding output is durably written), not committed eagerly on read.

Database writes. Use the database's native atomic primitives: an INSERT ... ON CONFLICT DO UPDATE (Postgres) or MERGE keyed by a business key plus version, for single-row idempotency; a local transaction for multi-row atomicity within that one database. If the write must be atomic with the Kafka consumer offset commit (a common payments pattern), the outbox pattern (write the outbox row in the SAME local database transaction as the business write) decouples that atomicity from needing Kafka and the database to share a distributed transaction, which they generally cannot.

REST calls. If the third-party API supports an idempotency-key parameter (Stripe-style), generate that key deterministically from the logical operation (not fresh per retry) and let the API's own deduplication handle it. If it does not, apply an idempotency-proxy pattern: put a proxy in front of the API (the strongest option, if worth building), or accept a compensating-transaction fallback for genuinely one-way, non-idempotent operations.

Object storage (S3). Native S3 operations are individually retry-safe (a PutObject retried with the same key and content just re-uploads the same bytes, no duplication), but a MULTI-OBJECT logical write (many files representing one dataset version) needs a manifest-based atomic commit: stage, then atomically swap a small manifest pointer, so a partial or duplicated multi-object write is never visible as "done."

Worked example

A pipeline reads Kafka, writes to a Postgres database (for a materialized view), calls a third-party fraud-check REST API, and archives raw events to S3, all per logical event, needing the whole chain to behave correctly under retries. Concrete wiring, in order:

1. Kafka consumer reads with read_committed, does not commit its offset yet.
2. Postgres write: INSERT ... ON CONFLICT (event_id) DO NOTHING (idempotent by event_id).
3. Fraud-check REST call: the API supports an idempotency-key header; the pipeline passes event_id as that key deterministically, so a retried call after a timeout is recognized and returns the original result.
4. S3 archive: PutObject keyed by event_id (content-addressed by logical identity), so a retried upload overwrites the identical object harmlessly.
5. Only after all three writes are confirmed does the Kafka consumer commit its offset.

If step 3 (the REST call) times out ambiguously and the whole event is retried from step 2: step 2's ON CONFLICT DO NOTHING is a safe no-op (already inserted), step 3's idempotency key correctly returns the cached fraud-check result rather than re-running it, and step 4's re-upload is harmless. The offset is committed only once all four steps are confirmed, so a crash before that point simply replays this exact same, now-fully-idempotent sequence, and a crash after commit never revisits this event again (correct, since it was already fully processed).

Trade-offs and pitfalls

• Common mistake: committing the Kafka offset before all downstream writes are confirmed. This is the single most common way to silently lose the "at-least-once" half of the guarantee: a crash between offset-commit and the last downstream write means that event is never retried, since the consumer believes it already handled it.
• Common mistake: assuming Kafka's idempotent producer alone gives end-to-end exactly-once. It only protects the Kafka WRITE from producer-side retries; it says nothing about the downstream database, REST call, or S3 write each independently needing their own idempotency discipline, exactly why this answer treats each connector type separately rather than claiming one mechanism covers the whole chain.
• Ordering the four connector writes matters for correctness, not just tidiness. Placing the offset commit last (as in the worked example) is deliberate: it is the one step in the chain that, if it happens too early, breaks the whole at-least-once guarantee; every other step being idempotent means their relative order among themselves is more flexible.
• Per-connector idempotency does not automatically give cross-connector atomicity. If the fraud-check call succeeds but the process crashes before the S3 archive, on retry the fraud-check idempotency key correctly avoids re-running (good), but there is a window where downstream state is partially applied; this is the same partial-failure-across-heterogeneous-sinks problem any multi-sink write faces, and the fix is the same: make every step both idempotent AND independently retriable, not build a fragile distributed transaction across all four.

Direct answer

A circuit breaker for a flaky sink used by MANY concurrent ingestion workers needs its state SHARED across all of them (not one independent breaker per worker, which would let each worker rediscover the same failure independently and keep hammering a down sink collectively even while individually "protected"), a standard three-state machine (closed/open/half-open), thresholds calibrated to the AGGREGATE failure signal across all workers rather than any single worker's own small sample, and circuit-status metrics exposed so operators can see the breaker's state directly rather than inferring it from downstream symptoms.

Structured elaboration

Why shared, not per-worker, circuit state. With many concurrent workers, a per-worker circuit breaker means each worker independently accumulates its OWN failure count before opening; with, say, 50 workers each needing 20 failed calls to trip their own local breaker, the sink absorbs up to 50 x 20 = 1,000 failed calls collectively before every worker has individually protected itself, a far larger and slower-to-react blast radius than a SHARED breaker state (a distributed counter, or a coordinating service) that opens once the AGGREGATE failure signal crosses the threshold, protecting all 50 workers simultaneously from the moment that shared threshold is crossed.

States. CLOSED: normal operation, all workers' calls pass through, the shared failure counter tracks recent outcomes. OPEN: once the shared threshold is crossed, ALL workers immediately start failing fast (no more calls to the sink from any worker) for the cooldown duration. HALF-OPEN: after cooldown, a SMALL, COORDINATED number of probe calls (not one probe per worker, which at 50 workers would send 50 simultaneous probes, itself a mini-thundering-herd) test recovery; typically implemented as one worker (or a small designated subset) being granted probe permission by the shared state, with the rest still failing fast until the probe's result is known.

Thresholds for opening/closing. Opening: a percentage or count of recent failures AGGREGATED across all workers (e.g., more than 50% of the last 200 total calls across all workers, not per-worker), since 5xx errors from a single flaky sink are a property of the SINK, not any individual worker, so the signal should be pooled. Closing (via half-open success): a small number of consecutive successful probe calls (e.g., 3-5) before fully reopening to all workers, avoiding a single lucky probe call declaring full recovery prematurely.

Reset policy. Cooldown duration before allowing probes; if probes fail, DOUBLE the cooldown for the next attempt (a backoff on the circuit's own reopening attempts, mirroring exponential backoff applied at the circuit level rather than the individual-call level), so a sink in a genuinely extended outage does not get probed at a fixed, wasteful interval indefinitely.

Integration with backoff retries. As with a single-caller circuit breaker, this shared one gates WHETHER to attempt a call at all; backoff-with-jitter governs individual retry timing for calls that DO proceed (circuit closed or a granted half-open probe). With many concurrent workers, jitter on the retry timing is specifically important to avoid synchronized retry waves across workers hitting the sink at the same moments.

Surfacing circuit status in metrics and alerts. Emit the shared circuit's current state (closed/open/half-open) as a first-class metric, plus the aggregate failure rate feeding the open/close decision and a count of calls currently being fast-failed (a direct measure of how much work is being deferred/dropped while open). Alert specifically on state TRANSITIONS (a page-worthy signal: "circuit just opened for sink X") separately from a slower-moving dashboard of the underlying failure rate, since the transition itself is the actionable, time-sensitive event.

Worked example

50 ingestion workers write to a sink returning 5xx errors intermittently. With a SHARED circuit breaker (a distributed counter backed by, say, Redis, incremented atomically by every worker's call outcome) and a threshold of "open once aggregate failures exceed 50% of the last 200 total calls": at 10,000 combined calls/sec across all 50 workers, that 200-call window fills in:

meaning the shared breaker reacts to a genuine sink-wide degradation within roughly 20 milliseconds of it beginning, versus a per-worker breaker (each worker seeing only its own $10{,}000/50 = 200$ calls/sec share) needing its OWN local 200-call window to fill, which happens at the same wall-clock rate per worker in this SPECIFIC symmetric example, but critically, each worker trips INDEPENDENTLY and at a DIFFERENT moment relative to when it happened to observe its own 100th failure, meaning some workers keep hammering the sink for a meaningfully longer tail after the shared approach would have already protected everyone uniformly, and the difference grows sharply once workers are NOT symmetric (a slow worker sending few calls per second takes proportionally much longer to accumulate its own local failure count, continuing to hit the sink long after a shared breaker would have already opened for the whole fleet).

Trade-offs and pitfalls

• Common mistake: implementing the circuit breaker as pure in-process state, which is exactly the per-worker anti-pattern above; a genuinely shared breaker requires either a coordinating external store (Redis, a dedicated service) or a broadcast mechanism, real infrastructure, not a drop-in library default in every framework.
• The shared state store itself becomes a new dependency and potential bottleneck. Every worker's call outcome updates shared state; this needs to be cheap and fast (an atomic increment, not a heavyweight transaction) or the circuit-breaker mechanism's own overhead becomes a meaningful tax on the very throughput it is meant to protect.
• Common mistake: uncoordinated half-open probing at scale. As noted above, 50 independent probe attempts the instant cooldown ends is itself a small thundering herd; coordinating probe permission (one or a few workers, not all) avoids re-triggering the exact overload the cooldown existed to prevent.
• Circuit-state metrics are only useful if alerted on the TRANSITION, not just visible on a dashboard. A circuit that silently opened and stayed open for an hour, visible only to someone who happened to check the dashboard, provides far less operational value than an explicit page the moment it opens.

Direct answer

An automated test harness for idempotency validates three distinct claims, each needing its own test shape: that a SINGLE replayed message produces no change to database state (basic idempotency), that the database's FINAL state after any number of replays matches a clean, single-pass run exactly (convergence correctness), and that CONCURRENT duplicate deliveries (not just sequential replays) do not introduce a race-condition-induced duplicate that sequential testing alone would miss (the same kind of concurrency proof any check-then-set race demonstration needs, generalized here into a reusable test harness rather than a one-off demonstration).

Structured elaboration

Generating duplicate messages. The harness needs a message generator producing three distinct duplicate shapes, since each exercises a different part of the idempotency mechanism: (1) exact sequential replay (the same message redelivered later, simulating a consumer restart replaying from an older offset), (2) concurrent duplicate delivery (the same message delivered to two consumer instances simultaneously, simulating a rebalance race or a dual-delivery bug), and (3) partial-batch replay (a batch of N messages where some subset was already applied and the REST of the batch is new, simulating a crash mid-batch).

Expected database state after replays. For each generated scenario, the harness computes the EXPECTED final state independently (a ground-truth computation directly from the distinct underlying messages, ignoring duplicates entirely) and asserts the actual post-replay database state matches EXACTLY, not approximately.

Verification queries. Beyond a final-state comparison, the harness should assert specific INVARIANTS that would catch a subtler bug even if the final aggregate happens to look right: a per-idempotency-key row/entry count of exactly 1 (proving the dedup KEY itself, not just the aggregate value, behaved correctly), and a write-count assertion on a wrapped/mocked sink (counting actual side-effect executions, not just checking the resulting state, since a coincidentally-correct final value can mask a bug that happened to cancel out).

Detecting latent duplicates introduced by concurrency. This is the harness's most valuable, most often-skipped component: run the SAME duplicate-delivery scenario under REAL concurrent execution (multiple threads/processes racing on the identical idempotency key) many times (since a race condition may only manifest probabilistically, not on every run) and assert the invariant holds across ALL runs, not just once. A test that only exercises SEQUENTIAL replay can pass cleanly while a genuine concurrency bug (a check-then-set race) remains completely undetected.

Worked example

A test suite for an idempotent order-processing pipeline includes: (1) a sequential-replay test asserting that processing order-123 twice, one after another, results in exactly one row in the orders table and exactly one call to the (mocked) payment-charge function; (2) a concurrent-duplicate test spawning 50 threads all processing order-456 simultaneously, asserting exactly one row and exactly one payment-charge call across all 50 (the same shape as a single-key concurrency proof, generalized here as a reusable harness pattern rather than a one-off answer); (3) a partial-batch-replay test where a batch of 20 orders is processed, 12 confirmed applied, then the SAME batch of 20 is replayed (simulating a crash-and-restart), asserting the final state shows exactly 20 distinct orders (not 32), with the 12 previously-applied ones correctly recognized via their idempotency key. Running the concurrent-duplicate test (2) 100 times in CI (not just once) is what actually catches a race-condition bug that might only manifest in, say, 3 out of 100 runs due to timing sensitivity, a single run passing would give false confidence.

Trade-offs and pitfalls

• Common mistake: testing idempotency ONLY via sequential replay. As a broken-store negative control demonstrates, a check-then-set race is completely invisible to sequential testing and requires genuine concurrent execution to surface; a harness that only replays messages one after another provides false confidence about production behavior under real concurrent load.
• Common mistake: asserting only the final aggregate value, not the underlying write count or key-level invariants. A bug that happens to produce the numerically correct final sum through a WRONG mechanism (e.g., one write incorrectly succeeded while another incorrectly failed, cancelling out) would pass a final-value-only assertion while masking a real defect; per-key row counts and actual side-effect call counts catch this class of false-positive-pass.
• Concurrency tests need to run MANY times, not once, in CI. A probabilistic race condition might only manifest in a small fraction of runs; a single passing CI run of a concurrency test is meaningfully weaker evidence than the same test run 100 times with zero failures.
• Ground-truth computation for the expected state must be INDEPENDENT of the pipeline's own logic, computed directly from the distinct underlying test messages (not by re-running a simplified version of the same pipeline code being tested), or a bug shared between the pipeline and its own "expected value" computation could pass undetected.

Direct answer

Treat this as two separate problems that need separate techniques: a statistical/structural fix to stop the duplicate firing in the first place, and a monitoring layer over the suppression itself so you can prove you are not accidentally hiding a real incident inside the noise you just silenced.

Structured elaboration

Statistical analysis to characterize the noise first. Before changing anything, break down the 1000 daily alerts by rule, by service, and by whether each historically corresponded to real action; this typically reveals a small number of rules or a small number of services responsible for a disproportionate share of the volume (a Pareto-shaped distribution is the common case), which tells you where to focus rather than tuning everything uniformly.

Suppression rules. For alerts that are genuine duplicates of an already-open, already-acknowledged incident (the same underlying condition firing repeatedly), suppress repeats within a defined window rather than re-paging for each occurrence; this is a deduplication pattern applied specifically to the "many alerts, one still-open cause" case.

Instrumentation changes. Where the root issue is that a single upstream event fans out into many downstream alerts (one dependency failing causes every service that calls it to also alert), correlate those into one incident using dependency-graph adjacency rather than suppressing them independently and separately, since blind suppression of each individually-firing alert risks losing the information that many services are affected, information a correlated single-incident view preserves.

Verifying suppressed alerts are not hiding real incidents. This is the step most alert-noise-reduction efforts skip, and it matters because a suppression rule that is slightly too aggressive fails silently: nobody notices a hidden real incident until it has already caused customer impact. Two concrete techniques: (1) track suppressed-alert volume as its own monitored metric with its own alerting, specifically watching for an anomalous SPIKE in the suppressed count, since a spike in what you are silencing is itself informative even though the individual alerts are not paging anyone; (2) periodically (not just once) sample a portion of suppressed alerts for manual review, checking whether any of them, in hindsight, corresponded to a real issue that should have been surfaced, which validates the suppression logic against ground truth on an ongoing basis rather than trusting it was tuned correctly once and left alone.

Worked example

Statistical breakdown of the 1000 daily alerts shows 60% come from just 4 rules, and of those 4, one (a flaky health check on a specific service) accounts for 35% of total volume alone with a near-zero historical action rate. Fixing that one rule (adding a sustained-window requirement) alone cuts total volume by roughly a third. For the remaining volume, correlation across dependency edges collapses a fan-out pattern (one shared cache's degradation triggering alerts on 15 downstream consumers) into one incident instead of 15 separate pages. After these changes, total volume drops from 1000/day to roughly 300/day. The team then instruments a dashboard tracking suppressed-alert count over time and finds, in week 3, an anomalous spike in suppressions for one rule; investigating that spike (rather than ignoring it because none of those suppressed alerts individually paged anyone) surfaces a genuine, slowly-building issue that the suppression logic had been correctly filtering as "looks like the usual noise" right up until it became a real, if still-contained, problem, catching it earlier than it would otherwise have been noticed.

Trade-offs and pitfalls

Aggressive suppression genuinely can hide a real incident, which is why the suppressed-volume monitoring and periodic sampling above are not optional add-ons but the mechanism that makes aggressive suppression safe to use at all; deploying suppression rules without that verification layer trades a visible noise problem for an invisible detection-gap risk, which is a worse trade, not a better one. A related pitfall: tuning suppression rules once against historical data and never revisiting them, when the traffic patterns and failure modes that justified a given rule can shift over time, silently turning a once-safe suppression rule into a real detection gap months later.

Direct answer

Four techniques trade consistency, latency, and complexity differently under a network partition, and the right choice depends on which side of that trade the workload actually needs: CRDTs (conflict-free replicated data types) accept writes on BOTH sides of a partition and merge deterministically afterward, at the cost of only supporting operations that have a well-defined, correct merge (an arbitrary object does not automatically have one); quorum writes (requiring acknowledgment from W of N replicas, reads from R of N, with W+R>N for strong consistency on that read) tune availability against consistency explicitly, but a partition that prevents EITHER side from reaching quorum makes that side fully unavailable for writes, not just slower; transactional (leader-based, synchronous) replication gives the strongest, easiest-to-reason-about consistency, but the side without the leader cannot accept writes at all during a partition; and application-level conflict resolution accepts writes on both sides like a CRDT but pushes correctness onto a hand-written merge function, trading a well-studied algebraic guarantee for full flexibility (and full responsibility for getting the merge logic right).

Structured elaboration

Why CRDTs are not a universal answer despite looking attractive on this table. A CRDT only exists for data structures with a well-defined, associative, commutative, idempotent merge operation (counters, sets, certain sequence types). Forcing an arbitrary business object into a CRDT shape can silently violate an invariant the object actually needs: a bank account balance modeled as a pure additive counter would happily merge to a negative balance across concurrent withdrawals from both sides of a partition, which is exactly the kind of business invariant a CRDT's merge rule has no concept of enforcing.

Why quorum sizing does not fully solve the partition problem. Choosing $W + R > N$ gives strong consistency for a read that genuinely satisfies that inequality, but it says nothing about what happens when a partition prevents either side from ASSEMBLING a quorum at all: a roughly even split of a replica set can leave BOTH sides unable to reach quorum, making the system unavailable for writes on both sides simultaneously, a harsher outcome than either the CRDT or transactional-replication approaches produce.

Application-level conflict resolution as a deliberate, not default, choice. This is the right tool specifically when the correct merge semantics are genuinely business-specific and do not map onto an existing CRDT (a shopping cart merge that needs to apply promotional-pricing rules to the merged result, for instance), not as a default alternative to reaching for a CRDT when one already exists for the data type in question.

Worked example

A G-Counter (grow-only counter CRDT, one increment slot per replica, merge takes the element-wise MAX per slot) is demonstrated below on the same additive workload as a naive last-write-wins (LWW) register, to show concretely why the two "look similar" (both are simple merge rules, both converge after a partition heals) but are not equally correct.

"""
Demonstrates a G-Counter CRDT correctly converging after a network partition
heals (both sides' concurrent increments are preserved), and contrasts it
against a naive last-write-wins (LWW) register on the SAME workload, which
DEMONSTRABLY LOSES one side's writes -- not a hypothetical, actually run
below.
"""


class GCounter:
    """Grow-only counter CRDT: each replica tracks its OWN increments in a
    per-replica slot; total = sum of all slots; merge = element-wise max
    per slot (never element-wise sum, which would double count after merge)."""

    def __init__(self, replica_id, all_replicas):
        self.replica_id = replica_id
        self.counts = {r: 0 for r in all_replicas}

    def increment(self, amount=1):
        self.counts[self.replica_id] += amount

    def merge(self, other):
        for r in self.counts:
            self.counts[r] = max(self.counts[r], other.counts[r])

    def value(self):
        return sum(self.counts.values())


class LWWRegister:
    """Naive alternative: a single shared 'total' overwritten directly.
    Under partition, each side keeps writing to its OWN local copy; merge
    picks 'whichever write has the higher logical timestamp wins', which
    means the LOSING side's writes vanish entirely, not just reorder."""

    def __init__(self):
        self.total = 0
        self.timestamp = 0

    def write(self, new_total, at_timestamp):
        if at_timestamp >= self.timestamp:
            self.total = new_total
            self.timestamp = at_timestamp


def main():
    replicas = ["A", "B"]

    # --- G-Counter: correct convergence under partition ---
    a = GCounter("A", replicas)
    b = GCounter("B", replicas)

    # Partition begins: A and B each accept writes independently, with NO
    # communication between them (this is the whole point of a partition).
    for _ in range(7):
        a.increment(1)   # A's replica independently processes 7 local writes
    for _ in range(4):
        b.increment(1)   # B's replica independently processes 4 local writes

    print(f"During partition: A.value()={a.value()} (A's own view), "
          f"B.value()={b.value()} (B's own view) -- diverged, as expected mid-partition.")

    # Partition heals: replicas exchange state and merge.
    a_copy_of_state = GCounter("A", replicas)
    a_copy_of_state.counts = dict(a.counts)
    b_copy_of_state = GCounter("B", replicas)
    b_copy_of_state.counts = dict(b.counts)

    a.merge(b_copy_of_state)
    b.merge(a_copy_of_state)

    print(f"After partition heals and merge: A.value()={a.value()}, B.value()={b.value()}")
    assert a.value() == b.value() == 11, "both replicas must converge to 7+4=11 after merge"
    assert a.counts == b.counts, "merged internal state must be IDENTICAL across replicas, not just the same total"
    print("CONFIRMED: G-Counter converges to 11 on both sides (all 7+4 writes preserved),")
    print("with byte-identical internal state on both replicas, not just a coincidentally")
    print("matching total.")

    # --- LWW register on the SAME workload: demonstrably LOSES writes ---
    lww_a = LWWRegister()
    lww_b = LWWRegister()

    # Same partition scenario: A applies 7 increments locally (each a
    # read-modify-write against ITS OWN local total, timestamps 1..7);
    # B applies 4 increments locally (timestamps 1..4, B's own local clock,
    # concurrently and independently -- this is what "partitioned" means).
    running = 0
    for t in range(1, 8):
        running += 1
        lww_a.write(running, at_timestamp=t)
    running = 0
    for t in range(1, 5):
        running += 1
        lww_b.write(running, at_timestamp=t)

    print(f"\nDuring partition: lww_a.total={lww_a.total} (ts={lww_a.timestamp}), "
          f"lww_b.total={lww_b.total} (ts={lww_b.timestamp})")

    # Partition heals: the two registers exchange their (total, timestamp)
    # state and each applies the OTHER's write via the SAME LWW merge rule.
    a_state, a_ts = lww_a.total, lww_a.timestamp
    b_state, b_ts = lww_b.total, lww_b.timestamp
    lww_a.write(b_state, b_ts)
    lww_b.write(a_state, a_ts)

    print(f"After partition heals and LWW merge: lww_a.total={lww_a.total}, lww_b.total={lww_b.total}")
    assert lww_a.total == lww_b.total, "LWW does converge to the SAME value on both sides..."
    assert lww_a.total == 7, "...but that value is 7, not 11: B's 4 increments are GONE, not merged"
    print("CONFIRMED WRONG: LWW converges (both sides agree), but to 7, not 11. B's four")
    print("increments were not lost due to a bug in this demo -- they were overwritten BY")
    print("DESIGN, because LWW has no way to represent 'both sides changed the same field'.")
    print("The two mechanisms 'look similar' (both are simple merge rules that converge),")
    print("but only the CRDT's convergence is also CORRECT for this additive workload.")


if __name__ == "__main__":
    main()

Output (actually executed with python3):

During partition: A.value()=7 (A's own view), B.value()=4 (B's own view) -- diverged, as expected mid-partition.
After partition heals and merge: A.value()=11, B.value()=11
CONFIRMED: G-Counter converges to 11 on both sides (all 7+4 writes preserved),
with byte-identical internal state on both replicas, not just a coincidentally
matching total.

During partition: lww_a.total=7 (ts=7), lww_b.total=4 (ts=4)
After partition heals and LWW merge: lww_a.total=7, lww_b.total=7
CONFIRMED WRONG: LWW converges (both sides agree), but to 7, not 11. B's four
increments were not lost due to a bug in this demo -- they were overwritten BY
DESIGN, because LWW has no way to represent 'both sides changed the same field'.
The two mechanisms 'look similar' (both are simple merge rules that converge),
but only the CRDT's convergence is also CORRECT for this additive workload.

Both mechanisms converge, in the sense that both replicas agree on a single value after the partition heals: the G-Counter converges to 11, correctly reflecting all 7 of side A's writes and all 4 of side B's writes; the LWW register also converges, but to 7, silently DISCARDING side B's 4 writes entirely, not because of a bug in the demonstration but because LWW has no representation for "both sides independently changed the same field" beyond picking one winner. "Converges" alone is not sufficient evidence of correctness; the merge rule has to actually match the workload's semantics (additive, in this case), which is exactly why the CRDT/quorum/replication/app-level choice in the table above is a semantics question, not a generic availability-versus-consistency dial.

Trade-offs and pitfalls

• Common mistake: reaching for a CRDT-shaped merge for a field that is not genuinely additive/mergeable, because the technique is fashionable. As the demonstration shows for the CONTRASTING failure case, the risk runs the other way too: an LWW register looks like a simpler CRDT and is dangerously plausible for anything additive, silently dropping writes rather than erroring loudly.
• A quorum-based design needs an explicit answer for the "neither side reaches quorum" case, not just for "the majority side stays available"; for a roughly even replica split, both sides can end up unavailable for writes simultaneously, worth stating explicitly in a design review rather than assuming quorum always leaves ONE side functional.
• Transactional replication's consistency guarantee is the easiest to reason about, but it is bought with the least availability during a partition; this is the right trade specifically when the correctness cost of a wrong or conflicting value (an actual ledger balance, not an approximate counter) outweighs the cost of one side being unable to write at all.
• The "complexity" column is not just about implementation effort. For application-level conflict resolution specifically, complexity means every edge case of the merge function's correctness rests on the team's own reasoning, with no algebraic proof backing it the way a well-studied CRDT has; this is a real, ongoing maintenance cost, not a one-time build cost.