Automated Incident Response and Cross-Phase Incident Scenarios Questions

Direct answer

Safe retry semantics have to be designed per connector type, because each one offers a different native primitive for idempotency or atomicity: Kafka producers get exactly-once via the idempotent producer plus transactions; Kafka consumers get it via read_committed isolation reading only committed transactional output; database writes get it via native upserts or local transactions; REST calls to a third-party get it via an idempotency-key header when the API supports one, or an outbox-plus-proxy pattern when it does not; and object storage like S3 gets it via content-addressed keys or an atomic manifest commit. There is no single mechanism that covers all four; the architecture's job is to pick the right one per connector and make sure they compose correctly end to end.

Structured elaboration

Kafka producers. Enable the idempotent producer (enable.idempotence=true), which assigns each producer a unique ID and each message a sequence number, letting the broker deduplicate retried sends from the SAME producer session automatically. For cross-partition or cross-topic atomicity (writing to multiple topics as one unit), wrap the writes in a Kafka transaction (initTransactions, beginTransaction, commitTransaction), which the broker either fully commits or fully aborts.

Kafka consumers. Reading a transactional producer's output requires setting the consumer's isolation level to read_committed, so aborted or in-flight transactions are invisible; a consumer left at the default read_uncommitted would see uncommitted, possibly-aborted data, silently breaking the exactly-once guarantee the producer side worked to provide. Consumer offset commits should be tied to downstream processing completion (commit the offset only after the corresponding output is durably written), not committed eagerly on read.

Database writes. Use the database's native atomic primitives: an INSERT ... ON CONFLICT DO UPDATE (Postgres) or MERGE keyed by a business key plus version, for single-row idempotency; a local transaction for multi-row atomicity within that one database. If the write must be atomic with the Kafka consumer offset commit (a common payments pattern), the outbox pattern (write the outbox row in the SAME local database transaction as the business write) decouples that atomicity from needing Kafka and the database to share a distributed transaction, which they generally cannot.

REST calls. If the third-party API supports an idempotency-key parameter (Stripe-style), generate that key deterministically from the logical operation (not fresh per retry) and let the API's own deduplication handle it. If it does not, apply an idempotency-proxy pattern: put a proxy in front of the API (the strongest option, if worth building), or accept a compensating-transaction fallback for genuinely one-way, non-idempotent operations.

Object storage (S3). Native S3 operations are individually retry-safe (a PutObject retried with the same key and content just re-uploads the same bytes, no duplication), but a MULTI-OBJECT logical write (many files representing one dataset version) needs a manifest-based atomic commit: stage, then atomically swap a small manifest pointer, so a partial or duplicated multi-object write is never visible as "done."

Worked example

A pipeline reads Kafka, writes to a Postgres database (for a materialized view), calls a third-party fraud-check REST API, and archives raw events to S3, all per logical event, needing the whole chain to behave correctly under retries. Concrete wiring, in order:

1. Kafka consumer reads with read_committed, does not commit its offset yet.
2. Postgres write: INSERT ... ON CONFLICT (event_id) DO NOTHING (idempotent by event_id).
3. Fraud-check REST call: the API supports an idempotency-key header; the pipeline passes event_id as that key deterministically, so a retried call after a timeout is recognized and returns the original result.
4. S3 archive: PutObject keyed by event_id (content-addressed by logical identity), so a retried upload overwrites the identical object harmlessly.
5. Only after all three writes are confirmed does the Kafka consumer commit its offset.

If step 3 (the REST call) times out ambiguously and the whole event is retried from step 2: step 2's ON CONFLICT DO NOTHING is a safe no-op (already inserted), step 3's idempotency key correctly returns the cached fraud-check result rather than re-running it, and step 4's re-upload is harmless. The offset is committed only once all four steps are confirmed, so a crash before that point simply replays this exact same, now-fully-idempotent sequence, and a crash after commit never revisits this event again (correct, since it was already fully processed).

Trade-offs and pitfalls

• Common mistake: committing the Kafka offset before all downstream writes are confirmed. This is the single most common way to silently lose the "at-least-once" half of the guarantee: a crash between offset-commit and the last downstream write means that event is never retried, since the consumer believes it already handled it.
• Common mistake: assuming Kafka's idempotent producer alone gives end-to-end exactly-once. It only protects the Kafka WRITE from producer-side retries; it says nothing about the downstream database, REST call, or S3 write each independently needing their own idempotency discipline, exactly why this answer treats each connector type separately rather than claiming one mechanism covers the whole chain.
• Ordering the four connector writes matters for correctness, not just tidiness. Placing the offset commit last (as in the worked example) is deliberate: it is the one step in the chain that, if it happens too early, breaks the whole at-least-once guarantee; every other step being idempotent means their relative order among themselves is more flexible.
• Per-connector idempotency does not automatically give cross-connector atomicity. If the fraud-check call succeeds but the process crashes before the S3 archive, on retry the fraud-check idempotency key correctly avoids re-running (good), but there is a window where downstream state is partially applied; this is the same partial-failure-across-heterogeneous-sinks problem any multi-sink write faces, and the fix is the same: make every step both idempotent AND independently retriable, not build a fragile distributed transaction across all four.

Direct answer

Key incoming events by (service, error_code), and use a session-style sliding window per key: an event extends the current incident for that key if it arrives within dedup_window seconds of the key's last event, otherwise the old incident closes and a new one starts. Track the running summary (unique hosts, first/last seen, total count) per open key and emit it when the window closes.

Structured elaboration

The core design decision is fixed bucket versus session window. A fixed bucket (e.g., "group everything in each 60-second wall-clock tick") is simpler but arbitrarily splits one continuous burst that straddles a bucket boundary. A session window, which resets the clock on every new event for the same key and only closes after dedup_window seconds of silence, correctly treats a continuous burst as one incident no matter how long it runs, and correctly splits two bursts of the same error separated by a genuine quiet period. Session windows are the standard choice for this problem and are what the worked example below demonstrates.

Data structure: a dict keyed by (service, error_code) mapping to a small summary record {hosts: set, first_seen, last_seen, total_events}. Using a set for hosts gives O(1) amortized insert and an exact unique-host count without needing a second pass.

Complexity: each event does O(1) work (a dict lookup, a set insert, a few comparisons), so processing N events is O(N) time and O(K) additional space where K is the number of distinct (service, error_code) keys concurrently open, which is bounded and small in practice (far smaller than N for a real alert stream).

Worked example

from dataclasses import dataclass, field

@dataclass
class IncidentSummary:
    service: str
    error_code: str
    hosts: set = field(default_factory=set)
    first_seen: float = None
    last_seen: float = None
    total_events: int = 0

    def to_dict(self):
        return {
            "service": self.service,
            "error_code": self.error_code,
            "unique_host_count": len(self.hosts),
            "first_seen": self.first_seen,
            "last_seen": self.last_seen,
            "total_events": self.total_events,
        }


def dedupe_alerts(events, dedup_window):
    """events arrive in non-decreasing timestamp order (a realistic stream assumption)."""
    open_incidents = {}
    closed = []
    for e in events:
        key = (e["service"], e["error_code"])
        cur = open_incidents.get(key)
        if cur is not None and e["timestamp"] - cur.last_seen > dedup_window:
            closed.append(cur.to_dict())
            cur = None
        if cur is None:
            cur = IncidentSummary(service=e["service"], error_code=e["error_code"],
                                   first_seen=e["timestamp"], last_seen=e["timestamp"])
            open_incidents[key] = cur
        cur.hosts.add(e["host"])
        cur.last_seen = e["timestamp"]
        cur.total_events += 1
    for cur in open_incidents.values():
        closed.append(cur.to_dict())
    return closed

Run against a stream where checkout/500 fires at t=0, t=10, t=20, then goes quiet until t=220 (a 200-second gap) and fires again at t=220, t=230, with dedup_window=60, plus one unrelated payments/429 event at t=15:

dedupe_alerts(events, dedup_window=60)
-> {'service': 'checkout', 'error_code': '500', 'unique_host_count': 2, 'first_seen': 0,  'last_seen': 20,  'total_events': 3}
-> {'service': 'checkout', 'error_code': '500', 'unique_host_count': 1, 'first_seen': 220,'last_seen': 230, 'total_events': 2}
-> {'service': 'payments', 'error_code': '429', 'unique_host_count': 1, 'first_seen': 15, 'last_seen': 15,  'total_events': 1}

This was executed and the actual output matched the three summaries above: because the 200-second gap exceeds the 60-second window, the algorithm correctly emits the t=0..20 burst as one incident and the t=220..230 burst as a second, separate incident for the same key, rather than incorrectly merging everything into one long-running incident.

Trade-offs and pitfalls

Assuming events arrive in timestamp order is realistic for a single ordered stream (e.g., one Kafka partition per alert source) but breaks in a naively fanned-in multi-producer stream where out-of-order arrival is possible; in that case you need either an upstream sort/watermark step or a small out-of-order tolerance buffer that holds each key's window open slightly past dedup_window before finalizing, trading a little latency for correctness. For very high cardinality of (service, error_code) pairs, memory for open incidents can grow; a background sweep that force-closes any incident whose last_seen is older than dedup_window bounds memory even if a producer stalls mid-stream and never sends a closing gap.

Direct answer

Contain immediately by disabling the test that performed the destructive call and confirming no further destructive calls are in flight, assess and begin recovering the deleted data from backups right away since that clock is the one that matters most to the customer, and scope the postmortem broadly enough to ask not just "why did this test do that" but "why did nothing stop a destructive call from ever reaching production."

Structured elaboration

Immediate incident response steps. First, disable or quarantine the specific automated test (and, if it runs on a schedule or trigger, ensure it cannot fire again before you understand it), since a test framework or CI credential capable of a destructive production API call is a class of danger that will recur immediately if left active. Second, confirm the full scope of what was deleted: which records, which customers, over what time window. Third, immediately begin backup/restore procedures for the affected data, since the customer-facing harm from a data-loss incident is proportional to how long the data stays gone, distinct from and usually more urgent than fully understanding how the test came to run against production in the first place.

Short-term mitigations. Beyond restoring the deleted data, revoke or scope down whatever credentials the test used, since the fact that a TEST had credentials capable of a destructive, customer-data-deleting call in production is itself the sharper, more urgent problem, separate from why this particular test happened to exercise that capability; a test suite should not hold production-destructive capability at all, regardless of what any individual test does with it. Audit whether any OTHER automated jobs share the same overly-broad credential or execution environment, since if one test could do this, others might be able to as well.

Postmortem scope. This needs two, not one, root-cause threads. Thread one: why did this specific test perform a destructive call (a bug in the test itself, a misconfigured target environment variable that pointed it at production instead of a test environment, a fixture that was supposed to be mocked and was not). Thread two, the more important one: why did CI pass a test capable of this at all, and why did the execution environment grant test code the credentials to make a real, destructive production call in the first place. A postmortem that only answers thread one and fixes the specific test's bug, without addressing thread two, leaves the door open for the next test (or the next bug in this same one, if only partially fixed) to do the same thing again.

Long-term changes to the test harness and CI policies. Enforce environment isolation structurally, not by convention: test execution environments should have no network path to production systems at all, or if some tests genuinely need to exercise real infrastructure, use environment-specific credentials scoped to a non-production account with no access to real customer data, verified by an automated check (not a code-review reminder) that flags any test attempting to reach a production endpoint or use production-scoped credentials. Add a policy gate that blocks any newly-introduced test capable of a genuinely destructive API call (delete, bulk-update, financial-transaction-triggering) from merging without a specific, elevated review, distinct from ordinary code review, given the severity class this incident demonstrates such a capability carries.

Worked example

Investigation reveals the destructive test was originally written and correctly scoped to run against a staging environment, but a recent CI configuration change intended to consolidate environment variables accidentally caused the staging-environment URL variable to fall back to the production URL when a particular new pipeline stage ran, and no automated check existed to catch a test targeting a production hostname before it executed. Immediate response: the test is disabled, the CI configuration bug is reverted, and the deleted customer records are restored from the most recent backup, with a gap-analysis of any legitimate writes made between that backup and the deletion that also need reconciling. Credentials: the CI service account used by this test pipeline is found to have broader production access than any test genuinely needs, and is immediately scoped down to a non-production-only credential while a proper least-privilege audit of all CI service accounts is scheduled. Postmortem, thread one: the specific environment-variable fallback bug that caused this test to target production. Postmortem, thread two, and the one the team treats as the higher-priority finding: no structural barrier existed to prevent a test from reaching a production endpoint at all, which is the gap that actually determined how bad this incident could get, independent of this specific configuration bug. Long-term fix: an automated pre-execution check now blocks any test run whose target hostname resolves to a production-tagged endpoint, regardless of how it got there, closing the class of bug rather than just this instance of it.

Trade-offs and pitfalls

The instinct to focus the postmortem entirely on "why did this specific test do this" is understandable but incomplete, because fixing only the proximate cause (the environment-variable bug in the worked example) leaves the deeper, more dangerous gap in place: nothing structurally prevented ANY test from reaching production with destructive capability, so the next bug of a completely different shape could reproduce the same class of incident. The cost of the structural fix (a hard, automated block on test code reaching production endpoints, tighter credential scoping for all CI service accounts) is real engineering investment beyond just patching the one test, but the alternative, relying on this not happening again through code review vigilance alone, is exactly the kind of process-only safeguard that already failed once here.

Direct answer

Poll the process's RSS (resident set size) memory on a timer, count consecutive checks over the threshold rather than reacting to the first one, and restart gracefully: send SIGTERM, wait for a bounded timeout, and only escalate to SIGKILL if the process ignores the polite request.

Structured elaboration

Why "two consecutive checks" instead of one. A single over-threshold reading can be a transient spike (a brief allocation burst that gets freed right after), and restarting on that would cause unnecessary downtime for a process that was never actually leaking. Requiring two consecutive over-threshold checks filters out that single-spike case while still catching a genuine, sustained growth trend within two polling intervals, a small and bounded detection delay.

Graceful restart. SIGTERM asks the process to shut itself down cleanly (flush buffers, close connections, finish an in-flight request); SIGKILL cannot be caught or ignored and terminates immediately, which can leave work half-done. Always try SIGTERM first with a bounded wait, and only fall back to SIGKILL if the process does not exit within that window, since a process wedged badly enough to leak memory might also be too wedged to respond to a polite signal.

Testability. Keep the polling loop, the decision logic (count consecutive over-threshold checks), and the restart action as separable pieces (a watch loop that calls out to a graceful_restart helper) so each can be tested independently: the restart logic against a real or fake process without waiting through real polling intervals, and the threshold/consecutive-check logic against a sequence of fake memory readings without needing a real process at all.

Worked example

import logging, subprocess, time
import psutil

log = logging.getLogger("watchdog")

def graceful_restart(proc, start_cmd, term_timeout=5.0):
    pid = proc.pid
    try:
        proc.terminate()          # SIGTERM
        proc.wait(timeout=term_timeout)
        log.info("pid %s exited cleanly after SIGTERM", pid)
    except psutil.TimeoutExpired:
        log.warning("pid %s did not exit within %.1fs, sending SIGKILL", pid, term_timeout)
        proc.kill()
        proc.wait(timeout=term_timeout)
    except psutil.NoSuchProcess:
        pass
    new_popen = subprocess.Popen(start_cmd)
    log.info("restarted process, new pid=%s", new_popen.pid)
    return new_popen

def watch(pid, mem_threshold_mb, interval_seconds, start_cmd, max_checks=None):
    over_count = 0
    checks_done = 0
    proc = psutil.Process(pid)
    threshold_bytes = mem_threshold_mb * 1024 * 1024
    while max_checks is None or checks_done < max_checks:
        checks_done += 1
        try:
            rss = proc.memory_info().rss
        except psutil.NoSuchProcess:
            log.info("process %s is gone, stopping watch", proc.pid)
            return
        log.info("check %d: pid=%s rss=%.1fMB threshold=%.1fMB", checks_done, proc.pid,
                  rss / 1024 / 1024, mem_threshold_mb)
        over_count = over_count + 1 if rss > threshold_bytes else 0
        if over_count >= 2:
            log.warning("pid %s exceeded %.1fMB for 2 consecutive checks, restarting",
                        proc.pid, mem_threshold_mb)
            new_popen = graceful_restart(proc, start_cmd)
            proc = psutil.Process(new_popen.pid)
            over_count = 0
        time.sleep(interval_seconds)

This was executed against a real spawned process (a small script that allocates 2MB per iteration on a timer, so its RSS crosses a low 20MB test threshold predictably): the actual run log showed the watchdog correctly tracking a growing RSS (0.0MB, 18.7MB, 22.7MB, 24.8MB), restarting after the 2nd consecutive over-threshold check (the 22.7MB and 24.8MB readings), and then correctly resuming monitoring the new process's PID afterward, with a fresh RSS baseline. An earlier draft of this code had a real bug caught only by running it: the log statements inside the loop referenced the outer pid parameter instead of the current proc.pid, so after a restart the logs kept reporting the OLD, now-dead PID even though the code was correctly monitoring the new one internally, a mismatch between what was logged and what was actually happening. Confirming this required an actual restart to occur during a test run, since the bug is invisible until you have watched the PID actually change.

Trade-offs and pitfalls

RSS is a reasonable, simple signal for "is this process using too much memory," but it is not the only one: RSS includes shared memory the process maps but does not own exclusively, so for a process that shares large memory-mapped files with siblings, RSS can overstate what would actually be freed by restarting it; a leak-specific signal (private, unshared memory) would be more precise but is more platform-dependent to obtain reliably. The consecutive-check requirement trades detection speed for false-positive resistance; for a process where even a brief memory spike is dangerous (a small container with a tight memory limit that will get OOM-killed by the kernel before two check intervals pass), a single-check trigger with a lower threshold might be the safer choice instead, so the "2 consecutive checks" policy should be tuned to the actual failure mode being protected against, not treated as a universal default.

Direct answer

Four techniques trade consistency, latency, and complexity differently under a network partition, and the right choice depends on which side of that trade the workload actually needs: CRDTs (conflict-free replicated data types) accept writes on BOTH sides of a partition and merge deterministically afterward, at the cost of only supporting operations that have a well-defined, correct merge (an arbitrary object does not automatically have one); quorum writes (requiring acknowledgment from W of N replicas, reads from R of N, with W+R>N for strong consistency on that read) tune availability against consistency explicitly, but a partition that prevents EITHER side from reaching quorum makes that side fully unavailable for writes, not just slower; transactional (leader-based, synchronous) replication gives the strongest, easiest-to-reason-about consistency, but the side without the leader cannot accept writes at all during a partition; and application-level conflict resolution accepts writes on both sides like a CRDT but pushes correctness onto a hand-written merge function, trading a well-studied algebraic guarantee for full flexibility (and full responsibility for getting the merge logic right).

Structured elaboration

Why CRDTs are not a universal answer despite looking attractive on this table. A CRDT only exists for data structures with a well-defined, associative, commutative, idempotent merge operation (counters, sets, certain sequence types). Forcing an arbitrary business object into a CRDT shape can silently violate an invariant the object actually needs: a bank account balance modeled as a pure additive counter would happily merge to a negative balance across concurrent withdrawals from both sides of a partition, which is exactly the kind of business invariant a CRDT's merge rule has no concept of enforcing.

Why quorum sizing does not fully solve the partition problem. Choosing $W + R > N$ gives strong consistency for a read that genuinely satisfies that inequality, but it says nothing about what happens when a partition prevents either side from ASSEMBLING a quorum at all: a roughly even split of a replica set can leave BOTH sides unable to reach quorum, making the system unavailable for writes on both sides simultaneously, a harsher outcome than either the CRDT or transactional-replication approaches produce.

Application-level conflict resolution as a deliberate, not default, choice. This is the right tool specifically when the correct merge semantics are genuinely business-specific and do not map onto an existing CRDT (a shopping cart merge that needs to apply promotional-pricing rules to the merged result, for instance), not as a default alternative to reaching for a CRDT when one already exists for the data type in question.

Worked example

A G-Counter (grow-only counter CRDT, one increment slot per replica, merge takes the element-wise MAX per slot) is demonstrated below on the same additive workload as a naive last-write-wins (LWW) register, to show concretely why the two "look similar" (both are simple merge rules, both converge after a partition heals) but are not equally correct.

"""
Demonstrates a G-Counter CRDT correctly converging after a network partition
heals (both sides' concurrent increments are preserved), and contrasts it
against a naive last-write-wins (LWW) register on the SAME workload, which
DEMONSTRABLY LOSES one side's writes -- not a hypothetical, actually run
below.
"""


class GCounter:
    """Grow-only counter CRDT: each replica tracks its OWN increments in a
    per-replica slot; total = sum of all slots; merge = element-wise max
    per slot (never element-wise sum, which would double count after merge)."""

    def __init__(self, replica_id, all_replicas):
        self.replica_id = replica_id
        self.counts = {r: 0 for r in all_replicas}

    def increment(self, amount=1):
        self.counts[self.replica_id] += amount

    def merge(self, other):
        for r in self.counts:
            self.counts[r] = max(self.counts[r], other.counts[r])

    def value(self):
        return sum(self.counts.values())


class LWWRegister:
    """Naive alternative: a single shared 'total' overwritten directly.
    Under partition, each side keeps writing to its OWN local copy; merge
    picks 'whichever write has the higher logical timestamp wins', which
    means the LOSING side's writes vanish entirely, not just reorder."""

    def __init__(self):
        self.total = 0
        self.timestamp = 0

    def write(self, new_total, at_timestamp):
        if at_timestamp >= self.timestamp:
            self.total = new_total
            self.timestamp = at_timestamp


def main():
    replicas = ["A", "B"]

    # --- G-Counter: correct convergence under partition ---
    a = GCounter("A", replicas)
    b = GCounter("B", replicas)

    # Partition begins: A and B each accept writes independently, with NO
    # communication between them (this is the whole point of a partition).
    for _ in range(7):
        a.increment(1)   # A's replica independently processes 7 local writes
    for _ in range(4):
        b.increment(1)   # B's replica independently processes 4 local writes

    print(f"During partition: A.value()={a.value()} (A's own view), "
          f"B.value()={b.value()} (B's own view) -- diverged, as expected mid-partition.")

    # Partition heals: replicas exchange state and merge.
    a_copy_of_state = GCounter("A", replicas)
    a_copy_of_state.counts = dict(a.counts)
    b_copy_of_state = GCounter("B", replicas)
    b_copy_of_state.counts = dict(b.counts)

    a.merge(b_copy_of_state)
    b.merge(a_copy_of_state)

    print(f"After partition heals and merge: A.value()={a.value()}, B.value()={b.value()}")
    assert a.value() == b.value() == 11, "both replicas must converge to 7+4=11 after merge"
    assert a.counts == b.counts, "merged internal state must be IDENTICAL across replicas, not just the same total"
    print("CONFIRMED: G-Counter converges to 11 on both sides (all 7+4 writes preserved),")
    print("with byte-identical internal state on both replicas, not just a coincidentally")
    print("matching total.")

    # --- LWW register on the SAME workload: demonstrably LOSES writes ---
    lww_a = LWWRegister()
    lww_b = LWWRegister()

    # Same partition scenario: A applies 7 increments locally (each a
    # read-modify-write against ITS OWN local total, timestamps 1..7);
    # B applies 4 increments locally (timestamps 1..4, B's own local clock,
    # concurrently and independently -- this is what "partitioned" means).
    running = 0
    for t in range(1, 8):
        running += 1
        lww_a.write(running, at_timestamp=t)
    running = 0
    for t in range(1, 5):
        running += 1
        lww_b.write(running, at_timestamp=t)

    print(f"\nDuring partition: lww_a.total={lww_a.total} (ts={lww_a.timestamp}), "
          f"lww_b.total={lww_b.total} (ts={lww_b.timestamp})")

    # Partition heals: the two registers exchange their (total, timestamp)
    # state and each applies the OTHER's write via the SAME LWW merge rule.
    a_state, a_ts = lww_a.total, lww_a.timestamp
    b_state, b_ts = lww_b.total, lww_b.timestamp
    lww_a.write(b_state, b_ts)
    lww_b.write(a_state, a_ts)

    print(f"After partition heals and LWW merge: lww_a.total={lww_a.total}, lww_b.total={lww_b.total}")
    assert lww_a.total == lww_b.total, "LWW does converge to the SAME value on both sides..."
    assert lww_a.total == 7, "...but that value is 7, not 11: B's 4 increments are GONE, not merged"
    print("CONFIRMED WRONG: LWW converges (both sides agree), but to 7, not 11. B's four")
    print("increments were not lost due to a bug in this demo -- they were overwritten BY")
    print("DESIGN, because LWW has no way to represent 'both sides changed the same field'.")
    print("The two mechanisms 'look similar' (both are simple merge rules that converge),")
    print("but only the CRDT's convergence is also CORRECT for this additive workload.")


if __name__ == "__main__":
    main()

Output (actually executed with python3):

During partition: A.value()=7 (A's own view), B.value()=4 (B's own view) -- diverged, as expected mid-partition.
After partition heals and merge: A.value()=11, B.value()=11
CONFIRMED: G-Counter converges to 11 on both sides (all 7+4 writes preserved),
with byte-identical internal state on both replicas, not just a coincidentally
matching total.

During partition: lww_a.total=7 (ts=7), lww_b.total=4 (ts=4)
After partition heals and LWW merge: lww_a.total=7, lww_b.total=7
CONFIRMED WRONG: LWW converges (both sides agree), but to 7, not 11. B's four
increments were not lost due to a bug in this demo -- they were overwritten BY
DESIGN, because LWW has no way to represent 'both sides changed the same field'.
The two mechanisms 'look similar' (both are simple merge rules that converge),
but only the CRDT's convergence is also CORRECT for this additive workload.

Both mechanisms converge, in the sense that both replicas agree on a single value after the partition heals: the G-Counter converges to 11, correctly reflecting all 7 of side A's writes and all 4 of side B's writes; the LWW register also converges, but to 7, silently DISCARDING side B's 4 writes entirely, not because of a bug in the demonstration but because LWW has no representation for "both sides independently changed the same field" beyond picking one winner. "Converges" alone is not sufficient evidence of correctness; the merge rule has to actually match the workload's semantics (additive, in this case), which is exactly why the CRDT/quorum/replication/app-level choice in the table above is a semantics question, not a generic availability-versus-consistency dial.

Trade-offs and pitfalls

• Common mistake: reaching for a CRDT-shaped merge for a field that is not genuinely additive/mergeable, because the technique is fashionable. As the demonstration shows for the CONTRASTING failure case, the risk runs the other way too: an LWW register looks like a simpler CRDT and is dangerously plausible for anything additive, silently dropping writes rather than erroring loudly.
• A quorum-based design needs an explicit answer for the "neither side reaches quorum" case, not just for "the majority side stays available"; for a roughly even replica split, both sides can end up unavailable for writes simultaneously, worth stating explicitly in a design review rather than assuming quorum always leaves ONE side functional.
• Transactional replication's consistency guarantee is the easiest to reason about, but it is bought with the least availability during a partition; this is the right trade specifically when the correctness cost of a wrong or conflicting value (an actual ledger balance, not an approximate counter) outweighs the cost of one side being unable to write at all.
• The "complexity" column is not just about implementation effort. For application-level conflict resolution specifically, complexity means every edge case of the merge function's correctness rests on the team's own reasoning, with no algebraic proof backing it the way a well-studied CRDT has; this is a real, ongoing maintenance cost, not a one-time build cost.