Data Consistency and Distributed Transactions Questions

Direct answer: When a single NoSQL store doesn't give you multi-entity distributed transactions, you approximate them at the application level with one of four patterns: giving each entity a single owning writer (so there's never real cross-writer contention to coordinate), per-entity optimistic or pessimistic locks used on their own without a full commit protocol, per-entity locks combined with a manual two-phase-style protocol at the application layer, or a consensus-backed external coordinator. Each trades complexity for stronger guarantees; the right choice depends on how much true multi-entity atomicity the workload actually needs.

Structured elaboration

Single-writer / entity ownership. If every entity that needs to change together is owned by exactly one logical writer (a single actor, partition, or aggregate boundary that serializes all writes to it), you avoid the coordination problem rather than solving it: writes to that entity happen one at a time, in order, with no concurrent conflicting writers to reconcile. This works well when your data model can be shaped so that "things that must change together" naturally live under one owner (e.g. an aggregate in a domain-driven design sense), but breaks down when a real business operation spans two independently-owned aggregates, exactly the case here.

Per-entity locks alone (no commit protocol). Before this escalates to a full application-level 2PC, the simpler version is just acquiring a pessimistic lock (or an optimistic version-check) on each entity in a fixed order, doing the work against each entity directly, and releasing as you go, with no shared "pending" record and no atomic all-or-nothing decision across entities. This is cheaper to build than app-level 2PC (no prepare phase, no cross-entity coordinator state) and is enough when the operation is really a sequence of independent per-entity updates that just need to avoid concurrent-writer corruption, not true cross-entity atomicity. Its failure mode is different from 2PC's: if the process crashes after updating entity A but before updating entity B, there is no "pending" marker anywhere recording that an operation was in flight, entity A is simply left updated and entity B is not, with nothing to detect or resolve that half-applied state short of comparing it against expected invariants elsewhere. This is strictly weaker than app-level 2PC, which at least leaves a pending record a sweeper can find.

Per-entity locks plus application-level two-phase commit. You build a lightweight prepare/commit protocol yourself on top of the store's single-entity atomic operations (conditional writes / compare-and-swap): write a "pending" record to each entity with a shared transaction ID (the prepare step, using the store's native conditional-write support to detect conflicts), then once all entities show "pending" for that transaction ID, flip each to "committed" (the commit step), each individual flip being a single-entity atomic operation the store natively supports. If any entity's prepare fails (a conflicting concurrent write got there first, or a validation check fails), abort by clearing the pending markers on whichever entities already got them, using the same idempotent-write discipline as a saga's compensation.

Consensus-backed coordinator. For workloads that need stronger, more general guarantees (arbitrary sets of entities per transaction, not knowable ahead of time), an external coordinator, itself backed by consensus for its own durability, drives the same prepare/commit protocol but tracks state independently of any single entity, giving you real 2PC semantics layered on top of a store that doesn't natively support them.

Comparing complexity and failure modes

Worked example. A NoSQL store with only single-item conditional writes (like DynamoDB's conditional PutItem) needs to move an item between two "bucket" entities atomically. Prepare: write {item_id, txn_id: T1, state: "pending_remove"} to bucket A's item list and {item_id, txn_id: T1, state: "pending_add"} to bucket B's item list, each a single-item conditional write (fails if a concurrent transaction already touched that item). If both prepares succeed, commit: atomically flip A's entry to "removed" and B's entry to "added", each again a single-item operation. If prepare on B fails (item already exists there under a different transaction), abort by deleting A's pending marker. A background sweeper periodically scans for pending markers older than some threshold (indicating the process that started the transaction died mid-flight) and resolves them by re-checking both entities' actual states and either completing or rolling back.

Trade-offs and pitfalls. The recurring failure in home-grown app-level 2PC is skipping the background sweeper: without it, a crash between "both prepared" and "both committed" leaves the data permanently in a pending state that nothing ever resolves, which is strictly worse than 2PC's blocking (at least a blocked 2PC participant is visibly stuck and holding a lock someone will eventually notice); a silently stuck "pending" record can go undetected for a long time.

Direct answer: To let clients choose consistency per operation, expose it explicitly in the API and data model, rather than baking one global choice into the whole service, tag each operation with its required consistency level, route strongly-consistent (linearizable) operations to a synchronously-coordinated path (a single-partition-owner or quorum write/read) and eventually-consistent ones to the cheaper, asynchronously-replicated path, and partition the underlying data so an operation's consistency need maps cleanly onto how it's stored and served.

Structured elaboration

API-level exposure. Rather than a single, undifferentiated GET/PUT, the API distinguishes operations by their consistency requirement, either through distinct endpoints (POST /orders/{id}/finalize implies strong consistency by its nature; GET /orders/{id}/status for casual polling can be served eventually-consistent) or an explicit parameter/header the client sets (Consistency: strong vs Consistency: eventual) for operations that could reasonably go either way depending on context.

Data partitioning to avoid corruption. The risk in mixing consistency levels isn't the READS, it's making sure a strongly-consistent WRITE and an eventually-consistent read of the SAME underlying data can't produce a genuinely corrupted result (as opposed to merely stale). The design partitions data so the strongly-consistent operations own a clear, authoritative write path (e.g. a single-partition-owner model, or a quorum write), and eventually-consistent reads are explicitly understood (by the client, via the API contract) to be a possibly-stale VIEW of that same authoritative data, not a second, independently-writable copy that could diverge and cause real corruption, only staleness, which is a fundamentally safer failure mode.

Routing. An internal routing layer directs strongly-consistent operations to the path that can actually provide that guarantee (a leader-only read/write, or a quorum-coordinated one), and eventually-consistent operations to a cheaper path (a local replica read, or an asynchronously-applied write), the client's declared consistency need drives which internal path handles the request, invisible to the client beyond the contract it opted into.

Versioning for compatibility. As the API evolves (e.g. adding a new consistency tier, like "bounded staleness" between full strong and full eventual), the consistency parameter/header needs its own versioning discipline so existing clients that only understand "strong" or "eventual" don't silently misinterpret a new tier as one they already know, an explicit, documented default (usually the SAFER, strongly-consistent option) for any client that doesn't specify a consistency preference at all avoids a client silently getting weaker guarantees than the service's default behavior.

Developer ergonomics. From the calling developer's point of view, the choice should be a simple, well-documented parameter or endpoint choice with clear guidance ("use finalize-order for anything that commits money or inventory; use order-status for a polling UI"), not a deep understanding of the underlying replication architecture, most application developers calling this API shouldn't need to reason about quorums or replication lag directly, the API's job is to translate their INTENT (I need this to be authoritative vs I'm fine with a quick, possibly-slightly-stale view) into the right internal behavior.

Worked example. finalize-order requires the caller to have already read a strongly-consistent inventory count (via a preceding strongly-consistent read the API forces as part of the flow) and commits atomically against the authoritative partition-owner, guaranteed no double-finalization even under concurrent requests. view-order-status (used by a status-polling UI) reads from the nearest, possibly-slightly-stale replica, fast and cheap, with an explicit as_of timestamp in the response so the UI can show "last updated Xs ago" rather than presenting the data as unconditionally current.

Trade-offs and pitfalls. The riskiest design mistake here is letting an EVENTUALLY-consistent read feed directly into a decision that then gets written WITHOUT going through the strongly-consistent write path's own validation, e.g. a client reading a stale "available" status and then calling finalize-order assuming that stale read is still accurate; the finalize operation itself must independently re-verify against the strongly-consistent source at write time, never trust a client-supplied eventually-consistent read as sufficient justification for a strongly-consistent action.

Direct answer: Since you can't use a single distributed database, the choice is between a saga that debits and credits as two independently-committing local transactions with a compensating credit-back if the second step fails, or a 2PC-style protocol where both shards prepare before either commits. Given the audit-trail and fund-conservation requirements, a saga with an idempotent, carefully sequenced compensation plus a durable audit log of every step is the more operationally realistic choice for most systems at this scale; 2PC is defensible only if the two services are both under your control and the transfer volume is low enough that its availability cost is acceptable.

Structured elaboration

Why not 2PC by default here. 2PC would give you genuine atomicity, but "moving money between accounts managed by separate services" strongly suggests these services can evolve independently, may not share an operational team, and the transfer needs to keep working even if one service is temporarily slow. 2PC's blocking failure mode (a coordinator crash stranding a participant holding a lock on someone's balance) is a worse operational risk here than a saga's eventual-consistency window, provided the saga is designed so no external party ever sees an impossible intermediate state.

Saga design for the transfer.

1. Reserve/hold on the source account (not an unconditional debit): decrement available balance, but keep it recoverable, this is itself a local ACID transaction on the source service, and only requires the source service's own database, no cross-service coordination.
2. Credit the destination account. If this succeeds, finalize the hold on the source (converting the reservation into a real, permanent debit) and record both legs as complete in an audit log entry keyed by a single transfer ID.
3. Compensating action if the credit fails (destination account invalid, destination service unavailable after retries, business rule violation): release the hold on the source account, restoring its available balance, and mark the transfer as failed in the audit log with the failure reason.

Idempotency at every step. Every step (reserve, credit, finalize, compensate) must be safe to retry: keyed on the transfer ID, so a retried "reserve" call after a network timeout doesn't double-reserve, and a retried "compensate" call after a crash doesn't double-restore the balance. Without this, a saga's retry-on-failure model becomes a source of the exact double-charge or lost-money bugs it's supposed to prevent.

Auditability and fund conservation. Every state transition (reserved, credited, finalized, compensated) is appended to a durable, transfer-ID-keyed audit log BEFORE the corresponding service call is made, mirroring 2PC's "log before you act" discipline even though there's no coordinator. This log is what lets you prove, for any transfer ID, exactly what happened, and lets an operator (or an automated reconciliation job) find and resolve transfers that got stuck mid-flight, e.g. reserved on the source but the credit call to the destination never got a definitive answer.

Proving conservation of funds specifically. Per-transfer audit history is necessary but not sufficient, proving conservation requires an aggregate check across the whole ledger, not just tracing one transfer ID. A periodic (e.g. hourly) reconciliation job sums, per currency, every FINALIZED debit recorded in the audit log against every FINALIZED credit recorded in the audit log for the same time window; a healthy system nets to zero (double-entry style: total debits == total credits). The same job independently re-derives each account's expected balance by replaying its audit-log entries and diffs that against the account service's own reported balance, any nonzero diff on either check (net debits/credits not zero, or a re-derived balance that disagrees with the service of record) is flagged for investigation rather than silently trusted. Separately, any transfer whose audit log shows a non-terminal state (reserved or credited without a following finalized or compensated) older than the expected completion SLA is surfaced as a stuck transfer, this is what the reconciliation job in the earlier paragraph resolves operationally; the aggregate debit/credit check is what proves fund conservation across the system, a distinct and additional guarantee from resolving individual stuck transfers.

Failure handling for the specific "stuck" case. If the destination service call times out (not a clean success or failure, an ambiguous outcome), the transfer must NOT immediately compensate, that risks a double-credit if the destination actually did apply it before the timeout. Instead, the orchestrator retries a status check (using the transfer ID) against the destination service until it gets a definitive answer, or escalates to a reconciliation job after a bounded number of attempts.

Worked example. Transfer T-9182 moves $500 from account A (service X) to account B (service Y). Orchestrator logs "T-9182: reserve started" then calls X's reserve API with idempotency key T-9182-reserve; X durably decrements A's available balance and returns success; orchestrator logs "T-9182: reserved". Orchestrator calls Y's credit API with idempotency key T-9182-credit; suppose the call times out with no response. Orchestrator does NOT immediately compensate; instead it polls Y's GET /transfers/T-9182-credit/status (idempotent read) until Y reports either "applied" (in which case the orchestrator finalizes on X's side and logs completion) or "not found / rejected" (in which case it compensates: calls X's release-reservation API with idempotency key T-9182-release, and logs the transfer as failed).

Applies beyond money transfers. The same reserve-then-confirm shape protects any "two things must both happen or neither should" cross-service operation, not just fund transfers. A global booking platform preventing double-bookings across continents (e.g. holding a seat on a flight and a room at a hotel as one logical reservation) uses the identical pattern: a reservation/hold on each resource first, a saga that finalizes both holds together, and a fallback to consensus-backed coordination only for the rare, genuinely latency-tolerant cases where the business is willing to pay 2PC's availability cost for airtight atomicity across the booking.

Trade-offs and pitfalls. The single most common design mistake here is compensating too eagerly on an ambiguous failure (a timeout is not the same as a confirmed failure), which is exactly how "double the money existed briefly" bugs happen: the destination actually applied the credit, then the source got its reservation released too, creating money from nothing until reconciliation catches it, if it ever does. A reservation/hold step (rather than an immediate unconditional debit) is what makes the source-side compensation safe and reversible in the first place.

Direct answer: To make eventual consistency between service A and B idempotent and reconciliation-friendly, service A publishes events with a stable event ID (or a monotonic sequence number per entity), service B's consumer deduplicates on that ID before applying any change, and a periodic reconciliation job independently compares A's and B's views to catch and repair anything that slipped through despite the idempotency guarantees.

Structured elaboration

Idempotent event processing on the consumer side. Every event from A carries a stable identifier; B's consumer checks (atomically, alongside applying the event) whether that ID has already been processed, using the same "dedup record plus the actual state change in one transaction" discipline as any idempotent write. This is what makes at-least-once delivery (which any reasonable messaging setup between A and B will actually provide) safe: redelivery is a no-op rather than a duplicate application.

Preserving ordering where necessary. If events for the same entity must be applied in order (e.g. "created" before "updated" before "deleted"), B's consumer needs either a strictly-ordered delivery channel per entity (partition by entity ID) or an explicit sequence number in each event that B checks against the last-applied sequence for that entity, rejecting or buffering an out-of-order arrival rather than applying it prematurely.

Supporting replays. Because B's state can, despite everything, still drift from A's (a bug, an extended outage, a schema-migration mistake), the design should support REPLAYING A's full event history into B from scratch (or from a checkpoint) to rebuild B's view, which requires A to retain (or be able to regenerate) its event history for at least as long as any realistic replay window, and requires B's apply logic to be safe to run repeatedly over the same events (which it already is, by the idempotency design above).

Reconciliation as the safety net, not the primary mechanism. A periodic job independently compares A's and B's data (via checksums, row counts, or a full diff on a schedule appropriate to the data's size and criticality) and either auto-repairs small, well-understood divergences or flags larger ones for human review. This is deliberately a SEPARATE mechanism from the event-driven sync path, its job is to catch failures of that path (a dropped event no retry ever recovered, a bug in the consumer's apply logic), not to be the primary way B stays in sync (that would defeat the point of event-driven propagation in the first place).

Worked example. Service A (an Orders service) publishes OrderUpdated{order_id, sequence, payload} events. Service B (a search index) consumes them, checking (order_id, sequence) against the last sequence it applied for that order, skipping (as an idempotent no-op) any event with a sequence it's already seen or older, and buffering (briefly) any event that arrives out of order, applying it once the gap is filled or timing it out into a "request full replay for this order_id" fallback if the gap doesn't close. Nightly, a reconciliation job compares a sample (or full set, for smaller datasets) of orders between A's source of truth and B's index, flagging any order where B's data doesn't match A's for investigation, this is how the team discovered a bug where B's consumer was silently dropping events during a brief scaling event, well before any customer noticed stale search results.

Trade-offs and pitfalls. Skipping the reconciliation job because "the event pipeline is reliable" is a common and risky shortcut, event-driven consistency mechanisms fail in ways that are often invisible until reconciliation (or a customer complaint) surfaces them, since a missed event usually produces no error, just quietly stale data.

Direct answer: 2PC can block indefinitely because the commit decision lives in exactly one place, the coordinator's durable log. If the coordinator crashes after collecting votes but before every participant has received the decision, a participant that already voted yes cannot safely guess the outcome, so it must sit holding its locks until the coordinator (or someone with equivalent information) comes back. Three-phase commit (3PC) tries to fix this by adding an extra round, but it depends on assumptions that don't hold in real networks, so it's essentially never used.

Structured elaboration

Why 2PC blocks. The failure scenario is specific: all participants voted yes, so none of them may unilaterally abort (that would break atomicity if the coordinator had already decided to commit). But without the coordinator's decision, a participant also doesn't know it's safe to commit. It's stuck in an "in-doubt" state that only resolves once it learns the real outcome, either the coordinator recovers, or another participant that happens to already know the answer tells it (this only works if such a participant exists and can be reached).

What 3PC changes. 3PC inserts a "pre-commit" phase between prepare and commit: after everyone votes yes, the coordinator broadcasts PRE-COMMIT and waits for acknowledgments before sending the final COMMIT. The idea is that once a majority of participants have seen PRE-COMMIT, they know a commit decision was reached and can safely commit even without hearing directly from the coordinator, because a pre-commit message could only have been sent after unanimous yes votes.

Why 3PC still doesn't solve it in practice. The non-blocking property of 3PC relies on synchronous system assumptions: a known upper bound on message delay and processing time, so that a timeout reliably distinguishes "the coordinator crashed" from "the coordinator is just slow." Real networks are asynchronous: you cannot tell a slow coordinator from a dead one purely by waiting. If participants time out and elect a new coordinator while the old one is actually still alive but partitioned, you can get two coordinators making conflicting decisions, a split-brain that can violate atomicity, exactly the thing the protocol exists to prevent. 3PC also costs an extra network round-trip on every transaction, for a safety property it only delivers under an assumption that doesn't hold in production.

Non-blocking alternatives actually used in practice

Worked example of the blocking window. Coordinator collects yes votes from participants P1 and P2, durably logs "commit", sends COMMIT to P1 (which applies it and moves on), then crashes before the message to P2 goes out. P2 is now holding its locks with no way to know the transaction committed. If P2 tries to reach P1, P1 can honestly tell it "I got COMMIT", which lets P2 also commit safely, that's the one case where a peer can rescue an in-doubt participant. If P1 is unreachable too, P2 has no choice but to keep waiting for the coordinator to restart.

Trade-offs and pitfalls. The most common mistake is treating "we compared timeouts and picked a value" as if it solves the blocking problem, it only bounds the WORST-case wait, it doesn't remove the possibility that the guess made after the timeout is wrong. Anyone proposing 3PC in an interview should be able to name the synchrony assumption it needs and explain why that's the actual reason it isn't deployed, not just "it's more complex."