Data Consistency and Distributed Transactions Questions

Direct answer: Migrating a monolith with a single ACID database into microservices without breaking correctness means splitting data ownership incrementally (the strangler pattern), routing writes through a transitional layer that keeps both the old monolith's transactions and the new services' local transactions consistent during the cutover, and using the transactional outbox plus change-data-capture to keep systems in sync without ever relying on a distributed transaction across the boundary.

Structured elaboration

Splitting data ownership. Identify the bounded contexts that should become independent services, and for each, decide which tables it will own outright once migrated. The hard part is tables currently touched by transactions that ALSO touch tables destined for a different service, these are exactly the places where the monolith's implicit ACID guarantee is doing work you'll need to replace explicitly (with a saga, or by keeping that specific operation inside the monolith longer).

The strangler pattern. Rather than a big-bang cutover, new functionality (and gradually, migrated old functionality) is routed to the new service while the monolith continues serving everything not yet migrated, with a routing layer (a proxy, feature flag, or API gateway rule) deciding which implementation handles a given request. This lets you migrate one bounded context at a time, validating each in production before moving to the next, rather than betting the whole system on one cutover.

Anti-corruption layer. During the transition, the new service and the still-live monolith need to exchange data without either directly depending on the other's internal schema, an anti-corruption layer translates between the monolith's data model and the new service's model, so schema changes on one side don't ripple uncontrolled into the other, and so the eventual removal of the monolith's involvement doesn't require rewriting the new service's own model.

Transactional outbox for dual writes. While a table is transitioning between being monolith-owned and service-owned, changes often need to reach both temporarily (e.g. the monolith's own downstream consumers still expect data in the old location, while the new service needs the same data flowing in). The outbox pattern, or CDC tailing the monolith's write-ahead log via a tool like Debezium, lets you propagate changes reliably without adding synchronous coupling between the monolith and the new service.

Handling cross-service transactions during the transition. Any operation that used to be one ACID transaction spanning tables now owned by different services needs to become a saga (with compensating actions) for the transition period, even if, post-migration, it will stay a saga permanently. This is usually the highest-risk part of the migration and deserves the most testing.

Testing strategy. Dual-write validation (compare the monolith's and the new service's view of the same data for a period, alerting on divergence) before fully cutting over reads to the new service; shadow traffic (routing a copy of production traffic to the new path without it affecting real responses) to validate correctness under real load before it's authoritative.

Backfill. Before the new service can serve any read traffic authoritatively, it needs the monolith's HISTORICAL data, not just newly dual-written changes going forward. A backfill job copies existing rows from the monolith's tables into the new service's store (typically in controlled batches, to avoid overloading either database), and needs to correctly interleave with the dual-write path already running concurrently: any row already updated by a dual-write after the backfill's snapshot point must not be overwritten by an older backfilled version, usually handled by having the backfill skip (or take the newer of) any row the dual-write path has already touched.

Rollback plan. Every migration phase needs an explicit "how do we go back" answer: keep the monolith's code path live and the routing flag flippable back to it for a defined window after each phase, and keep dual-writing (or replicating back from the new service to the monolith's tables) until you're confident enough to remove that safety net.

Worked example. Migrating an Orders bounded context: phase 1, the monolith keeps writing orders as before, but a CDC-based outbox also streams every order change to the new Orders service's own database (read-only shadow, validated against the monolith for a few weeks). Phase 2, new order CREATION is routed through the new service (strangler cutover for writes), which now writes to its own database first and publishes events the monolith consumes via an anti-corruption adapter to keep its own legacy views current. Phase 3, once nothing else depends on the monolith's order tables, they're decommissioned, and any operation that used to be one transaction (e.g. "create order and reserve inventory," if inventory was also migrated) is now a saga with its own compensating actions.

Trade-offs and pitfalls. The riskiest, most commonly underestimated part is operations that span the migration boundary during the transition (a table half-migrated, half still monolith-owned), teams that skip explicit dual-write validation for these tend to discover divergence only when a customer reports incorrect data, well after the fact.

Direct answer: Since you can't use a single distributed database, the choice is between a saga that debits and credits as two independently-committing local transactions with a compensating credit-back if the second step fails, or a 2PC-style protocol where both shards prepare before either commits. Given the audit-trail and fund-conservation requirements, a saga with an idempotent, carefully sequenced compensation plus a durable audit log of every step is the more operationally realistic choice for most systems at this scale; 2PC is defensible only if the two services are both under your control and the transfer volume is low enough that its availability cost is acceptable.

Structured elaboration

Why not 2PC by default here. 2PC would give you genuine atomicity, but "moving money between accounts managed by separate services" strongly suggests these services can evolve independently, may not share an operational team, and the transfer needs to keep working even if one service is temporarily slow. 2PC's blocking failure mode (a coordinator crash stranding a participant holding a lock on someone's balance) is a worse operational risk here than a saga's eventual-consistency window, provided the saga is designed so no external party ever sees an impossible intermediate state.

Saga design for the transfer.

1. Reserve/hold on the source account (not an unconditional debit): decrement available balance, but keep it recoverable, this is itself a local ACID transaction on the source service, and only requires the source service's own database, no cross-service coordination.
2. Credit the destination account. If this succeeds, finalize the hold on the source (converting the reservation into a real, permanent debit) and record both legs as complete in an audit log entry keyed by a single transfer ID.
3. Compensating action if the credit fails (destination account invalid, destination service unavailable after retries, business rule violation): release the hold on the source account, restoring its available balance, and mark the transfer as failed in the audit log with the failure reason.

Idempotency at every step. Every step (reserve, credit, finalize, compensate) must be safe to retry: keyed on the transfer ID, so a retried "reserve" call after a network timeout doesn't double-reserve, and a retried "compensate" call after a crash doesn't double-restore the balance. Without this, a saga's retry-on-failure model becomes a source of the exact double-charge or lost-money bugs it's supposed to prevent.

Auditability and fund conservation. Every state transition (reserved, credited, finalized, compensated) is appended to a durable, transfer-ID-keyed audit log BEFORE the corresponding service call is made, mirroring 2PC's "log before you act" discipline even though there's no coordinator. This log is what lets you prove, for any transfer ID, exactly what happened, and lets an operator (or an automated reconciliation job) find and resolve transfers that got stuck mid-flight, e.g. reserved on the source but the credit call to the destination never got a definitive answer.

Proving conservation of funds specifically. Per-transfer audit history is necessary but not sufficient, proving conservation requires an aggregate check across the whole ledger, not just tracing one transfer ID. A periodic (e.g. hourly) reconciliation job sums, per currency, every FINALIZED debit recorded in the audit log against every FINALIZED credit recorded in the audit log for the same time window; a healthy system nets to zero (double-entry style: total debits == total credits). The same job independently re-derives each account's expected balance by replaying its audit-log entries and diffs that against the account service's own reported balance, any nonzero diff on either check (net debits/credits not zero, or a re-derived balance that disagrees with the service of record) is flagged for investigation rather than silently trusted. Separately, any transfer whose audit log shows a non-terminal state (reserved or credited without a following finalized or compensated) older than the expected completion SLA is surfaced as a stuck transfer, this is what the reconciliation job in the earlier paragraph resolves operationally; the aggregate debit/credit check is what proves fund conservation across the system, a distinct and additional guarantee from resolving individual stuck transfers.

Failure handling for the specific "stuck" case. If the destination service call times out (not a clean success or failure, an ambiguous outcome), the transfer must NOT immediately compensate, that risks a double-credit if the destination actually did apply it before the timeout. Instead, the orchestrator retries a status check (using the transfer ID) against the destination service until it gets a definitive answer, or escalates to a reconciliation job after a bounded number of attempts.

Worked example. Transfer T-9182 moves $500 from account A (service X) to account B (service Y). Orchestrator logs "T-9182: reserve started" then calls X's reserve API with idempotency key T-9182-reserve; X durably decrements A's available balance and returns success; orchestrator logs "T-9182: reserved". Orchestrator calls Y's credit API with idempotency key T-9182-credit; suppose the call times out with no response. Orchestrator does NOT immediately compensate; instead it polls Y's GET /transfers/T-9182-credit/status (idempotent read) until Y reports either "applied" (in which case the orchestrator finalizes on X's side and logs completion) or "not found / rejected" (in which case it compensates: calls X's release-reservation API with idempotency key T-9182-release, and logs the transfer as failed).

Applies beyond money transfers. The same reserve-then-confirm shape protects any "two things must both happen or neither should" cross-service operation, not just fund transfers. A global booking platform preventing double-bookings across continents (e.g. holding a seat on a flight and a room at a hotel as one logical reservation) uses the identical pattern: a reservation/hold on each resource first, a saga that finalizes both holds together, and a fallback to consensus-backed coordination only for the rare, genuinely latency-tolerant cases where the business is willing to pay 2PC's availability cost for airtight atomicity across the booking.

Trade-offs and pitfalls. The single most common design mistake here is compensating too eagerly on an ambiguous failure (a timeout is not the same as a confirmed failure), which is exactly how "double the money existed briefly" bugs happen: the destination actually applied the credit, then the source got its reservation released too, creating money from nothing until reconciliation catches it, if it ever does. A reservation/hold step (rather than an immediate unconditional debit) is what makes the source-side compensation safe and reversible in the first place.

Direct answer: To make eventual consistency between service A and B idempotent and reconciliation-friendly, service A publishes events with a stable event ID (or a monotonic sequence number per entity), service B's consumer deduplicates on that ID before applying any change, and a periodic reconciliation job independently compares A's and B's views to catch and repair anything that slipped through despite the idempotency guarantees.

Structured elaboration

Idempotent event processing on the consumer side. Every event from A carries a stable identifier; B's consumer checks (atomically, alongside applying the event) whether that ID has already been processed, using the same "dedup record plus the actual state change in one transaction" discipline as any idempotent write. This is what makes at-least-once delivery (which any reasonable messaging setup between A and B will actually provide) safe: redelivery is a no-op rather than a duplicate application.

Preserving ordering where necessary. If events for the same entity must be applied in order (e.g. "created" before "updated" before "deleted"), B's consumer needs either a strictly-ordered delivery channel per entity (partition by entity ID) or an explicit sequence number in each event that B checks against the last-applied sequence for that entity, rejecting or buffering an out-of-order arrival rather than applying it prematurely.

Supporting replays. Because B's state can, despite everything, still drift from A's (a bug, an extended outage, a schema-migration mistake), the design should support REPLAYING A's full event history into B from scratch (or from a checkpoint) to rebuild B's view, which requires A to retain (or be able to regenerate) its event history for at least as long as any realistic replay window, and requires B's apply logic to be safe to run repeatedly over the same events (which it already is, by the idempotency design above).

Reconciliation as the safety net, not the primary mechanism. A periodic job independently compares A's and B's data (via checksums, row counts, or a full diff on a schedule appropriate to the data's size and criticality) and either auto-repairs small, well-understood divergences or flags larger ones for human review. This is deliberately a SEPARATE mechanism from the event-driven sync path, its job is to catch failures of that path (a dropped event no retry ever recovered, a bug in the consumer's apply logic), not to be the primary way B stays in sync (that would defeat the point of event-driven propagation in the first place).

Worked example. Service A (an Orders service) publishes OrderUpdated{order_id, sequence, payload} events. Service B (a search index) consumes them, checking (order_id, sequence) against the last sequence it applied for that order, skipping (as an idempotent no-op) any event with a sequence it's already seen or older, and buffering (briefly) any event that arrives out of order, applying it once the gap is filled or timing it out into a "request full replay for this order_id" fallback if the gap doesn't close. Nightly, a reconciliation job compares a sample (or full set, for smaller datasets) of orders between A's source of truth and B's index, flagging any order where B's data doesn't match A's for investigation, this is how the team discovered a bug where B's consumer was silently dropping events during a brief scaling event, well before any customer noticed stale search results.

Trade-offs and pitfalls. Skipping the reconciliation job because "the event pipeline is reliable" is a common and risky shortcut, event-driven consistency mechanisms fail in ways that are often invisible until reconciliation (or a customer complaint) surfaces them, since a missed event usually produces no error, just quietly stale data.

Direct answer: A saga coordinator that supports long-running transactions with human approvals needs an explicit, durable state machine (not just a sequence of automated steps) with a distinct "awaiting approval" state, timeout and escalation policies for approvals that never come, and the same durable-state-plus-idempotent-compensation discipline as any other saga, extended to cover a human-in-the-loop step that can take hours or days rather than milliseconds.

Structured elaboration

State machine design. Beyond the usual step states (pending, in_progress, completed, compensating, compensated), a human-approval step introduces awaiting_approval as a first-class state, entered when the automated part of a step completes and a human decision gates the next one (e.g. a large refund needing manager sign-off). The saga sits in this state, potentially for a long time, without any automated retries firing.

Durable state storage. The full saga state (current step, all prior steps' outcomes, any approval requests issued and their IDs) is persisted to a durable store on every transition, exactly like an automated saga, so an orchestrator restart doesn't lose track of a saga that's been sitting in awaiting_approval for two days.

Visibility for operators. Since a human has to act on the saga, there needs to be a queryable view (a dashboard or queue) listing every saga currently awaiting_approval, who it's assigned to or eligible to act on it, how long it's been waiting, and enough context (what's being approved, why) for the approver to decide without digging through logs.

Timeout and escalation policies. An approval request that sits unanswered past a threshold should escalate, notify a different approver, page an on-call, or auto-escalate to a higher authority level, rather than silently stalling forever. Some designs also define an explicit "default" outcome for a timed-out approval (auto-deny is usually safer than auto-approve for anything consequential).

Auditability and idempotent retries of compensating steps. Every approval decision (who approved, when, any notes) is recorded immutably alongside the saga's state, both for compliance and so a later dispute can be traced. Compensating actions triggered after a rejected approval must be idempotent exactly like any other saga compensation, since an operator might reject the same request twice (e.g. a UI double-click), or a retry might be triggered by an infrastructure blip rather than a genuine second action.

Worked example. A saga for a $50,000 vendor payment reaches step "approve large payment," transitions to awaiting_approval, and creates an approval request visible in an approvals queue with the amount, vendor, and originating order context. Two approvers are eligible; the saga waits. After 24 hours with no response, an escalation job fires: it notifies a backup approver and pages a manager. The manager approves at hour 30; the saga transitions to approved and proceeds to the payment-execution step, using the approval-request ID as part of the idempotency key for that step so a duplicate approval click doesn't trigger two payments.

Trade-offs and pitfalls. A common design mistake is treating the approval step like any other automated step with a short timeout and automatic retry, which either spams the approver with duplicate notifications or, worse, silently drops the request if a retry policy assumes failure after a short window. Human-in-the-loop steps need their OWN timeout scale (hours/days, not seconds) and their own explicit escalation path, not an automated saga's default retry/backoff policy.

Direct answer: Monotonic reads means a client never sees data go backward in time across successive reads (each read is at least as current as the last one it saw); monotonic writes means a client's own writes are applied in the order it issued them, even if they land on different nodes. Both are properties about a SINGLE client's own sequence, not about how fast the whole system converges. Testing them means deliberately injecting the conditions that would violate them (routing successive reads to different, differently-lagging replicas; sending writes with induced delay/reordering) and asserting the guarantee holds despite that.

Structured elaboration

Monotonic reads, precisely. If client C reads value V1 at time T1, then reads again at T2 > T1, the second read must return V1 or something NEWER, never something OLDER, even if the second read happens to be served by a different, more-lagging replica than the first. Without this, a user could refresh a page and see data revert, confusing regardless of eventual correctness.

Monotonic writes, precisely. If client C issues write W1 then write W2, the system guarantees W1 is applied before W2 EVERYWHERE, even if W1 and W2 are processed by different nodes and could otherwise race. Without this, W2 could be visible while W1 is still pending (or never lands, if a naive "last write wins" scheme picks W1 as later due to clock skew), an out-of-order result from the client's own perspective.

Test design

Injected failures/conditions. Partitions (temporarily cut connectivity between specific replicas to control which one a client's next request gets routed to and how stale it is), induced delay (deliberately slow down message delivery for one of a client's writes to test whether the system still enforces the intended order), and controlled replica staleness (a test replica deliberately held back from applying recent writes, to serve as a "known-stale" target for a monotonic-reads assertion).

Automated assertions.

• For monotonic reads: perform read R1, record its version/timestamp; force the NEXT read to be routed to a replica with LOWER known freshness than what R1 returned; assert the second read either returns data at least as fresh as R1 (correct: the routing layer detected the staleness and either waited or redirected) or the test explicitly confirms the system PREVENTS routing to that stale replica for this client's session in the first place.
• For monotonic writes: issue W1, then immediately issue W2 with W1 deliberately delayed at the network layer so it could arrive at its target AFTER W2 does; assert that after both eventually land, a read reflects W1 applied before W2 (i.e. the system enforced ordering rather than applying whichever arrived first).

Test harness components. A controllable network-simulation layer (able to inject delay, partition, or reordering between specific client-to-replica or replica-to-replica links), a way to pin a specific test read to a specific, known-state replica (bypassing normal load-balancing for the test), and an assertion library that compares returned data against expected VERSION/ordering, not just content, since two reads could have the "same" content while still violating a freshness guarantee if the test doesn't check versions explicitly.

Worked example. Test: client writes profile update W1 (bio = "A"), the test harness delays W1's replication to replica R2 by 5 seconds; client immediately writes W2 (bio = "B") which, without monotonic-write enforcement, could reach R2 (via a different path) before the delayed W1. The harness then forces a read from R2 and asserts the final visible bio is "B" (both applied, in the correct order) not "A" (only W1 applied, arriving late and incorrectly overwriting W2 due to naive last-write-wins on arrival time rather than causal order). The passing assertion confirms the system used causal/session ordering (e.g. a per-client sequence number) rather than raw arrival-time ordering.

Trade-offs and pitfalls. A common testing gap is asserting only on the FINAL converged state (eventually, W1 and W2 both apply and content looks right) without checking the INTERMEDIATE observable behavior a real client session would see, monotonic guarantees are specifically about what's visible ALONG THE WAY, not just the eventual steady state, so tests need to assert on reads taken DURING the injected delay/partition window, not only after everything settles.