Data Consistency and Distributed Transactions Questions
Maintaining correctness of state across services and replicas: eventual consistency, conflict resolution (last-write-wins, CRDTs, vector clocks), the saga pattern, two-phase commit, and idempotency keys for exactly-once effects. Covers when to trade strict consistency for availability and how to reason about read-your-writes and monotonic guarantees. Focuses on the application/service layer rather than storage-engine internals.
An enterprise needs eventual consistency between service A and service B using events. Design an idempotent event processing and reconciliation strategy that guarantees convergence and supports replays, while preserving ordering where necessary.
Sample Answer
Direct answer: To make eventual consistency between service A and B idempotent and reconciliation-friendly, service A publishes events with a stable event ID (or a monotonic sequence number per entity), service B's consumer deduplicates on that ID before applying any change, and a periodic reconciliation job independently compares A's and B's views to catch and repair anything that slipped through despite the idempotency guarantees.
Structured elaboration
Idempotent event processing on the consumer side. Every event from A carries a stable identifier; B's consumer checks (atomically, alongside applying the event) whether that ID has already been processed, using the same "dedup record plus the actual state change in one transaction" discipline as any idempotent write. This is what makes at-least-once delivery (which any reasonable messaging setup between A and B will actually provide) safe: redelivery is a no-op rather than a duplicate application.
Preserving ordering where necessary. If events for the same entity must be applied in order (e.g. "created" before "updated" before "deleted"), B's consumer needs either a strictly-ordered delivery channel per entity (partition by entity ID) or an explicit sequence number in each event that B checks against the last-applied sequence for that entity, rejecting or buffering an out-of-order arrival rather than applying it prematurely.
Supporting replays. Because B's state can, despite everything, still drift from A's (a bug, an extended outage, a schema-migration mistake), the design should support REPLAYING A's full event history into B from scratch (or from a checkpoint) to rebuild B's view, which requires A to retain (or be able to regenerate) its event history for at least as long as any realistic replay window, and requires B's apply logic to be safe to run repeatedly over the same events (which it already is, by the idempotency design above).
Reconciliation as the safety net, not the primary mechanism. A periodic job independently compares A's and B's data (via checksums, row counts, or a full diff on a schedule appropriate to the data's size and criticality) and either auto-repairs small, well-understood divergences or flags larger ones for human review. This is deliberately a SEPARATE mechanism from the event-driven sync path, its job is to catch failures of that path (a dropped event no retry ever recovered, a bug in the consumer's apply logic), not to be the primary way B stays in sync (that would defeat the point of event-driven propagation in the first place).
Worked example. Service A (an Orders service) publishes OrderUpdated{order_id, sequence, payload} events. Service B (a search index) consumes them, checking (order_id, sequence) against the last sequence it applied for that order, skipping (as an idempotent no-op) any event with a sequence it's already seen or older, and buffering (briefly) any event that arrives out of order, applying it once the gap is filled or timing it out into a "request full replay for this order_id" fallback if the gap doesn't close. Nightly, a reconciliation job compares a sample (or full set, for smaller datasets) of orders between A's source of truth and B's index, flagging any order where B's data doesn't match A's for investigation, this is how the team discovered a bug where B's consumer was silently dropping events during a brief scaling event, well before any customer noticed stale search results.
Trade-offs and pitfalls. Skipping the reconciliation job because "the event pipeline is reliable" is a common and risky shortcut, event-driven consistency mechanisms fail in ways that are often invisible until reconciliation (or a customer complaint) surfaces them, since a missed event usually produces no error, just quietly stale data.
Compare last-writer-wins (LWW) conflict resolution against CRDTs for achieving eventual consistency. In what scenarios is LWW acceptable, and when do CRDTs become the better fit? What operational costs does adopting CRDTs add (metadata growth, merge cost)?
Sample Answer
Direct answer: LWW is acceptable for fields where losing one of two concurrent updates silently is a genuinely fine outcome, typically single-valued fields with no natural merge semantic; CRDTs become the better fit once you need EVERY concurrent update's information to survive the merge, at the real cost of extra implementation complexity and ongoing metadata management that LWW doesn't require.
Structured elaboration
Decision criteria. Ask, for the specific field or data type in question: does losing one of two concurrent writes represent real information loss a user or the business would care about? If no (the field is naturally "replace with newest," like a status flag or a cached external value), LWW is the simpler, sufficient choice. If yes (both concurrent writes carry information that should survive, like a counter, a set of tags, or collaborative document content), you need something that can actually MERGE both updates rather than pick one, which is what CRDTs are designed for.
Recommended approach by scenario.
- Collaborative document edits: CRDTs (specifically a sequence CRDT for the text itself), since LWW at any granularity coarser than individual characters would discard entire concurrent edits, exactly the failure this domain can't tolerate.
- Offline mobile updates to a simple status field (e.g. "is this task marked complete"): LWW is often fine here, since a boolean "complete" flag genuinely has a "replace with the latest state" semantic, there's no meaningful way to "merge" two different completion states, one has to win, and losing the earlier one is the intended behavior, not a bug.
- Offline mobile updates to a LIST of items (tags, cart contents): CRDT (OR-Set), for the same reason argued in the shopping-cart example elsewhere in this topic, concurrent additions from different devices need to all survive.
Operational cost of adopting CRDTs. CRDTs require modeling your data as CRDT-compatible types (which sometimes means restructuring a field that was a simple scalar into a CRDT-native shape), implementing or adopting a CRDT library correctly (subtle correctness bugs are easy to introduce, see the CRDT test-strategy discussion elsewhere in this topic), and managing metadata growth over time (tombstones, per-replica vectors) that a plain LWW-timestamped field never accumulates. This is real, ongoing engineering investment, not a one-time setup cost, worth weighing explicitly against the value of not losing concurrent updates for a specific field.
Merge cost comparison. LWW's "merge" is a single timestamp comparison, O(1), trivial. A CRDT merge, depending on type, ranges from cheap (a G-Counter's element-wise max) to more involved (a sequence CRDT's structural merge, which can be non-trivial for large documents with long edit histories), a real performance consideration at scale that LWW simply doesn't have to think about.
Worked example. A team building a multi-device note-taking app initially applied LWW to the WHOLE note object (title + body + tags combined). Users editing the same note from two devices while both were briefly offline (a common case: editing on a phone during a commute, then opening a laptop before the phone had synced) would lose one device's entire set of edits. The team's fix: keep LWW for the title (a single-valued field where "replace with latest" is genuinely fine, and users rarely have GENUINE conflicting title edits from two devices), but move the body to an operational sequence CRDT (needed to make character-level concurrent edits, not just whole-note replacement, actually merge) and tags to an OR-Set. This targeted, per-field decision (rather than a blanket choice either way) resolved the actual complaint (lost body content) without over-engineering the title field, which never needed CRDT machinery.
Trade-offs and pitfalls. Adopting CRDTs uniformly across an entire object "to be safe," rather than deciding per-field based on the criteria above, is a common over-engineering response to a real LWW data-loss incident, it solves the problem but at unnecessary implementation and metadata cost for fields that never needed it.
Tell me about a time you were on-call for a production data-consistency incident. Using the STAR format: what was the symptom and impact, what did you investigate, what immediate mitigation did you take, and what long-term fixes and postmortem actions did you implement?
Sample Answer
Direct answer: A strong STAR-format answer here names a specific incident (real symptom, real investigation, real fix), not a generalized description of what one WOULD do; the structure is: what broke and who noticed, what you actually looked at to find the cause, what you did immediately to stop user impact, and what changed afterward so it doesn't recur.
Structured elaboration
Situation/Task. Set the scene concretely: what system, what the symptom looked like from the outside (e.g. "customers reported seeing stale order statuses" or "a reconciliation alert fired showing balances didn't reconcile across two services"), and how it was first noticed (a customer report, an automated alert, an internal dashboard anomaly), since HOW an incident is detected shapes the urgency and the first investigative steps.
Action, investigation. What signals were checked, in what order, and why: logs correlated by a request or transaction ID, metrics showing when the anomaly started (to narrow down what changed around that time, a deploy, a config change, a dependency's own incident), traces showing where in a multi-service flow things diverged from expected. A strong answer explains the REASONING behind each investigative step, not just a list of tools checked.
Action, immediate mitigation. What was done to stop user-visible impact BEFORE the root cause was fully understood, this is often different from the eventual fix: routing traffic away from a bad region, disabling a feature flag, manually correcting a small number of affected records, or putting a system into a degraded-but-safe mode. Naming a concrete mitigation (not just "we investigated and fixed it") is what distinguishes a real incident story from a rehearsed generality.
Result and long-term fixes. What the actual root cause turned out to be, what was changed to prevent recurrence (a code fix, a new alert, a process change, an architectural change like adding a missing idempotency check or reconciliation job), and what came out of the postmortem, ideally including something that DIDN'T go perfectly (a delay in detection, a wrong initial hypothesis that wasted time), since acknowledging that shows genuine reflection rather than a polished-but-shallow narrative.
Worked example (illustrative shape, not a template to recite verbatim). "A reconciliation job alerted that a subset of orders showed 'paid' in the payments service but 'unconfirmed' in the orders service, a saga stuck mid-flight. I first checked the saga's audit log for the affected order IDs and found the payment step had completed successfully but the final 'confirm order' step had never fired, the orchestrator process had been restarted during a deploy at almost the exact time those sagas were mid-flight, and its resume-on-restart logic had a bug that skipped sagas already past the payment step. Immediate mitigation: I manually re-ran the confirm-order step for the ~40 affected orders, verified against the payment service's own record before doing so, and disabled further deploys of that orchestrator until the resume logic was fixed. Root cause: the resume logic filtered on the WRONG saga state field after a recent schema change. Long-term fix: corrected the resume filter, added an integration test specifically covering resume-after-restart for a saga past its midpoint, and added a dedicated alert for 'stuck saga older than expected duration' so this class of issue would be caught automatically rather than only via the periodic reconciliation job."
Trade-offs and pitfalls. A common weak answer describes the FIX in detail but glosses over the investigation (how you actually narrowed down the cause) and the immediate mitigation (what stopped user impact before the fix shipped), both of which are usually what an interviewer is actually trying to assess: judgment under uncertainty, not just eventual code-level correctness.
Design a scalable approach to support atomic increments for a counter that's sharded across many keys (for example, a global 'likes' count). Compare a few approaches (per-shard counters with periodic aggregation, CRDT counters, a central counter service, optimistic CAS-based increments) on accuracy, throughput, read latency, and reconciliation cost.
Sample Answer
Direct answer: For a sharded counter that needs to keep incrementing correctly under high concurrency without coordination, a CRDT counter (a PN-Counter, sharded by key with one counter object per key) gives exact eventual convergence with no central bottleneck; per-shard counters with periodic aggregation trade a small aggregation delay for simpler infrastructure; a central counter service gives immediate global consistency at the cost of becoming a write bottleneck; and optimistic CAS (compare-and-swap)-based increments work for moderate contention but degrade under very high concurrent write rates on the same key.
Structured elaboration
CRDT counter (PN-Counter per key). Each region/replica maintains its own local increment count for a given counter key; merging (element-wise max plus sum) converges to the exact correct total with no coordination required at write time. Accuracy is eventually exact (once all increments have propagated and merged, the total is precisely correct, nothing is approximated or lost). Storage overhead is one entry per replica per counter key, and merge/read cost scales with the number of replicas, not the number of increments, since increments to the same replica's entry just add to that entry rather than each needing its own record.
Per-shard counters with periodic aggregation. Instead of a CRDT's continuous merge, each shard just accumulates its own local count independently, and a periodic batch job sums all shards' counts to produce the displayed total. Simpler to build (no CRDT library or merge logic needed), but the displayed total is only as fresh as the last aggregation run, real-time reads see a stale, under-counted total between aggregation cycles, an explicit trade-off, not a bug, if the use case tolerates it.
Central counter service. A single service (backed by its own strongly-consistent store) owns the counter and serializes all increments through it. Gives immediate, always-accurate reads with no aggregation delay, but throughput is capped by that single service's write capacity, and it becomes a single point of contention (and, without careful design, a single point of failure) as write volume grows, exactly the scaling problem sharding elsewhere in the system was meant to avoid.
Optimistic CAS-based increments. Each increment reads the current value, computes the new value, and writes it back with a compare-and-swap (only succeeding if the value hasn't changed since the read); a losing CAS retries. Works well at moderate contention (few concurrent writers per key), but retry rate grows sharply as concurrent writers to the SAME key increase, under very high contention (many writers hammering one popular key, a "hot key"), this can degrade into a large fraction of writes retrying repeatedly, hurting both latency and throughput.
Comparison
| Approach | Accuracy | Throughput under high concurrency | Read latency | Reconciliation needed |
|---|---|---|---|---|
| CRDT (PN-Counter) | Eventually exact | High, no coordination on write | Read = merge of current replica states, cheap | None; merge IS the reconciliation |
| Per-shard + periodic aggregation | Exact as of last aggregation, else stale | Very high, purely local writes | Fast but stale between cycles | The aggregation job itself |
| Central counter service | Always exact, immediately | Bounded by the service's own capacity | Fast, single source of truth | None needed, but scaling requires sharding the service itself eventually |
| Optimistic CAS | Always exact, immediately | Degrades under high same-key contention | Fast when uncontended | None; but retry storms under contention are a real operational risk |
Worked example. A "post likes" counter under a viral post might receive thousands of concurrent increments per second from a single popular key. A central counter service or CAS-based approach would see heavy contention specifically on that one hot key; a PN-Counter sharded across regions handles it gracefully since each region's writers only contend with OTHER writers in the SAME region (much lower local contention), converging the true global total asynchronously with no single bottleneck.
Trade-offs and pitfalls. Teams sometimes default to a central counter service for simplicity and only discover the bottleneck once a specific key goes viral or otherwise becomes hot, worth explicitly asking during design "what's the expected max increment rate on a SINGLE key," not just the aggregate system-wide rate, since that's what determines whether central/CAS approaches will hold up.
Explain how vector clocks detect concurrent updates in an eventually-consistent key-value store. Give a small example where two nodes concurrently update the same key and produce divergent versions, and contrast that with last-write-wins (LWW) conflict resolution: where is LWW acceptable, and where does it cause data loss?
Sample Answer
Direct answer: A vector clock is a per-replica counter vector that lets a distributed system detect when two updates happened concurrently (neither caused the other) rather than one clearly happening after the other, which last-write-wins (LWW) can't distinguish at all, it just picks whichever write has the later timestamp, even if that timestamp is wrong due to clock skew.
Structured elaboration
How a vector clock works. Each replica keeps a vector of counters, one per replica in the system, e.g. {A: 0, B: 0} for a two-replica system. When replica A processes a local update, it increments its OWN entry: {A: 1, B: 0}. When A sends this update to replica B, B merges the incoming vector with its own (taking the element-wise max) and increments its own entry for the local receive event. Comparing two vectors tells you their relationship: if every entry in vector V1 is <= the corresponding entry in V2 (and at least one is strictly less), V1 happened-before V2. If neither vector dominates the other (some entries higher in V1, others higher in V2), the two updates are CONCURRENT, neither caused the other, and there's a real conflict to resolve.
Worked example. Two replicas, A and B, both start with a key's vector clock at {A:0, B:0}.
- Replica A updates the key locally: its clock becomes
{A:1, B:0}. - Before A's update reaches B, replica B ALSO updates the same key locally (independently): its clock becomes
{A:0, B:1}. - When these two updates are compared (e.g. during replication or a read that sees both), neither
{A:1, B:0}nor{A:0, B:1}dominates the other, A's entry is higher in the first, B's is higher in the second. This is detected as a genuine concurrent conflict: two updates happened without either replica having seen the other's change first.
Contrast with last-write-wins. LWW attaches a wall-clock timestamp to each write and, on conflict, simply keeps whichever timestamp is later, discarding the other write entirely. In the example above, LWW would compare A's and B's write timestamps and silently drop one of the two updates, with NO signal that a conflict even happened, and worse, if the two replicas' clocks are skewed (a common reality, NTP, Network Time Protocol, drift, VM pauses), the "later" timestamp might not even correspond to the update that actually happened more recently in real time.
Where LWW is acceptable. Low-stakes data where losing a concurrent update silently is an acceptable cost, and where the data model doesn't naturally support merging (a single scalar field with no sensible "combine both" semantic, e.g. a user's display name, there's no meaningful way to "merge" two different names, so picking one, even somewhat arbitrarily, is a reasonable trade for simplicity).
Where LWW causes real data loss. Anything where BOTH concurrent updates carry information that matters and shouldn't be silently dropped: a shopping cart where two devices concurrently add different items (LWW would keep only one device's additions), a counter (incrementing from two replicas concurrently under LWW loses one increment entirely, rather than correctly summing both), or any financial or inventory field where silently discarding a concurrent update is a correctness bug, not a UX nuisance.
Trade-offs and pitfalls. Vector clocks tell you a conflict EXISTS, they don't tell you how to RESOLVE it, that's a separate design decision (merge the two values, prompt the user, apply a domain-specific rule). This is the single most important nuance to state explicitly: reaching for vector clocks alone doesn't solve conflict resolution, it's the detection mechanism, not the resolution mechanism.
Unlock Full Question Bank
Get access to all 42 Data Consistency and Distributed Transactions interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.