Data Consistency and Distributed Transactions Questions
Maintaining correctness of state across services and replicas: eventual consistency, conflict resolution (last-write-wins, CRDTs, vector clocks), the saga pattern, two-phase commit, and idempotency keys for exactly-once effects. Covers when to trade strict consistency for availability and how to reason about read-your-writes and monotonic guarantees. Focuses on the application/service layer rather than storage-engine internals.
Compare last-writer-wins (LWW) conflict resolution against CRDTs for achieving eventual consistency. In what scenarios is LWW acceptable, and when do CRDTs become the better fit? What operational costs does adopting CRDTs add (metadata growth, merge cost)?
Sample Answer
Direct answer: LWW is acceptable for fields where losing one of two concurrent updates silently is a genuinely fine outcome, typically single-valued fields with no natural merge semantic; CRDTs become the better fit once you need EVERY concurrent update's information to survive the merge, at the real cost of extra implementation complexity and ongoing metadata management that LWW doesn't require.
Structured elaboration
Decision criteria. Ask, for the specific field or data type in question: does losing one of two concurrent writes represent real information loss a user or the business would care about? If no (the field is naturally "replace with newest," like a status flag or a cached external value), LWW is the simpler, sufficient choice. If yes (both concurrent writes carry information that should survive, like a counter, a set of tags, or collaborative document content), you need something that can actually MERGE both updates rather than pick one, which is what CRDTs are designed for.
Recommended approach by scenario.
- Collaborative document edits: CRDTs (specifically a sequence CRDT for the text itself), since LWW at any granularity coarser than individual characters would discard entire concurrent edits, exactly the failure this domain can't tolerate.
- Offline mobile updates to a simple status field (e.g. "is this task marked complete"): LWW is often fine here, since a boolean "complete" flag genuinely has a "replace with the latest state" semantic, there's no meaningful way to "merge" two different completion states, one has to win, and losing the earlier one is the intended behavior, not a bug.
- Offline mobile updates to a LIST of items (tags, cart contents): CRDT (OR-Set), for the same reason argued in the shopping-cart example elsewhere in this topic, concurrent additions from different devices need to all survive.
Operational cost of adopting CRDTs. CRDTs require modeling your data as CRDT-compatible types (which sometimes means restructuring a field that was a simple scalar into a CRDT-native shape), implementing or adopting a CRDT library correctly (subtle correctness bugs are easy to introduce, see the CRDT test-strategy discussion elsewhere in this topic), and managing metadata growth over time (tombstones, per-replica vectors) that a plain LWW-timestamped field never accumulates. This is real, ongoing engineering investment, not a one-time setup cost, worth weighing explicitly against the value of not losing concurrent updates for a specific field.
Merge cost comparison. LWW's "merge" is a single timestamp comparison, O(1), trivial. A CRDT merge, depending on type, ranges from cheap (a G-Counter's element-wise max) to more involved (a sequence CRDT's structural merge, which can be non-trivial for large documents with long edit histories), a real performance consideration at scale that LWW simply doesn't have to think about.
Worked example. A team building a multi-device note-taking app initially applied LWW to the WHOLE note object (title + body + tags combined). Users editing the same note from two devices while both were briefly offline (a common case: editing on a phone during a commute, then opening a laptop before the phone had synced) would lose one device's entire set of edits. The team's fix: keep LWW for the title (a single-valued field where "replace with latest" is genuinely fine, and users rarely have GENUINE conflicting title edits from two devices), but move the body to an operational sequence CRDT (needed to make character-level concurrent edits, not just whole-note replacement, actually merge) and tags to an OR-Set. This targeted, per-field decision (rather than a blanket choice either way) resolved the actual complaint (lost body content) without over-engineering the title field, which never needed CRDT machinery.
Trade-offs and pitfalls. Adopting CRDTs uniformly across an entire object "to be safe," rather than deciding per-field based on the criteria above, is a common over-engineering response to a real LWW data-loss incident, it solves the problem but at unnecessary implementation and metadata cost for fields that never needed it.
Describe a zero-downtime migration strategy to change a service's consistency model from strong to eventual. Include feature flags, dual-writes, read-path toggles, monitoring to verify correctness, and a rollback path if anomalies appear. Explain how you would validate data correctness throughout the migration.
Sample Answer
Direct answer: A zero-downtime migration from strong to eventual consistency uses feature flags to control which consistency path is active, dual-writes to keep both models correct during the transition, a read-path toggle so you can validate the new (eventually-consistent) reads against the old (strong) ones before trusting them, ongoing monitoring to catch correctness regressions early, and a rollback plan that doesn't require another migration to execute.
Structured elaboration
Feature flags. A flag (ideally per-tenant or percentage-rollout capable, not a single global switch) controls whether a given request path uses the old strongly-consistent read/write logic or the new eventually-consistent one, letting you migrate gradually and limit the blast radius of any problem to whatever slice of traffic the flag currently covers.
Dual-writes. During the transition, writes go to BOTH the old strongly-consistent path and the new eventually-consistent path (e.g. writing synchronously to the primary datastore as before, while also publishing to whatever asynchronous replication or eventual-consistency mechanism the new path relies on). This keeps both models populated and comparable while neither is fully trusted alone yet.
Read-path toggle plus shadow reads. Before actually SERVING reads from the new eventually-consistent path, run it in shadow mode: for a sample of requests, read from both paths, compare the results, and log any divergence, without letting the new path's result affect the response the user actually sees. This surfaces correctness bugs (staleness beyond expected bounds, a merge bug, a missed write) before they're customer-visible.
Monitoring to verify correctness. Track divergence rate between the two paths (from shadow reads), staleness distribution on the new path (how far behind is a typical eventually-consistent read, and what's the tail), and business-relevant anomaly signals specific to the domain (e.g. for inventory, oversell events). These metrics are what tell you when it's actually safe to flip more traffic to the new path, not just elapsed time or gut feel.
Rollback path. Because writes are dual-written throughout the migration, rolling back is just flipping the feature flag back to serve reads from the still-current strongly-consistent path, no data migration or backfill needed, since that path was never stopped. This is why dual-writing throughout the transition (rather than a hard cutover) is the key enabler of true zero-downtime rollback.
Worked example. A product-catalog service migrates from a single strongly-consistent primary to a multi-region, eventually-consistent replicated store. Phase 1: dual-write catalog updates to both the existing primary and the new replicated store; reads still come exclusively from the primary. Phase 2: enable shadow reads for 5% of traffic, comparing primary reads against the new store's reads, this surfaces that replication lag occasionally exceeds 2 seconds during peak write bursts, more staleness than the product team is comfortable with, so the team adds a bounded-staleness check (reject a shadow-read comparison, and hold back cutover, if lag exceeds a threshold) before proceeding. Phase 3: once divergence and staleness metrics are within agreed bounds for a sustained period, gradually flip the read-path flag to serve real traffic from the new store, monitoring the same metrics the whole time, with the flag remaining flippable back to the primary for weeks afterward as a safety net.
Trade-offs and pitfalls. Skipping the shadow-read validation phase (going straight from dual-write to serving real traffic from the new path) is the most common shortcut that causes incidents, correctness bugs in an eventually-consistent path often only show up under real production write patterns and timing, not in staging, and shadow reads are what let you catch them without a customer-visible impact.
Design a scalable approach to support atomic increments for a counter that's sharded across many keys (for example, a global 'likes' count). Compare a few approaches (per-shard counters with periodic aggregation, CRDT counters, a central counter service, optimistic CAS-based increments) on accuracy, throughput, read latency, and reconciliation cost.
Sample Answer
Direct answer: For a sharded counter that needs to keep incrementing correctly under high concurrency without coordination, a CRDT counter (a PN-Counter, sharded by key with one counter object per key) gives exact eventual convergence with no central bottleneck; per-shard counters with periodic aggregation trade a small aggregation delay for simpler infrastructure; a central counter service gives immediate global consistency at the cost of becoming a write bottleneck; and optimistic CAS (compare-and-swap)-based increments work for moderate contention but degrade under very high concurrent write rates on the same key.
Structured elaboration
CRDT counter (PN-Counter per key). Each region/replica maintains its own local increment count for a given counter key; merging (element-wise max plus sum) converges to the exact correct total with no coordination required at write time. Accuracy is eventually exact (once all increments have propagated and merged, the total is precisely correct, nothing is approximated or lost). Storage overhead is one entry per replica per counter key, and merge/read cost scales with the number of replicas, not the number of increments, since increments to the same replica's entry just add to that entry rather than each needing its own record.
Per-shard counters with periodic aggregation. Instead of a CRDT's continuous merge, each shard just accumulates its own local count independently, and a periodic batch job sums all shards' counts to produce the displayed total. Simpler to build (no CRDT library or merge logic needed), but the displayed total is only as fresh as the last aggregation run, real-time reads see a stale, under-counted total between aggregation cycles, an explicit trade-off, not a bug, if the use case tolerates it.
Central counter service. A single service (backed by its own strongly-consistent store) owns the counter and serializes all increments through it. Gives immediate, always-accurate reads with no aggregation delay, but throughput is capped by that single service's write capacity, and it becomes a single point of contention (and, without careful design, a single point of failure) as write volume grows, exactly the scaling problem sharding elsewhere in the system was meant to avoid.
Optimistic CAS-based increments. Each increment reads the current value, computes the new value, and writes it back with a compare-and-swap (only succeeding if the value hasn't changed since the read); a losing CAS retries. Works well at moderate contention (few concurrent writers per key), but retry rate grows sharply as concurrent writers to the SAME key increase, under very high contention (many writers hammering one popular key, a "hot key"), this can degrade into a large fraction of writes retrying repeatedly, hurting both latency and throughput.
Comparison
| Approach | Accuracy | Throughput under high concurrency | Read latency | Reconciliation needed |
|---|---|---|---|---|
| CRDT (PN-Counter) | Eventually exact | High, no coordination on write | Read = merge of current replica states, cheap | None; merge IS the reconciliation |
| Per-shard + periodic aggregation | Exact as of last aggregation, else stale | Very high, purely local writes | Fast but stale between cycles | The aggregation job itself |
| Central counter service | Always exact, immediately | Bounded by the service's own capacity | Fast, single source of truth | None needed, but scaling requires sharding the service itself eventually |
| Optimistic CAS | Always exact, immediately | Degrades under high same-key contention | Fast when uncontended | None; but retry storms under contention are a real operational risk |
Worked example. A "post likes" counter under a viral post might receive thousands of concurrent increments per second from a single popular key. A central counter service or CAS-based approach would see heavy contention specifically on that one hot key; a PN-Counter sharded across regions handles it gracefully since each region's writers only contend with OTHER writers in the SAME region (much lower local contention), converging the true global total asynchronously with no single bottleneck.
Trade-offs and pitfalls. Teams sometimes default to a central counter service for simplicity and only discover the bottleneck once a specific key goes viral or otherwise becomes hot, worth explicitly asking during design "what's the expected max increment rate on a SINGLE key," not just the aggregate system-wide rate, since that's what determines whether central/CAS approaches will hold up.
Design a multi-region user profile service that must support 100M users, 50k profile updates per second globally, and 1M reads per second. Requirements: users see their own updates immediately (read-your-writes) within a region, other users see updates eventually (within a bounded window), and 99th-percentile read latency stays low per region. Sketch the high-level architecture, replication strategy, and how you provide the read-your-writes guarantee without strong global coordination.
Sample Answer
Direct answer: At 100M users and 50k updates/sec, you need per-user data partitioned by region with a "home region" per user for writes, asynchronous cross-region replication for eventual global visibility, and a session-scoped mechanism (sticky routing to the user's home region, or a causal token) so each user sees their own updates immediately without requiring global synchronous coordination.
Structured elaboration
Partitioning and write routing. Each user is assigned a home region (based on signup location or explicit preference), and writes for that user's profile are always directed there, giving each user's writes a single, consistent, low-latency write path with no cross-region coordination needed per write. This is what makes 50k writes/sec globally distributed and still individually cheap: it's 50k INDEPENDENT single-region writes, not 50k globally-coordinated ones.
Cross-region replication. Each region's writes replicate asynchronously to every other region (a standard multi-region replication topology, e.g. a change stream fanning out from each region's primary store to the others). This is what provides "eventually visible to other users" within the required bound (here, one minute), the replication pipeline's own throughput and lag characteristics need to comfortably clear that bound under peak load, with monitoring on actual observed lag, not just a theoretical budget.
Read-your-writes without global coordination. Since the user's OWN writes always land in their home region, and their own subsequent reads can be routed to that SAME home region (sticky, based on the user's identity, not their current network location) for some bound (or indefinitely), the user always sees their own latest update, at native single-region latency, no waiting for cross-region replication for their OWN view. Other users reading this profile from a different region see it once replication catches up, within the required window, satisfying "others see it eventually within 1 minute" without that requirement touching the write or same-user-read path at all.
Achieving sub-50ms P99 read latency per region. Reads (for other users viewing a profile, not the owner's own reads) are served from the LOCAL region's replica, so they never cross a region boundary, keeping latency to local-network/local-disk numbers rather than being bounded by cross-region round-trip time (which alone would often exceed 50ms). This is only possible because the read doesn't need to be perfectly fresh, it needs to be fresh within a minute, which the async replication path already provides.
Handling the home-region-unavailable case. If a user's home region has an outage, either the user experiences degraded write availability (a real, honest trade-off of this design, since the write path is intentionally NOT cross-region-coordinated) or the system fails over the user's home-region assignment to another region (a more complex, operationally significant decision that itself needs careful handling to avoid conflicting writes from the old and new home regions during the transition).
Worked example. User in Region A updates their bio. The write commits to Region A's primary (their home region) as a purely local operation, with no cross-region network hop on the critical path, so its latency is bounded by local disk/network characteristics rather than by inter-region round-trip time, the same reasoning that keeps the P99 read latency for other users' LOCAL reads low. The response confirms success, and if the user immediately reloads their own profile, that read is also routed to Region A (sticky by user identity), showing the update instantly, own-write visibility achieved with zero cross-region dependency. Meanwhile the change replicates asynchronously to Regions B and C; a different user in Region B viewing this profile sees the OLD bio for however long replication takes (comfortably under the 1-minute bound under normal load, monitored explicitly for tail cases during regional replication backlogs).
Trade-offs and pitfalls. A common design mistake at this scale is routing the OWNER's reads based on their CURRENT network location rather than their home region (e.g. "nearest region" routing applied uniformly to all reads), which breaks read-your-writes the moment a user travels or is routed to a different region than the one their write landed in, own-write reads need identity-based (not network-proximity-based) routing specifically for this reason.
Compare orchestration versus choreography when implementing sagas across microservices. Produce a decision matrix covering coupling, observability, error handling, and versioning, and give criteria for when you would recommend each approach.
Sample Answer
Direct answer: Choreography fits a small number of loosely-coupled services where each step's failure handling is simple, since there's no central process to build or operate; orchestration fits workflows with more than a handful of steps, complex or conditional failure handling, or a need for centralized visibility into where every saga instance currently stands.
Structured elaboration
Decision matrix
| Dimension | Choreography | Orchestration |
|---|---|---|
| Coupling | Low: each service only needs to know which events to listen for and emit, not the identity or existence of other services | Higher: services expose commands the orchestrator calls directly, and the orchestrator needs to know about every participant |
| Observability | Hard: the workflow's state is implicit, spread across every service's event log; understanding "where is saga X right now" requires correlating events across services | Easy: the orchestrator holds the saga's explicit state machine in one place, straightforward to query and dashboard |
| Error handling | Gets tangled quickly as the number of steps grows, each service has to know what to do on every relevant failure event, and cyclic or conditional failure logic becomes hard to express cleanly | Centralized: the orchestrator's state machine can express arbitrary branching and compensation logic explicitly, in one place |
| Versioning | Changing the workflow (adding a step, changing order) means coordinating event-contract changes across every affected service | Changing the workflow is mostly a change to the orchestrator's logic; participant services just implement the same command/compensation interface |
| Number of participants | Scales well with few services (2-4); each new participant multiplies the event-contract surface every existing service might need to know about | Scales better as participant count grows, adding a step is adding one more call from the orchestrator, not touching every existing participant |
When to recommend choreography. A 2-3 service flow with straightforward, largely linear failure handling (if step 2 fails, always compensate step 1, no branching), where teams value not having a shared piece of orchestration infrastructure everyone depends on, and where the participating services are otherwise fully independent (no team already owns cross-service workflow logic).
When to recommend orchestration. More than about 4-5 services, conditional or branching compensation logic (different failure handling depending on WHICH step failed or WHY), a compliance or operational need to see "where is this specific transaction right now" without correlating distributed logs, or a workflow that changes often enough that touching every participating service's event contract on each change would be a real cost.
Worked example. A 2-service "reserve inventory then send confirmation email" flow is a reasonable choreography candidate: Inventory service reserves and publishes Reserved, Notification service (subscribed to Reserved) sends the email; if reservation fails, Inventory publishes ReservationFailed and nothing else needs to react. A 6-service order-fulfillment flow (Order, Inventory, Payment, Fraud-check, Shipping, Notification) with different compensation logic depending on which step failed (a fraud rejection needs different handling than a payment decline) is a stronger orchestration candidate: expressing that branching logic as a web of event subscriptions across 6 services becomes hard to reason about and debug.
Trade-offs and pitfalls. Teams sometimes start with choreography for its lower initial coupling and later regret it as the workflow grows past 3-4 services and failure handling gets more conditional, at which point migrating to orchestration means introducing a new piece of shared infrastructure AND unwinding event contracts multiple teams already depend on. It's usually cheaper to start orchestrated for anything expected to grow past a handful of steps, even though it means one more service to build up front.
Unlock Full Question Bank
Get access to all 46 Data Consistency and Distributed Transactions interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.