Distributed Systems Fundamentals Questions

Decide per feature, not per product: for each piece of state, ask what a stale or lost read or write actually costs, in both directions. Features whose operations are naturally commutative or idempotent, like a like count or a view count, tolerate eventual consistency cheaply, because being briefly wrong self-heals and nobody's safety depends on the exact number. Features where a stale or lost update directly causes an incorrect, hard-to-reverse outcome, money, a limited resource, or an invariant like at least one thing must remain true, need strong consistency or a convergent structure specifically engineered not to lose updates, even though that costs latency and availability during a partition.

The three named features

• Like counter: pure eventual consistency is fine. It is a simple, non-negative, additive count; a grow-only-counter-style commutative merge, or even just an approximate cache, means a brief undercount self-corrects on the next sync, and no one's correctness depends on the exact number at any instant.
• Timeline feed: needs causal consistency, not full linearizability. A reply must never be visible before the post it replies to, but unrelated posts from different authors can be shown in different orders to different viewers without breaking anything.
• A user's own settings: needs read-your-writes, or session consistency, for that user, not global linearizability. If a user just changed a setting, their own very next read must reflect it, or the product looks broken to them, but there is no requirement that every other user's session see that change instantly.

Feature store: per-user causal consistency, not global

In a machine learning feature store, a user's own online feature update, say their most recent click, must be visible to their own next inference request; that is the same read-your-writes requirement as the settings example, scoped per user. It does not need to be globally linearizable across all users' sessions, since one user's features have no bearing on another user's inference.

One service, different conflict-resolution policy per preference type

Within a single settings service, the right conflict-resolution policy varies by preference type, not just by feature:

• A boolean toggle preference, say dark mode on or off, is naturally last-write-wins-safe: whichever value wins is still a valid state, and there is nothing to lose except which of two valid values stuck.
• A set-valued preference, a list of blocked users, is not last-write-wins-safe. Concretely: a user's phone, offline with edits queued, sets blocked_users to {X}; concurrently, the same user's laptop sets blocked_users to {Y}, unaware of the phone's change. If the laptop's clock happens to run a few minutes fast, a naive last-write-wins merge picks the laptop's write purely because its timestamp looks later, giving blocked_users = {Y} and silently unblocking X, an actual correctness bug the user never asked for. An observed-remove-set merge, unioning the adds while respecting only the removes each device actually observed, instead gives blocked_users = {X, Y}, preserving both edits.

Billing and metering aggregation: under-counting vs over-counting, both cost money

A usage counter feeding billing must never silently under-count, that is straightforward revenue leakage, and ideally should not over-count either, since that produces customer complaints and refund credits. Both are direct cost consequences of picking the wrong merge strategy, not just an abstract correctness concern.

Worked example: plain overwrite vs a G-Counter, same events, different outcomes

Two shards independently record usage events for the same customer and need to combine into one total.

Plain mutable counter, naive approach:

• Shared counter starts at 0.
• Shard 1 reads the counter (0), adds 3 new usage events, writes 3.
• Shard 2, concurrently, also reads the counter before shard 1's write lands (0), adds 5 new usage events, writes 5.
• Final stored value: 5. Shard 1's update was overwritten and lost.

True total is 3 + 5 = 8, but the stored value is 5: 3 units of usage vanished, a direct case of under-counting and revenue leakage.

G-Counter approach, same events:

• Each shard keeps its own slot, both starting at 0.
• Shard 1 increments its own slot by 3.
• Shard 2 increments its own slot by 5.
• Read = sum of slots.

$\text{total} = c_1 + c_2 = 3 + 5 = 8$

No event is lost, because each shard only ever writes to its own slot; there is no shared mutable field for a concurrent write to overwrite.

This is the concrete cost of getting the direction wrong: assuming a plain field is fine because writes are rare quietly loses exactly the increments that happen to race, and the fix is not more locking, it is picking a data structure whose merge cannot lose an update in the first place.

Trade-offs & pitfalls

• Getting it too eventual: silent lost updates, as in the plain-counter example above, and user-visible correctness bugs, as in the blocked-users example, both compounded by a debugging nightmare, since the bug is nondeterministic and only shows up when two writes race.
• Getting it too strong: unnecessary coordination latency and reduced availability during a network partition for state that never needed it. A like counter does not need to block its write path on a quorum round trip.
• Common wrong turn: picking one consistency model for the whole product instead of reasoning feature by feature. A senior answer explicitly separates what needs strong or linearizable behavior, what needs causal or session guarantees, and what tolerates pure eventual consistency, rather than defaulting the entire system to one setting.

Direct answer

For a workflow spanning around 30 services, the real choice is not orchestration versus choreography as a single binary decision for the whole workflow; it is which steps need a component that can prove ordering and drive compensations (orchestration), and which steps can react to events with no central authority at all (choreography). Orchestration puts one coordinator in charge of calling each step and firing compensations in a known sequence; choreography has each participant publish an event when its own step completes and react to others' events, with no single place holding the overall plan.

Orchestration

A coordinator persists the saga's state as an explicit record (an event-sourced log or a saga_state table with a status per step), calls each participant directly, and on a failure at step k issues compensating calls for steps 1..k-1 in reverse order. Because the plan lives in one place, ordering and auditability are straightforward to reason about; the coordinator itself must be made durable and, typically, run as a small number of replicas, since it is now a component the whole workflow depends on.

Choreography

No coordinator exists. Participant N completes its local step and emits a domain event; participant N+1 subscribes to that event and reacts; a failure is just another event (e.g. ShippingFailed) that any interested participant can subscribe to and use as its own trigger to compensate. This removes the central dependency but means "what state is this workflow in" is a property of the whole event graph rather than one component's state, which is harder to reconstruct when debugging.

Compensating actions

A compensating action is the business-meaning inverse of a step, not a literal undo: refunding a settled charge is not "un-charging" it, and cancelling a shipped order needs a return flow, not a rollback. Compensations must be idempotent (safe to invoke more than once with the same effect), because a coordinator restart or a redelivered event can cause the same compensation to be issued twice.

Idempotency of each step

Every forward and compensating action is invoked with a natural key, typically (saga_id, step), that the receiving service stores alongside the resulting effect. If the same key arrives again, the service returns the already-recorded result instead of re-applying the effect (charging twice, releasing stock twice). This is what makes it safe for either a restarted orchestrator or a redelivered choreography event to retry a step it cannot be sure completed.

Detecting and recovering a mid-protocol crash

Orchestration: the coordinator's saga state is durable, so on restart it scans for sagas stuck in an in-flight status past an expected time bound, reads the last completed step from that record, and resumes forward execution or begins compensation from there. Because every action is idempotent, resuming is safe even in the worst case (crash after a participant executed but before the coordinator recorded it): the only possible cost is one duplicate no-op call.

Choreography: there is no single resume point. Each participant instead needs its own local timeout: for example, the inventory service reserves stock with an expiry, and if it never receives a downstream "payment confirmed" event within that window, it independently emits its own "reservation expired" event to trigger compensation across whatever already acted. Detecting "stuck" is decentralized and has to be designed per-participant rather than once, centrally.

Worked example: order O-500 across Payment, Inventory, Shipping

Orchestration trace:

sequenceDiagram
    participant C as Coordinator
    participant P as Payment
    participant I as Inventory
    participant S as Shipping
    C->>P: charge(step=1)
    P-->>C: success
    C->>I: reserve(step=2)
    I-->>C: success
    Note over C: crash before calling Shipping
    Note over C: restart, reads saga_state
    C->>S: schedule(step=3)
    S-->>C: fail
    C->>I: release(step=2)
    C->>P: refund(step=1)

1. saga_state(saga_id=S-500, step=1, status=STARTED).
2. Coordinator calls Payment.charge(saga_id=S-500, step=1, key=S-500:1); succeeds; saga_state updated to step=1, status=DONE.
3. Coordinator calls Inventory.reserve(saga_id=S-500, step=2, key=S-500:2); succeeds; saga_state updated to step=2, status=DONE.
4. Coordinator crashes before calling Shipping (step 3).
5. Coordinator restarts, reads saga_state for S-500: last DONE step is 2, step 3 was never started, so it resumes at step 3 and calls Shipping.schedule(saga_id=S-500, step=3, key=S-500:3).
6. Shipping fails permanently (undeliverable address).
7. Coordinator runs compensations in reverse for the completed steps: Inventory.release(saga_id=S-500, step=2), then Payment.refund(saga_id=S-500, step=1).
8. If the coordinator crashes again mid-compensation and retries Inventory.release(step=2) a second time, Inventory recognizes the key S-500:2 was already applied and returns the recorded result instead of releasing stock twice.

Choreography, same scenario: Payment emits PaymentCharged(S-500); Inventory, subscribed to it, reserves stock and emits InventoryReserved(S-500); Shipping, subscribed to that, tries to schedule and fails, emitting ShippingFailed(S-500); Inventory and Payment, both subscribed to ShippingFailed, independently run their own compensations on receiving it. If Shipping crashes before ever publishing ShippingFailed, no coordinator exists to notice the gap; Inventory only recovers because its own reservation carries a TTL (time-to-live, an expiry after which it self-cancels; say 15 minutes), and on expiry with no follow-up event it self-triggers its own compensation.

Trade-offs & pitfalls

A common pitfall is writing a compensation that isn't actually the semantic inverse of the forward action, which produces a technically-completed rollback that is still wrong for the business. In practice, a workflow like this is often a hybrid: strict, auditable steps (payment, billing) run under orchestration because ordering matters and correctness is expensive to get wrong, while more tolerant downstream steps (inventory, shipping) are choreographed since they are naturally eventual and cheaper to compensate if something goes wrong.

A CRDT-based counter (CRDT: Conflict-free Replicated Data Type, a data structure whose replicas can be updated independently and merged with a deterministic rule that always converges to the same value) lets every region increment locally with no coordination, so writes never block and never fail because of a remote outage. A consensus-backed counter (built on Raft or Paxos, where a majority of replicas must durably agree on each state transition before it counts as committed) gives you a single, linearizable value where every acknowledged read reflects every previously acknowledged write, at the cost of needing a live majority and at least one round trip per operation. The concrete thing the CRDT approach gives up is that read guarantee: a client can observe a stale, undercounted value during the window before a remote increment has propagated, with no signal that the value is incomplete, whereas consensus makes that undercounting structurally impossible.

What each approach actually guarantees

Worked example: where the freshness guarantee actually breaks

Take a G-Counter (a grow-only counter CRDT) tracking a global increment total across three regions, A, B, and C, each holding its own local counter; a read sums the counters a replica currently knows about.

1. Start: A = 0, B = 0, C = 0.
2. Region A processes one local increment: A = 1.
3. Region C processes two local increments: C = 2.
4. Before anti-entropy has propagated A's and C's updates to B, a client reads the counter at region B. B's local view is still A = 0, B = 0, C = 0, so the read returns 0, even though three increments are already durably accepted somewhere in the system.
5. An anti-entropy round runs: B receives the vectors {A: 1} and {C: 2}, merges by taking the elementwise maximum, and B's state becomes A = 1, B = 0, C = 2. A read at B now returns 3, the correct eventual total.

No increment was lost between steps 2 and 5, which is the CRDT's core promise. But the read at step 4 was not just a little stale, it was materially undercounted with no indication of that to the caller. A consensus-backed counter cannot produce this outcome, because an increment is not considered committed until a majority has durably logged it, and any successful read after that point is defined to include it.

A related but distinct choice: automatic CRDT merge vs. application-level conflict resolution

The same coordination-free idea shows up one level up in shared-document collaboration, where two users edit the same paragraph while offline. There a third option exists beyond CRDT and consensus: application-level conflict resolution, where conflicting edits are detected and handled by explicit business logic (show both versions to the user, prefer the longer edit, run a custom three-way merge) instead of a mathematically guaranteed merge function. A CRDT gives the same coordination-free availability as the counter case, with a merge that is provably correct for that specific data type; application-level resolution can encode arbitrary rules a generic CRDT cannot express, such as preferring the document owner's edit, but only for data types someone is willing to hand-write a merge function for, and it carries no formal convergence guarantee if that logic has a bug.

Trade-offs and pitfalls

This is not a universal ranking of CRDT versus consensus, it is a question of which correctness property a specific use case cannot live without. A monitoring signal or a rolling feature aggregate feeding a dashboard, or a non-blocking model feature, can absorb a bounded, self-correcting undercount, so a CRDT is the right default: no leader, no quorum loss during a regional outage, cheaper per write. A signal that gates an irreversible action, such as a billing or quota threshold, a fraud rule, or an inventory decrement, cannot absorb that undercount, because the moment a threshold decision fires is exactly the moment staleness turns from cosmetic delay into a correctness bug, so it belongs on a linearizable path even though every write costs more there. A common mistake is defaulting to CRDTs everywhere for their operational simplicity and only discovering the freshness gap when a threshold check fires on stale data; the fix is rarely to abandon the CRDT for everything, but to route the one decision that needs freshness through a linearizable read, or a periodic consensus-backed reconciliation snapshot, while leaving the bulk of the aggregation coordination-free.

Direct Answer

A saga coordinates a business transaction that spans multiple services by breaking it into a sequence of local transactions. Each service commits its own step immediately with no cross-service lock held, and if a later step fails, the saga undoes the steps that already succeeded by running a compensating action for each one, in reverse order. This trades strict, all-or-nothing atomicity for eventual, recoverable consistency and loose coupling between services.

Choreography vs. Orchestration

• Orchestration: a central coordinator issues each step as a command to the relevant service and decides, based on that service's response, what to do next, including which compensations to trigger if something fails. The whole workflow lives in one place, which makes it easier to see, test, and reason about end to end.
• Choreography: there is no central coordinator; each service publishes an event when it finishes its local step, and whichever service is subscribed to that event reacts by doing its own step and publishing its own event in turn. This avoids coupling every service to a central coordinator's command contract, but it scatters the workflow logic across services, so understanding or changing the whole sequence means tracing through several services' event subscriptions instead of reading one place.

Worked Trace: Order Fulfillment When Shipment Fails (Orchestration Style)

Order O123, three steps: reserve inventory, charge payment, schedule shipment.

1. Orchestrator sends ReserveInventory(O123, sku=42, qty=1) to Inventory. Inventory reserves the unit and replies Reserved.
2. Orchestrator sends ChargePayment(O123, $50) to Payment. Payment captures the charge and replies Charged.
3. Orchestrator sends ScheduleShipment(O123) to Shipping. Shipping tries to allocate a carrier slot and replies Failed: no carrier capacity.
4. The orchestrator now runs compensations in reverse order. It sends RefundPayment(O123, $50) to Payment, undoing step 2. Payment replies Refunded.
5. It sends ReleaseReservation(O123, sku=42, qty=1) to Inventory, undoing step 1. Inventory replies Released.
6. The orchestrator marks order O123 as Failed and notifies the customer.

The same sequence in choreography looks like this instead: Inventory reserves and emits InventoryReserved(O123). Payment, subscribed to that event, charges and emits PaymentCharged(O123). Shipping, subscribed to PaymentCharged, tries to schedule and, on failure, emits ShipmentFailed(O123). Both Payment and Inventory are subscribed to ShipmentFailed: Payment independently issues its own refund and emits PaymentRefunded(O123), and Inventory independently releases its reservation and emits ReservationReleased(O123). No single component ever holds the full picture of the workflow; each service only knows what to do when it sees an event it's subscribed to.

Trade-offs and Pitfalls

• Every forward step and every compensating action has to tolerate being retried, since at-least-once delivery means ChargePayment could be delivered twice; this is a system-property requirement on the saga's steps, a separate concern from how an external API exposes idempotency to its own callers.
• The saga's state, meaning which steps have completed and which compensations are pending, needs to be durably persisted, whether by a central orchestrator or by each participant in a choreography, so that a crash and restart can resume the saga correctly instead of leaving it stuck partway.
• Not every action has a true inverse. Compensating a shipment step after the package has physically left the warehouse can't undo the physical fact, only correct the system's record and possibly trigger a real-world return process; a senior design puts the hardest-to-compensate steps as late as possible in the sequence.
• Choose a saga when the steps naturally live in separate services or databases and each one can be given a real, working compensating action. Reach for a real distributed transaction only when an intermediate, partially-applied state genuinely cannot be tolerated and you can afford a synchronous locking protocol across every participant, which a saga specifically avoids.

Direct answer

PACELC, short for "if Partition, Availability vs. Consistency; Else, Latency vs. Consistency", says that CAP's dilemma, choose Consistency or Availability when a network Partition is happening, is only half the story. Even when there is no partition at all, a system still has to choose between Latency and Consistency for every write it replicates, because making a write durable on every replica before acknowledging it takes longer than acknowledging it once it's durable on a single node. PACELC packages this as: if Partition occurs, trade off Availability against Consistency (exactly what CAP already says); Else, meaning no partition, trade off Latency against Consistency.

Restating CAP precisely first

CAP says that during an actual network partition, a distributed system can guarantee only one of Consistency (every read sees the latest completed write) or Availability (every request gets a non-error response) for the nodes on either side of the split, not both. A common misreading treats CAP as "pick two of three, always"; it isn't. CAP's teeth are specifically about behavior during a partition. Most systems are both consistent and available almost all of the time, precisely because a true network partition is a rare event relative to total uptime, not something happening continuously.

flowchart TD
    Start[Write occurs] --> P{Partition active?}
    P -->|Yes| AC[Choose Availability or Consistency]
    P -->|No| LC[Choose Latency or Consistency]

What PACELC adds

PACELC names the trade-off CAP is silent about: during normal operation, with no partition, you still choose between Latency (L) and Consistency (C), because synchronous replication that waits for a majority of replicas costs a round trip before it can acknowledge a write, while asynchronous or single-node-acknowledged replication returns faster but risks a reader seeing stale data, or the acknowledged write being lost outright if that one node fails before it propagates. Systems are commonly labeled by both branches together, for example PA/EL (favor Availability under partition, favor Latency otherwise, the Cassandra/Dynamo-style default) or PC/EC (favor Consistency in both cases, the HBase-style default).

Worked example: a decision with no partition occurring

A write to a piece of user data must be replicated to three nodes: R1 in the local region, and R2, R3 in two remote regions. All three are reachable; no partition is happening anywhere in this example.

• Favor consistency (the "C" side of the Else branch): the write path waits for acknowledgment from a majority, at least two of the three replicas, say R1 and R2, before returning success to the caller. Any subsequent read from a majority quorum is now guaranteed to see this write. Cost: the caller's write waits on the round trip to R2, a remote replica, even though R1, the local one, already has it durably.
• Favor latency (the "L" side of the Else branch): the write path acknowledges as soon as R1 has it durably, and replicates to R2 and R3 asynchronously in the background. Cost: the caller gets a fast, local acknowledgment, but a read served from R2 immediately afterward, before the async replication catches up, will not see the write yet. If R1 crashes before that background replication completes, the already-acknowledged write can be lost entirely, with zero partition ever occurring.

This decision, wait for two of three versus acknowledge on one, is made on every single write regardless of whether any partition is happening, which is exactly the trade-off PACELC's Else branch names and CAP alone has nothing to say about, since CAP only speaks to a system that is not fully connected.

Trade-offs & pitfalls

A common misconception is treating a database's PACELC label as a fixed law of the software rather than a description of its typical default: most systems let you tune the replication wait per request (via quorum size), so "Cassandra is PA/EL" describes its usual configuration, not something it's incapable of changing. It's also easy to blur this Else-branch trade-off with an availability discussion; in the worked example above, no node was ever unreachable, so the trade being made is purely about how long the write path waits before acknowledging, not about surviving an outage, which is a separate concern belonging to the partition branch of the theorem.