Cloud Architecture Design Principles and Trade-offs Questions
The cross-pillar reasoning skill for architecting cloud systems: weighing reliability, scalability, security, performance, and cost against each other to justify ONE architectural choice over another under real constraints (budget, team size, timeline, existing systems). Covers well-architected-style design reviews, resilience and failure-mode reasoning (blast radius, graceful degradation, idempotency), consistency-versus-availability trade-offs (CAP/PACELC), and scenario-based decisions such as choosing a managed versus self-hosted component or an architectural style (monolithic, microservices, or serverless) for one system. Provider-agnostic: no specific cloud vendor's service catalog. This topic is the JUSTIFICATION layer, not a subsystem deep dive: a full design of observability, disaster recovery, identity and access management, networking, caching, or Kubernetes orchestration belongs to that subsystem's own topic. Comparing compute abstractions (VM versus container versus serverless versus GPU/TPU) belongs to compute options and trade-offs. Choosing an architectural style is covered here, but the internal implementation patterns of that style (service mesh, sagas, two-phase commit, event sourcing) belong to microservices architecture and service design. Multi-year roadmaps, vendor evaluation, and governance belong to infrastructure strategy and technology selection. Spanning multiple cloud providers or bridging on-premises and cloud belongs to multi-cloud and hybrid cloud architecture. The IaaS/PaaS/SaaS delivery-model taxonomy belongs to cloud service and deployment models. Region-crossing replication and failover design belongs to multi-region and geo-distributed systems.
Architect a cross-region system that must provide strong consistency for payments and eventual consistency for user profile updates. Explain how you would partition the data, where to place strongly consistent data vs eventually consistent data, which class of database or service to use for each (consensus-based stores vs geo-replicated stores), how to implement transactional guarantees for payments, and the resulting performance trade-offs.
Sample Answer
Direct answer
Partition the data by consistency requirement, not by convenience: payments go on a consensus-based store, one that uses a quorum protocol to agree on every write, giving strong consistency at the cost of latency and availability during a partition, and user-profile updates go on a geo-replicated store optimized for availability and low latency, using conflict-free replicated data types (CRDTs, data structures designed so concurrent updates from different regions merge automatically) or last-writer-wins semantics to resolve concurrent edits.
Structured elaboration
Where to place each kind of data
- Payments, strong consistency: use a database or coordination layer that requires a quorum, a majority of replicas, for example 2 of 3 or 3 of 5, to agree before a write is acknowledged. This guarantees that once a payment is confirmed, every subsequent read sees it, and two conflicting writes, like two concurrent debits from the same account, cannot both succeed silently.
- User profile updates, eventual consistency: use a geo-replicated store that accepts writes locally in each region and asynchronously propagates them, prioritizing low write latency and availability, even during a network partition between regions, over immediate global consistency.
PACELC as the deeper model here
CAP, consistency, availability, partition tolerance, only describes behavior during a network partition. PACELC extends it: if there is a Partition, choose between Availability and Consistency, as CAP describes; Else, during normal operation with no partition, choose between Latency and Consistency. This matters here because payments and profiles differ not just in their partition-time behavior but in their everyday latency-versus-consistency trade-off: the payments path accepts higher normal-operation latency, waiting for a quorum, to get consistency, while the profile path accepts lower consistency, eventual convergence, to get lower everyday latency, even when there is no partition happening at all.
Consensus and CRDT mechanisms
The payments store's quorum protocol, the specific algorithm varies but Raft and Paxos-family protocols are the common approach, means a write to an account balance requires a majority of replicas, often spanning at least two regions or availability zones, to persist it before acknowledging the client. During a partition that isolates a minority of replicas, those replicas correctly refuse writes rather than risk a split-brain double-spend, consistency winning over availability for that data, by design. The profile store can use CRDTs for fields where concurrent updates need to merge automatically without a human resolving a conflict: a last-write-wins register, the concurrent edit with the latest timestamp wins, for a display name, or a proper CRDT set type for something like a list of favorited items, where two regions' concurrent additions should both survive the merge rather than one silently overwriting the other.
Transactional guarantees for payments
A payment, debit A, credit B, needs to be atomic across both operations even when A and B are served by different partitions or shards. This typically uses either a distributed transaction protocol, two-phase commit, within the consensus-based store, or, more commonly in practice, a saga pattern: a sequence of local transactions with compensating actions, if the credit to B fails after the debit from A succeeded, an automatic compensating credit reverses A, plus an idempotency key on the whole payment operation so a client retry after a timeout cannot cause a double-debit.
Client and API impact, latency, and testing
The client-facing API surface should make the consistency difference explicit rather than hiding it: a payment confirmation only returns success after the quorum write is durable, higher latency but a trustworthy confirmation, while a profile update can return immediately after the local region accepts it, lower latency, with the understanding that another region might briefly show the old value. Expect the payments path's p99 latency, the 99th-percentile response time, to be materially higher than the profile path's, driven by the quorum round trip, often crossing multiple regions, which needs to be reflected in the SLA (service-level agreement, the latency and uptime commitment made to whoever depends on this API) the payments path commits to, not held to the same latency target as the profile path. Testing has to explicitly exercise partition scenarios, using fault injection to simulate a region becoming unreachable, to confirm the payments store correctly refuses writes from a minority partition rather than silently accepting them, and to confirm the profile store's CRDT merge logic produces a sane result when both regions have concurrently edited the same field, a case that is easy to get subtly wrong and needs a dedicated conflict-scenario test suite, not just a happy-path test.
flowchart LR
subgraph RegionA["Region A"]
AppA[App tier]
end
subgraph RegionB["Region B"]
AppB[App tier]
end
AppA -->|"quorum write, waits for majority"| Consensus[(Consensus-based store: payments)]
AppB -->|"quorum write, waits for majority"| Consensus
AppA -->|"local write, async propagate"| GeoA[(Geo-replicated store: profiles, Region A copy)]
AppB -->|"local write, async propagate"| GeoB[(Geo-replicated store: profiles, Region B copy)]
GeoA -.->|"CRDT merge"| GeoB
Worked example
A customer in Region A updates their display name at the same moment a support agent in Region B updates the same field for a different reason, say correcting a typo. Both writes land locally and propagate; the CRDT's last-write-wins merge resolves to whichever carries the later timestamp, and both regions converge to the same final value within the replication propagation window, no manual conflict resolution needed and no error shown to either writer. Meanwhile that same customer's payment of $50 is processed: the debit and credit are wrapped in a saga with an idempotency key. If the network between regions partitions after the debit succeeds but before the credit is confirmed, the payments store's consensus layer holds the transaction in a pending state rather than letting a minority-side replica guess at the outcome, and once the partition heals, the saga either completes the credit or runs its compensating reversal on the debit, never leaving the system in a state where money vanished or was created.
Trade-offs and pitfalls
- Using CRDTs for a field that actually needs strong consistency, an account balance, instead of just display metadata, would silently allow a lost-update or double-spend style bug; CRDTs solve concurrent-merge problems, they do not provide transactional correctness.
- Treating the payments path's higher latency as a bug to be optimized away, rather than the deliberate cost of the consistency guarantee it needs, is a common mistake; the fix is setting the right SLA for that path, not trying to make quorum writes as fast as local writes.
- Skipping partition-injection testing means the "correctly refuses writes during a partition" property is never actually verified until a real partition happens in production, the worst possible time to discover a split-brain bug.
Explain the difference between strong consistency, causal consistency, and eventual consistency. A game leaderboard needs to serve reads with sub-millisecond latency: which of these three consistency models would you choose for it, and why? Walk through the user-facing trade-offs of that choice.
Sample Answer
Strong consistency guarantees that every read reflects the latest completed write, as if there were only ever a single copy of the data. Causal consistency guarantees only that operations one client's actions causally depend on are seen in that order, while operations with no causal relationship to each other can appear in a different order to different readers. Eventual consistency guarantees only that, once writes stop arriving, every replica eventually converges on the same value, with no bound on how stale any individual read can be until then.
For a leaderboard that has to serve reads in under a millisecond, eventual consistency is the only one of the three that can meet the requirement. Strong and causal consistency both need the reader's request to check or coordinate with something beyond its own local copy of the data, and that coordination alone costs more time than a sub-millisecond budget allows, before any application logic runs at all. The design that follows is to keep the write path (the score submission itself) strongly consistent against one source of truth, so a score is never lost or double-counted, while serving every leaderboard read from a separate, eventually consistent, purely local cache.
Strong, causal, and eventual consistency compared
| Model | What every reader is guaranteed | What it costs to provide | Where it fits |
|---|---|---|---|
| Strong (linearizable) | Every read sees the single most recent committed write; all clients agree on one global order of operations | A quorum (a majority, more than half of replicas) must coordinate on the write, usually via a consensus protocol (an algorithm such as Raft or Paxos that gets multiple replicas to agree on one ordered log), and a read from a follower must confirm it is not stale | Bank balances, inventory counts, anything where a stale or out-of-order read is a correctness bug, not just an inconvenience |
| Causal | A reader who is causally connected to a write (made it, or read something derived from it) always sees it, and in order; unrelated concurrent writes can appear in different orders to different readers | Needs a way to track "happened before," such as vector clocks (a small per-write counter set that lets a node detect which operations causally precede which) or a session token, but not global agreement | Comment threads or activity feeds, where a reply must appear after the comment it replies to, but the exact order of two unrelated users' actions does not matter |
| Eventual | Every replica converges to the same value once writes stop, with no guaranteed order or freshness bound while writes are still happening | The cheapest of the three: a read is served from whichever local copy is nearest, with zero cross-node coordination at read time | Anything where a bounded staleness window is acceptable and raw read speed matters more than a strict ordering guarantee, which is exactly the leaderboard case here |
Why a sub-millisecond budget decides this before anything else does
It is worth putting an actual number on what "coordination costs time" means. Light in optical fiber travels at roughly two-thirds its vacuum speed, because of the fiber's refractive index (about 1.5):
vfiber=nc≈1.53×108 m/s=2×108 m/sA round trip has to cover the distance out and back, so for a full latency budget of one millisecond (0.001 s), the farthest a single round trip can physically reach is:
d=2vfiber×tbudget=22×108 m/s×0.001 s=1×105 m=100 kmThat is the physical floor, before adding any serialization, queueing, or protocol overhead, and before accounting for the fact that both strong and causal consistency typically need more than one such round trip (a consensus protocol commits a write only once a quorum acknowledges it, and a linearizable read from a follower has to separately confirm it is not stale). Availability zones, the isolated data-center groups a cloud region is built from, are frequently already tens of kilometers apart, and a cross-region hop is hundreds to thousands of kilometers. Any read that must coordinate beyond its own local node is competing for a budget that a single physical hop can already exhaust on its own. A read that never leaves a local, in-memory cache pays none of this cost: its latency is bounded only by local compute and local network, which is routinely and comfortably under a millisecond.
The leaderboard architecture: split the write path from the read path
flowchart LR
subgraph WRITE["Write path (authoritative, strongly consistent)"]
P[Player submits score] --> W[Game service]
W --> DB[(Primary score store: atomic max-update)]
end
subgraph READ["Read path (eventual, sub-millisecond)"]
C[(In-memory sorted leaderboard cache)]
R[Leaderboard read request] --> C
C --> UI[Rendered rank list]
end
DB -. async fan-out on bounded refresh interval .-> C
The write path is where correctness lives, so it stays strongly consistent: a score submission is an atomic "set my score to X only if X beats what is stored" update against a single primary store, which stops two near-simultaneous submissions (including a cheating client racing its own requests) from corrupting the record. The read path is where the latency requirement lives, so it is eventually consistent: an asynchronous process fans the primary store's changes out to a precomputed, sorted, in-memory structure on a bounded refresh interval, and every leaderboard read is served from that local structure with no coordination at read time. This is the same idea as CQRS (Command Query Responsibility Segregation: separating the path that accepts writes from the path that serves reads so each can be optimized on its own terms), applied specifically along the consistency dimension.
User-facing trade-offs of choosing eventual consistency for reads
- A player who just posted a new high score may not see their own updated rank until the next refresh cycle runs, because their read is served from the same eventually consistent cache as everyone else's. This is the specific cost of applying eventual consistency uniformly: it breaks the one guarantee a user notices immediately, seeing the result of their own action. It is usually patched at the interface layer rather than the consistency layer: show the player's own new score optimistically the instant the write is acknowledged, while the shared leaderboard view around them stays eventually consistent.
- Two players can briefly see different top-ten lists, or different ranks for the same third player, depending on which cache replica or edge location served each of them. For a leaderboard this is usually an acceptable cost, since a rank is inherently a snapshot in time, but it has to be a deliberate call. If the actual requirement were "every player sees an identical ranking at every instant," that pushes back toward strong consistency and a completely different latency conversation.
- Anti-cheat or audit tooling must never read from the eventually consistent cache to decide something consequential, such as banning a player or voiding a score, because that data is stale or momentarily wrong by design. It should read from the strongly consistent primary store, which is exactly why the two paths were split apart in the first place instead of trying to make one path serve both needs.
- The refresh interval feeding the cache needs an explicit staleness bound, a service-level objective (SLO) such as "the cache is never more than a few hundred milliseconds behind the primary store," plus monitoring against that bound, rather than just running "as fast as the background job happens to go." An eventually consistent system with no stated limit on how eventual is not a design decision, it is an outage waiting to be noticed.
- The most common wrong turn here is reaching for strong consistency "to be safe" on the read path, which is precisely where the stated requirement (latency) lives, while the operation that genuinely needs a strong guarantee, the write, gains nothing extra from that choice. Matching the consistency model to the specific operation, instead of applying one model uniformly across the whole feature, is the actual skill this question is testing.
You're designing the compute architecture for a medium-sized, customer-facing web application that needs to ship features frequently and handle unpredictable traffic growth. Explain the cloud-native design principles you would apply, and for each one, give a concrete example of how it shapes a compute architecture decision for this application.
Sample Answer
For a customer-facing app that must ship features often and absorb traffic growth it cannot forecast, the right compute architecture rests on six cloud-native design principles: statelessness, loose coupling, design-for-failure, elastic horizontal scaling, immutable infrastructure with automated delivery, and decoupling deploy from release. These are not abstract hygiene. Each one determines a specific, concrete decision about how the compute fleet is built, scaled, and rolled forward, and skipping any one of them reintroduces exactly the bottleneck the others exist to remove.
The six principles and the compute decision each one forces
1. Statelessness: nothing that matters lives on one instance
Application and session state is kept outside the instance, in a shared store, so any instance can serve any request. Compute decision: run the app tier as identical, interchangeable instances behind a load balancer (a component that spreads incoming requests across many backend instances), with session data in a shared cache such as Redis rather than in instance memory. That is what lets an autoscaler add or remove instances in the middle of a spike without dropping a single user's session, because "add capacity" no longer means "provision a machine that already knows about this specific user."
2. Loose coupling: decouple whatever scales on a different curve
Components talk through well-defined boundaries such as queues or APIs, not shared memory or a shared deploy unit. Compute decision: separate background and batch work (sending emails, processing images, running analytics) from the request-serving web tier, connected by a message queue. A spike in checkout traffic then never forces the unrelated email workload to scale, and each piece can be released and rolled back on its own schedule, which is the compute-side precondition for shipping features frequently at all.
3. Design for failure: assume an instance dies mid-request
Build in automated checks that an instance is still responding correctly, and let an orchestrator replace unhealthy instances without paging a human. Compute decision: configure automated health checks so a struggling instance is pulled out of rotation and replaced automatically, and make request handling idempotent (safe to retry without a side effect happening twice) so a mid-request instance replacement cannot double-charge a customer or duplicate an order.
4. Elastic, horizontal scaling driven by a leading signal
Capacity tracks demand automatically, and it grows by adding more identical instances (horizontal) rather than making one instance bigger (vertical), since vertical scaling has a hard ceiling and usually needs a restart. Compute decision: pick a compute platform whose autoscaler reacts to a leading indicator, such as request rate or queue depth, instead of a lagging one like a five-minute average of CPU load. By the time a lagging average notices an unpredictable spike, users have already timed out.
5. Immutable infrastructure and automated delivery
Every release is a new, versioned artifact (a machine image or container image) deployed fresh, never a running server edited in place. Compute decision: build one artifact per commit through a continuous integration and continuous delivery (CI/CD) pipeline, and roll it out through automated blue-green or canary gates (running the new version alongside the old and shifting traffic to it gradually) instead of logging into servers to patch them. Shipping ten times a day this way does not accumulate configuration drift nobody can reproduce, and a bad release is a one-command rollback rather than an incident.
6. Decouple deploy from release
Getting new code running in production and exposing it to users are two separate events. Compute decision: ship code behind feature flags (a runtime switch that turns a code path on or off without a new deploy), so the whole compute fleet runs one build while only a chosen slice of traffic sees the new behavior. That lets the team deploy continuously and release on a separate, business-driven schedule, which is what "ship features frequently" actually requires without tying every release to fleet-wide risk.
Worked example: absorbing an unplanned spike
Assume, purely for illustration, that each stateless web instance is sized to handle about 50 requests per second before p99 (99th-percentile) latency degrades. On a normal day the fleet runs 5 instances, covering 250 requests per second. The product gets an unplanned surge of attention and traffic jumps to 5,000 requests per second within a few minutes: 5,000 / 50 = 100 instances are needed, so the autoscaler has to add 95 instances (100 minus the 5 already running) on top of the existing fleet.
This only works cleanly because principles 1 and 4 act together. The 95 new instances are stateless, so the load balancer can route any request to any of them the moment they pass their health check, and the autoscaler is watching request rate rather than a slow-moving CPU average, so it starts adding capacity within the first tens of seconds of the spike instead of after a multi-minute averaging window. Loose coupling (principle 2) means the checkout-page spike does not also force the email-sending workers to scale by 20x, since they are a separate pool sized against their own queue depth. Because the fleet is built from immutable images (principle 5), the 95 new instances are bit-for-bit identical to the 5 already running: no first-boot configuration step to fail under load, no drift between old and new capacity.
Trade-offs and pitfalls
- Statelessness carries a migration cost: an app built around sticky sessions or local file storage has to be re-architected to push that state out, and the shared session store becomes a new dependency that must itself be made highly available, or it becomes the new single point of failure.
- Elastic scaling that only scales up quietly inflates the bill. The same automation needs a scale-in policy, and scale-in has its own failure mode (terminating an instance mid-request), which is exactly why design-for-failure and statelessness have to already be in place before elasticity is safe to add.
- Feature flags accumulate. A flag meant to decouple deploy from release for a few weeks is easy to leave in the codebase for two years, and an app with hundreds of stale flags is harder to reason about than the deploy risk the flag was solving.
- The common wrong turn on this question is treating "cloud-native" as a synonym for one specific compute product, serverless functions being the usual guess. These six principles hold whether the fleet is virtual machines, containers, or managed functions. What matters is that whichever compute abstraction gets chosen is stateless, loosely coupled, self-healing, horizontally elastic, and deployed immutably, not which vendor implements it.
Explain the CAP theorem in the context of cloud services. Provide one practical example of a managed cloud service that favors availability over consistency and one example of a service that favors consistency over availability. Discuss the operational consequences of each design choice.
Sample Answer
Direct answer
The CAP theorem says a distributed data system can only guarantee two of three properties during a network partition: consistency, every read sees the most recent write; availability, every request gets a response even if it might not be the latest data; and partition tolerance, the system keeps working despite a network split between nodes. Because partition tolerance is not really optional for any real distributed system spanning more than one machine or data center, CAP in practice is a choice between consistency and availability specifically during a partition; everyday, non-partition behavior is a separate trade-off that CAP does not actually describe.
Structured elaboration
The three properties, precisely
- Consistency: every node that gets a read returns the most recent successfully written value, or an error, never a stale value silently.
- Availability: every request that reaches a working node gets a non-error response, even if that response might be based on stale data.
- Partition tolerance: the system continues operating even when network messages between some nodes are lost or delayed.
Since a real distributed system can and eventually will experience network partitions, partition tolerance is treated as a given, and the actual design decision is what the system does during a partition: does it refuse to answer some requests to guarantee correctness, choosing consistency (CP), or does it keep answering with whatever data it has locally, accepting that answer might be stale, choosing availability (AP)?
One example favoring availability over consistency
A geo-distributed shopping cart or session store, the same shape as Amazon DynamoDB: DynamoDB's reads are eventually consistent by default (a strongly consistent read is available only as an explicit, pricier option, and is not offered at all on global secondary indexes or streams), and its multi-Region global tables replicate asynchronously by default, so each region keeps accepting reads and writes locally even if that region is cut off from the others during a partition. This favors availability: a customer can keep adding items to their cart even if their region cannot currently talk to other regions, at the cost that if they somehow interact with two partitioned regions during the outage, their view of the cart could briefly diverge until the partition heals and the system reconciles.
Operational consequence: the system needs a defined conflict-resolution strategy for when the partition heals, which value wins if both sides accepted writes, and application logic has to tolerate showing a customer a view that might be a few seconds or minutes stale during a partition, usually an acceptable trade for a shopping cart.
One example favoring consistency over availability
A configuration or coordination service used by other infrastructure to elect a leader or store critical cluster configuration, such as etcd, the Raft-based key-value store that backs the control plane of most managed Kubernetes offerings, or Google Cloud Spanner, which by default provides external consistency, its strictest guarantee, using a Paxos-based quorum for every read and write. This favors consistency: during a partition, the minority side of the split deliberately refuses to serve writes, and often refuses certain reads too, rather than risk a split-brain cluster, two nodes both believing they are the authoritative leader, which is a far worse outcome than a temporary unavailability.
Operational consequence: during a partition, some clients on the minority side will see errors or timeouts instead of a stale-but-working response, and operators need monitoring and alerting specifically for whether the system is currently in a minority partition and refusing traffic, since that is a real, if hopefully rare, production event that needs a runbook, not a surprise.
Worked example
A three-node cluster running a CP-style coordination service such as etcd splits so that one node is isolated from the other two. The two-node majority side continues accepting writes normally, since it can still form a quorum; the isolated single node refuses writes, and depending on configuration refuses reads too, to avoid serving stale data with false confidence, until the partition heals and it can rejoin and catch up. Any client that happens to be talking only to the isolated node experiences an outage during this window, that is availability being deliberately sacrificed to guarantee that no two nodes can independently believe they are both the current leader.
Trade-offs and pitfalls
- A very common mistake is treating CAP as describing everyday behavior: CAP only says something about what happens during a partition. A system's normal-operation latency-versus-consistency trade-off, the far more common case since a full network partition is relatively rare, is a different trade-off, often better described by PACELC (if there is a Partition, choose between Availability and Consistency, as CAP already describes; Else, during normal operation, choose between Latency and Consistency), which extends CAP to cover the non-partition case explicitly.
- Assuming a chosen availability-favoring design means consistency does not need to be thought about at all is a mistake; availability-favoring systems still need a defined conflict-resolution or reconciliation strategy for when the partition heals, skipping that design step leaves the system with an undefined, effectively random, outcome for conflicting writes.
- Treating partition tolerance as something you can simply opt out of is a misunderstanding: any system with more than one node connected by an unreliable network will eventually face a partition, so choosing consistency and availability while skipping partition tolerance is not a real long-term option for a genuinely distributed system, it just means the system has not been tested against a partition yet.
A payments service must keep operating correctly through a partial regional network partition, and it must never double-charge a customer or lose a transaction. Design the resilience patterns you would put across the request path (client retries, the write path, and cross-service coordination) to guarantee correctness under partition and retries, and explain how you would audit for and reconcile any anomalies afterward.
Sample Answer
Direct answer
Preventing a double-charge or lost transaction under a partial regional network partition comes down to three things working together: the client must retry safely, an idempotency key attached to every payment attempt, the write path must make the idempotency key the actual source of truth, reject or return the original result for a duplicate key, never process it twice, and cross-service coordination must use a pattern, a saga with compensating actions, or a durable outbox, that survives a partition without leaving money in a half-applied state. On top of that, an independent reconciliation process has to audit for anomalies afterward, because no design is provably perfect against every partition timing.
Structured elaboration
Client side: idempotency keys
Every payment request carries a client-generated idempotency key, typically a version-4 universally unique identifier, generated once per user action, not once per network attempt. If the client's request times out and it retries, the retry carries the same key. The client never generates a new key for a retry of the same logical action; that single discipline is what makes retries safe instead of dangerous.
Write path: the idempotency key as source of truth
The payment service persists the key and its result as an atomic part of the same transaction that processes the payment, not as a separate step that could itself fail out of sync. On receiving a request, it first checks whether this key has been seen before. If yes, and the prior attempt completed, it returns the original result, not a new charge. If the prior attempt is still in progress, it returns a "still processing, retry shortly" response rather than starting a second concurrent attempt. If the key is new, it processes it and records the result atomically with the charge itself. This makes "processed twice" structurally impossible for any request carrying the same key, regardless of how many times the client retries.
Cross-service coordination under partition
A payment usually spans at least two operations, debit the payer, credit the payee, maybe notify a ledger service, which cannot be a single database transaction across services. Use a saga: each step is a local transaction, and each step has a compensating action defined up front, if the credit fails after the debit succeeds, run an automatic compensating credit back to the payer. Pair this with a durable outbox pattern: the debit step and the "please credit the payee" event are written atomically in the same local transaction, so a partition or crash between "debit succeeded" and "credit event published" cannot happen, either both happened or neither did, and a background process reliably delivers the outbox event once connectivity is restored, retrying safely because the credit step is also idempotent by its own key.
Handling the partition specifically
If the network partition isolates the credit step's target service, the debit's local transaction already committed and the outbox event is durably stored; it simply cannot be delivered yet. This is safe: money is not lost, the debit is real and recorded, and no double-charge occurs, since the outbox event, when eventually delivered, carries the same idempotency key, so even if it is redelivered multiple times after the partition heals, the credit step processes it exactly once. The customer sees "payment processing" rather than a false success or a false failure, both worse than an honest "still working on it."
Auditing and reconciliation after the fact
Run a periodic reconciliation job that compares the payment ledger against actual money-movement records, bank or card-network settlement data, and flags any mismatch: a debit with no matching credit past a time threshold, a credit with no matching debit, or two records sharing what should have been a unique idempotency key. Every flagged anomaly should be investigated and, once understood, either resolved automatically, replaying a stuck outbox event, or escalated to a human for manual correction, with the resolution logged for audit purposes. This step exists precisely because "we designed it to be correct" is not the same claim as "we verified it stayed correct," and partitions are exactly the condition most likely to expose a subtle bug in the design.
sequenceDiagram
participant C as Client
participant P as Payment service
participant O as Outbox / event log
participant L as Ledger service (payee credit)
C->>P: Charge request (idempotency key K)
P->>P: Check key K, not seen before
P->>P: Debit payer and write outbox event (K) atomically
P-->>C: "Processing" (or success once confirmed)
Note over P,L: Network partition isolates L
O--xL: Credit event (K) cannot be delivered yet
Note over O,L: Partition heals
O->>L: Redeliver credit event (K)
L->>L: Check key K, apply credit exactly once
L-->>O: Acknowledge
C->>P: Client retries with same key K (thinks it failed)
P->>P: Key K already recorded, return original result
P-->>C: Original result, no second debit
Worked example
A customer's payment times out on their phone during a regional partition and they tap "pay" again. The second tap generates a request with the same idempotency key, because the client library correctly treats it as a retry of the same action, not a new one. The payment service sees the key already recorded from the first attempt, which had already debited the payer and written the outbox event, and returns that original result instead of debiting again. Meanwhile the outbox event to credit the payee, blocked by the partition, is redelivered once connectivity returns and applied exactly once because the ledger service checks the same key. The customer is charged once, the payee is credited once, and the reconciliation job later confirms the debit and credit match, closing the loop with evidence rather than assumption.
Trade-offs and pitfalls
- The single most common bug in real systems is a client that generates a new idempotency key on every retry, say from an auto-generated request ID, instead of reusing the key for the same logical action. That silently defeats the entire idempotency mechanism and reintroduces the double-charge risk it was built to prevent.
- Storing the idempotency-key check in a different transaction than the charge itself, a common shortcut, reintroduces a race: two near-simultaneous requests with the same key can both pass the "not seen before" check before either has recorded the key, causing a double charge. The check and the charge must be atomic together.
- Skipping the reconciliation step because "the design is provably correct" is a mistake; partitions have edge-case timings, a crash at the exact moment between two writes, a bug in the compensating-action logic, that a design review will not catch but a reconciliation audit will.
Unlock Full Question Bank
Get access to all 19 Cloud Architecture Design Principles and Trade-offs interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.