Multi-Region and Geo-Distributed Systems Questions

Direct answer

Accept writes locally in whichever region is nearest the client, give every user or session one
home shard that is the ordering authority for that user's events, and reconcile the two
asynchronously so that a write never waits on another region. Resolving the concurrent writes that
this local-accept window creates, with a small explicit rule (last-writer-wins or a conflict-free
data type) rather than by blocking the write to check with other regions, is what makes the model
hold together.

Structured elaboration

Partitioning, and where a conflict can actually come from. Partition by user_id (or session_id
for anonymous traffic) so all of one user's events belong to the same logical shard, home-assigned
to whichever region that user usually writes from. In the common case the user is writing from
their home region, so accepting the write locally and owning it in the home shard are the same
action and nothing crosses a region boundary. The two come apart when the user writes from
somewhere else. That write is still accepted locally, into the nearest region's own append log, and
shipped asynchronously to the home shard afterwards rather than being forwarded synchronously
across an ocean, because synchronous forwarding would put the cross-region round trip straight back
onto the 150ms budget. The window between local accept and home-shard merge is the only place in
this design where two writes for one user can be concurrent, and it is why the next section exists
at all. State the alternative explicitly, because an interviewer will ask for it: forwarding every
out-of-region write synchronously to the home shard gives exactly one writer per user and no
conflicts to resolve, and the reason to reject it here is the latency budget, not correctness.

Conflict resolution. When a user's device writes from two regions in quick succession, for
example switching networks mid-flight, use last-writer-wins (LWW: the write carrying the later
timestamp overwrites the earlier one) for simple fields. For fields where losing an update is
unacceptable, use a conflict-free replicated data type (CRDT: a data structure with a mathematically
defined merge rule, so any two replicas' concurrent updates combine into the same result regardless
of arrival order), for example a grow-only counter CRDT for "items added to cart" so neither write
clobbers the other.

Clock synchronization. LWW is only as good as the clocks behind it. Use a synchronized time
source (network time synchronization) plus a per-event logical clock component to break exact ties
deterministically, since millisecond-level clock skew across regions is common and can otherwise
flip which write "wins."

Session affinity. Pin a session's writes, and its own subsequent reads, to the region it
started in for the session's duration via a sticky routing token, so a user never reads their own
just-written event from a stale replica in a different region.

Ordering guarantees. Assign each event a per-shard sequence number (strict order within one
user's shard) plus a wall-clock timestamp, and state the real guarantee explicitly: events for one
user are strictly ordered; events across different users are not, and are not meant to be, since
guaranteeing that would need exactly the cross-region synchronous coordination the latency budget
rules out.

Worked example

The 150ms local write budget: a cross-region synchronous write, say a majority acknowledgment from
the nearer of two other regions at roughly 90ms round trip, would already eat 60% of that budget
before any disk write or business logic runs, which is why the local-write design reserves that
headroom instead. Against the 10-second eventual consistency target (all regions' copies must converge on the same
value within that window, even though they can briefly disagree), replicating in batches every
2-3 seconds clears the bound with margin even for one retried batch.

Trade-offs & pitfalls

Choosing last-writer-wins for a field that actually needs "every write counts," like the cart
counter, silently drops user data, the single most common bug in this design. Audit each field for
whether "last write wins" or "all writes merge" is the correct semantic before defaulting to LWW
everywhere.

A partial outage is the hard case precisely because a single rolled-up health score can look fine while the specific piece that actually matters, say the control plane that handles leader election or configuration changes, is down. Detection has to check the control plane and the data plane as separate signals, and the failover response may need to be partial too, moving only the operations that actually depend on the broken piece rather than evacuating everything.

Detecting a partial outage. Run distinct synthetic probes per plane: a control-plane probe that specifically exercises "can I create or update a resource, elect a leader, or renew a lease," and a data-plane probe that exercises "can I read or write an existing record." A partial outage shows up as one probe failing while the other still succeeds. A dashboard that averages these into a single "region health: 80%" number hides the exact signal that matters, so alert on each plane independently.

Automated failover for user traffic. Since the data plane may still be genuinely serving correctly, don't necessarily evacuate all traffic. Route only the operations that depend on the broken control-plane function, for example new resource creation or leader-dependent writes, to a healthy region, while continuing to serve reads locally if the data plane still works. A blunt full-region failover when only the control plane is down needlessly moves healthy traffic and adds unnecessary load elsewhere.

Automated failover for data replication. If the control plane can't perform leader election or renew a lease, replication itself may stall even though the underlying data store is technically reachable, since there's no leader to coordinate from. The failover here specifically targets restoring a working coordinator, for example temporarily electing a leader for that shard in a healthy region, rather than triggering a full data migration for data that was never actually unreachable.

Avoiding split-brain during a partial failover. This is subtler than the full-outage case, because the degraded region isn't fully down, from its own (correct, but incomplete) point of view its data plane looks fine. The safeguard is the same idea used for full split-brain prevention: a fencing check that only increases and is enforced at every write path, but applied narrowly, per operation type, so the specific control-plane-dependent function that failed over gets fenced against the degraded region's old instance, while data-plane operations that never moved don't need any fencing change at all, since they were never actually ambiguous.

Direct answer

I would run the migration as a series of small, reversible waves validated against explicit go/no-go metrics, rather than one big cutover (a single all-at-once switch of all traffic to the new setup), with a named owner for every risk and a rollback trigger defined in advance for each wave, because the entire point of going slow is to make every step survivable if it goes wrong.

Stakeholder communication

Map stakeholders up front: customers and support, executives who need status, and the operations team who will live with the new topology daily. Give a pre-migration briefing that lays out the plan and rollback criteria, keep a running status update at every wave boundary, and tell support and ops what changes for them before each wave starts, not after something breaks and they are caught off guard.

Migration waves, cutover, and rollback

Split waves by blast radius (how much of the system, and how many users, would be affected if this particular wave goes wrong), not by convenience: internal or synthetic traffic first, then a single low-risk customer segment's read traffic, then broader read traffic, then a small slice of write traffic, then full cutover. Every wave routes through a mechanism that can be dialed back to zero without a redeploy (a traffic-percentage switch or feature flag), and the rollback trigger for each wave is a specific number decided in advance, for example error rate more than a defined percentage above baseline, or replication lag above a defined threshold sustained for a set number of minutes, so the decision is not made emotionally in the middle of an incident.

A trigger is only as good as the volume it is evaluated on, and this is where carefully written migration plans quietly fail. Deciding the threshold in advance is necessary but not sufficient: the same threshold that is precise on a wave carrying half the traffic is pure coin-flipping on a wave carrying 1% of it, because at low request counts ordinary random variation produces the threshold on its own. So every trigger needs a companion condition, which is the minimum number of requests that must land in the evaluation window before the trigger is allowed to fire at all. Below that count the correct action is to widen the window or extend the soak, not to roll back on noise, and a plan that rolls back on noise teaches the team to ignore the trigger, which is worse than having no trigger.

Testing and validation

Before wave one touches real traffic, run shadow or dark-launch testing: send real traffic to the new path and compare its output without actually serving the response, to validate correctness with zero user risk. Rehearse the rollback path itself in a game day (a scheduled, deliberately-triggered rehearsal of the failure scenario run against a real or production-like environment), not just the forward migration path, since a rollback nobody has ever exercised is the least trustworthy part of the whole plan. After each wave, run a data-reconciliation check to confirm consistency actually held.

Runbooks and escalation

Write a runbook (a step-by-step written procedure covering exactly what to do, and in what order) per wave with the exact cutover and rollback steps, contact information, and known-failure-mode responses, reviewed before the wave starts rather than improvised during it. Name an incident commander (the single person with authority to make real-time decisions during the wave, so nobody is guessing who is in charge) and a backup for each wave window, and define in advance the threshold at which a wave issue graduates from "the migration owner is handling it" to a formal incident with its own escalation.

Updating SLOs and error budgets during the migration

Temporarily widen or separately track the team's error budget (its SLO, or service-level objective, is the internal reliability target the team commits to, and the error budget is the amount of unreliability that target already allows it to spend before breaching) during active migration windows, and communicate that policy change to stakeholders explicitly, so migration-induced turbulence is not silently mistaken for an unrelated production regression, and so it also does not quietly eat the budget the team needs for everything else happening at the same time.

Mentoring and preparing the operations team

Run the operations team through a tabletop rehearsal (walking through the failure scenario as a discussion, with no real actions taken) or a real game-day rehearsal of the new topology's failure modes before they are on point for it live. Pair anyone new to the runbooks with an experienced on-call engineer for the first real wave, and only complete the ownership handoff once they have exercised a failover themselves, not just read the document.

Worked example

Take a service handling 500 requests/second migrating from a single-region primary to a three-region topology, and assume writes are 10% of that, so 50 writes/second and 450 reads/second. Spending those numbers rather than only naming percentages is what turns the plan into a schedule.

• Wave 1, days 1-2: internal synthetic traffic only, no real users.
• Wave 2, days 3-5: 5% of read traffic, which is about 22 requests/second on the new path. Over a 48-hour soak that is roughly 3.9 million requests, so a gate of "error rate within 0.1 percentage points of baseline" is measured against a band about 3,900 errors wide and is comfortably resolvable. Second gate: p95 replication lag (the lag value the worst 5% of measurements exceed) under 5 seconds.
• Wave 3, week 2: 50% of read traffic, about 225 requests/second.
• Wave 4: 10% of write traffic, which is about 5 requests/second, and this is the wave where the gate has to be written differently. In a 5-minute window that path carries about 1,500 requests. If the baseline write error rate is 0.1%, the window is expected to contain 1.5 errors, so a "more than twice the baseline sustained for 5 minutes" rule fires on 3 or more errors, and ordinary Poisson variation alone produces that in roughly 19% of windows, which is a spurious rollback about every 26 minutes of a healthy wave. At a 0.01% baseline it is worse in relative terms: 0.15 expected errors, any single error trips the rule, and noise fires it in about 14% of windows. So for wave 4 the trigger becomes: at least 9,000 requests observed in the window (a 30-minute window at this rate), and the observed error rate more than twice baseline over that window, with an immediate-halt override on any single hard-failure class such as a replication error or a write rejected by the new primary, where one occurrence is already a verdict and volume is irrelevant.
• Wave 5, week 3: full cutover.

A full rollback rehearsal happens once, before wave 1 begins. The general rule the wave-4 arithmetic illustrates is that as traffic share goes down, the evaluation window has to go up to hold statistical power constant, and rare-but-unambiguous failure classes should be watched by count rather than by rate so that low volume does not blind the gate to the failures that matter most.

Trade-offs and pitfalls

Compressing waves to hit a deadline erodes the entire safety rationale the plan was built on. Treating the runbook as a formality that gets written but never rehearsed means it is most likely to fail exactly when it is needed. Setting a percentage threshold without checking the request volume it will be evaluated on produces a trigger that fires on noise, and the real damage is not the wasted rollback but that the team learns to override it. Forgetting to communicate the temporary error-budget policy leads someone to page for what looks like a regression but is actually expected migration noise. And skipping a real rehearsal for the operations team means their first live failover happens during the migration itself, which is the worst possible time for a first attempt.

Direct answer

Active-active means two or more regions concurrently serve live production traffic, so losing one just means the survivors, already warm and already handling real load, absorb the rest. Active-passive means one region actively serves traffic while one or more standbys replicate quietly in the background, and losing the active region requires an explicit promotion step before the standby can take over.

Structured elaboration

• Failover behavior: active-active can, with enough spare capacity already provisioned, fail over close to instantly, since the surviving regions are already serving real traffic and need no warm-up. Active-passive requires detection, promotion, and traffic redirection, a process that takes real minutes, not seconds, and the standby may need to warm caches or scale up since it wasn't handling live load beforehand.
• Complexity: active-active must solve write conflicts, if multiple regions can accept writes, or cleanly partition write ownership, a genuinely hard, continuously-running operational burden. Active-passive concentrates its complexity into the failover mechanism itself (detection, promotion, and fencing to prevent the old primary from coming back and causing split-brain), which is easier to reason about day to day since nothing unusual is happening most of the time.
• Typical use cases: active-passive fits systems needing one unambiguous source of truth where conflict-free concurrent writes are hard to guarantee, such as a payments or billing ledger service, since "who's authoritative right now" must never be in question. Active-active fits systems that can tolerate partitioned ownership or eventual reconciliation and where global capacity or latency matters more than a single writer, such as a global content-delivery-backed catalog service, or a collaborative document editor built on conflict-free replicated data types (CRDTs).

The SRE ownership angle

As the SRE (site reliability engineer) responsible for reliability, the two topologies demand genuinely different day-to-day attention. Active-passive requires you to own and regularly drill the failover runbook, since an untested promotion path is the single most common reason a disaster-recovery plan fails exactly when it's needed: standby regions silently drift out of readiness over time (stale configuration, unpatched dependencies, cold caches) if they're never actually exercised. Active-active requires you to own conflict and partition monitoring plus capacity headroom discipline: each region must be able to absorb a peer's traffic if that peer fails, which means day-to-day utilization has to be kept below the ceiling needed to absorb one more region's load, not run at a seemingly efficient near-100%.

Worked example

For a 3-region active-active layout, each provisioned for 100 capacity units and running at a steady-state 50% utilization (50 units), losing one region shifts its roughly 50 units of traffic across the two survivors, about 25 units each:

$$75 \text{ units used} \div 100 \text{ units capacity} = 75\% \text{ utilization}$$

still comfortably under each region's ceiling. Contrast that with active-passive: the standby normally runs at little to no utilization and must absorb 100% of the primary's load the moment it's promoted, meaning it has to be provisioned for the primary's full capacity even though it contributes nothing to day-to-day throughput, a pure insurance cost that active-active avoids by putting all regions to productive use.

Trade-offs and pitfalls

A common pitfall in active-active systems is assuming they don't need a drilled failover runbook at all, since traffic is "already live everywhere." In practice they still need an explicit, tested procedure for draining a failed region's traffic cleanly, and separately, for re-admitting that region once it's healthy again: a region that's quietly behind on replication and rejoins the write set too early can itself trigger a fresh round of conflicts, exactly the kind of self-inflicted incident a drilled re-admission procedure is meant to prevent.

Direct answer

Track two separate SLO (service-level objective, a target for a measurable aspect of the service) families for a globally replicated service: an availability SLO (did the request get a successful response at all) and a data-correctness SLO (was the response fresh or accurate enough), because these two trade off against each other precisely when a region gets cut off from the others, and a system can be fully "available" while quietly serving stale or wrong data.

Defining the SLOs and computing partition-tolerant SLIs

• Availability SLO: percentage of requests receiving a successful response within a latency budget, tracked per region and globally.
• Data-correctness SLO: percentage of reads observing writes within a defined freshness bound (using the same lag measurement approach used for replication monitoring), or percentage of writes that reach durable, conflict-free state within a target window.
• Partition-tolerant SLIs (service-level indicators, the actual measured values behind an SLO): the naive approach only measures the happy path and silently undercounts what happens during a network partition between regions. Instrument explicitly for the degraded state: if a region loses contact with the others, does it reject writes (protecting correctness at the cost of availability) or accept them locally and reconcile later (protecting availability at the cost of temporary correctness)? Count and label "degraded but available" responses separately from normal ones so the SLI reflects what users actually experienced, not just an aggregate that erases the distinction.
• Error budgets triggering automated failover: define a burn-rate policy using multiple rolling windows (for example a fast 1-hour window and a slower 6-hour window), and only trigger automated regional failover once the fast-burn condition holds for a minimum duration, to avoid flapping on a brief blip.
• Decomposing by region and customer impact: never rely on one global number alone. Weight SLIs by both region traffic share and customer tier, since a small error rate concentrated on your largest enterprise customer's shard is a bigger deal than the same rate spread thin across free-tier traffic, even though a blended global number could look identical in both cases.

Worked example

Take a global availability SLO of 99.95% measured monthly. A 30-day month has 30 x 24 x 60 = 43,200 minutes, so the allowed error budget is:

$$\text{monthly\_budget\_min} = (1 - 0.9995) \times 43{,}200 = 21.6 \text{ minutes}$$

For a fast-burn alert, suppose the policy pages if the service is on pace to consume 2% of that monthly budget within a single hour: 2% of 21.6 minutes is 0.432 minutes, or about 25.9 seconds of allowed downtime-equivalent within that hour, which corresponds to an error rate of 25.9 / 3,600 = 0.72% sustained for that hour. Crossing that threshold is the automated-failover trigger for the affected region, not a raw "any error occurred" trigger.

Trade-offs and pitfalls

Automated failover on error-budget burn needs hysteresis (a minimum sustained duration in the bad state before acting) or it will flap between regions on noisy but transient spikes. Correctness SLIs are genuinely harder to instrument than availability ones because "was this response correct" requires a reference point, so many teams under-invest here relative to availability. Per-customer decomposition also has an operational-ownership consequence: it only works if incident response has a clear model for who is in charge across regions. a single-region degradation is usually owned and worked by that region's on-call, but the moment error-budget burn crosses into automated cross-region failover, someone needs to hold the whole-system view so two regions do not independently take conflicting recovery actions at once.