Designing systems for resilience and availability across geographic regions, including strategies for cross region replication, failover, and operational recovery. Candidates should understand deployment models such as active active and active passive and the trade offs they imply for availability, consistency, cost, and operational complexity. Discuss replication topologies and the differences between synchronous and asynchronous replication and how those choices affect consistency and the recovery point objective. Cover leader election and failover coordination mechanisms, conflict resolution approaches including last write wins, version vectors, and convergent data types, and implications for transactional guarantees and global transactions. Include global traffic routing and failover techniques such as DNS based routing, global load balancing, health checks, and the impact of routing and time to live on failover behavior. Address data partitioning and cross region latency trade offs, strategies for orchestrating data recovery and region seeding, backup and restore practices, and testing approaches such as planned failovers, rehearsal drills, and chaos testing. Explain how to derive and meet recovery time objective and recovery point objective from business requirements, and consider monitoring, observability, automation, runbooks, cost considerations, and compliance and data residency requirements.
MediumTechnical
65 practiced
For an eventually consistent multi-region key-value service, propose a conflict-resolution approach using version vectors. Describe write & read behavior, metadata stored per key, and how you'd reconcile when vectors indicate concurrent updates.
Sample Answer
Metadata per key:- Value payload- Version Vector (VV): map region_id → integer counter- Last-writer timestamp (wall-clock) for heuristics/TTL only- Optional tombstone flag + VV for deletes- Optional causal metadata (e.g., dependency list) if multi-op transactions neededWrite behavior:- Client sends write to local region; region increments its counter in the key's VV (VV[local] += 1), attaches updated VV, stores value locally, and asynchronously replicates the (value, VV) to other regions.- Writes are accepted locally (low latency). Replication is reliable but eventually delivered; use a write-ahead log for durability and retry on failure.Read behavior:- Read from local replica returns the locally stored value + VV. Optionally read-repair: on read, fetch remote replicas' VVs and values to detect divergence and trigger reconciliation.- For strong read-after-write within region, read local; for cross-region, expose options: "read latest" triggers read-repair/merge; "read local" returns local state.Reconciliation when vectors indicate concurrency:- If incoming VV dominates stored VV (all counters ≥ and one >), accept incoming value (causal successor).- If stored VV dominates incoming, ignore incoming.- If neither dominates (concurrent updates): retain both versions as siblings (multi-value), exposing to client or invoking automatic merge: - Automatic merge strategies: CRDT-like merges (if data type supports), last-writer-win by comparing logical counters+timestamp as heuristic, or application-level merge via user-defined merge function. - If siblings kept, future writes must include merged VV = element-wise max of sibling VVs + local increment to reflect causality; garbage-collect resolved siblings once merged.Operational notes (SRE):- Monitor VV growth and prune/compact occasionally.- Ensure clocks are only heuristic; never rely on wall-clock for correctness.- Instrument replication lag, conflict rates, and read-repair frequency; alert on rising conflicts.- Provide tooling to inspect siblings and run manual reconciliation during incidents.
EasyTechnical
72 practiced
Describe the concept of region seeding for a new geographic region. Provide a step-by-step plan to seed a read-write service (including data snapshot, transfer, applying deltas, validation, and cutover) while minimizing RPO impact for live traffic.
Sample Answer
Situation: We need to seed a fully read-write service in a new region with minimal RPO impact on the live primary region.Step-by-step plan:1. Clarify constraints & safety: RPO target, expected data volume, network bandwidth, maintenance window, consistency model (strong vs eventual), and rollback criteria.2. Prepare target infra: provision compute, storage, networking, DNS/LB entries, IAM, monitoring, and capacity for expected peak plus buffer.3. Take a consistent snapshot: - For databases: take a physical snapshot (storage snapshot / filesystem snapshot) or logical dump with a point-in-time marker (e.g., pause writes briefly or use DB consistent snapshot tools). - Record changelog position: binlog/GCS/WAL/CDC cursor (LSN).4. Bulk transfer snapshot: encrypt and transfer snapshot to target (parallel streaming, rsync, or cloud transfer). Monitor throughput and resume on failure.5. Restore snapshot on target and bootstrap service components. Start read-only instance(s).6. Start delta application: - Begin streaming deltas from recorded changelog position using CDC/replication (binlog, WAL streaming, Debezium). - Apply deltas continuously to keep target caught up; use backpressure/throttling to avoid overwhelming target.7. Validation & integrity checks: - Run checksum verification (pt-table-checksum, row counts, hash sampling). - Smoke tests: API read/write semantics on a shadow namespace with synthetic traffic. - Monitor replication lag and error rates, alert on divergence.8. Final cutover planning: - Choose low-traffic window and advertise expected brief write freeze if unavoidable. - Option A (zero-freeze): wait until replication lag < threshold, then switch traffic with weighted DNS/LB shift and promote target to writable atomically. - Option B (short freeze): briefly pause writes on primary, drain outstanding transactions, ensure last LSN applied, promote target, resume writes.9. Execution & rollback: - Execute cutover steps, monitor SLOs, latency, error rates. - Have rollback plan: switch traffic back, re-enable primary as writable, or failover to previous region.10. Post-cutover validation & cleanup: - Run end-to-end tests, reconcile metrics and logs, remove temporary replication if desired, and document the process.Key considerations:- Use CDC for low RPO; physical snapshot + WAL/binlog streaming is common.- Automate steps, include health checks, and enforce access controls.- Validate with checksums and production-like probes to ensure correctness before declaring completion.
MediumSystem Design
84 practiced
Design a leader election scheme across regions using an external KV store (e.g., etcd or consul) for a globally distributed service. How would you handle network partitions, leases, clock skew, and preventing split-brain? What lease durations and probing behavior would you recommend for high-latency WAN?
Sample Answer
Requirements and constraints:- Single active global leader at any time (no split-brain)- Operates across high-latency WAN regions- Uses external KV store (etcd/Consul) as coordination primitive- Tolerates partitions, clock skew, and transient network jitterHigh-level design:- Use the KV store's native leader-election primitives: a single well-known key (e.g., /service/leader) acquired via a Compare-And-Swap / session-lease mechanism.- Each candidate creates a session with a TTL (lease) and attempts to Acquire the key atomically. The KV returns a unique, monotonically increasing revision/fence token for the acquiring client.- The leader must continuously renew its session before TTL expiry. Followers watch the key for changes and read the fence token on leadership change.Core components & responsibilities:1. Candidate process (in each region): attempts Acquire, renews lease, exposes health, serves when leader.2. KV cluster: single logical source-of-truth. Prefer a geo-aware deployment that provides strong quorum semantics (etcd with cross-region clusters is high-latency; better to run regional clusters + a global consensus/relay if possible).3. Observers/Clients: before performing leader-only actions, check current revision/fence token to prevent acting on stale knowledge.Handling network partitions and preventing split-brain:- Rely strictly on KV-managed leases (not local clocks). If leader loses connectivity to KV quorum, it cannot renew its lease and the lease will expire — preventing split-brain.- Use fencing tokens: the KV returns a monotonically increasing revision or token when a new leader obtains the key. All resources (databases, message queues) require the leader to present the current token before allowing leader-only operations. This prevents an old leader (partitioned) from continuing work if it regains network but its token was superseded.- Quorum requirement: ensure KV cluster's quorum is maintained and that Acquire semantics require majority. If you must span KV across regions, accept higher latency or instead use a hierarchical election: elect a regional leader locally, then region leaders run a lightweight global election (fewer participants) to reduce latency while preserving quorum semantics.Leases, probing, and timeouts for high-latency WAN:- Pick TTL >> max RTT * safety factor. Rule of thumb: TTL = max_round_trip_ms * 10. Example: if WAN RTT worst-case 200ms, TTL ≈ 2s; to be conservative use 5–15s for inter-region setups.- Renew frequency: heartbeat interval ≈ TTL / 3 with random jitter (±10–30%) to avoid thundering herd. For a 10s TTL, heartbeat ~3s.- Renewal retries: on transient failure, retry renew immediately with small backoff; allow up to TTL/2 of cumulative retry time before giving up to avoid rapid flapping.- Election backoff: when followers detect leader absence, stagger candidate attempts with randomized backoff windows to prevent simultaneous contests.Clock skew:- Do not rely on system clocks for lease validity. Use KV TTL and server-side timestamps/tokens. Use monotonic timers locally only for scheduling renewals.Operational considerations:- Metrics & alerts: expose lease state, renew latency, number of renew retries, leader churn rate, KV request latency, and quorum-health.- Testing: chaos test partitions and delayed KV responses; simulate leader isolation and verify fencing prevents stale actions.- Trade-offs: longer TTL reduces false failovers but increases recovery time; shorter TTL increases sensitivity to transient network blips. Choose TTL based on SLO for failover time vs. acceptable risk of unnecessary leader re-elections.Example recommended settings (high-latency WAN):- Worst-case RTT ~200ms–300ms: TTL = 10s, heartbeat = 3s ± jitter, retry window up to 5s before relinquish.- Worst-case RTT ~400ms–800ms: TTL = 15s–30s, heartbeat = TTL/3 ± jitter, retry window TTL/2.Summary:- Use KV-managed leases + atomic Acquire + fencing tokens.- Never trust local clocks for lease expiry; rely on server-side TTL and monotonic timers for renewals.- Choose TTLs conservatively for WAN, probe at TTL/3 with jitter, and implement retries/backoff to avoid flapping.- Monitor and chaos-test to validate behavior under partitions and high latency.
EasyTechnical
60 practiced
Explain how DNS TTL affects multi-region failover behavior for web services. Provide an example: DNS TTL = 60s, client resolver caches, and primary region fails. What factors determine how quickly traffic moves to a healthy region and how would you design around client-side caching?
Sample Answer
DNS TTL controls how long a client or recursive resolver may cache a DNS answer before re-querying authoritative servers. In multi-region failover, TTL is a key lever for how quickly clients will discover a new healthy endpoint after the primary region becomes unavailable.Example (TTL = 60s):- At t=0 a client resolves service.example.com → primary-region IP and caches it for up to 60s.- At t=10s the primary region fails. The authoritative DNS / traffic manager updates records to point to a healthy region.- The client will continue using the cached IP until its TTL expires (up to t=60s) or until the client/OS/app actively re-resolves earlier. Thus, the worst-case cutover for that client is the remaining TTL; average cutover depends on when the client cached it.Factors that determine how quickly traffic moves:- TTL value (shorter → faster but more DNS load)- Behavior of recursive resolvers (many ignore very low TTLs, or normalize TTLs)- Client OS and application caching (some browsers/SDKs cache longer or don't re-resolve on connection failures)- Connection-level behavior (TCP keep-alives, long-lived connections keep sending to failed endpoints)- DNS propagation and how quickly the authoritative service updates records and health checksDesign patterns to mitigate client-side caching:- Use a balanced TTL (e.g., 30–60s) considering DNS query capacity and desired RTO.- Combine DNS failover with anycast or global load balancers that shift traffic at the network layer (faster than DNS).- Implement active health checks and automatic failover in authoritative/traffic manager with pre-warmed endpoints.- Use short-lived connection strategies (HTTP/2 connection pooling limits, retry logic) and client-side retry/backoff to re-resolve on errors.- Prefer traffic managers that support connection draining and gradual cutover (weighted DNS or geo-pools) to reduce blast radius.- Monitor resolver behaviors and measure real-world failover times; document expected RTO in runbooks.As an SRE, choose TTL and complementary mechanisms to balance failover speed, DNS load, and realistic client behavior.
HardTechnical
67 practiced
Design a split-brain detection mechanism for a multi-region database with asynchronous replication. What signals would you use (e.g., divergent commit histories, conflicting write sets), and how would you automate containment to prevent further divergence while preserving availability where possible?
Sample Answer
Requirements & constraints:- Multi-region DB with asynchronous replication.- Goal: detect split-brain quickly, contain divergence automatically, preserve availability where safe, and make reconciliation deterministic/minimal-impact.- Assume arbitrary network partitions and possibility of concurrent writes.Detection signals (combine multiple for high confidence):1. Divergent commit histories: detect non-linear replication graph (e.g., region A’s latest LSN not ancestor of region B’s). Use vector clocks or hybrid logical clocks (HLC) per write.2. Conflicting write-sets / key-level conflicts: same key modified in different regions with overlapping timestamps/HLCs.3. Replication lag & missing acks: sustained bi-directional replication failures beyond threshold.4. Control-plane heartbeats + quorum loss: region cannot reach majority of write-ack replicas.5. Application-layer SLO signals: sudden error-rate or stale-read spikes.Automated containment strategy:- Decision policy: graded, rule-based combining signals with severity scores (confidence). Prefer minimally-disruptive options first.- If low-confidence (transient lag): backoff, increase heartbeat cadence, pause non-essential background work.- If medium-confidence (divergence detected but limited keys): enable per-key write fencing — convert conflicting keys to read-only or to causal-merge workflows; route reads to last-good region; flag keys for manual/automated reconciliation.- If high-confidence widespread split-brain or quorum loss: perform region-level write-fencing using distributed lease (e.g., Paxos/RAFT-backed controller or cloud leader election). Only the lease-holder accepts writes; other regions switch to read-only or queue local writes for deferred merging.- Preserve availability: allow reads in all regions; allow writes in the lease-holder/majority region. For geo-local low-latency needs, provide "local-write with conflict flagging" operative only when eventual reconciliation is acceptable.Automation details:- Implement detectors as streaming jobs that consume replication metadata (LSNs, HLCs), write-audit logs, and heartbeat metrics; emit incidents with confidence score.- Use an orchestration runbook engine (e.g., self-healing operator) to atomically apply containment actions: set feature flags in DB (fence tokens), update load-balancer/write-router rules, and notify downstream services.- All state transitions logged with causally-linked tokens for safe rollback.Reconciliation & post-containment:- Record conflicting writes in a conflict store with metadata for automated resolution rules (last-writer-wins only if safe, CRDT merge for supported datasets, or human-in-the-loop for critical keys).- Run gradual catch-up: replicate leader’s history into other regions, apply resolved changes, then revoke fences.- Post-incident: automated postmortem checklist, adjust thresholds or add more signals.Trade-offs:- Aggressive fencing reduces divergence risk but impacts write availability in some regions.- Per-key containment is finer-grained but more complex.- Use confidence-scored, staged automation to balance availability vs correctness.This approach gives fast detection using multiple signals, staged containment to minimize impact, and clear reconciliation paths to restore consistency.
Unlock Full Question Bank
Get access to hundreds of Multi Region Disaster Recovery interview questions and detailed answers.
