Designing systems for resilience and availability across geographic regions, including strategies for cross region replication, failover, and operational recovery. Candidates should understand deployment models such as active active and active passive and the trade offs they imply for availability, consistency, cost, and operational complexity. Discuss replication topologies and the differences between synchronous and asynchronous replication and how those choices affect consistency and the recovery point objective. Cover leader election and failover coordination mechanisms, conflict resolution approaches including last write wins, version vectors, and convergent data types, and implications for transactional guarantees and global transactions. Include global traffic routing and failover techniques such as DNS based routing, global load balancing, health checks, and the impact of routing and time to live on failover behavior. Address data partitioning and cross region latency trade offs, strategies for orchestrating data recovery and region seeding, backup and restore practices, and testing approaches such as planned failovers, rehearsal drills, and chaos testing. Explain how to derive and meet recovery time objective and recovery point objective from business requirements, and consider monitoring, observability, automation, runbooks, cost considerations, and compliance and data residency requirements.
MediumSystem Design
70 practiced
Design a global traffic routing strategy using a combination of DNS, global load balancers, and health checks for an API service across three regions. Explain how you would configure health checks, failover priorities, and TTLs to balance failover speed and stability. Also discuss how to handle long-lived TCP or WebSocket connections during failover.
Sample Answer
Requirements & constraints:- Global API service in 3 regions (primary in region A, secondaries B/C).- Minimize downtime, avoid split-brain, balance failover speed vs DNS caching.- Preserve correctness for data ingestion; handle long-lived TCP/WebSocket sessions.High-level design:- DNS: Route53/Cloud DNS with latency-based or geolocation routing + health-check-based failover records.- Global Load Balancer: Use cloud global LB (GCP Global LB / AWS ALB+Global Accelerator) in front of regional backends for global anycast and session affinity when needed.- Regional backends: regional autoscaling groups / instance pools behind regional LBs. Each regional LB performs local health checks; the global LB polls regional LBs.Health checks & failure detection:- Two-layer checks: 1. Regional LB → backend instances: TCP + application-level HTTP health endpoint (/health) that verifies process, disk, queue lag, and downstream datastore connectivity. Frequency 10s, healthy after 2 successes, unhealthy after 3 failures. 2. Global LB/DNS health probes → regional LB: application-level check including a lightweight write/read (or status of ingestion pipeline) to detect partial failures. Frequency 10–15s, unhealthy after 2–3 failures.- Use rising/falling thresholds to avoid flapping. Add synthetic checks for downstream dependencies (DB, message queues).Failover priority & routing:- Configure primary preference: prefer region A for clients when healthy (via weighted DNS or LB priority). Regions B/C carry smaller weight for load distribution and as failover targets.- On detecting region A unhealthy at global LB/DNS, shift traffic to B then C based on priority/weights. Use health-check status and automated traffic shifting (not manual).DNS TTL & tradeoffs:- Set DNS TTL to 30–60s for failover responsiveness while limiting global DNS churn. Use slightly higher TTL (120s) for stable endpoints under normal operation.- Combine DNS with global LB/anycast so many clients keep a stable IP even when origin region changes—reduces reliance on aggressive TTLs.- Implement jitter/randomization and gradual weight shifts (traffic ramp-down over 30–120s) to avoid sudden surges on failover targets.Connection draining / long-lived connections:- Use graceful connection draining on LB/backends: when a node/region marked unhealthy, stop accepting new connections and allow existing connections to drain for a configurable timeout (e.g., 5–10 minutes depending on typical connection length). For WebSocket/TCP: - Signal clients via application-layer message (if protocol allows) to reconnect. - Return 503 + Retry-After for new requests during graceful window. - Encourage client reconnection logic with exponential backoff and ability to re-resume or replay idempotent ingestion.- For very long streams, prefer sticky routing via global accelerator or session affinity so clients maintain a stable path; on failover, rely on client reconnects to re-establish on healthy region.- If exactly-once semantics required, design ingestion to be idempotent (dedup keys) or use sequence checkpoints so reconnect can continue without data loss.Operational considerations:- Monitor failover metrics, health-check flapping, and reconnection rates. Alert on increased reconnects or backlog growth.- Test failover regularly (chaos testing) to validate TTLs, drain time, and client behavior.- Document recovery expectations and client SDK best practices (reconnect, idempotency, backoff).This approach balances fast failover (short TTLs, aggressive global LB routing) with stability (health thresholds, gradual weight shifts, connection draining) while protecting correctness for data ingestion workloads.
EasyTechnical
84 practiced
Describe basic leader election patterns for a distributed coordination service in a multi-region architecture. List commonly used tools (e.g., etcd, Zookeeper, Consul) and explain how quorum placement and latency influence leader stability and failover election time across regions.
Sample Answer
Basic patterns- Single global leader: one consensus group spans regions (all replicas participate). Simple correctness but high cross-region latency for heartbeats/elections.- Region-local leaders with global coordination: each region runs its own cluster; a global coordinator or control plane elects an active region (failover between region leaders). Low latency in-region, slower cross-region failover.- Hybrid (read-local, write-primary): local followers serve reads; a single primary (possibly in preferred region) accepts writes. Useful when reads dominate.Common tools / protocols- ZooKeeper (Zab): leader election via ephemeral znodes and notifications; requires majority quorum.- etcd (Raft): leader using Raft heartbeats and election timeouts; supports learners (non-voting) to reduce quorum size.- Consul (RAFT + sessions): uses Raft for leader and sessions for locks; supports WAN federation (multiple datacenters).- Additional: Apache Curator (client patterns for ZooKeeper), Patroni (Postgres HA using etcd/ZK), observers/witnesses patterns.Quorum placement and latency effects- Quorum must be a majority of voting nodes. Placing voting nodes across regions increases availability against region failure but raises median RTTs for heartbeats and append entries.- Higher RTT -> longer election timeouts required to avoid false positives. That increases failover time (time to detect failure + new election).- Cross-region quorums also lengthen commit latency for writes (synchronous replication to majority), affecting throughput and tail latency.Trade-offs & recommendations for data engineers- For low-latency local control planes, keep a voting majority in one region and use read-only learners in others; cross-region failover handled as planned outage/failover.- For true multi-region availability, distribute voting nodes across regions (odd number), accept higher write latency and tune election timeouts.- Use non-voting observers/witnesses to reduce quorum write cost while preserving availability.- Tune Raft/Zab timeouts proportional to inter-region RTT, monitor heartbeat loss, and implement graceful region-failover playbooks.- Test failure scenarios to measure realistic election/failover times and ensure pipelines handle short leader unavailability.
MediumSystem Design
86 practiced
You manage a distributed metadata service (e.g., Hive Metastore or a central schema registry) used by analytics clusters in multiple regions. Design a leader election and failover coordination approach so only one active leader manages writes to the metadata store at a time. Explain how you avoid split-brain, how you handle network partitions, and how clients discover the current leader.
Sample Answer
Requirements & constraints:- Single active writer globally; multi-region readers allowed- Tolerate region failures and network partitions without split-brain- Low-latency leader discovery for analytics clustersHigh-level approach:1. Use a consensus-backed coordinator (Raft-based cluster: etcd/Consul or a small ZooKeeper ensemble) deployed across regions with an odd number of voting nodes and quorum placement tuned to expected failure domains. This cluster performs leader election and issues short-lived leader leases/fencing tokens.2. Leader role:- The elected leader is the only instance allowed to perform metadata writes. It obtains a lease (timestamped token) from the consensus cluster.- On each write, leader attaches its monotonically increasing fencing token (term+index) so replicas can reject stale-writes.3. Avoid split-brain:- Rely on quorum-based elections: a candidate must acquire majority votes to be leader. If network partition prevents a majority, no leader is elected—safe by construction.- Use lease expiry + fencing tokens: if a leader loses quorum or its clock drifts, its lease will expire; other nodes will refuse writes carrying an older token.4. Network partitions & failover:- If leader loses majority (consensus detects), it relinquishes lease and stops accepting writes.- A new leader only becomes active after acquiring quorum and a fresh lease.- Implement graceful handoff: leader requests transfer via consensus; if impossible, wait for lease expiry to avoid overlap.- For transient cross-region latency, tune lease TTL (long enough to avoid churn; short enough to bound failover time).5. Client discovery:- Clients query the consensus service’s key (e.g., /metadata/leader) which stores current leader address + fencing token. Use local caching with TTL and watch/long-poll to get updates. Provide SDK helper: on stale leader error (fencing mismatch), refresh immediately.- Optionally expose a DNS SRV with low TTL backed by a proxy that resolves current leader.6. Reads & performance:- Allow reads from followers (stale reads acceptable) or route through the leader for strongly consistent reads.- For high-read volumes, maintain regional read-only caches/replicas; ensure writes flow to leader.Trade-offs:- Cross-region quorum increases latency; mitigate by placing quorum nodes across regions strategically or use leader-in-region affinity with read replicas.- Simplicity vs availability: smaller TTLs favor faster failover but risk flapping during transient network issues.Operational safeguards:- Automated health monitors, alerting on leader flaps- Chaos testing (partition/failover) to validate lease/ fencing behavior- Safe migration plan for cluster topology changesThis design uses proven consensus primitives + fencing leases so only one active writer exists, preventing split-brain and enabling predictable failover and low-latency leader discovery for analytics clients.
EasyTechnical
69 practiced
Explain Recovery Time Objective (RTO) and Recovery Point Objective (RPO) in the context of multi-region disaster recovery for data systems. For a data pipeline ingesting 1 TB/day, provide two concrete examples of business requirements that would map to different RTO and RPO targets (one aggressive, one relaxed). For each example, describe the architectural trade-offs (cost, replication strategy, operational complexity) required to meet those targets.
Sample Answer
RTO (Recovery Time Objective) is the maximum acceptable downtime after a failure — how long systems can be unavailable. RPO (Recovery Point Objective) is the maximum acceptable data loss measured backward from the time of failure — how much recent data you can lose.Example 1 — Aggressive (real-time analytics for fraud detection)- Business requirement: zero/near-zero business impact for fraud signals; lose at most 5 minutes of data; system back online within 15 minutes.- Targets: RPO = 5 minutes (~3.5 GB lost max), RTO = 15 minutes.- Architecture & trade-offs: - Replication: synchronous or near-synchronous streaming replication (Kafka MirrorMaker/Confluent Replicator, cross-region CDC to replicas), multi-region active-passive or active-active with global load balancing. - Storage: hot replicas in a different region (streamed to object store + real-time materialized views). - Cost: high — double running clusters, cross-region egress, lower-latency storage. - Operational complexity: high — failover automation, consistency checks, conflict resolution for active-active, monitoring and runbooks. - Benefit: minimal data loss and fast recovery for critical use-cases.Example 2 — Relaxed (daily reporting / historical analytics)- Business requirement: daily aggregated reports acceptable; losing up to one day of ingest is tolerable; recovery within 24–48 hours.- Targets: RPO = 24 hours (up to 1 TB), RTO = 24–48 hours.- Architecture & trade-offs: - Replication: scheduled daily backups or asynchronous cross-region replication (periodic snapshots of object store / EBS / HDFS, incremental backups). - Storage: cold replicas or archived snapshots in secondary region. - Cost: low — no hot standby, cheaper storage classes in DR region, reduced cross-region transfer. - Operational complexity: moderate — restore testing, orchestrated restore playbooks, longer manual steps. - Benefit: lower cost and simpler ops at expense of potential data loss and longer downtime.Summary trade-offs: tighter RTO/RPO require synchronous/real-time cross-region strategies, higher cost and complexity; relaxed targets let you use asynchronous snapshots and cold recovery to save cost but accept more data loss and slower recovery. Choose based on business impact and cost tolerance, and validate with regular DR drills.
HardSystem Design
126 practiced
Design a cross-region consensus strategy using etcd spanning three regions such that leader elections and majority operations remain reliable with RTO <= 2 minutes after regional failures. Explain quorum placement, latency implications, how you handle slow nodes, and how to avoid split-brain during network partitions.
Sample Answer
Requirements & constraints:- 3 regions (A, B, C). RTO ≤ 2 minutes for recovery after regional failure. Strong consistency for control-plane ops (etcd) across regions to coordinate data pipelines.- Minimize cross-region latency impact on common operations while preserving safety (no split-brain).High-level approach:- Run an odd number of etcd members across regions: 5 members total (2 in Region A, 2 in Region B, 1 in Region C) or preferably 7 if capacity allows. Use weighted/multi-site quorum placement so majority survives a single-region outage and leader can be elected quickly.- Example chosen: 5 nodes: A1, A2; B1, B2; C1. Quorum = 3. Losing any single region still leaves at least 3 nodes in two-region failure scenarios? (Single-region failure: if C fails, A+B = 4 nodes → quorum ok. If A fails, B+C = 3 → quorum ok.)Quorum placement rationale:- Place at least two nodes in the two lowest-latency regions to serve as primary read/leader candidates. Put the single node in the higher-latency/backup region.- Ensure infrastructure (instance types, IOPS) and availability zones within regions for intra-region redundancy.Latency implications & mitigation:- Synchronous Raft commits require majority acknowledgements → cross-region writes pay cross-region round-trip. To reduce impact: - Keep control-plane writes infrequent; batch non-critical operations. - Use local read-only caches for heavy read patterns (etcd read-index with lease-based caching). - Prefer reads from leader region; set follower-read mode for low-latency reads when acceptable.Handling slow nodes:- Configure conservative election and heartbeat settings tuned for cross-region latencies: - Increase ElectionTimeout to e.g., 3–5s and HeartbeatInterval to ~500–1000ms, balancing faster recovery vs spurious elections. - Monitor follower performance; use automatic failover: mark persistently slow nodes unhealthy and remove them from cluster (or demote) via automation, then replace from healthy regions. - Use raft learner nodes for slower regions so they don't affect quorum until caught up.Avoiding split-brain:- Rely on Raft majority: no partition can form a leader without majority. Ensure cluster only accepts leadership with quorum (etcd default).- Prevent asymmetric network visibility: ensure inter-region connectivity is routed via reliable links and use network-level health checks.- Use static cluster membership (avoid simultaneous multi-master manual changes); require operator approval for membership changes during partitions.- Enable pre-vote (etcd supports) to avoid disruptive elections caused by transient latency.Recovery & RTO considerations:- Automate health checks and replacement: when a region fails, automation detects missing members, triggers leader election among surviving nodes immediately. With tuned election timeouts and good monitoring, leader election should complete within seconds; data-plane reconnection and pipeline restarts targeted < 2 minutes.- For full-region failover drills: have playbooks to scale a new member in another region quickly (AMI/terraform ready) and use snapshot/backup restore if needed.Trade-offs:- 5-node multi-region gives availability across single-region failure with limited write latency increase. 7 nodes increases survivability (can survive a region + single node failure) but increases write latency and operational cost.- Using learners or read-only proxies reduces latency but delays full consistency.Monitoring & operational practices:- Track leader location, commit latency, follower catch-up time, election count. Alert on repeated elections or > 500ms commit latencies.- Run periodic failover drills and validate RTO ≤ 2 minutes.This design uses Raft majority guarantees, tuned timeouts, automation for slow-node handling, and careful quorum placement to ensure reliable leader elections and maintain RTO under 2 minutes after regional failures.
Unlock Full Question Bank
Get access to hundreds of Multi Region Disaster Recovery interview questions and detailed answers.
