Multi Region and Geo Distributed Systems Questions
Designing and operating systems and infrastructure that span multiple geographic regions and cloud or on premise environments. Candidates should cover data placement and replication strategies and trade offs such as synchronous versus asynchronous replication, single primary versus multi master topologies, read replica placement, quorum selection, conflict detection and resolution, and techniques for minimizing replication lag. Discuss consistency models across regions including strong, causal, and eventual consistency, cross region transactions and the trade offs of two phase commit versus compensation patterns or eventual reconciliation. Explain latency optimization and traffic routing strategies including read and write locality, routing users to the nearest region, domain name system based routing, anycast, global load balancers, traffic steering, edge caching and content delivery networks, and deployment techniques such as blue green and canary rollouts across regions. Cover network and interconnect considerations such as direct private links, virtual private network tunnels, internet based links, peering strategies and internet exchange points, bandwidth and latency implications, and how they influence failover and replication choices. Describe availability zones and their role in fault isolation, how to design for high availability within a region using multiple availability zones, and when to use multi region active active or active passive topologies for resilience. Plan for disaster recovery and resilience including failover detection and automation, backup and restore, recovery time objectives and recovery point objectives, cross region failover testing, run books, and operational playbooks. Include security, identity, and compliance concerns such as data residency and sovereignty, regulatory constraints, cross border encryption and key management, identity federation and authorization across regions, and cost and legal implications of region selection. Discuss operational practices including monitoring and alerting for region health and replication metrics, capacity planning, deployment automation, observability, run book procedures, and testing strategies for simulated region failures. Finally reason about workload partitioning and state localization, replication frequency, read and write locality, cost and complexity trade offs, and provide concrete patterns or examples that justify chosen architectures for global user bases.
HardSystem Design
20 practiced
For an e-commerce platform, checkout must be strongly consistent while product browsing can be eventually consistent across regions. Design the database architecture and transactional flow to achieve this hybrid consistency model, minimizing checkout latency for users far from the primary region.
Sample Answer
**Hybrid consistency goals**- Strong consistency for checkout; eventual consistency for browsing to scale globally and improve read latency.**Database architecture**- Partition data into two stores/patterns: 1) Checkout-critical data (orders, payments, inventory reservation): single-writer-per-partition with strongly-consistent store (regional primary with cross-region synchronous or semi-sync replication to a small quorum), or a central transaction service. 2) Catalog/browsing data: globally replicated read-optimized stores (eventual) using async replication or CDN.**Transactional flow**- Checkout path: - Step 1: Reserve inventory via a strongly-consistent service located in a nearby write-capable region. Use short-lived reservations (hold tokens) with TTL. - Step 2: Authorize payment using regional payment gateways. - Step 3: Commit order in strongly-consistent DB (use local quorum write or route to primary owning partition). On success, publish an event to global event bus to update catalog and analytics asynchronously.- Browsing path: - Reads served from regional replicas / CDN; updates from event stream propagate eventually.**Latency optimization**- Place checkout service endpoints in multiple regions; use partition ownership so user writes go to nearest write-capable partition; if partition primary is remote, use reservation proxy to avoid full cross-region commit latency.- Use optimistic local reservations (fast local ACK) and finalize with strong commit in the background; expose order state to user (pending -> confirmed) to maintain UX.**Consistency guarantees & compensations**- Keep reservation TTL and automated release to avoid oversell.- Compensating workflows to reconcile inconsistencies discovered during final commit.This architecture minimizes checkout latency by localizing reservation and using strong commits on a per-partition basis while allowing browsing to be fast and eventually consistent.
EasyTechnical
27 practiced
Define Recovery Time Objective (RTO) and Recovery Point Objective (RPO). When designing multi-region architectures, describe a process to map business requirements to RTO/RPO targets for multiple services and give examples of choices (and trade-offs) you might make for a customer-facing API vs an analytics pipeline.
Sample Answer
**RTO and RPO mapping for multi-region architectures****Definitions**- RTO (Recovery Time Objective): Maximum allowable downtime before service restored.- RPO (Recovery Point Objective): Maximum acceptable data loss (time window of lost data).**Process to map business needs to targets**1. Inventory services and rank by business impact (revenue, regulatory, customer experience). 2. For each service, interview stakeholders: acceptable downtime and acceptable data loss.3. Translate to RTO/RPO SLAs (e.g., Tier 1: RTO < 1 min, RPO = 0; Tier 2: RTO < 1 hour, RPO < 15 min).4. Select architecture options (sync replication, async replication, backups, multi-region) to meet targets and estimate cost.**Examples & trade-offs**- Customer-facing API: Business-critical, low latency and no data loss → Aim for RTO < 1 min, RPO = 0 using active-active with synchronous or quorum-based writes; trade-off: higher latency and cost.- Analytics pipeline: Tolerates some delay/loss → RTO hours, RPO minutes to hours; use async replication, periodic snapshots, cheaper storage; trade-off: possible data reprocessing and eventual consistency.**Outcome**: Map SLAs to clear architecture and cost decisions, document monitoring and tests (failover drills) to validate RTO/RPO.
EasyTechnical
26 practiced
What is quorum selection in distributed replication systems? Explain majority quorums, configurable (read/write) quorums, and how quorum size and selection across regions affect write latency, availability, and partition tolerance.
Sample Answer
**Definition & purpose**- Quorum selection determines how many nodes must agree for reads/writes in distributed replication to ensure consistency and availability.**Majority quorums**- Requirement: more than half nodes must respond (N/2+1). Guarantees at most one commit leader set and strong consistency.- Effect: higher write latency across regions because write must wait for majority acknowledgements; better consistency and partition safety.**Configurable read/write quorums (R/W)**- Use parameters R (reads) and W (writes) with N replicas. Consistency when R + W > N.- Tuning: choose smaller W to reduce write latency (at cost of potential stale reads) or smaller R for faster reads.**Cross-region impacts**- Large quorum spread across regions increases latency (inter-region RTTs) but improves fault tolerance. Placing quorums within a latency-aware subset (region-majority) reduces latency but risks availability during region outages.**Trade-offs (CAP prism)**- Strong consistency (large quorum) reduces availability under partitions. Smaller quorums increase availability and lower latency but can return stale data.Choose quorum size/location based on workload: critical financial writes prefer majority with low-latency links; read-heavy global services may favor eventual consistency with tuned R/W.
HardSystem Design
26 practiced
You have a legacy stateful monolith with session state that must be migrated to multi-region microservices without downtime. Provide a migration plan with concrete phases, describe how you'd sync session state, ensure consistency during cutover, and outline rollback and validation procedures.
Sample Answer
**Phased migration plan**1) Assess & instrument: map session data shape, TTLs, access patterns; add telemetry.2) Introduce session store layer: deploy a globally-available session service (Redis cluster with regional replicas or a durable session DB) alongside monolith.3) Dual-write & sync: - Implement proxy in monolith to write session to new store while retaining local sessions. - Run continuous background sync to copy existing session state into new store; ensure idempotency and conflict resolution (last-write wins with timestamps or version vectors).4) Gradual reads switch: - Route a small percentage of reads to new service, validating parity. Increase rollout progressively.5) Cutover: - Once parity high and validation passes, switch all reads/writes to microservice-based session service and decommission monolith session code path.6) Cleanup & rollback: - Keep monolith capable of re-enabling old path for a rollback window. Use feature flags and traffic split for rapid rollback.**Consistency during cutover**- Use sticky session tokens and ensure new store supports TTL semantics identical to monolith.- For in-flight sessions, dual-read (read from both sources and reconcile) during cutover period.**Rollback & validation**- Validation: checksum of session counts, random session spot-checks, integration tests replicating user flows.- Rollback steps: flip feature flag to re-enable monolith session path and halt sync jobs; verify successful failback.This approach minimizes downtime by parallel-running session stores, incremental traffic migration, and robust rollback via feature flags and dual-write sync.
HardTechnical
21 practiced
Propose a capacity planning approach for inter-region bandwidth when continuous replication of large datasets is required: 1 TB per hour between regions with variable inter-region RTT (~100ms ± 50ms). Describe throttling, burst handling, replication scheduling, and cost trade-offs including choosing between dedicated links and internet transfer.
Sample Answer
**Inter-region bandwidth planning for 1 TB/hr continuous replication (RTT ~100ms ±50ms)****Baseline capacity**- 1 TB/hr ≈ 2.22 Gbps sustained. Add overhead (encryption, protocol) → plan for ~2.5–3 Gbps.**Throttling & scheduling**- Implement token-bucket throttling to enforce sustained throughput and allow controlled bursts.- Schedule heavy bulk transfers during off-peak windows with higher allowed throughput; limit peak rate to avoid congestion.**Burst handling**- Allow short bursts (e.g., 10s–1min) above sustained rate by consuming burst tokens; monitor RTT to back off.- Use congestion-aware transport (TCP tuning, BBR) and parallel streams to utilize available window given RTT variability.**Replication strategies**- Continuous streaming for near-real-time, with chunked checkpointing and resumable transfers.- For large daily deltas, use incremental snapshots during low-cost windows.**Cost trade-offs: dedicated links vs internet**- Dedicated link (Direct Connect/Interconnect): predictable bandwidth, lower egress variability, better SLA, higher fixed cost — choose when predictable replication urgency and compliance matters.- Internet transfer: cheaper, flexible, but variable latency, potential throttling and egress cost unpredictability.- Recommendation: baseline via dedicated link sized for sustained 3 Gbps; overflow and bulk window via encrypted internet transfers to lower cost if budget constrained.**Operational controls**- Instrument bandwidth, RTT, retransmit rates; autoscale replica ingestion workers.- Alert on sustained lag or backlog thresholds; implement backpressure from consumer side to prevent OOM/backlog.
Unlock Full Question Bank
Get access to hundreds of Multi Region and Geo Distributed Systems interview questions and detailed answers.
