Design summary (goal): low read latency worldwide using local read replicas, eventual consistency for product-catalog data, single writable primary region for writes, fast failover and minimized write latency.
High-level architecture
- One writable Primary region (A) holding the authoritative DB.
- Multiple Read Replica regions (B, C, D) hosting read-only replicas and a caching layer (CDN + regional cache).
- Asynchronous geo-replication pipeline from Primary -> replicas (CDC / log shipping).
- Global traffic routing via DNS latency-aware policy + Anycast / regional edge proxies.
Replication strategy
- Use async physical or logical replication (WAL shipping or CDC like Debezium) to stream changes from Primary to regional replicas.
- Apply changes in near-real-time (sub-second to low seconds) — tune batch/window for throughput vs freshness.
- For non-critical catalog data, favor async replication to avoid write latency hit.
Traffic routing
- Read path: clients resolve to nearest region via geo-DNS or Global Load Balancer; edge proxies serve from regional cache first, then read replica.
- Write path: clients send writes to nearest regional edge which forwards to Primary via:
- Persistent optimized connection (GRPC, HTTP/2) with connection pooling and regional egress peering to reduce RTT.
- Optionally use a write-proxy fleet (stateless) in each region to batch/route writes to Primary, reducing handshake overhead.
- Health checks + latency-based DNS fallback to route around unavailable regions.
Conflict resolution / consistency
- Since writes go only to Primary, conflicts are minimized. For the small window during failover or simultaneous local writes (e.g., cached optimistic updates), use:
- Monotonic timestamps (NTP-synced or hybrid logical clocks) + Last-Write-Wins for simple fields.
- For richer merges (inventory, attributes), use application-level merge rules or CRDTs where appropriate.
- Periodic background reconciliation jobs (read-compare-fix) to detect replication lag or divergence and repair.
Minimizing write latency
- Keep Primary in a region with highest write volume and good network connectivity.
- Use:
- Persistent multiplexed connections and HTTP/2 or gRPC from edge to Primary.
- Batching small updates where acceptable.
- Fast path acks: Primary can acknowledge receipt once committed locally (not after replicas apply) for speed; use monitoring to track replication lag.
- For UX improvements, allow local optimistic responses (UI shows change) with eventual correction if write fails.
Failover & promotion
- Automated leader election using a consensus mechanism (external coordination via etcd/consul or cloud-managed regional failover) to promote a new Primary if current Primary becomes unreachable.
- Steps on failover:
- Quiesce writes, ensure last WAL segments are shipped/applied, cut-over to promoted region after ensuring no split-brain (use majority / lease-based locks).
- Reconfigure read routing and update DNS/global LB.
- Mitigations: cross-region quorum for safe promotion, manual intervention window for risky promotions, and automated rollback mechanisms.
Observability & operations
- Monitor replication lag, error rates, traffic patterns.
- Alert on >X seconds replication lag, write queue growth, or proxy failures.
- Run regular DR drills to validate promotion and rollback procedures.
This design trades immediate strong consistency for low-latency global reads and operational simplicity; it minimizes write latency by optimizing network paths and using async replication, while safe failover relies on coordinated promotion and reconciliation.
