Requirements (clarified): multi-datacenter platform, low-latency cross-DC calls where possible, robust failover across DCs, consistent service routing for stateful services, operable by ops teams, and secure discovery.
High-level options compared:
- DNS-based discovery
- How it works: Services register A/CAA/CNAME records or use external DNS (Route 53) with health checks.
- Latency: Very low per-request cost (native DNS resolution cached), but TTLs introduce propagation delays.
- Consistency: Eventual; changes take up to TTL to propagate -> stale records.
- Failover: Good for coarse-grained failover (switch entire DC via DNS failover), poor for per-instance fast failover.
- Operational complexity: Low — familiar tooling, simple automation to update records. Harder to achieve fine-grained health-aware routing and mTLS.
- Client-side discovery with registry (Consul/etcd)
- How it works: Services register with registry; clients query registry (or use client-side library) and pick endpoints (round-robin, sticky).
- Latency: Slight request overhead to registry lookups but clients cache lists; faster than frequent DNS lookups when cached.
- Consistency: Stronger if using quorum reads; across DCs requires federation or WAN gossip leading to eventual consistency or higher latency.
- Failover: Fast local failover if clients use health-checked lists; cross-DC failover needs policy (local-first then remote).
- Operational complexity: Medium — run/elect cluster per DC, handle federation, backfill service health checks and ACLs.
- Service-mesh-based discovery (e.g., Istio + Envoy)
- How it works: Sidecars handle discovery via control plane; central control plane pushes config; service-to-service via sidecar proxies.
- Latency: Small added hop (sidecar) but optimized; supports intelligent routing (circuit breaking, retries) to reduce tail latency.
- Consistency: Control plane provides consistent routing policies; discovery state pushed, so fast convergence within mesh domain; multi-DC adds control-plane federation.
- Failover: Excellent: retries, outlier detection, weighted/binary failover, traffic shifting across DCs.
- Operational complexity: High — run control plane, manage sidecars, observability, mTLS, and multi-DC federation; higher learning curve.
Trade-offs summary and recommendation:
- If you need simple, low-op overhead and coarse failover: DNS-based is acceptable (use short TTLs + active health checks, but expect possible flapping and DNS caching issues).
- If you require fast local failover, client resiliency, and moderate operational cost: Client-side registry (Consul) per DC with federation for cross-DC discovery. Use local-first resolution, service intentions, and TTLs; implement fallback policies for cross-DC calls.
- If you require advanced traffic control, security (mTLS, authz), observability, and smooth canary/traffic shift across DCs: adopt a service mesh with federated control planes. Mitigate complexity by phased rollout (platform infra team owns mesh), automation, and robust testing.
Practical hybrid pattern:
- Use per-DC Consul/etcd for registration + local DNS (or Consul DNS) for simple resolution; place a service mesh in front of critical service groups. For cross-DC failover, use weighted DNS or global load balancer to route to preferred DC, with mesh-level traffic shifting for controlled failover.
Operational notes:
- Monitor metrics: resolution latency, stale-count, failover times.
- Automate registrations, health checks, ACLs, and policy propagation.
- Test cross-DC partitions and recovery regularly.
