Data Consistency During Failover and Multi Region Replication Questions

Handling consistency challenges when failing over between regions. Understand synchronous replication (slower, consistent) vs. asynchronous replication (faster, potential data loss). Discuss split-brain scenarios (if communication between regions breaks, how do you prevent two independent systems each thinking they're the primary?). At Staff level, show understanding of tradeoffs and practical operational considerations.

MediumSystem Design

25 practiced

Design a multi-region read strategy that provides low read latency and strong read-after-write consistency for users who may move between regions frequently (e.g., mobile users traveling). Describe caching, write-forwarding, session tokens, and the trade-offs in latency, availability, and complexity.

Sample Answer

Requirements:- Low read latency from whichever region the user is in- Strong read-after-write (RAW) consistency for a user who may move between regions frequently- High availability and reasonable complexity for SRE operations

High-level approach:- Route reads to a local read cache/replica for low latency.- Ensure writes are globally consistent by forwarding writes to a single primary region or to a strongly-consistent global datastore with linearizable writes.- Use session tokens (sticky session metadata) to guarantee RAW consistency by routing reads to a region that is guaranteed to have the latest write or by validating cache freshness.

Architecture:- Global load balancer → regional ingress → local read cache (L1: in-memory like Redis or CDN edge) → regional read replica (R2) → write-forwarding path to primary/global store.- Writes received in any region are synchronously forwarded to the primary (or to a strongly-consistent global datastore using consensus). Primary applies write and returns an acknowledgement.- On write ack, the origin region updates local cache and issues a session token (version vector or logical timestamp) to the client.- Client sends session token on subsequent reads; regional layer checks token freshness: - If local replica’s applied timestamp >= token timestamp → serve from cache/replica. - Else either block/wait for replication (bounded wait) or proxy the read to primary.

Caching, session tokens and mechanisms:- Session token = (user_id, write_ts). Keep compact logical clock (Lamport or per-user increment).- Cache keyed by user and object; include version metadata.- Background replication pushes updates to replicas; use change streams to minimize lag.- If client moves to a new region, the session token tells new region whether it can serve locally or must validate.

Trade-offs:- Latency: - Best: reads served locally when replica is up-to-date. - Worst: reading after recent write in different region may incur extra hop/proxy or bounded wait to ensure RAW.- Availability: - Write-forwarding to a primary introduces a single-write-path dependency; if primary is down, writes need failover (consensus-based leader election) which adds complexity. - Option: use strongly-consistent global datastore (e.g., Spanner) to avoid manual leader logic but with higher write latency.- Complexity: - Session tokens and validation logic add state and SRE operational burden (monitoring replication lag, token correctness). - Implementing bounded waits vs proxy reads is a design choice: blocking preserves strictness but increases tail latency; proxying increases cross-region traffic but keeps latency bounded.- Consistency vs performance: - Favoring RAW means some reads incur cross-region cost; to optimize, make writes apply locally after ack (read-your-writes fast path) by updating local cache post-ack. - For softer availability, provide an “eventual” fast-read fallback when strict RAW cannot be met (feature toggle).

Operational considerations:- Monitor replication lag metrics, per-region cache hit rate, session-token failures.- SLOs: define read latency percentiles for local hits vs remote validations, and acceptable staleness window.- Automation: leader failover playbooks, cache invalidation tools, canary deployments for replication changes.

This design balances low-latency local reads with strong per-user RAW using session tokens and a single consistent write path; trade-offs are increased complexity and potential write/write-failover latency.

HardTechnical

17 practiced

As SRE lead during a major cloud provider outage in RegionA, craft a decision framework to choose between failing over to RegionB or waiting for RegionA recovery. Use SLO/error budget, data durability guarantees, traffic impact, legal/data-residency constraints, rollback risk, and estimated recovery timelines. Provide a checklist usable under pressure.

Sample Answer

Situation: RegionA (primary) is down from a major cloud provider incident. Decision: fail over to RegionB (active/passive) or wait for RegionA recovery.

Decision framework (prioritized, quantifiable)1. SLO / error budget- Current SLOs (p99 latency, availability). If projected regional unavailability will consume >50% of monthly error budget within the next 2 hours, favor failover.2. Data durability & consistency guarantees- Verify replication lag/tail of writes to RegionB. If RPO <= our SLA (e.g., <5 min) and no single-writer constraints, failover is safer. If writes are single-region-only or there’s risk of data loss/inconsistency, favor wait or partial read-only mode.3. Traffic impact & capacity- Can RegionB absorb peak+margin (at least 120% expected traffic)? If not, consider partial failover (edge routing, degraded features) or wait.4. Legal / data residency- Identify customer segments with residency constraints. If critical customers’ data cannot legally be served from RegionB, do not failover for those tenants — use circuit breakers / tenant routing instead.5. Rollback and reconciliation risk- Estimate complexity to reconcile split-brain or duplicate writes and time to rollback. High reconciliation cost → prefer waiting unless availability trumps consistency.6. Estimated recovery timeline & confidence- Use provider status + telemetry to estimate ETA and confidence band. If recovery likely within acceptable window (per SLO/error budget) with high confidence, wait.7. Business impact & stakeholder alignment- Quantify revenue/CSAT impact; consult product/legal/security leads for go/no-go.

Checklist to use under pressure- [ ] Read current SLO consumption and projected burn rate (dashboard)- [ ] Confirm replication health: last-replicated timestamp, lag, failed shards- [ ] Capacity check: RegionB headroom, autoscale policies, DB R/W limits- [ ] Compliance check: list affected tenants with residency/legal restrictions- [ ] Risk assessment: estimated data-loss window, reconciliation steps, rollback time- [ ] Recovery ETA & confidence from provider and internal telemetry- [ ] Communication plan: stakeholders notified, customer-facing message drafted- [ ] Execute safe mode if partial: route read-only / degrade non-critical features- [ ] If failing over: run pre-approved runbook (DNS/Load-balancer steps, health checks, throttles)- [ ] Post-action: enable enhanced telemetry, start reconciliation plan, schedule postmortem

Decision rule example (can be pre-approved):- If error budget burn >50% in 2h OR customer-impacted revenue > X% AND RegionB can meet durability/legal/capacity checks → failover.- Else wait and mitigate (rate limit, degrade, notify).

This framework gives fast, auditable yes/no criteria and a compact checklist teams can follow under incident command.

HardTechnical

25 practiced

How would you manage and reconcile transaction IDs or monotonic timestamps across regions to support ordering and deduplication? Discuss wall-clock synchronization (NTP), logical clocks, hybrid logical clocks (HLC), watermark strategies, and how to handle clock skew and reordering during failover.

Sample Answer

Start by clarifying requirements: strong ordering per partition/key, global deduplication, and acceptable ordering latency under cross-region failover.

1) Wall-clock sync (NTP/chrony): keep OS clocks close (ms-level) for observability and sane timestamps. But don’t rely on perfect sync for correctness—network partitions and leaps can break monotonicity.

2) Logical clocks: use per-partition monotonic counters (or Lamport clocks) for strict ordering within a single leader/partition. They’re simple and perfect for local ordering and dedupe but don’t map to real time.

3) Hybrid Logical Clocks (HLC): combine physical timestamp with a logical counter. Use physical time when causality permits; bump counter when receiving remotely-timestamped events that would violate monotonicity. HLC gives largely real-time sortable IDs while preserving causality and monotonicity across regions—good for SRE needs (debugging, retention policies) and for deduplication.

4) Watermarks and stable-time: publish advancing watermarks (per partition or globally) indicating that no older transactions will arrive after watermark T. Consumers can buffer events until watermark passes, then safely emit/commit and garbage-collect dedupe state. Watermarks can be computed by leader election or via gossip/consensus.

5) Handling clock skew & failover:- Keep HLC tolerant: on failover, the new leader initializes its physical time but will set logical counter to max(local HLC, incoming HLC)+1 when processing older timestamps—preventing regressions.- During failover, buffer incoming events and use HLC+watermark: delay final commit until watermark catches up or until configured timeout (tradeoff: latency vs correctness).- Ensure IDs include region/leader identifier to avoid accidental collisions.- For deduplication, keep idempotency tokens keyed by transaction ID + window based on watermark; expire tokens only after watermark advances.

6) Practical SRE policies:- Monitor clock drift, NTP offsets, and HLC counter anomalies; alert on large physical/logical divergence.- Define SLA for maximum buffering delay and watermark lag.- Test failover paths with injected skew and partition to validate ordering and dedupe behavior.

Trade-offs: pure logical clocks lose real-time semantics; pure physical rely on sync. HLC + watermarks offers practical balance: near-real-time ordering, causal correctness, and operational observability—suitable for cross-region production systems.

MediumTechnical

18 practiced

During an incident you observe replication lag in a cross-region setup spike to 5 minutes and catching up slowly. Walk through the immediate mitigations you would perform as an SRE, how you would prioritize actions, which teams to involve, and longer-term fixes you would propose to prevent recurrence.

Sample Answer

Situation: Cross-region DB replication lag spiked to ~5 minutes and is catching up slowly during peak traffic.

Immediate mitigations (first 0–15 mins) — prioritized by impact on availability and data integrity:1. Triage & confirm: verify alerts, dashboard metrics (replica lag, apply/sec, network RTT, CPU, I/O, replication queue length). Correlate with recent deploys, maintenance, or traffic spikes.2. Pause non-essential writes: if read replicas are used for analytics/batch jobs, throttle or pause those to reduce load on primary and network.3. Promote reads to healthier replicas: update routing/traffic split to direct read traffic away from the lagging replica(s) to up-to-date ones or region-local caches.4. Stabilize replication pipeline: if async replication backlog is huge, consider temporarily increasing replica apply throughput (tune replica SQL worker threads, apply batch size) and ensure network QoS to replication channels.5. Mitigate risk of split-brain: prevent any accidental failovers while lag is high; set failover automation to manual.

Who to involve (in order):- On-call SRE (lead)- DB/DBA team (replication internals, tuning)- Network/infra team (cross-region links, packet loss)- Service owners/PM (customer impact, communication)- App engineers (if schema or writes contributed)

Short-term diagnostics (15–60 mins):- Collect logs (primary and replica), check I/O wait, lock contention, long-running transactions on primary, binlog generation rate.- Run targeted queries to find large transactions blocking replication apply.- If network is bottleneck, shift traffic or use compression for replication.

Longer-term fixes:- Capacity: provision faster links, dedicated replication lanes, scaled IOPS on storage.- Reliability: implement multi-threaded or parallel replication where supported.- Observability: add finer-grained replication metrics, alerts on growth rate of backlog, and automated runbooks.- Traffic shaping: rate-limit heavy batch jobs and enforce write quotas or backpressure.- Automation: safe throttling automation to pause heavy jobs when lag > threshold and circuit-breaker for failovers.- Testing: yearly chaos tests for cross-region replication and playbooks for recovery.- Post-incident: run blameless RCA, quantify impact vs SLOs, implement prioritized action items and track to completion.

Result-focus: stabilize traffic to protect reads, eliminate backlog causes, and implement monitoring + automation to prevent recurrence and reduce MTTR.

MediumTechnical

17 practiced

You're using etcd clusters in each region for leader election and metadata. Explain a safe, automated procedure to migrate leadership and primary responsibilities from RegionA to RegionB for planned maintenance, detailing steps to avoid split-brain, how to drain traffic, update leases, validate application-level consistency, and automate rollback if the migration fails.

Sample Answer

Goal: move primary responsibilities (leader/primary metadata) from RegionA to RegionB for maintenance with zero split‑brain, validated consistency, and automated rollback.

Plan (high level)1. Preconditions & safety:- Ensure RegionB etcd cluster healthy and has quorum + enough capacity.- Ensure clock sync (NTP), consistent etcd versions, network routes allowed.- Create maintenance window & increased alerting.

2. Drain traffic from RegionA to RegionB:- Update load‑balancer weights gradually (canary shifts e.g., 10%→50%→100%) with health checks.- Quiesce RegionA-only writes by flipping feature-flag to read-only or blocking new leader leases there.

3. Controlled leadership transfer:- Use etcd’s lease/leader mechanisms with fencing. Implement leader resignation from RegionA by revoking its leadership key using a compare-and-swap (only if current leader) and waiting for RegionB to acquire.- Prefer a coordinated, transactional handoff: create a "primary-transfer" key with TTL and signed token. RegionA sets transfer intent; RegionB attempts atomic CAS to become leader if it sees intent and healthy quorum.

Automation snippet (pseudo-python):

python

# simplified: use python-etcd3
from etcd3 import Etcd3Client
etcdA = Etcd3Client(host='regionA-endpoint')
etcdB = Etcd3Client(host='regionB-endpoint')

# 1. Precheck
if not etcdB.status().leader:
    raise Exception("RegionB no leader/quorum")

# 2. Request RegionA leader to resign
etcdA.delete_if_value('service/leader', current_leader_id)

# 3. Wait for RegionB to acquire leader
wait_until(lambda: etcdB.get('service/leader') == expected_id, timeout=120)

4. Update leases & metadata:- Migrate long‑lived leases/tokens by copying keys with version checks (ETCD txn: get-and-put) so you don’t overwrite newer data.- Use etcd transactions: if key.version == X then write new owner; else abort and reconcile.

5. Validate application-level consistency:- Run end-to-end smoke tests: read/write workflows, cross-region replication checks, compare checksums/hashes of critical metadata.- Monitor metrics: error rates, latency, write success rates, quorum changes.

6. Rollback automation:- Define failure criteria: leader not acquired within timeout, error rate spike > SLO, failed validation tests.- Rollback steps: stop client writes to RegionB, revert LB weights to RegionA, re-create leader key in RegionA using saved lease token, restore leases from snapshots.- Automate with orchestrator (Argo/Ansible/Runbook service) that executes stepwise with checkpoints and can revert based on health checks.

7. Post-migration:- Keep increased monitoring for cooldown period.- Take etcd snapshot of RegionB, document lessons.

Key safety points / why:- Use atomic etcd transactions and fencing tokens to avoid split‑brain.- Gradual traffic shifts allow quick rollback if anomalies.- Version/lease checks prevent overwriting newer data during migration.- Automated timeouts and health checks ensure deterministic rollback.

This procedure balances safety (quorum, fencing, transactions), automation (scripted steps + canary LB shifts), and validation (tests + metrics) to migrate leadership reliably.

Unlock Full Question Bank

Get access to hundreds of Data Consistency During Failover and Multi Region Replication interview questions and detailed answers.

Join thousands of developers preparing for their dream job.