Designing systems for resilience and availability across geographic regions, including strategies for cross region replication, failover, and operational recovery. Candidates should understand deployment models such as active active and active passive and the trade offs they imply for availability, consistency, cost, and operational complexity. Discuss replication topologies and the differences between synchronous and asynchronous replication and how those choices affect consistency and the recovery point objective. Cover leader election and failover coordination mechanisms, conflict resolution approaches including last write wins, version vectors, and convergent data types, and implications for transactional guarantees and global transactions. Include global traffic routing and failover techniques such as DNS based routing, global load balancing, health checks, and the impact of routing and time to live on failover behavior. Address data partitioning and cross region latency trade offs, strategies for orchestrating data recovery and region seeding, backup and restore practices, and testing approaches such as planned failovers, rehearsal drills, and chaos testing. Explain how to derive and meet recovery time objective and recovery point objective from business requirements, and consider monitoring, observability, automation, runbooks, cost considerations, and compliance and data residency requirements.
MediumSystem Design
82 practiced
You operate a microservices ecosystem where services have different RPO requirements: user profiles (RPO 30s), payment transactions (RPO 0), analytics (RPO 24h). Design a cross-region replication strategy that meets these service-level RPOs. Specify replication technologies (CDC, streaming, snapshotting), storage choices, and how you would orchestrate recovery of dependent services during a region failure.
Sample Answer
Requirements summary:- User profiles: RPO 30s (near-real-time)- Payment transactions: RPO 0 (zero data loss)- Analytics: RPO 24h (batch acceptable)- Cross-region replication and orchestrated recovery of dependent services on region failureDesign overview:1) Service-by-service replication choices- Payment transactions (RPO 0): Synchronous multi-region write with distributed transaction or durable single-writer + WAL-based replication. Recommended: primary region writes with geo-synchronous WAL shipping + quorum commit using a strongly consistent datastore (e.g., CockroachDB, Spanner) OR for existing relational DB: synchronous replication to standby replicas in remote region using storage-level replication (e.g., cloud provider cross-region persistent disk replication) combined with distributed consensus for leader election. Also enable write-ahead log (CDC) to an immutable archive for auditing.- User profiles (RPO 30s): Asynchronous near-real-time replication using CDC -> change stream (Debezium -> Kafka / Confluent Replicator) to remote-region read replicas. Apply consumer groups with low-latency connectors to the target DB. Use logical CDC to preserve transactions and conflict resolution.- Analytics (RPO 24h): Periodic snapshotting / bulk export (daily ETL) to object storage (S3/GS) replicated across regions (cross-region replication buckets) and loaded into analytics datastore in target region (BigQuery, Redshift, Snowflake).2) Storage choices- Payment: strongly-consistent distributed DB (Spanner/CockroachDB) or primary relational DB with synchronous replication and synchronous commit semantics. WAL archived in cold storage (S3) and replicated.- User profiles: operational DB (Postgres/MySQL) in primary region + read-replica in secondary region updated via CDC-stream into local DB. Kafka (cluster per region with MirrorMaker2 or Confluent Replicator) for streaming backbone.- Analytics: Object storage snapshots (S3) + data warehouse in each region.3) Replication tech details- CDC for user profiles: Debezium -> Kafka topic (compact) -> connector to apply to regional DB. Configure max.latency 5–10s and batching to meet 30s RPO.- Streaming for payment audit & fast failover: If not using truly synchronous DB, use transactional CDC + exactly-once Kafka streams with idempotent sinks and ensure producer acks=all, replication.factor=3 across AZs and region connect to remote cluster or MirrorMaker2 with minimal lag.- Snapshotting for analytics: daily scheduled ETL writing Parquet to CRR-enabled buckets.4) Orchestrated recovery & dependency ordering- Recovery coordinator (control plane): Kubernetes operator or runbook automation (e.g., HashiCorp Nomad/Consul + Terraform + Step Functions). Maintain a dependency graph (service -> upstream data services).- On primary region failure: a) Detect via health checks/region heartbeat and automated policy. b) Promote target region DBs in order: 1. Payment datastore: immediate promotion of standby to primary (must be first). Use automated leader election (consistent quorum) to avoid split-brain. Require manual approval only if split-brain risk. 2. User profiles: switch consumers to regional Kafka topics, apply any remaining CDC backlog, then promote read replica to writable (if needed). Ensure conflict resolution policy (last-write-wins with vector clocks or CRDTs) if dual writes possible. 3. Analytics: resume daily ETL in target region. c) Bring up stateless services with feature flags: - Start critical payment-processing services first, pointed at promoted payment DB. - Start user-facing services next, pointed to promoted profile DB; until CDC catch-up, return cached or degraded profile data where acceptable. - Start analytics last.- Data gating: Use feature flags and traffic shifting (API gateway / traffic manager) to control which services accept writes. Enforce schema compatibility checks before connectors apply changes.5) Operational considerations- Monitoring & SLOs: replicate metrics (lag, replication lag, last-applied LSN), alert thresholds (payment lag >0s triggers immediate failover readiness; profile CDC lag >20s).- Tests & runbooks: Chaos testing (simulated region failover), DR runbooks, automated DR drills.- Security & compliance: Encrypt in transit and at rest; ensure cross-region data residency policies followed.- Trade-offs: Synchronous replication for payments adds latency—if unacceptable, use synchronous local commit + guaranteed durable WAL plus immediate async replication with inbox/outbox pattern and idempotent consumers.This design meets per-service RPOs by using the strongest consistency for payments, low-latency CDC streaming for profiles, and daily snapshots for analytics, plus an orchestrated, dependency-aware failover process to restore services safely.
EasyTechnical
84 practiced
List and describe the essential components of a multi-region disaster recovery runbook for both planned and unplanned failovers. Include automated playbooks, manual steps, verification checks, communications with stakeholders, service-level rollback, and post-failover validation. Explain which steps you would automate and which you would keep manual due to risk or regulatory requirements.
Sample Answer
Overview: A multi-region DR runbook must be a concise, authoritative playbook covering triggers, roles, automation, manual actions, verification, stakeholder comms, rollback, and post-failover validation. Below I list essential components and indicate automation vs manual decisions.1. Trigger & Scope- Clear criteria for planned (maintenance, migrations) vs unplanned (region outage, data corruption).- Automated: health metrics thresholds (alerts) to flag unplanned failovers.- Manual: executive approval threshold for business-impacting decisions.2. Roles & Responsibilities- RACI: Incident Commander, Network, DB, App, Security, Communications, Legal, Customer Success.- Manual assignment & escalation (human judgment).3. Pre-failover checks / Preconditions- Inventory of failover-capable resources, replication lag, licensing, compliance constraints.- Automated checks: replication status, DNS propagation readiness, circuit health.- Manual checks: regulatory holds, change freezes.4. Automated Playbooks- Automated steps executed via runbook automation (Terraform/Ansible/RunDeck/Cloud-runbooks): - Promote read-replicas / failover DB, update routing tables, reconfigure LB target groups, BGP withdraw/announce, update DNS TTL, scale target region infra.- Include safe-guards: canary routing, feature flags, circuit breakers.- Automation requires automated rollback hooks.5. Manual Steps (risk/regulatory)- Manual: data reconstitution for corruption events, legal/ compliance sign-offs before restoring PII-sensitive datasets, transactional cutover for systems that require strict ordering.- Manual: final go/no-go for cross-border data transfer.6. Verification Checks (automated + manual)- Automated smoke tests: API health, basic transactions, synthetic user flows, DB consistency checks.- Manual: deep functional tests, reconciliation of financials, security review.7. Communications- Templates for internal and external messages, cadence (initial, status, resolution).- Automated: status page updates and incident ticketing.- Manual: customer calls, executive briefings, regulatory notifications (legal involvement).8. Service-Level Rollback- Define metric-based rollback criteria (error rate, latency, business KPIs).- Automated rollback triggers for clear thresholds; manual rollback if cause ambiguous or regulatory involvement needed.9. Post-Failover Validation & Remediation- Full regression suite, data integrity audits, performance baselines, security scans, and post-incident RCA.- Automated: test execution, logs aggregation, metrics dashboards.- Manual: RCA facilitation, lessons learned, policy updates.10. Audit, Logging, and Compliance- Immutable logs of all actions (automated playbook runs and manual approvals).- Store approvals for audits (manual signatures recorded).11. Runbook Maintenance- Scheduled drills (tabletop and live failovers), update owners, versioning.Why automate vs keep manual- Automate repeatable, high-frequency, low-ambiguity tasks to reduce MTTR and human error (promote replicas, DNS updates, smoke tests).- Keep manual anything with high business/legal risk, unclear state interpretation, or regulatory approvals (PII transfers, financial reconciliation, executive go/no-go). Where regulation demands, combine automation with manual approval gates (human-in-loop).This structure ensures fast, repeatable failovers while preserving control for high-risk decisions and auditability for compliance.
MediumSystem Design
77 practiced
Design an active-active multi-region architecture for a global e-commerce platform with 10M users and a maximum 100ms latency requirement for shopping flows. The system must maintain inventory correctness (no oversells) while offering low-latency reads for product browsing. Outline the architecture (regional services, data replication patterns, read/write routing), describe how payments and inventory updates will be handled to avoid oversell, and explain expected operational procedures for failover and reconciliation.
Sample Answer
Requirements:- Active-active across multiple regions, <100ms shopping-flow latency, 10M users, no oversells, low-latency product reads.High-level architecture:- Global DNS + Anycast LB → Regional API Gateways. Each region hosts: - Read-optimized product catalog (replicated, eventually consistent) - Regional Inventory Service (sharded by SKU via consistent hashing / geo-partitioning) - Order Service, Payment Gateway integration, Local cache/CDN for assets - Message bus (Kafka/NSQ) for async replication and events- Global Control Plane: metadata, routing policies, reconciliation service.Data replication & read/write routing:- Catalog: multi-master async replication (materialized views in each region) for low-latency reads; stale reads acceptable for browsing.- Inventory: partitioned by SKU; assign a primary region (based on demand/ownership) for each partition. Inventory writes (reservations/commits) go to the partition’s primary; reads served locally from replicated cache with short TTL.- Orders: created in region of request; writes that affect inventory routed to inventory primary via synchronous RPC (pinned by SKU) or via fast Paxos/consensus for hot SKUs.Preventing oversell (recommended approach):- Use reservation-based SAGA: 1. Reserve inventory (strongly consistent) at inventory primary with expiry (e.g., 5–15 minutes). 2. On successful reserve, proceed to authorize payment (payment auth can be regional). 3. On capture, commit reservation (decrement) at primary. If auth fails or TTL expires, release reservation.- Inventory primary implements optimistic concurrency (CAS on versioned counters) or lightweight leader-based decrement operations. For very hot SKUs use local in-memory counter with write-ahead log and quorum replication (Raft group) per partition to keep write latency low while ensuring correctness.Payments & inventory coordination:- Payment Authorization occurs after successful reservation. Use idempotent events and correlation IDs.- Capture triggers final commit; if capture is in a different region, forward commit to inventory primary.- Use asynchronous retries and compensating transactions (refunds/re-restores) if cross-region inconsistencies occur.Operational procedures: failover & reconciliation- Regional failure: traffic shifts via DNS/LB to other regions. Inventory primary election: if primary region is down, promote standby replicas (Raft leader election) for affected partitions. Use automated health checks and predefined promotion policy.- Reconciliation: continuous background anti-entropy job compares committed transactions (orders/payments) vs inventory ledger; mismatches create compensating actions (restores, refunds) and alerts. Periodic global inventory journaling and watermark checkpoints to speed reconciliation.- Runbooks: promote replicas, run compensating SAGA flows, bleed orders into safe queue, enable read-only mode for catalog if needed.- Monitoring: end-to-end latency, reservation TTL saturation, discrepancy rates, lost/failed commits.Trade-offs:- Strong correctness via primary-per-SKU keeps oversell risk minimal but introduces cross-region hops for non-owned SKUs; mitigated by smart ownership, geo-affinity, and caching.- Alternatives: fully distributed CRDT counters risk transient oversells; two-phase commit adds latency—SAGA balances correctness and user experience.This design provides <100ms browsing by local reads and keeps inventory correctness via per-SKU strong operations, reservation-based SAGA, and robust failover/reconciliation procedures.
EasyTechnical
80 practiced
You are working with a client who provides online video streaming. The business requires 'no more than 5 minutes of outage for any region' and 'no more than 1 minute of data loss' per 24-hour period. Explain the difference between Recovery Time Objective (RTO) and Recovery Point Objective (RPO), and show step-by-step how you would derive concrete RTO and RPO targets from these business requirements. For each derived target, explain which technical mechanisms (replication mode, backup cadence, snapshot frequency, and failover automation) map to meeting that requirement.
Sample Answer
RTO vs RPO (short):- RTO (Recovery Time Objective) = maximum acceptable outage duration (how long service can be down).- RPO (Recovery Point Objective) = maximum acceptable data loss window (how much new data may be lost).Derivation from business requirements (step-by-step):1. Business: "no more than 5 minutes of outage for any region" → RTO_target = 5 minutes. - Add operational buffer: allocate subcomponents (detection + decision + failover). Example split: detection 30s, orchestration 60s, failover & warmup 3m, DNS/edge convergence 30s = ~5m.2. Business: "no more than 1 minute of data loss per 24h" → RPO_target = 1 minute. - This implies replication/ingestion must preserve ≤60s of writes at worst.Technical mechanisms mappingFor RPO = 1 minute:- Replication mode: synchronous or semi-sync for critical metadata; for high-throughput video metadata use intra-region sync and cross-region near-sync with max lag <60s. Technologies: database with WAL shipping/CDC (e.g., Postgres streaming replication, Aurora Global DB, multi-master or CDC to mirror writes).- Backup cadence / snapshot: periodic backups remain daily/hourly for durability, but continuous replication/CDC provides the RPO. If using snapshots for quick restore, snapshot frequency must be ≤1 minute (rare) — usually not practical; instead rely on WAL retention and point-in-time-recovery (PITR) with continuous WAL archiving.- Monitoring: replication lag alarms (alert when >30s) and automated throttling/failover triggers.For RTO = 5 minutes:- Failover automation: automated health detection (active probes), automatic leader election or orchestrated failover runbook. Use orchestrators (Kubernetes with automated replica promotion, managed DB failover, Route53 with health checks and low TTL).- DNS/edge: reduce TTL (e.g., 30s) and use CDN with instant origin switch to minimize client-visible outage.- Warm standby / multi-AZ or multi-region active-passive: keep standby warmed (ready to accept traffic) to avoid long cold starts. Example: active-active for reads, active-passive with automated promotion for writes.- Snapshot frequency: useful for recovery beyond replication scope; not primary for RTO but ensure snapshot restore time fits RTO if used—otherwise use warm warm-standby.Concrete example architecture:- Primary in Region A, synchronous intra-region replicas, async replicated to Region B with monitored lag <30s. WAL-based continuous replication + PITR.- Automated failover: healthcheck → promote Region B writer via orchestration script (1–2 min) + CDN/origin switch + DNS TTL 30s = total <5m.- Alerts & runbooks: replication-lag >30s triggers pre-emptive mitigation; CPI tests to validate RTO/RPO periodically.Trade-offs: synchronous cross-region replication guarantees RPO=0 but increases write latency. Near-sync async with tight SLAs gives low latency and meets 1-minute RPO with good monitoring.
MediumTechnical
78 practiced
Design an IAM and secrets-management plan for region failovers that ensures least privilege, rapid revocation of compromised secrets if a region is breached, and automated re-binding of service principals and secret rotation in the secondary region. Include considerations for regional KMS separation versus cross-region key replication, audit trails, and emergency access procedures.
Sample Answer
Requirements & constraints:- Least privilege, fast revocation if a region is breached, automated re-binding and secret rotation in secondary region, clear audit trails, emergency access.High-level design:1. Multi-region tenancy model- Primary and Secondary regions each have their own IAM namespace and KMS key material (preferred) to limit blast radius.- Optionally use cross-region key replication only for read-only backup keys where operationally necessary (trade-off: replication eases failover but increases risk surface).2. IAM & principals- Use short-lived, role-based credentials (OIDC or STS-issued tokens) for services; avoid long-lived keys.- Define roles (service roles, operator roles) with narrowly scoped policies (resource, action, condition by region, VPC).- Assign service principals to roles via identity federation (e.g., cloud-managed service accounts) so you can re-map in automation.3. Secrets management & KMS- Store secrets in a secrets manager per region encrypted by regional KMS keys. Secrets have versioned metadata and TTL.- Use envelope encryption: secrets encrypted with data keys; data keys encrypted by KMS CMKs.- Key strategy: regional CMKs by default. Cross-region replication of ciphertext only if business requires, keeping CMKs separate to allow immediate revocation of CMKs in a breached region.4. Automation for failover, re-bind, rotation- Maintain a control plane (serverless workflow or orchestrator) that: - Detects regional outage or breach (via health checks, Security Center incidents). - Triggers rotation workflows in secondary: generate new data keys, create new secret versions, update service principal bindings to point to secondary-role ARNs or federated identity mapping. - Use Infrastructure-as-Code (IaC) templates that parameterize region-specific ARNs for fast re-deployment. - All operations use ephemeral credentials and require MFA for human operator steps.5. Rapid revocation on compromise- Revoke by: - Key revocation: disable/rotate regional CMK to immediately make secrets unreadable in that region. - Token revocation: revoke active sessions by invalidating STS tokens or rotating trust relationships. - Secrets: mark compromised secret versions as revoked; ensure applications fetch fresh secrets on errors.- Pre-scripted emergency playbook automates revocation and rotates to pre-provisioned secondary keys/secrets.6. Audit, monitoring, and detection- Centralized immutable audit logs (write-once) fed from both regions to a separate, tamper-evident storage (e.g., encrypted object store in a third region or SIEM).- Log events: IAM role assumptions, KMS Decrypt/Encrypt, secret access, rotation events, binding changes.- Alerts for anomalous accesses, CMK policy changes, or mass secret reads.7. Emergency access & break-glass- Break-glass accounts with strict controls: locked in vault, require multi-party approval (vaulted OTP + scheduled approver), short-lived elevated sessions.- Use just-in-time (JIT) elevation tooling and record every action.Trade-offs and rationale:- Regional CMKs maximize isolation and allow immediate cryptographic revocation; increases complexity for cross-region workloads. Cross-region replication simplifies failover but widens compromise blast radius.- Short-lived credentials + automation minimize the window of exposure and make revocation effective.Operational practices:- Regular drills: run failover and rotation rehearsals quarterly.- Automated CI pipelines for secret rollouts with canary validation.- Periodic access reviews, key rotation cadence aligned with risk model, and retention of audit logs beyond compliance windows.This plan gives least-privilege control, rapid cryptographic revocation per region, automated re-binding/rotation, and strong audit and emergency controls while making explicit trade-offs between isolation and operational convenience.
Unlock Full Question Bank
Get access to hundreds of Multi Region Disaster Recovery interview questions and detailed answers.
