High Availability and Disaster Recovery Questions

Designing systems to remain available and recoverable in the face of infrastructure failures, outages, and disasters. Candidates should be able to define and reason about Recovery Time Objective and Recovery Point Objective targets and translate service level agreement goals such as 99.9 percent to 99.999 percent into architecture choices. Core topics include redundancy strategies such as N plus one and N plus two, active active and active passive deployment patterns, multi availability zone and multi region topologies, and the trade offs between same region high availability and cross region disaster recovery. Discuss load balancing and traffic shaping, redundant load balancer design, and algorithms such as round robin, least connections, and consistent hashing. Explain failover detection, health checks, automated versus manual failover, convergence and recovery timing, and orchestration of failover and reroute. Cover backup, snapshot, and restore strategies, replication and consistency trade offs for stateful components, leader election and split brain mitigation, runbooks and recovery playbooks, disaster recovery testing and drills, and cost and operational trade offs. Include capacity planning, autoscaling, network redundancy, and considerations for security and infrastructure hardening so that identity, key management, and logging remain available and recoverable. Emphasize monitoring, observability, alerting for availability signals, and validation through chaos engineering and regular failover exercises.

EasyTechnical

83 practiced

Explain common leader election algorithms and split-brain mitigation techniques (including Raft/Paxos basics, quorum-based election, fencing tokens, and lease mechanisms). As a Cloud Architect, state which approach you would recommend for a distributed metadata service and why.

Sample Answer

**Brief overview (Raft vs Paxos)**- Raft: leader-based consensus with log replication, leader election via randomized timeouts, easier to understand and implement; provides strong consistency and single leader for linearizable writes.- Paxos: family of protocols (Multi-Paxos common) that achieve consensus via proposers/acceptors/learners; more flexible but historically harder to reason about.

**Quorum-based election**- Requires majority of nodes for decisions (N/2 + 1). Prevents split-brain because two disjoint majorities cannot exist.- Use heartbeats and election timeouts to detect leader failure; candidate wins if it collects quorum votes.

**Fencing tokens & leases**- Fencing tokens: issue monotonically increasing token to a leader; external resources check token prevents stale leader from acting after partition.- Leases: time-bounded leadership grant; if lease expires, leader must stop. Good for reducing coordination but needs clock-synchronization considerations.

**Split-brain mitigation techniques**- Enforce strict majority quorum for writes/leader election.- Use fencing tokens when interacting with external state (storage, network appliances).- Combine leases with monotonic tokens for faster failover while preventing stale writes.- Prefer leader stickiness + randomized election timeouts to reduce unnecessary churn.

**Recommendation as Cloud Architect**I’d choose Raft (managed or well-tested implementation) with strict majority quorum plus fencing tokens for any external mutable resources, and short leases for performance-sensitive paths. Rationale: Raft’s clarity speeds operational troubleshooting; majority quorum prevents split brain; fencing tokens/leases protect external systems; cloud environments favor predictable leader behavior and easy debugging — important for a distributed metadata service requiring strong consistency and low operational risk.

**Trade-offs**- Paxos can be more flexible at scale but adds complexity.- Leases reduce availability under clock skew — mitigate with hybrid fencing.

MediumTechnical

70 practiced

As Cloud Architect, design a disaster recovery testing schedule and validation criteria for a portfolio of services. Include types of drills (table-top, component failover, full-region failover), frequency by service criticality, success criteria, rollback plans, metrics to capture, and stakeholder responsibilities.

Sample Answer

**Overview (approach)** I propose a risk-based DR testing schedule mapped to service criticality, combining table-top, component failover, and full-region failover drills. Tests validate recovery objectives, runbooks, automation, and people readiness.

**Service criticality & frequency**- P0 (customer-facing, RTO < 1 hr): Table-top monthly, component failover quarterly, full-region failover annually - P1 (internal critical, RTO 1–4 hrs): Table-top quarterly, component failover biannual, full-region failover every 18 months - P2 (non-critical, RTO > 4 hrs): Table-top biannual, component failover annual, full-region failover every 2–3 years

**Drill types & scope**- Table-top: validate runbooks, communication, decision points (2–4 hrs) - Component failover: simulate AZ/network/db instance failure; test automation, data replication, routing (partial traffic) - Full-region failover: shift production traffic to DR region, validate consistency, security, compliance (blue/green or cutover window)

**Success criteria**- RTO/RPO met within tolerance (e.g., 90% of services within target) - Automated playbooks execute without manual intervention beyond approved steps - No data loss beyond RPO; integrity checks pass - Customer-impact window within SLA; stakeholder sign-off

**Rollback plans**- Predefined automated rollback scripts or DNS/traffic re-route plan; runbook decision tree with clear abort/rollback triggers (e.g., data divergence, >X% error rate) - Reconciliation steps and post-rollback validation

**Metrics to capture**- Time to detect, time to failover start, time to full service recovery (RTO), data delta (RPO), error rates, user-transaction success rate, rollback occurrences, manual intervention time, runbook step durations

**Stakeholder responsibilities**- Cloud Architect: design tests, approve scope, postmortem lead - SRE/Platform: execute drills, automation, metrics collection - App Owners: verify functional correctness and approve go/no-go - Security/Compliance: validate controls during failover - Product/Business: accept customer-impact trade-offs, approve schedule, communicate externally

**Post-drill**- Formal postmortem within 48 hours, action items tracked to closure, update runbooks and SLAs.

MediumTechnical

83 practiced

Design a logging and monitoring pipeline so that logs, metrics, and traces remain available and recoverable during a full-region failure. Include ingestion endpoints, durable storage, cross-region replication strategies, key management for encrypted logs, and a plan for replaying or rehydrating logs post-recovery.

Sample Answer

**Overview / goals**I would design a geo-resilient observability pipeline so logs, metrics, and traces remain writable, recoverable, and decryptable during a full-region failure, and can be replayed/rehydrated after recovery.

**Ingestion endpoints**- Regional ingest APIs (HTTP/gRPC) fronted by global DNS + Anycast or global load balancer that routes to healthy regions.- Local edge agents (Fluentd/Vector/OTel collector) batch to nearest region; fall back to local disk buffer if network/region is down.

**Durable storage & replication**- Primary storage: nearline object store (S3/Blob/GCS) in each region with lifecycle tiers.- Cross-region replication: use asynchronous cross-region replication (CRR) per-bucket with versioning; for metrics/traces write to regional TSDB (Cortex/M3DB) with multi-cluster replication (write to local cluster and replicate to secondary clusters).- Write-ahead logs: store append-only WALs in replicated object store for guaranteed replay.

**KMS / Key management**- Use a multi-region KMS (customer-managed keys) with automatic key replication or a primary/secondary key pair rotation strategy.- Encrypt at-rest with envelope encryption: data key per object encrypted by KMS key; keep key metadata replicated to secondary KMS region.- Ensure KMS IAM policies and emergency key escrow exist so secondary region can decrypt when primary KMS unavailable.

**Replay / rehydration plan**- On recovery, promote secondary region storage + KMS access.- Replay pipeline: read WALs/objects from replicated buckets in time order, feed into batch ingestion jobs or direct re-injection into collectors, and backfill TSDB via import tools.- Validate integrity using checksums and sequence numbers; reconcile duplicates by idempotent ingestion or dedupe keys.

**Operational considerations**- SLAs: RPO = minutes (buffer size), RTO = hours.- Monitoring: health probes, alerting for replication lag, KMS health.- Tests: periodic chaos tests (region failover), restore drills, KMS failover drills, and runbooks for key promotion and rehydration.

MediumTechnical

86 practiced

Propose a capacity planning and autoscaling strategy to ensure high availability during sudden traffic spikes and during failover events. Discuss warm versus cold standbys, pre-warming, predictive/scheduled scaling, scaling cooldowns, and how you would load-test and validate the strategy.

Sample Answer

**Situation & goal**Design capacity and autoscaling to maintain high availability during sudden spikes and failovers while controlling cost.

**Strategy overview**- Use a hybrid approach: baseline reserved capacity + dynamic autoscaling + regional warm standbys.- Baseline (reserved/commit) covers typical peak; autoscaling covers variance and bursts.

**Warm vs Cold standbys**- Cold: minimal cost, longer RTO (minutes to tens of minutes). Use for non-critical regions.- Warm: pre-provisioned instances/containers kept healthy and behind health checks but scaled to low traffic—RTO seconds to <2 mins. Use for primary failover targets and for services with long startup times (JVM, ML models).

**Pre-warming & predictive/scheduled scaling**- Pre-warm before planned events (marketing, Black Friday) by scaling warm pools and priming caches, DB connections, and load balancer capacity.- Predictive scaling: use historical telemetry + ML forecast to schedule scale-outs ahead of expected spikes; tie to business calendar and feature flags.

**Scaling policy details**- Multi-dimensional metrics: request rate, CPU, latency p99, queue length, error rate.- Use target-tracking + step scaling: rapid step-outs for sudden spikes, conservative step-ins.- Implement cooldowns: short (30–60s) for scale-out to allow rapid response; longer (5–10min) for scale-in to avoid thrash.- Safety caps and surge quotas: prevent noisy neighbors from exhausting quotas; implement circuit breakers.

**Failover considerations**- Health-checked DNS with low TTL, multi-region load balancing, and automated traffic-shift playbooks.- Maintain warm cross-region pools and continuously replicate state (async + DR snapshots).- Run automated failover exercises and rollback plans.

**Load-testing & validation**- Staged testing: unit, canary, chaos/limit tests.- Synthetic traffic with realistic mixes (ramp rates, sustained peaks, tail latency). Tools: k6, Gatling, cloud-native traffic generators, distributed load injectors.- Validate autoscaling: run spike tests to verify RTO, scale actions, cooldown behavior, and downstream contention (DB, caches).- Observe SLOs (error rate, p99 latency), scaling timelines, cost impact. Run regular DR runbooks and game days.

**Metrics & monitoring**- Dashboards for capacity, scaling events, queue depth, latencies, cost.- Alerts on unexplained scaling, quota exhaustion, or failed health checks.

This plan balances availability, cost, and operational risk; I’d iterate with production telemetry and regular game-day validation.

EasyTechnical

92 practiced

Contrast multi-AZ and multi-region topologies for high availability and disaster recovery. Describe the typical fail-scenarios each protects against, the additional latency and complexity multi-region introduces, and the compliance, networking, and operational considerations a Cloud Architect should evaluate when recommending one over the other.

Unlock Full Question Bank

Get access to hundreds of High Availability and Disaster Recovery interview questions and detailed answers.

Join thousands of developers preparing for their dream job.