High Availability and Disaster Recovery Questions

Designing systems to remain available and recoverable in the face of infrastructure failures, outages, and disasters. Candidates should be able to define and reason about Recovery Time Objective and Recovery Point Objective targets and translate service level agreement goals such as 99.9 percent to 99.999 percent into architecture choices. Core topics include redundancy strategies such as N plus one and N plus two, active active and active passive deployment patterns, multi availability zone and multi region topologies, and the trade offs between same region high availability and cross region disaster recovery. Discuss load balancing and traffic shaping, redundant load balancer design, and algorithms such as round robin, least connections, and consistent hashing. Explain failover detection, health checks, automated versus manual failover, convergence and recovery timing, and orchestration of failover and reroute. Cover backup, snapshot, and restore strategies, replication and consistency trade offs for stateful components, leader election and split brain mitigation, runbooks and recovery playbooks, disaster recovery testing and drills, and cost and operational trade offs. Include capacity planning, autoscaling, network redundancy, and considerations for security and infrastructure hardening so that identity, key management, and logging remain available and recoverable. Emphasize monitoring, observability, alerting for availability signals, and validation through chaos engineering and regular failover exercises.

MediumSystem Design

92 practiced

Design a multi-region disaster recovery plan for a customer-facing web application with primary in us-east-1 and warm standby in eu-west-1. Requirement: RTO 1 hour, RPO 5 minutes. Describe replication approach, networking and routing choices (DNS failover, BGP anycast, or traffic manager), certificate/key distribution, health checks, runbook steps during failover, and how session continuity or graceful degradation would be handled.

Sample Answer

**Clarify requirements & constraints**- Primary: us-east-1; Warm standby: eu-west-1- RTO ≤ 1 hour, RPO ≤ 5 minutes- Network engineer lens: connectivity, routing, health, certificates, runbook, session continuity.

**Replication approach**- Data: primary does continuous WAL streaming (Postgres streaming replication) to a near-real-time replica in eu-west-1 with async replication tuned for <5min lag; enable WAL compression and monitoring. For critical writes, use logical replication for critical tables if needed.- Object store: replicate S3 via cross-region replication (CRR).- Cache/session store: cross-region Redis replication (primary in us-east, replica in eu-west) + fallback persistence to DB for critical session state.- Config/Secrets: use Secrets Manager/Parameter Store with cross-region replication.

**Networking & routing choices**- Primary routing: use DNS-based active/passive with low TTL (60s) in Route53 + health checks. Route53 Failover + weighted records for gradual shift.- For faster cutover and stable client IPs, recommend AWS Global Accelerator or BGP Anycast (Cloud CDN/PROVIDER) in front—accelerator directs traffic to healthy region and reduces DNS propagation issues.- Use dedicated inter-region VPC peering or Transit Gateway for replication/control-plane connectivity; ensure IPsec/Direct Connect if private connectivity required.

**Certificates & key distribution**- Use ACM for public certs with regional certificates issued in both regions; automate provisioning via Terraform/ACM PCA if private CA needed.- Store TLS private keys and app secrets in Secrets Manager with cross-region replication and strict IAM roles. Rotate keys and test retrieval.

**Health checks**- Multi-layer checks: - Network layer: TCP/ICMP probe to ALB/NLB - Application layer: HTTPS health endpoint verifying DB connectivity and business-critical checks - Data lag: monitor replication lag metric (WAL lag) with alert threshold <5m- Automated CloudWatch/Prometheus alerts integrated with PagerDuty.

**Runbook (failover steps — engineered for ≤1 hour)**1. Detect: automated health alarm triggers; verify manual/automatic failover policy.2. Promote replica DB in eu-west-1 to primary (promote replica, stop replication).3. Scale up warm standby app fleet and attach to eu-west ALB/NLB.4. Validate app connectivity to promoted DB and run smoke tests.5. Update routing: - If using Global Accelerator: Update endpoint weights / failover policy (near-instant). - If DNS-only: change Route53 failover record to eu-west and lower TTL pre-configured; verify propagation.6. Promote Secrets/Certs: ensure Secrets Manager endpoints in eu-west have latest secrets and ACM cert active.7. Monitor and roll back if failures.

**Session continuity & graceful degradation**- Aim for session continuity: - Prefer stateless app design using JWTs so sessions survive region switch. - If sticky sessions: store session state in Redis with replica promoted on failover; accept possible up to 5min RPO. - For write-heavy sessions, degrade to read-only mode for non-essential features and queue writes (SQS) for replay after recovery.- Graceful degradation: - Serve static assets from CDN (CloudFront) cached edge—unaffected by origin switch. - Feature flags to disable non-critical subsystems to reduce load during failover.

**Testing & validation**- Quarterly DR drills: automated failover exercises, measure RTO/RPO, iterate.- Runbook rehearsals, certificate/key retrieval tests, and BGP/DNS cutover dry-runs.

Trade-offs: DNS failover is simple but slower; Global Accelerator/BGP is faster and more deterministic at cost/complexity. Design favors Global Accelerator + Route53 failover fallback to reliably meet 1-hour RTO and 5-minute RPO.

MediumTechnical

109 practiced

Compare asynchronous, semi-synchronous, and synchronous replication models for replicating state between clusters across regions. Discuss impact on RPO and RTO, write latency, client-perceived performance, split-brain risk, and scenarios where each model is appropriate for networked services such as control-plane state or session stores.

Sample Answer

**Overview (approach)** Compare three replication models by trade-offs: RPO/RTO, write latency, client experience, split‑brain risk, and suitable use-cases for control-plane state or session stores.

**Synchronous replication** - RPO/RTO: RPO ≈ 0 (no data loss), low RTO if failover is automated. - Write latency: Increases to include cross‑region commit (RTT), blocking until replicas ack. - Client-perceived performance: Higher tail latency; depends on WAN RTT and pipelining. - Split‑brain risk: Low if leader election + fencing is strong; but network partitioning can cause availability loss (CP in CAP). - When appropriate: Critical control-plane state requiring zero loss (e.g., lease/master metadata) when network latency is acceptable or via dedicated low-latency links.

**Semi-synchronous (ack to a subset / async background to others)** - RPO/RTO: Tunable — RPO small if waiting for at least one remote ack; RTO moderate. - Write latency: Moderate (waits for a chosen replica set), better than full sync. - Client-perceived performance: Balanced: lower tail latency than full sync, more durability than async. - Split‑brain risk: Medium; needs quorum and fencing to avoid divergent primaries. - When appropriate: Session stores or regional control-plane replicas where some durability matters but you need responsiveness.

**Asynchronous replication** - RPO/RTO: RPO > 0 (data loss possible), RTO depends on catch-up; long for cross-region. - Write latency: Minimal (local writes return immediately). - Client-perceived performance: Best for latency-sensitive clients; risk of stale reads unless read routing used. - Split‑brain risk: Higher risk of divergence if concurrent primaries; mitigated by single-writer patterns. - When appropriate: Noncritical session caches, analytics, read replicas, or when network latency is high and eventual consistency is acceptable.

**Network-engineer considerations / mitigations** - Use WAN QoS, dedicated circuits, or SD‑WAN for synchronous needs. - Monitor replication lag, RTT, and implement health/fencing (BGP/EVPN considerations for failover paths). - Choose model by SLAs: strict RPO→synchronous; balanced SLA→semi-sync; latency-first or cheap cross-region→async.

MediumTechnical

92 practiced

As a network engineer what metrics, logs, and traces would you collect to detect availability degradation early? Define thresholds and alerting rules for packet loss, latency, TCP retransmissions, BGP route changes, interface errors, and load-balancer health. Describe dashboard structure, golden signals, and techniques to reduce alert noise while ensuring rapid incident detection.

Sample Answer

**Approach (role perspective)**I’d instrument packet-, flow- and device-level telemetry (NetFlow/sFlow, SNMP/IF-MIB, IP SLA/RTT probes, BGP RIB/updates, syslog, eBPF/tcp_info where possible) and export to an observability backend (Prometheus/Influx + Grafana / ELK / Honeycomb).

**Metrics, logs & traces to collect**- Packet loss: ICMP/IP SLA per-hop and per-path (1s & 1m windows). Alert: >0.5% sustained 1m (warn), >2% sustained 1m (critical).- Latency: p50/p95/p99 of RTT from probes and TCP handshake timings. Alert: p95 RTT > 100ms (WAN) or >20ms (DC) for 5m.- TCP retransmissions: device TCP/flow counters or tcp_info. Alert: retransmit rate >1% of packets over 5m or sudden spike >5x baseline.- BGP route changes: number of updates/withdrawals per minute and prefix flaps. Alert: >50 updates/min (critical) or >10 flaps/prefix in 15m.- Interface errors: rx_errors/tx_errors/CRC/FCS. Alert: any interface error rate >0.1% of traffic or absolute errors >1000 in 5m.- LB health: backend health checks failed, active connections, error-rate 4xx/5xx. Alert: backend failed >2/3 probes or error-rate >5% for 2m.

**Dashboard structure & golden signals**- Top row (Golden signals): Availability (health check pass%), Latency p95, Error rate, Traffic volume.- Second row: Packet loss heatmap per path, TCP retransmit trend, BGP updates graph.- Third row: Interface errors and utilization, per-DC LB health, recent syslog/BGP message tail.Golden signals prioritized: Latency, Traffic (throughput), Errors (loss/retransmit), Saturation (utilization).

**Alerting rules & tuning**- Multi-tier alerts: Info (single probe failure), Warn (2 probes/1 device), Critical (service impact / cross-device).- Use aggregation and anomaly detection: require at least 2 independent probes or correlated signals (e.g., loss + retransmits) before paging.- Dynamic baselines: use rolling 7d baseline and z-score to detect spikes instead of static thresholds where traffic varies.

**Noise reduction techniques**- Deduplicate alerts by grouping (prefix, device, region).- Alert suppression during planned maintenance via automation (change manager hooks).- Use alert severity and runbooks; auto-escalate if not acknowledged.- Correlate logs + metrics to suppress single-source flaps (e.g., transient ICMP vs sustained TCP failures).- Implement short “wait-for” windows (e.g., 60–120s) and require sustained violation to avoid flapping.

This setup provides early detection while minimizing false positives and supports quick triage (link → device → path → BGP → app).

MediumTechnical

65 practiced

Propose a set of DR test exercises and chaos-engineering experiments targeted at validating network availability and failover logic. Include small-scope unit experiments, progressive blast-radius plans, safety abort conditions, telemetry and KPIs to observe (RPO, RTO, error rates), and frequency of exercises. Also describe how to roll back and generate post-mortem action items.

Sample Answer

**Situation & goal**Validate network availability and failover logic across links, devices, and cloud-region routing so RTO/RPO targets are met and failovers are deterministic.

**Small-scope unit experiments**- Interface flap: administratively down/up a single non-critical switch port to verify BFD/OSPF/ISIS adjacency reconvergence and forwarding.- Route withdraw: withdraw a test BGP prefix on one PE and confirm route propagation and route-map/RTBH behavior.- Firewall path test: simulate policy denial for a test CIDR to validate fail-open/deny behaviors and telemetry.- Link saturation (micro): generate controlled burst traffic on lab link to exercise QoS and congestion policies.

**Progressive blast-radius plan**1. Canary: single dev/leaf in staging (unit tests).2. Small: single AZ production isolation — traffic steering to secondary AZ.3. Medium: entire region link failover — route and cross-region peering.4. Large: multi-region failover (only after prior levels green).

**Safety & abort conditions**- Abort if packet loss > X% for > Y minutes, control-plane CPU > 80%, BGP flaps > N/min, or RTO/RPO thresholds breached.- Predefined circuit breakers: instant rollback script, route re-injection, interface unblock.

**Telemetry & KPIs**- RTO (time to restore service), RPO (data lost window)- Control-plane convergence time (BGP/IGP)- Packet loss, latency, jitter per path- Error rates: interface errors, packet drops, TCP retransmits- Application-level success rate and SLA % per minute- Alerts: BFD/heartbeat misses, route flaps

**Frequency**- Unit tests: weekly automated in CI.- Canary chaos: monthly.- Medium blast-radius: quarterly.- Full DR drills: biannual.

**Rollback & post-mortem**- Automated rollback playbooks (Ansible/Netconf): revert configs, re-advertise routes, re-enable links.- Post-mortem template: timeline, hypotheses, root cause, affected services, metrics, mitigation, owner, ETA.- Action items: fix detection gaps, add pre-checks, tune timers (BFD/OSPF), improve runbooks, add synthetic tests.

EasyTechnical

79 practiced

Compare load balancing algorithms: round-robin, weighted round-robin, least-connections, and consistent hashing. For each algorithm explain typical use-cases, how it affects cache locality and session affinity, and pros/cons when used for stateful services such as session-based web apps or distributed caches.

Unlock Full Question Bank

Get access to hundreds of High Availability and Disaster Recovery interview questions and detailed answers.

Join thousands of developers preparing for their dream job.