Designing systems to remain available and recoverable in the face of infrastructure failures, outages, and disasters. Candidates should be able to define and reason about Recovery Time Objective and Recovery Point Objective targets and translate service level agreement goals such as 99.9 percent to 99.999 percent into architecture choices. Core topics include redundancy strategies such as N plus one and N plus two, active active and active passive deployment patterns, multi availability zone and multi region topologies, and the trade offs between same region high availability and cross region disaster recovery. Discuss load balancing and traffic shaping, redundant load balancer design, and algorithms such as round robin, least connections, and consistent hashing. Explain failover detection, health checks, automated versus manual failover, convergence and recovery timing, and orchestration of failover and reroute. Cover backup, snapshot, and restore strategies, replication and consistency trade offs for stateful components, leader election and split brain mitigation, runbooks and recovery playbooks, disaster recovery testing and drills, and cost and operational trade offs. Include capacity planning, autoscaling, network redundancy, and considerations for security and infrastructure hardening so that identity, key management, and logging remain available and recoverable. Emphasize monitoring, observability, alerting for availability signals, and validation through chaos engineering and regular failover exercises.
EasyTechnical
65 practiced
Explain the purpose of chaos engineering in the context of high availability and disaster recovery. Give two example experiments you would run against a production-like environment to validate DR readiness and what success criteria you would measure.
Sample Answer
**Purpose (brief)** Chaos engineering validates that systems meet high-availability (HA) and disaster recovery (DR) goals by proactively injecting realistic failures in production-like environments to reveal weaknesses in automation, runbooks, and observability. For a Cloud Engineer this proves failover automation, data durability, and measurable RTO/RPO.**Experiment 1 — Cross-region failover (compute+network)** - What: Simulate an entire primary region outage by disabling routing and instance health (e.g., withdraw routes in Route53 weighted records, take primary autoscaling group out of service). - Why: Validates DNS failover, global load balancing, autoscaling in the DR region, and IAM/secret availability. - Success criteria: automated failover completes within target RTO (e.g., <5 min); application 95% request success rate during failover; no data loss beyond RPO; monitoring/alerts triggered and runbooks exercised.**Experiment 2 — Database failure & backup restore** - What: Force primary DB failure (promote read-replica or simulate storage corruption) and perform a point-in-time restore from backups to a new instance in DR region. - Why: Validates replication, snapshot integrity, RPO, and application reconnection logic. - Success criteria: RPO met (e.g., <15 min of data loss), restored DB reachable and consistent; application can read/write after failover; backup verification checksums pass; post-recovery latency within SLA.**Observability & Runbooks** - Ensure metrics/logs (latency, error rates, replica lag), end-to-end synthetic tests, and alerting are in place before experiments. After each run capture timeline, gaps, and remediation steps to improve automation and reduce manual steps.
MediumTechnical
82 practiced
Explain the consistency and latency trade offs between synchronous and asynchronous replication in multi region database deployments. Give two real world examples of when to prefer synchronous and when asynchronous replication is preferable, and explain how those choices affect RPO and application throughput.
Sample Answer
**Consistency vs Latency (summary)** Synchronous replication writes are acknowledged only after replicas confirm commit, giving strong consistency (read-after-write) and minimal data loss. Cost: higher write latency (network round-trips) and lower write throughput. Asynchronous replication returns success immediately; replicas lag, yielding lower latency and higher throughput at the cost of eventual consistency and possible data loss.**How choices affect RPO and throughput** - Synchronous: RPO ≈ 0 (no acknowledged data loss) but application write throughput and latency suffer proportional to inter-region RTT and slowest replica. - Asynchronous: Higher RPO (window of lost commits if primary fails) but much higher write throughput and lower tail latency.**Two real-world examples**1) Prefer synchronous - Use case: Inter-bank transaction system or primary DB + local DR within the same metro (cross-AZ or metro fiber). - Why: Regulatory need for zero-loss and immediate consistency. - Effect: RPO ~0, write latency increases slightly (tens of ms within metro), throughput limited by replica ack speed.2) Prefer asynchronous - Use case: Global read replicas for user-facing content or analytics in other continents (e.g., AWS RDS read replicas in another region). - Why: Prioritize low local read latency and write throughput at primary; occasional acceptable lag. - Effect: RPO = replication lag window, primary throughput high, writes fast.**Operational considerations (Cloud Engineer)** - Mitigate synchronous latency by limiting synchronous replicas to low-RTT links (same region/metro). - For async, monitor replication lag, set retention/alerts, and plan failover with possible data reconciliation. - Hybrid: synchronous local replicas + async cross-region copies for DR and global reads.
MediumTechnical
78 practiced
A startup with tight monthly budget needs to choose between pilot light, warm standby, and active-active cross region disaster recovery strategies. Compare these three options based on cost to operate, expected RTO and RPO, operational complexity, and time to scale to full traffic. Then recommend the best approach for a B2B SaaS startup with moderate traffic and justify your choice.
Sample Answer
**Summary comparison**- **Pilot light** - Cost: low (minimal standby infra like base AMIs, replicated data) - RTO/RPO: RTO = hours; RPO = minutes-to-hours (depends on replication) - Complexity: low (simple scripts to spin up) - Time to scale: tens of minutes → hours to bring full fleet online- **Warm standby** - Cost: medium (small-sized instances/services running in DR region) - RTO/RPO: RTO = minutes → ~1 hour; RPO = near‑zero with continuous replication - Complexity: moderate (scaled-down pipelines, DNS and load balancer prep) - Time to scale: minutes to tens of minutes to scale to full traffic- **Active–active** - Cost: high (full capacity duplicated cross‑region) - RTO/RPO: RTO = seconds; RPO = near‑zero - Complexity: high (global traffic routing, data consistency, conflict resolution) - Time to scale: immediate (already serving traffic)**Recommendation (for a B2B SaaS startup, moderate traffic)**I recommend **warm standby**. It balances predictable costs and fast recovery: you run scaled-down application components and continuously replicate databases (cross‑region read replicas or CDC to a standby DB). On failover, autoscaling + IaC (Terraform) and Route53/Cloud DNS updates let you absorb traffic in minutes. Implement automated health checks, documented runbooks, and quarterly failover drills to validate RTO/RPO. Warm standby keeps monthly spend reasonable while meeting customer SLAs typical for B2B SaaS.
MediumSystem Design
64 practiced
Design a redundant load balancer topology for an internet-facing service in AWS that provides high availability and DDoS resilience. Discuss the use of ALB or NLB, Route53 health checks and failover, cross zone load balancing, and how to manage SSL termination and session affinity across redundant LB instances.
Sample Answer
**Requirements & goals**- Internet-facing, highly available, resilient to DDoS, zero single-point-of-failure, preserve SSL security and session affinity.**High-level topology**- Use a multi-AZ pair of load balancers in each AWS Region (ALB for HTTP/S app-layer, NLB for TLS passthrough or UDP). Put them behind Route 53 with health-based routing across Regions for DR.- Protect with AWS Shield Advanced + WAF (ALB) and AWS Global Accelerator or CloudFront in front for global DDoS absorb and lower latency.**ALB vs NLB guidance**- Choose ALB when you need HTTP/S features (host/path routing, WAF, HTTP/2, advanced metrics) and want TLS termination at the LB.- Choose NLB when you need extreme performance, preserve client IPs, or need TCP/TLS passthrough (terminate at backend).**Cross-zone & multi-AZ**- Enable cross-zone load balancing on ALB/NLB so traffic is evenly spread across instances in all AZs.- Deploy target groups across AZs; autoscaling groups ensure capacity per AZ.**Route 53 health checks & failover**- Configure Route 53 health checks for each Regional endpoint (ALB DNS names). Use latency or weighted routing plus failover records to shift traffic to healthy Regions.- Set health check to include app-level (HTTP 200) endpoint behind the ALB to avoid false positives.**SSL termination & certificate management**- Terminate TLS at ALB using ACM certificates (auto-renewed). For NLB passthrough, terminate at EC2/containers using ACM PrivateCA or certs on backends via ACM with instance profiles or Secrets Manager.- Offload TLS at ALB to inspect traffic with WAF and to reduce backend CPU.**Session affinity**- Use ALB sticky sessions (duration-based) when required; store session state in a distributed store (DynamoDB, ElastiCache Redis) or use JWTs to avoid affinity.- For multi-region failover, prefer stateless tokens or cross-region datastore (DynamoDB global tables) so failover doesn't break sessions.**Operational practices**- Monitor LB metrics (ALB/NLB) and WAF/Shield alerts in CloudWatch; use autoscaling and warm pools for scale.- Test failover regularly and run DDoS runbooks. Automate infra with IaC (Terraform/CloudFormation).This design balances availability, DDoS resilience, and secure SSL handling while minimizing session disruption during failover.
EasyTechnical
88 practiced
Describe the following load balancing algorithms and give one scenario where each is the best choice: round robin, least connections, and consistent hashing. Also explain implications for session stickiness in cloud load balancers.
Sample Answer
**Round robin — description & best use**- Distributes requests sequentially across healthy backends in a rotating order.- Best when instances are homogeneous and requests have similar load (e.g., stateless web servers autoscaled behind an ALB/NLB for a microservice with uniform request cost).**Least connections — description & best use**- Sends traffic to the server with the fewest active connections.- Best for long-lived or highly variable request workloads (e.g., WebSocket servers, FTP, or upstreams handling long polling) where connection count correlates with load.**Consistent hashing — description & best use**- Hashes client identifier (IP, session ID, or key) to pick a backend; minimizes remapping when nodes change.- Best for cache/shard placement (e.g., distributed cache like Redis cluster, or stateful sessions stored locally) to reduce cache misses on scaling events.**Session stickiness implications**- Sticky sessions (session affinity) bind a client to a backend using cookies or source IP. Works with round robin/least connections by adding affinity layer but reduces effective load distribution and complicates autoscaling/failover.- Consistent hashing provides affinity without a central store and minimizes reassignments on node changes, but still risks uneven load if hot keys exist.- Cloud recommendation: prefer stateless services with external session stores (Redis, DynamoDB) or use consistent hashing for caches; use stickiness only when necessary and monitor tail-load and failover behavior.
Unlock Full Question Bank
Get access to hundreds of High Availability and Disaster Recovery interview questions and detailed answers.
