Designing systems to remain available and recoverable in the face of infrastructure failures, outages, and disasters. Candidates should be able to define and reason about Recovery Time Objective and Recovery Point Objective targets and translate service level agreement goals such as 99.9 percent to 99.999 percent into architecture choices. Core topics include redundancy strategies such as N plus one and N plus two, active active and active passive deployment patterns, multi availability zone and multi region topologies, and the trade offs between same region high availability and cross region disaster recovery. Discuss load balancing and traffic shaping, redundant load balancer design, and algorithms such as round robin, least connections, and consistent hashing. Explain failover detection, health checks, automated versus manual failover, convergence and recovery timing, and orchestration of failover and reroute. Cover backup, snapshot, and restore strategies, replication and consistency trade offs for stateful components, leader election and split brain mitigation, runbooks and recovery playbooks, disaster recovery testing and drills, and cost and operational trade offs. Include capacity planning, autoscaling, network redundancy, and considerations for security and infrastructure hardening so that identity, key management, and logging remain available and recoverable. Emphasize monitoring, observability, alerting for availability signals, and validation through chaos engineering and regular failover exercises.
MediumSystem Design
119 practiced
You are a Systems Administrator asked to design an HA and DR architecture for a customer-facing web application that must meet an SLA of 99.99% and an RPO of under 5 minutes. Describe the components, redundancy strategy, storage and database replication approach, failover mechanics, and trade-offs you would choose to meet these targets.
Sample Answer
**Context & objectives (I):** design HA + DR so customer web app meets 99.99% availability and RPO < 5 minutes. I propose a primary multi-AZ active deployment plus a warm standby in a different region for DR.**Architecture & components**- Multi-AZ active region: load balancer (cloud ALB/NGINX LB) + autoscaling web/app layer in 3 AZs, health checks and session stickiness via Redis session store (multi-AZ).- DB tier: primary cluster in Region-A with synchronous or semi-synchronous replicas across AZs (read replicas) for HA.- DR region (Region-B): warm replica of DB + app fleet on autoscale (kept patched, smaller size).**Storage & replication**- Block storage: use replicated network block (EBS multi-AZ or equivalent) for instance persistence.- Database replication: synchronous (or synchronous commit to at least one AZ replica) within region for zero/near-zero data loss; asynchronous cross-region replication (logical replication / WAL shipping / managed cross-region replica) with monitoring to keep lag < 5 minutes. Continuous backups and point-in-time recovery (PITR) to object storage.**Failover mechanics**- AZ failure: LB + health checks remove instances, autoscaler replaces automatically — transparent.- Region failover (DR): automated runbook + DNS failover (Route53 health checks with low TTL) or BGP/Anycast for faster cutover. Promote Region-B DB replica to primary (scripts), scale app, update DNS. Use orchestration (Terraform/Ansible) and runbook with automated checks.- RPO enforcement: continuous WAL shipping and frequent snapshots; test failover drills monthly.**Monitoring & testing**- End-to-end synthetic checks, replication lag metrics, alerting thresholds (<1 min for lag warning).- Quarterly full DR failover test; weekly partial tests.**Trade-offs**- Synchronous cross-AZ DB increases write latency and cost but ensures low RPO in-region. Cross-region async reduces latency impact but risks up to configured replication lag — must be tuned to stay <5 minutes.- Warm-standby DR costs more than cold but reduces RTO significantly.- Complexity and operational overhead increase with automation and frequent testing — justified by SLA.I would document runbooks, automate failover steps, and instrument replication/health metrics to prove SLA/RPO compliance.
MediumTechnical
80 practiced
Design a capacity planning and autoscaling strategy for a public API that experiences unpredictable traffic spikes due to external events. Describe baseline capacity, scaling triggers (metrics and thresholds), warm pool strategies, cooldowns, cross-AZ scaling considerations, and how to prevent cold-start issues affecting availability.
Sample Answer
**Situation & goal**I’d ensure the public API stays available and performant during unpredictable spikes while minimizing cost.**Baseline capacity**- I’d run a steady baseline equal to 95th-percentile normal traffic plus buffer (e.g., 20% headroom).- Reserve capacity across AZs: baseline instances in each AZ to tolerate AZ failure.**Scaling triggers (metrics & thresholds)**- Primary metric: request concurrency / RPS per instance and latency (P95).- Trigger scale-out when concurrency > 70–75% of instance capacity OR P95 latency > target (e.g., 250 ms) for 30s.- Scale-in when concurrency < 40% and P95 latency < target for 5m.**Warm pool strategies**- Maintain a warm pool of pre-initialized instances/containers (e.g., 10–20% of peak or N instances per AZ) ready to attach to load balancer.- Use pre-baked AMIs/containers with app warmed (JIT caches loaded) via synthetic warm requests.**Cooldowns**- Short cooldown for scale-out evaluation (30–60s) to react quickly to spikes.- Longer cooldown for scale-in (5–10m) to avoid thrashing.**Cross-AZ considerations**- Distribute baseline and warm pool evenly across AZs.- Scale policies per-AZ with global coordination to prevent one AZ from becoming hot; use load-balancer-level metrics to rebalance.**Preventing cold-start issues**- Pre-warm JVM/containers, lazy-load only noncritical modules on first request.- Health-check and probe new instances with synthetic requests before adding to LB.- Use connection pooling and keep-alive to reduce overhead.**Monitoring & testing**- Alert on scaling failures, throttling, and error rates.- Regular chaos tests and spike-replay drills to validate thresholds and warm-pool effectiveness.I’d iterate thresholds from load test data and real incidents, automating runbook steps for emergency capacity increases.
MediumTechnical
89 practiced
Describe availability and disaster recovery considerations for a central key management service (KMS). Include replication and backup of key material, access control and least privilege, key rotation procedures, offline key escrow, and how to ensure encrypted data remains accessible after regional failover.
Sample Answer
**Situation overview**As a systems administrator I treat KMS as a critical, highly-available service: loss of key material = permanent data loss, so availability and DR are first-order requirements.**Replication & backup of key material**- Use HSM-backed keys with built‑in multi-region replication (or cloud multi‑region keys). If unavailable, export and securely import keys into a secondary region’s HSM (where policy allows).- Keep encrypted, integrity‑protected backups of key material in immutable storage (WORM) and air‑gapped/offline escrow.**Access control & least privilege**- Enforce least privilege with IAM roles and key policies: separate “use” vs “manage” permissions; require MFA and Just‑In‑Time elevation for admin ops.- Separation of duties and approval workflows for key export/restore. Enable logging (KMS, CloudTrail) and alerting.**Key rotation procedures**- Use envelope encryption: data encrypted with DEKs; DEKs wrapped by CMKs. Rotate CMKs regularly (e.g., quarterly), create new CMK and configure automatic rewrap of DEKs or maintain overlap period so both old and new keys can decrypt.- Test rotation in staging; keep previous keys for a retention window before deletion.**Offline key escrow**- Store an encrypted copy of master key material in a physically secure vault (HSM module backup, safe deposit), with documented access and dual custody.**Ensuring data access after regional failover**- Replicate ciphertext and DEKs (wrapped) alongside data to secondary region. Ensure secondary region has either replicated CMK or ability to unwrap via exported/escrowed key.- Predefine failover playbook: promote replicated CMK or restore from escrow, validate integrity, rewrap DEKs if needed, and run availability tests.**Operational best practices**- Regular backup/restore drills, automated monitoring, strict change control, and retention policies to avoid accidental destruction.
MediumTechnical
76 practiced
You manage a leader-based coordination system. Outline split-brain mitigation strategies including fencing/STONITH, lease-based leadership, strict quorum rules, and safe leader promotion workflows. Describe recovery steps after a network partition heals to ensure no data divergence or double-leadership remains.
Sample Answer
Situation overviewI manage a leader-based coordination cluster (e.g., HAProxy/Consul/Zookeeper style). The key risk is split‑brain: two nodes simultaneously believing they’re leader, causing conflicting writes. My mitigation plan covers prevention and safe recovery.Prevention strategies- Fencing / STONITH: Configure out‑of‑band power/network fencing (IPMI, iLO, cloud API) so any preempted leader is forcibly powered off or isolated before promoting a new leader.- Lease-based leadership: Use short, renewable leader leases (e.g., TTL) stored in a durable store; leadership expires if lease not renewed, preventing stale leaders holding control.- Strict quorum rules: Require majority/quorum for any leader election and for committing writes. Refuse leadership when node cannot see quorum.- Safe promotion workflows: Implement multi-step promotion: 1) verify quorum, 2) acquire lease atomically, 3) run health checks, 4) announce leadership only after durable write and replication caught up.Recovery after partition heals- Do not auto‑merge leaders. Detect divergent leaders and block writes until reconciliation.- Re‑establish quorum: ensure a majority of nodes are reachable.- Determine authoritative data source: prefer the quorum-backed leader or the node with highest term/last-applied index.- Fencing stale leaders: use STONITH or network ACLs to isolate any node that might act as leader but lost lease.- Replay/merge logs: apply replicated logs from authoritative leader to bring followers current; run checksums and conflict resolution for divergent entries.- Verify system health: run end-to-end tests, monitor metrics, and only then re-enable write traffic.Operational practices- Automate fencing and lease renewal scripts; test partitions in staging.- Maintain runbook: step‑by‑step commands for detecting split brain, fencing, promoting leader, and rollback.- Regular backups and immutable audit logs to recover from unexpected divergence.This approach minimizes window for double‑leadership and provides clear, tested recovery steps for safe rejoin.
EasyTechnical
74 practiced
Describe common health-check types (TCP connect, HTTP status probe, script/command) and design a practical health-check strategy for a critical internal API that must remain available across two availability zones. Include check frequency, retry thresholds, and the action to take when checks fail.
Sample Answer
**Common health‑check types (brief)**- **TCP connect** — attempts TCP handshake on service port. Good for basic liveness (process listening + network path).- **HTTP status probe** — GET/HEAD to endpoint (e.g., /health) expecting 200/204. Checks app-level readiness and simple downstream dependencies.- **Script/command** — custom shell/script run on host that can check DB connectivity, message queues, disk space, config, etc. Most expressive but costlier and must be secured.**Practical strategy for a critical internal API across 2 AZs**- **Checks** - Primary: HTTP status probe to /health (must validate app + key dependency responses). - Secondary: TCP connect to port as a fast fallback. - Periodic deep: local script every 30s that validates DB ping, cache, and disk <80%.- **Frequency & thresholds** - HTTP probe: every 10s, fail after 3 consecutive failures (30s). - TCP probe: every 10s, fail after 5 consecutive failures (50s). - Script: every 30s, fail after 2 consecutive failures (60s). - Recovery: require 2 consecutive successful checks to mark healthy.- **Actions on failure** - Instance-level failure: mark instance unhealthy in LB and remove from rotation immediately. - AZ-level degraded ( >50% instances unhealthy in AZ ): shift traffic away via weighted LB/DNS to healthy AZ; trigger autoscaling to spin replacement instances in healthy AZ first. - Automated remediation: try graceful restart of app service once per failure window; if restart fails, recreate instance via auto-replace. - Alerting & escalation: page on SRE/Oncall immediately when instance recreate or AZ shift occurs; create ticket with failure diagnostics (logs, metrics). - Post-recovery: run replayed health script and run full integration test before returning traffic.This balances low-latency detection with deeper checks and safe automated remediation to keep the API available across both AZs.
Unlock Full Question Bank
Get access to hundreds of High Availability and Disaster Recovery interview questions and detailed answers.
