Designing systems to remain available and recoverable in the face of infrastructure failures, outages, and disasters. Candidates should be able to define and reason about Recovery Time Objective and Recovery Point Objective targets and translate service level agreement goals such as 99.9 percent to 99.999 percent into architecture choices. Core topics include redundancy strategies such as N plus one and N plus two, active active and active passive deployment patterns, multi availability zone and multi region topologies, and the trade offs between same region high availability and cross region disaster recovery. Discuss load balancing and traffic shaping, redundant load balancer design, and algorithms such as round robin, least connections, and consistent hashing. Explain failover detection, health checks, automated versus manual failover, convergence and recovery timing, and orchestration of failover and reroute. Cover backup, snapshot, and restore strategies, replication and consistency trade offs for stateful components, leader election and split brain mitigation, runbooks and recovery playbooks, disaster recovery testing and drills, and cost and operational trade offs. Include capacity planning, autoscaling, network redundancy, and considerations for security and infrastructure hardening so that identity, key management, and logging remain available and recoverable. Emphasize monitoring, observability, alerting for availability signals, and validation through chaos engineering and regular failover exercises.
HardTechnical
75 practiced
You run a Raft-based cluster across three regions where inter-region latency is high. Propose changes to leader placement, election timeouts, read path optimization, and quorum rules to reduce availability impact from network flaps while preserving safety.
Sample Answer
Approach: balance availability and safety by making leader placement and election behavior latency-aware, using lease/ReadIndex optimizations for reads, and carefully relaxing operational quorum behaviour (not consensus quorum) with strong monitoring and fallback. Changes must never violate Raft’s majority-safety.1) Leader placement- Prefer stable, low-latency region as primary leader candidate (e.g., region with strongest connectivity and most capacity). Use deterministic leader preference (e.g., region ranking + stickiness) so leaders don’t bounce between regions during transient flaps.- Make leadership “sticky”: after election, increase the leader’s priority to avoid unnecessary re-election unless a definitive failure is detected.2) Election timeouts- Increase election timeouts and tune them per-region using measured RTTs: election_timeout = base + k * max_inter_region_RTT. Use randomized jitter range proportional to RTT to reduce split votes across high-latency links.- Detect chronic high-latency: dynamically widen timeouts when inter-region latency spikes are observed to avoid elections triggered by transient delays.3) Read-path optimization- Use lease-based leader reads or ReadIndex where possible: - If leader holds a valid lease (clock-based with bounded clock drift), serve linearizable reads locally without contacting a majority, reducing cross-region RTTs. - If lease cannot be guaranteed, use ReadIndex which at worst requires a round-trip to majority but avoid full leader heartbeat blocking.- Allow stale (follower) reads with an explicit option and SLO tradeoff: serve local reads with max-staleness policy (e.g., “read-your-writes” or T milliseconds), with clear client-side flags and metrics.4) Quorum and write behavior- Preserve Raft majority quorum for safety on committed writes. Do NOT change core majority requirement.- Operational flexibility: introduce read-only witnesses/observers in remote regions to improve read locality without counting toward commit quorum.- For planned degraded operations (e.g., region outage), support conditional forwarding: clients in minority regions forward writes to leader region to maintain single-writer safety.- Optionally use a “preferred majority” policy: bias leader elections to choose leaders located in a predefined set of regions forming a stable majority (e.g., two preferred regions). This is an operational policy, not a protocol change.Implementation & monitoring- Instrument: leader changes, election start/completion times, RTT distributions, ReadIndex/lease hit ratio, client-perceived latency and error rates.- Rollout: enable leader preference and timeout increases gradually, observe election frequency; enable lease reads once clock drift bounds and lease renewal telemetry are validated.- Alerts: elevated election rate, lease renewal failures, majority connectivity loss.Trade-offs & safety- Increasing timeouts raises failover latency but reduces flapping—acceptable for availability if SLOs allow slightly longer failover.- Lease reads improve latency but rely on clock drift assumptions; fall back to ReadIndex if lease safety is uncertain.- Never reduce commit quorum below majority—safety preserved.Expected outcomes & metrics- Fewer unnecessary elections, lower write/read latency for most clients, reduced availability blips during transient network flaps.- Track: elections/hour ↓, leader churn ↓, median read/write latency ↓, linearizability violations = 0 (verified), and incident count for region flaps ↓.
HardSystem Design
63 practiced
Design a secure backup and restore strategy for encrypted secrets and identity materials such that keys remain confidential and recoverable after a full-region outage. Consider key rotation, offline escrow, multi-party approval workflows, and automation to reduce recovery time while preserving security.
Sample Answer
Requirements & constraints:- Recover all encrypted secrets and identity materials after a full-region outage while keeping keys confidential.- Support regular key rotation, multi-party approval for critical recovery actions, offline escrow, automated recovery to meet RTO/RPO targets.- Threats: compromised KMS in-region, insider threats, physical theft of escrow, and availability zone/region failures.High-level architecture:- Active KMS cluster in primary region + geographically separated, offline escrow in two cold sites (different legal jurisdictions).- Hardware Security Modules (HSMs) for root keys at each escrow site; KMS instances use ephemeral data keys derived from root via KMIP/HSM.- Secrets stored encrypted with envelope encryption: secret_data = Enc_DEK(secret), DEK encrypted with KMS: Enc_Root(DEK).Core components & workflows:1. Key hierarchy: - Root Key (RK) in offline HSM escrow. Dual HSM shards optionally via Shamir Secret Sharing across two escrow sites. - Regional KMS holds WRAPPED_DEKs; KMS uses RK via secure signing/unwrap during recovery only.2. Offline escrow & multi-party: - RK split into n-of-m shares; physical smartcards + HSMs. Recovery requires m operators with hardware tokens and a quorum service (e.g., threshold sig). - All recover actions require multi-party approval recorded in immutable audit (WORM storage) and time-locked policy.3. Rotation: - Regular automatic DEK rotation per secret; RK rotated annually or on major event. Rotation pipeline: create new RK in escrow, rewrap active DEKs, update metadata, with staged rollout and audit.4. Automation for recovery: - Playbooks implemented as signed automation jobs triggered after verified region-loss event; jobs validate multi-party approvals, retrieve escrow shares via secure channel, reconstruct RK in ephemeral HSM environment, unwrap DEKs, restore secrets to standby region KMS. - Use Infrastructure-as-Code with canary steps, continuous integrity checks, and tamper-evident logs.5. Security controls: - Hardware-backed key material, least-privilege IAM, strong authentication (MFA + hardware tokens), air-gapped channels for escrow retrieval, encryption-in-transit with mutual TLS, attestation of HSM images. - Threshold cryptography prevents single-point compromise.6. Observability & testing: - Regularly scheduled disaster recovery drills that reconstruct RK in isolated test environment; cryptographic proof-of-possession checks; golden-record verification; metrics on MTTR.7. Trade-offs: - Offline escrow + multi-party increases recovery time and operational complexity but substantially reduces compromise risk. - Shamir vs. HSM quorum: Shamir is flexible but requires secure custody; HSM quorum gives stronger tamper resistance.Result: Confidential root keys remain offline and split across jurisdictions; automated, auditable recovery reduces human error and MTTR, while rotation and continuous testing keep security posture robust.
EasyTechnical
89 practiced
List common health check types used to detect service failure (for example TCP probe, HTTP probe, application-specific checks). Provide example checks for an API that depends on a database and cache, and explain ways to avoid false positives and negatives.
Sample Answer
Common health-check types- TCP probe: verifies port is accepting connections (network-level).- HTTP(S) probe: GET/HEAD to an endpoint; checks status code, latency, headers.- Command/exec probe: runs a script/binary inside the host/container for custom logic.- Application-specific (semantic) check: verifies business logic (e.g., DB query, cache read).- Dependency checks: lightweight checks for downstream services (message queue, auth).- Synthetic/transactional test: exercises a full user-flow end-to-end.Example checks for an API that depends on DB and cache- Readiness probe (before routing traffic): - HTTP /ready returns 200 only if: DB connection pool healthy, simple SELECT 1 succeeds, cache reachable (PING), schema migrations applied.- Liveness probe (detect hung process): - TCP on app port OR an internal exec that checks event loop/thread liveness and recent request timestamps.- Background dependency check (periodic, non-blocking): - Execute a small read from DB and a cache set/get; emit metrics and alarms if error rate > threshold.- Synthetic external check: - From synthetic runner, perform an authenticated API call that reads a cached object; verify correct response within SLA.Avoiding false positives/negatives- Don’t overload probes: make DB/cache probes lightweight (SELECT 1, PING) and short timeouts.- Separate readiness vs liveness: readiness fails prevent traffic; liveness restarts only on unrecoverable hangs.- Use thresholds and retries: require N consecutive failures before alert/restart to avoid transient network glitches.- Circuit-break and degrade gracefully: if cache temporarily fails, API can fallback to DB to remain healthy.- Monitor probe latency, not just success: rising latencies indicate degradation before errors.- Ensure probes are representative but not identical to production load to avoid impacting performance.
HardSystem Design
122 practiced
Design network redundancy for cross-region traffic using multiple transit providers, BGP failover, and private links. Discuss detection and failover mechanisms, route convergence characteristics, how to test provider-level failover, and automation to orchestrate provider switchovers.
Sample Answer
Requirements and constraints:- Cross-region traffic must remain available with <1–3 min failover target for provider outages, preserve predictable latency for critical paths, and use multiple transit providers + private links (MPLS/EVPN/Direct Connect) with BGP for control-plane.High-level architecture:- Each region has two or more upstream transit providers (A/B) and at least one private inter-region link to a peer/co-lo (Direct Connect, MPLS).- Edge routers run iBGP with regional route reflectors and eBGP sessions to each provider. Traffic engineered with BGP local-preference, AS-path prepending, MED, and communities.- Data plane prefers private link for intra-service cross-region traffic; internet-bound leaves via preferred provider.Detection & failover mechanisms:- Fast detection: BFD sessions on eBGP neighbors (sub-50ms timers) + BGP neighbor state. Supplement with active probes (HTTP/TCP) to regional service endpoints and on-path synthetic latency checks.- On detection of provider circuit loss or BFD down, BGP withdraws routes; local-preference adjustments or prepends on standby provider cause traffic to shift. For private-link failures, use BGP session over alternate path and reroute via transit providers.Route convergence characteristics:- Use aggressive BFD and tuned BGP timers for fast detection; expect control-plane convergence in <100–300ms for edge withdraws but end-to-end convergence often dominated by upstream transit policy and AS-path propagation (seconds). Use graceful restart and add-path carefully to avoid flap storms. Monitor convergence with RIB/FIB delta metrics.Testing provider-level failover:- Controlled tests: scheduled maintenance windows with staged failures—disable BFD, then admin-shut eBGP, then simulate blackhole using ACLs. Canary tests with small % of traffic (traffic-splitting via MED/local-pref or ECMP). Use Chaos engineering (chaos scripts that cut physical links) and validate end-to-end SLOs and telemetry.Automation & orchestration:- Playbooks in IaC (Ansible/Terraform + vendor APIs) to change BGP policies (local-pref, prepend, communities) and to toggle route-maps. Orchestrator service (k8s microservice) exposes safe runbooks: dry-run, canary, rollback, and automatic remediation (on BFD/bgp-down events). Integrate with monitoring/incident system to trigger automated cutovers when thresholds met, with pre-checks (capacity, latency). Keep all changes in Git, CI for config linting, and staged promotion.Operational considerations & trade-offs:- Faster timers reduce detection time but increase false positives—use multi-signal detection. Pre-configured partial failover (shift a fraction of traffic) reduces risk. Work with providers to coordinate community-based prepends for quicker convergence. Maintain capacity headroom and thorough observability (per-prefix telemetry, flow logs, latency histograms) to validate behavior during failovers.
HardTechnical
80 practiced
Explain split-brain detection and mitigation techniques including quorum-based decisions, fencing (for example STONITH), compare-and-swap leader leases, and external arbitration. For a disk-backed leader service, recommend a preferred approach and justify how it prevents stale-writer scenarios.
Sample Answer
Split-brain occurs when network partitions or failures let multiple replicas believe they are the leader and accept writes — causing divergence and stale-writer problems. Below I explain detection/mitigation techniques, trade-offs, and a recommended approach for a disk-backed leader service.Quorum-based decisions- Principle: require a majority (or configured quorum) for leader election and for write commits. Only nodes that can form quorum can accept leader role or commit changes.- Strengths: simple, proven (Paxos/Raft); prevents two disjoint majorities.- Weaknesses: availability drops when quorum unreachable; minority may become read-only.Fencing (STONITH)- Principle: forcibly cut off or reboot a suspected-stale leader (Shoot The Other Node In The Head) via out-of-band power/network control before allowing new leader to act.- Strengths: strongly prevents stale-writer from accessing shared resources (disks, NICs).- Weaknesses: requires reliable out-of-band control, operator privileges, and careful handling to avoid collateral damage.Compare-and-swap leader leases- Principle: leader holds a time-bound lease stored in shared storage (e.g., on-disk token). New leader obtains lease using atomic CAS; lease expiry prevents indefinite ownership.- Strengths: fast; uses atomic primitives to avoid races.- Weaknesses: requires clock stability or monotonic leases; risk if a holder continues writing after lease expiry due to clock skew or pause.External arbitration- Principle: an external arbitrator (witness, quorum service, or cloud metadata service) breaks ties when partitioned nodes cannot form quorum.- Strengths: increases availability in split scenarios; useful for two-node clusters.- Weaknesses: adds dependency and potential single point of failure if not redundant.Recommended approach for a disk-backed leader service- Use quorum-based Raft-style elections combined with strong fencing via STONITH for any leader that had direct access to the shared disk. Specifically: 1. Elect leaders only when a majority agrees (Raft). 2. Before granting new leader control of the shared disk, perform an out-of-band fencing action against the previous node identity that held disk mounts (power-cycle or revoke SAN LUN access via storage API). 3. Record leadership lease atomically on disk (CAS) as an additional safety check and require lease validation before writes.Why this prevents stale-writer scenarios- Quorum prevents two majorities from electing conflicting leaders.- STONITH ensures the previous node cannot continue issuing IO to the disk after partition healing — even if it still thinks it’s leader, its write path is cut off at the device level.- The on-disk CAS lease provides a durable check: any writer that lacks a valid lease or fails CAS is rejected, preventing accidental writes by a partitioned node that regained network visibility but lost storage access.- Combined, these layers (consensus + durable lease + out-of-band fencing) eliminate windows where a stale writer can write to shared persistent state, trading some complexity/availability for correctness — the right trade for disk-backed systems where data corruption is catastrophic.Operational notes- Test fencing repeatedly in staging; ensure storage APIs and orchestration integrate with SRE runbooks.- Monitor lease expiry/renewal, election rates, and fencing failures; alert on any failed fencing attempts for immediate human intervention.
Unlock Full Question Bank
Get access to hundreds of High Availability and Disaster Recovery interview questions and detailed answers.
