Designing systems to remain available and recoverable in the face of infrastructure failures, outages, and disasters. Candidates should be able to define and reason about Recovery Time Objective and Recovery Point Objective targets and translate service level agreement goals such as 99.9 percent to 99.999 percent into architecture choices. Core topics include redundancy strategies such as N plus one and N plus two, active active and active passive deployment patterns, multi availability zone and multi region topologies, and the trade offs between same region high availability and cross region disaster recovery. Discuss load balancing and traffic shaping, redundant load balancer design, and algorithms such as round robin, least connections, and consistent hashing. Explain failover detection, health checks, automated versus manual failover, convergence and recovery timing, and orchestration of failover and reroute. Cover backup, snapshot, and restore strategies, replication and consistency trade offs for stateful components, leader election and split brain mitigation, runbooks and recovery playbooks, disaster recovery testing and drills, and cost and operational trade offs. Include capacity planning, autoscaling, network redundancy, and considerations for security and infrastructure hardening so that identity, key management, and logging remain available and recoverable. Emphasize monitoring, observability, alerting for availability signals, and validation through chaos engineering and regular failover exercises.
HardTechnical
82 practiced
Explain techniques to avoid split-brain in a multi-master, multi-region replication topology. Compare quorum-based writes, lease/term-based leaders, vector clocks, deterministic conflict resolution, and CRDTs. Recommend an approach specifically for a collaborative document-editing application and justify your choice.
Sample Answer
Situation/context: In multi-master, multi-region systems, "split-brain" occurs when replicas diverge and accept conflicting writes concurrently. Avoiding or resolving it requires trade-offs between availability, consistency, latency, and complexity. Below I compare five techniques and then recommend a fit for collaborative document editing.Quorum-based writes- How: Read/write require majority (or configurable) quorum across replicas.- Pros: Stronger consistency, deterministic conflict avoidance if quorums overlap.- Cons: Higher cross-region latency; risk of reduced availability during partitions; operational complexity tuning quorums.Lease/term-based leaders- How: Elect a single leader (per shard) with a time-bound lease (Raft-style terms).- Pros: Simple linearizability, low-latency local writes directed through leader, clear single-writer semantics.- Cons: Leader failover complexity, requires reliable clocks or clock skew handling; cross-region leader placement affects latency.Vector clocks- How: Track causal histories per object; detect concurrent divergent updates.- Pros: Precise detection of concurrency; lightweight metadata.- Cons: Requires application-level merge logic; vector size grows with participants; only detects conflicts, doesn't resolve them.Deterministic conflict resolution- How: Use deterministic rules (last-writer-wins with timestamps, merge policies) to auto-resolve conflicts.- Pros: Simple, predictable; avoids manual reconciliation.- Cons: Can lose user intent (e.g., overwriting edits), relies on synchronized clocks or logical clocks.CRDTs (Conflict-free Replicated Data Types)- How: Data types designed so concurrent updates commute and converge without coordination.- Pros: Strong eventual consistency, no coordination or leader required, preserves intent for many operations (e.g., sequences, sets).- Cons: Data-type design complexity, possible metadata growth, some semantics (complex document edits) require advanced CRDTs.Recommendation for collaborative document-editing- Use CRDTs (sequence CRDTs like RGA/Logoot/WOOT or optimized versions like Yjs/Automerge/YATA) for the document model, combined with opportunistic leader/lease for presence/real-time session coordination and durability via multi-region replication with quorum for checkpoints.- Justification: Collaborative editing demands low-latency local responsiveness, concurrent multi-user edits, and preservation of user intent. CRDTs provide convergence without central coordination and excellent offline/partition behavior. Pairing CRDTs with a lightweight lease for session master (for tasks like cursor ownership, autosave orchestration) gives predictable coordination where needed. Periodic quorum-backed checkpoints or tombstone compaction ensures durability and consistent garbage collection across regions.- Trade-offs: Requires careful CRDT choice to bound metadata growth and to support rich text/operational transforms. If strong linearizability for document-wide operations is required (e.g., final publish with legal audit), use a quorum/consensus boundary for that workflow.Summary: For collaborative editing prioritize CRDT-based models for convergence and UX, augment with targeted leader leases for coordination and quorum-backed durability for critical state.
HardTechnical
80 practiced
Explain how to orchestrate health-checks across a microservices mesh to avoid cascading failures: include propagation of dependency health, computing health-weighted routing decisions, strategies for graceful degradation, and how to pause automated rollouts when dependent services are degraded.
Sample Answer
Requirements:- Prevent cascading failures across microservice mesh- Expose per-service and aggregated dependency health- Drive routing decisions by health-weighted scores- Support graceful degradation strategies and pause CI/CD rollouts when dependencies are degradedHigh-level design:- Use service-mesh (e.g., Istio/Linkerd) + sidecars for probe collection + central Health Aggregator service + Control Plane for routing and rollout control.Key components & flow:1. Sidecar probes: each service reports liveness, readiness, and dependency-level checks (calls to downstreams measured with latency, error-rate, SLO breach flags). Emit metrics (Prometheus) and structured health events (gRPC/HTTP) to the Health Aggregator.2. Health Aggregator: maintains dependency graph and computes rolling health scores per node using weighted factors (error-rate, p95 latency, saturation). Propagate upstream impact via graph propagation (e.g., exponential decay): upstreamHealth = f(ownHealth, weighted sum of downstream impacts).3. Control Plane: subscribes to aggregator; exposes health-weighted routing policies to mesh (via Envoy/Istio APIs). Routing weight = normalized healthScore^alpha (alpha tunes sensitivity). Fast path: set traffic-splits, mirror or divert traffic; slow path: circuit-breakers and retry budgets.4. Rollout Controller (integrated with CI/CD): watches aggregated health. On significant downstream degradation beyond thresholds, pause progressive rollout, mark canary unhealthy, and optionally rollback.Graceful degradation strategies:- Feature flags + progressive fallbacks (serve cached/read-only, reduced UI features)- Degraded-response templates and client-side feature toggles- Tiered routing: route critical traffic to healthy instances; non-critical to degraded pool- Backpressure: apply rate-limits at edge and token-bucket quotas per consumer to protect core servicesOperational details & trade-offs:- Use short-lived, aggregate windows (30s–2m) to be responsive, but include hysteresis to avoid flapping.- Compute health scores with explainability (store contributing metrics) so SAs can tune alpha, decay, thresholds.- Security/scale: stream health events over mTLS, shard aggregator for large graphs.Example thresholds:- If downstream error-rate >5% and impacts >20% of request paths => reduce routing weight by 50%, pause non-critical rollouts, enable circuit-breakers for affected calls.This design prevents cascades by propagating dependency impact, making routing decisions health-aware, enabling graceful fallbacks, and halting automated rollouts until dependent services recover.
EasyTechnical
82 practiced
Explain failover detection mechanisms used in distributed systems: heartbeats/leases, monitoring-based alerts, request-failure thresholds, and leader-lease expiration. Compare automated (automatic) failover with manual failover decision-making and identify scenarios where human approval should remain in the loop.
Sample Answer
Failover detection mechanisms — how they detect a failed node/service — typically include:- Heartbeats/leases: periodic lightweight pings (heartbeat) or time-limited ownership (lease). If heartbeats stop or lease expires, the system treats the node as unavailable. Good for low-latency detection; must tune heartbeat interval and grace period to balance sensitivity vs false positives (network partitions).- Monitoring-based alerts: telemetry (metrics, logs, health endpoints) raised into alerting systems (Prometheus, CloudWatch). Slower but richer context—useful to correlate partial degradation, resource exhaustion, or cascading failures.- Request-failure thresholds: trigger when error rate, latency, or request timeouts exceed configured thresholds (e.g., 5xx rate > 5% for 1 minute). Effective for detecting degraded service that still responds.- Leader-lease expiration: leaders hold a timed lease; when it expires (no renewal), other nodes can elect a new leader. Protects against split-brain when combined with quorum checks.Automated vs manual failover:- Automated failover: fast, reduces MTTR, appropriate for stateless services or systems with strong consistency/quorum guarantees and well-tested procedures (load balancers, stateless microservices, read replicas). Risk: false positives causing unnecessary failovers and potential data divergence.- Manual (human-in-loop): slower but safer for stateful systems where failover has high operational or business impact (databases without synchronous replication, billing systems, large transactions), or when root cause is unclear.When to require human approval:- Risk of data loss or split-brain (primary DB promotion)- Complex multi-step recovery affecting billing/security/compliance- Cross-region failover with significant cost/latency/regulatory effects- When alerts indicate ambiguous root cause or cascading failures where automated action could worsen stateBest practice: combine mechanisms (heartbeats + metrics + request thresholds), use automatic failover for low-risk paths, and require gated/manual approval (runbooks + playbooks + “canary” steps) for high-risk stateful recoveries. Include observability and post-failover audits.
MediumTechnical
84 practiced
Compare disaster-recovery strategies: cold-standby, pilot-light, warm-standby, and active-active. For each approach describe typical RTO/RPO ranges, cost implications, operational complexity, and when you would recommend it to a medium-sized SaaS customer with limited budget.
Sample Answer
Brief summary: choose trade-offs between recovery speed (RTO), data loss tolerance (RPO), cost, and operational complexity. Recommendations below assume a medium-sized SaaS with limited budget.Cold-standby- RTO/RPO: RTO = hours–days (typically 12–72h); RPO = hours–days (last full backup).- Cost: Lowest — pay for backups and storage; minimal compute reserved.- Complexity: Low (simple backups and restore runbooks) but restores and testing can be slow.- Recommend when: Non-critical services or components where downtime is tolerable and cost must be minimal.Pilot-light- RTO/RPO: RTO = 1–6 hours; RPO = minutes–hours (incremental replication).- Cost: Low–moderate — core infra (DB replica, core configs) running; app servers absent until failover.- Complexity: Moderate — automated provisioning scripts, replication, tested failover steps.- Recommend when: SaaS with moderate SLA needs wanting low cost but faster restart than cold-standby.Warm-standby- RTO/RPO: RTO = minutes–1 hour; RPO = seconds–minutes (near-real-time replication).- Cost: Moderate — smaller-sized active instances live in DR region, scaled down.- Complexity: Higher — synchronous/async replication, health checks, scaling policies, regular failover rehearsals.- Recommend when: Customer needs quick recovery and low data loss but still mindful of costs.Active-active- RTO/RPO: RTO ≈ near-zero (automatic routing); RPO ≈ near-zero (multi-region replication).- Cost: Highest — fully provisioned capacity across regions, complex networking.- Complexity: High — global consistency, conflict resolution, distributed DB patterns, traffic routing.- Recommend when: Mission-critical services with strict SLAs. For budget-constrained medium SaaS, only use selectively for critical subsystems (e.g., auth, billing) while other parts stay warm or pilot-light.Operational notes:- Always automate failover and rehearse runbooks; measure actual RTO/RPO in drills.- Consider hybrid: use pilot-light for most components, warm or active-active for critical paths to balance cost and risk.
EasyTechnical
83 practiced
List and briefly describe common load-balancing algorithms (round-robin, weighted round-robin, least-connections, consistent-hashing, IP-hash). For each algorithm, give an example use-case, describe how it affects failover and session affinity, and note one potential pitfall.
Sample Answer
Round-robin- Description: Rotate requests evenly across healthy backends in order.- Use-case: Simple stateless web servers behind an L4/L7 load balancer.- Failover & session affinity: Good failover if health checks remove failed nodes; no built-in session affinity (sticky sessions require additional mechanism).- Pitfall: Unequal capacity—treats all servers equally, so overloaded smaller instances.Weighted round-robin- Description: Like round-robin but assigns weights so higher-capacity nodes receive more traffic.- Use-case: Mixed-instance clusters (e.g., some VMs with more CPU/RAM).- Failover & session affinity: Weights adjusted when nodes removed; still no native affinity unless combined with cookies/IP.- Pitfall: Static weights may become suboptimal as load or instance performance changes.Least-connections- Description: Send new requests to the backend with the fewest active connections.- Use-case: Long-lived connections (WebSocket, gRPC) or heterogeneous request durations.- Failover & session affinity: Reacts quickly to hotspots; affinity not provided unless tied to a consistent key.- Pitfall: Connection counts may not reflect CPU/IO usage—short vs long requests skew decisions.Consistent-hashing- Description: Hash request key to a ring so keys map to specific backends; minimizes remapping when nodes change.- Use-case: Cache sharding (Redis/memcached) or sticky routing by resource ID.- Failover & session affinity: Provides strong affinity by key; when a node fails, only a subset of keys remap (rebalanced).- Pitfall: Uneven key distribution if hash or virtual node configuration is poor; complexity in rebalancing stateful caches.IP-hash- Description: Hash client IP to pick a backend, keeping same client routed consistently.- Use-case: Simple session affinity without cookies (legacy clients).- Failover & session affinity: Provides basic affinity but when a node fails, clients get remapped unpredictably; affinity breaks for clients behind NAT/proxies.- Pitfall: Clients behind shared IPs (proxies) concentrate traffic to one backend causing hotspots.Practical note: Combine algorithms with health checks, autoscaling, and application-level session storage (or distributed caches) to mitigate pitfalls and achieve resilient architectures.
Unlock Full Question Bank
Get access to hundreds of High Availability and Disaster Recovery interview questions and detailed answers.
