Reliability, High Availability, and Tradeoffs Questions

Design patterns and decision making for ensuring availability correctness and graceful behavior under failure while balancing technical trade offs. Topics include redundancy and failover strategies active passive and active active deployments; fault isolation using bulkheads and circuit breaker patterns; graceful degradation and feature gating strategies; defining and mapping service level objectives and service level agreements to recovery point and recovery time objectives; multi region and multi availability zone deployment considerations; testing for reliability including chaos engineering and fault injection; and reasoning about consistency versus availability trade offs and the operational cost of stronger guarantees. Candidates should be able to choose reliability patterns to meet business objectives and to explain their implications for cost performance and maintainability.

EasyTechnical

22 practiced

Why do stateless services typically improve availability and simplify failover? Provide two examples of how making a service stateless reduces recovery time and two counter-examples where stateful behavior is unavoidable and must be handled carefully.

MediumSystem Design

34 practiced

Design a failover strategy for a critical stateless API service serving 10 million monthly active users with 99.99% SLO. Your design should consider multi-AZ deployments, detection time, DNS/traffic switching mechanisms, and expected failover RTO. Describe sequence of steps during failover.

Sample Answer

Requirements & constraints:- Stateless API, HA across AZs, serving 10M MAU, SLO 99.99% (~4.38 min monthly downtime).- RTO target: ≤ 60 seconds for partial outage; ≤ 5 minutes for full-region failover.- Minimize user impact, preserve session affinity via tokens, and avoid DNS TTL pitfalls.

High-level design:- Deploy identical API pods in an orchestrator (K8s/ECS) across 3 AZs in primary region (N AZs≥2). Use autoscaling to maintain desired capacity per AZ (N+1 redundancy).- Active-active per-AZ load balancing using regional ALB/ELB or API Gateway with health checks to AZ-target groups.- Global traffic management: use a managed global DNS/traffic manager (Route53 latency-based + health checks, or GSLB like NS1/Traffic Director) with health probes to regional endpoints.- Observability: granular SLO/SLI metrics (latency, error-rate, availability), synthetic probes from multiple locations, per-AZ instance-level health, and on-call alerting.

Failure detection & decision:- Multi-tier health checks: - L4 probe: TCP connect per instance (1s interval). - L7 probe: lightweight /health (checks dependencies, DB connectivity, config) — 5s interval. - Synthetic external probes: global every 10s to detect user-impacting failures.- Automated detection rules: if >X% instances in an AZ fail L7 probe for >10s OR external probes to region degrade beyond SLO thresholds for 30s, trigger failover workflow.

Traffic switching mechanisms:- Fast local failover (AZ-level): ALB removes unhealthy targets via health checks automatically; traffic rebalanced across healthy AZs within seconds.- Region-level failover: GSLB uses health-check results and failover policy: - Prefer primary region; on primary unhealthy (sustained for 30–60s), shift traffic to secondary region using weighted-routes with quick convergence (DNS TTL=30s or 60s plus partial TCP-level redirection via Anycast/Global Accelerator). - Use global accelerator (Anycast) or Cloud CDN fronting to get sub-second failover for TCP/UDP where supported; otherwise accept DNS-based RTO ~ TTL + propagation.

Failover sequence (ordered steps):1. Detection: internal L7 alarms + external synthetic failures cross threshold (30s).2. Automatic mitigation: - Remove failing instances from ALB target groups (seconds). - Scale up healthy AZs if capacity drop >20% (automated ASG/K8s HPA).3. If remaining capacity insufficient or region-wide degradation sustained (60s): - GSLB switches weights: primary→0, secondary→100 (atomic if Global Accelerator; DNS if not). - Deploy warmed secondary region stack (IaC pre-provisioned), verify health via orchestration hooks.4. Traffic ramp: ramp traffic to secondary over 30–60s (weighted routing) while monitoring errors.5. Validation: synthetic and user metrics return to SLO; keep primary in degraded state for investigation.6. Post-failover: run rollback or cutback once root cause fixed; use blue-green principles to return traffic gracefully.7. Postmortem: capture timeline, metrics, and update runbooks.

Expected RTO:- AZ failure (local): ~5–15 seconds (load balancer health checks + autoscaling).- Region failover with Global Accelerator/Anycast: ~1–30 seconds.- Region failover using DNS-only (TTL 30s): ~30–120 seconds depending on client TTL caching; plan RTO ≤5 minutes.

Trade-offs & notes:- Short DNS TTLs speed failover but increase DNS traffic and potential cache variability.- Pre-warmed secondary increases cost but reduces RTO.- Anycast/Global Accelerator reduces RTO and session disruption for TCP-based APIs.- Test regularly: chaos engineering exercises, game days, and rehearsed runbooks to ensure RTO meets SLO.

EasyTechnical

37 practiced

Describe how to choose Recovery Time Objective (RTO) and Recovery Point Objective (RPO) for services with differing business criticality. Give two concrete examples (e.g., a customer profile store vs. a payments ledger) including recommended RTO/RPO and rationale.

HardTechnical

21 practiced

Explain how to design service-level dependency graphs and use them to prioritize reliability investments. Describe how dependency criticality, failure probability, and impact on user-facing SLIs combine into a scoring model to guide mitigation efforts.

Sample Answer

Requirements & goals:- Identify upstream/downstream service relationships, quantify risk to user-facing SLIs, and prioritize investments to reduce user-visible outages within a fixed budget/error‑budget.

Designing the dependency graph:- Nodes = services/components (APIs, infra pieces, third-party). Edges = synchronous or asynchronous dependency with direction and call characteristics (QPS, latency, timeout).- Enrich nodes/edges with metadata: ownership, SLOs, recent incident history, capacity/cost to mitigate.

Scoring model (combine three factors):1. Dependency criticality (C): how essential is the node to a user SLI. Compute via weighted fan-in/fan-out and request-path coverage. Normalize 0–1.2. Failure probability (P): estimated from historical MTTR/MTBF, change frequency, test coverage, and black‑box error rates. Use exponential smoothing to forecast; normalize 0–1.3. SLI impact (I): how much the node’s failure reduces the user SLI (e.g., % latency tail increase or error-rate delta). Measured by chaos experiments, canary failures, or failure-mode analysis; normalize 0–1.

Combined score: Risk = C * P * I (or Risk = w1*C + w2*P + w3*I for additive preferences). Multiplicative emphasizes that low value in any factor reduces overall risk.

Example:- Auth service: C=0.9 (on auth path), P=0.2 (stable), I=0.8 (auth failures block users) → Risk = 0.144.- Cache tier: C=0.7, P=0.6, I=0.5 → Risk = 0.21 → higher priority despite lower criticality.

Prioritization & investment rules:- Rank by Risk; compute cost-to-reduce-risk for mitigation options (redundancy, retries, better monitoring, runbooks). Use ROI = ΔRisk / Cost.- Target high-risk, low-cost fixes first; maintain some spend on reducing P for high-C components.- Allocate a buffer for systemic mitigations (shared infra) vs. service-level fixes.

Validation & governance:- Continuously recalibrate P and I from incidents, SLI telemetry, and chaos testing.- Use dashboards to show Risk and ROI; tie mitigation projects to error‑budget burn rates and quarterly reliability OKRs.- Periodically simulate prioritized outages and measure SLI delta to validate model fidelity.

Trade-offs:- Multiplicative model helps find single points of failure; additive lets policy tune sensitivity. Choose based on organizational risk tolerance and data quality.

MediumTechnical

19 practiced

Discuss design patterns to maintain availability when multiple services must perform a distributed transaction. Compare saga pattern vs two-phase commit (2PC) in terms of availability, complexity, and operational cost, and give an example where saga is preferred.

Sample Answer

Start with the problem: distributed transactions require atomicity across services, but coordinating them can reduce availability. Two common patterns are Two-Phase Commit (2PC) and the Saga pattern.

2PC (coordinator-driven commit)- How it works: a coordinator asks all participants to prepare, then either commits or aborts based on unanimous prepare responses.- Availability: low — blocking protocol: if coordinator or participants fail during commit, resources may be locked and progress stalls.- Complexity: conceptually simple but requires durable locking, careful failure/recovery logic, and support in each service.- Operational cost: higher — more DBA-like operational burden, complex recovery procedures, slower latency, transactional locks across services.

Saga (choreography or orchestrator)- How it works: a long-lived transaction is split into a sequence of local transactions; on failure, compensating transactions undo prior steps.- Availability: higher — services proceed independently; no global locks so system remains responsive even if some services are degraded.- Complexity: higher design complexity — need idempotency, compensating actions, and careful ordering; reasoning about consistency is harder.- Operational cost: moderate — more application-level logic and testing, but simpler runtime (no distributed lock manager), easier to scale and observe.

SRE trade-offs- Prefer saga when you prioritize availability and loose consistency (eventual consistency acceptable), need horizontal scalability, and want to avoid cross-service blocking during failures.- Prefer 2PC only when strict strong consistency and atomic commit across heterogeneous systems are mandatory and you can accept reduced availability and higher ops cost.

Concrete example where Saga is preferred- E-commerce order placement: steps — reserve inventory, charge payment, create shipment. Use an orchestrator that issues commands and, on payment failure, runs compensating actions to release inventory and cancel shipment. This keeps checkout available under partial failures and simplifies rolling back without global locks.

Operational recommendations- Implement idempotency and retries, circuit breakers, clear monitoring/trace for sagas, and run compensator workflows as separate, observable jobs. For critical financial transfers requiring strict ACID, evaluate 2PC or transactional middleware but plan for increased operational automation and recovery runbooks.

Unlock Full Question Bank

Get access to hundreds of Reliability, High Availability, and Tradeoffs interview questions and detailed answers.

Join thousands of developers preparing for their dream job.