Reliability, High Availability, and Tradeoffs Questions

Design patterns and decision making for ensuring availability correctness and graceful behavior under failure while balancing technical trade offs. Topics include redundancy and failover strategies active passive and active active deployments; fault isolation using bulkheads and circuit breaker patterns; graceful degradation and feature gating strategies; defining and mapping service level objectives and service level agreements to recovery point and recovery time objectives; multi region and multi availability zone deployment considerations; testing for reliability including chaos engineering and fault injection; and reasoning about consistency versus availability trade offs and the operational cost of stronger guarantees. Candidates should be able to choose reliability patterns to meet business objectives and to explain their implications for cost performance and maintainability.

EasyTechnical

22 practiced

Explain the differences between SLI, SLO and SLA. Provide a concrete example: given an SLO of 99.95% monthly availability for a payments API, calculate allowable downtime, suggest which SLIs you would instrument, and describe what consequences you would define in an SLA with a paying customer.

MediumSystem Design

24 practiced

Design a failover strategy for a stateful service (session store) that requires an RPO of under 30 seconds and an RTO under 2 minutes in the event of an availability zone failure. Describe replication approach, failover detection, cutover sequence, and operational steps to meet the objectives while keeping costs reasonable.

Sample Answer

Requirements clarification:- RPO < 30s (max data loss 30s), RTO < 2min for AZ failure. Stateful session store (low write volume per session, high read). Keep costs reasonable—avoid active-active cross-region complexity.

High-level approach:- Single-writer primary in AZ-A, synchronous-ish replication to warm replica in AZ-B and async replicas in AZ-C. Use a semi-synchronous replication mechanism (primary waits for ACK from at least one cross-AZ replica) to bound RPO while avoiding full-sync latency on every write.

Replication:- Use primary + at least one cross-AZ replica in a different AZ configured for semi-sync (ack within write path). Additionally keep 1-2 async replicas for read scaling and Backup.- Persist write-ahead log (WAL/AOF) with frequent flush (fsync every 1s or tuned) so replica catch-up window ≤30s.- Network: provision redundant low-latency links between AZs (same region).

Failover detection:- Multi-layer detection: - Local instance health (process, I/O, heartbeat). - Cluster coordinator (lightweight quorum service running across AZs—3 nodes, one per AZ) monitors heartbeats and replication lag metrics. - External load balancer/route53 health checks confirm AZ outage.

Cutover sequence (automated; targeted RTO <2min):1. Detection: coordinator sees primary dead/unreachable and replication lag <=30s on candidate replica → triggers failover.2. Promote: Stop replica read-only mode, apply pending WAL entries, promote to primary. Ensure all committed transactions acknowledged by semi-sync replica are present.3. Update clients: update service discovery/ELB target group and DNS TTL (low TTL e.g., 10s) so app nodes switch to new primary. Optionally use a connection proxy (VIP/HAProxy/Route 53 weighted) to reduce client reconnection complexity.4. Start new warm-replica: Spin a replacement replica in another AZ and bootstrap from new primary (WAL snapshot), bring semi-sync online.

Operational steps & runbook:- Pre-failure: run periodic failover drills; monitor replication lag, WAL sizes, disk I/O, latency; alerts when lag >20s.- During failover: run automated playbook — verify promoted node has writeable status, run smoke tests (write/read), flip LB, scale replicas if needed.- Post-failover: reintroduce recovered AZ as replica only (avoid split-brain). Rebalance reads gradually.- Backups: continuous incremental backups + periodic snapshots to enable recovery beyond replication window.

Trade-offs / Cost control:- Semi-sync limits RPO without paying the latency of full sync for all replicas.- Keep only one semi-sync replica to balance cost vs safety; additional async replicas for scaling.- Use smaller instance types for warm replicas with faster attach/boot scripts to reduce cost.- Use managed services (e.g., ElastiCache Redis replication group with Multi-AZ) to reduce operational overhead—may cost more but lowers risk and implementation time.- If stricter RPO needed, move to fully sync replication at higher latency/cost.

Why this meets objectives:- Semi-sync + WAL tuning bounds data loss <30s.- Automated detection + fast promotion + low TTL client routing keep RTO <2min.- Warm replicas and thoughtful instance sizing keep costs reasonable while preserving reliability.

EasyTechnical

26 practiced

Given an SLO of 99.95% availability per 30-day period, calculate the total allowable downtime in minutes for that period and explain how that limit maps to operational practices like maintenance windows, release windows, and emergency fixes.

HardTechnical

19 practiced

For an internal inventory service used by an e-commerce platform, describe how you would choose between strong consistency, causal consistency, and eventual consistency. Discuss availability, latency, operational complexity, and cost trade-offs and provide guidance on which guarantee to pick for inventory quantity reads/writes vs reporting use cases.

Sample Answer

Start by clarifying requirements: correctness (no oversell), latency SLA, read/write ratio, tolerance for stale reads, failure modes, and business cost of stock misrepresentation (lost sale vs oversell).

Consistency options (brief):- Strong (linearizability): every read reflects the latest completed write.- Causal: operations that are causally related are seen in order; concurrent unrelated ops may reorder.- Eventual: updates propagate asynchronously; replicas converge over time.

Trade-offs:- Availability: Eventual > Causal > Strong. Strong requires quorum/mutexing which reduces availability under partition.- Latency: Eventual lowest (local reads/writes), causal moderate (dependency tracking), strong highest (global coordination/quorum).- Operational complexity: Strong highest (consensus protocols like Raft/Paxos), causal medium (version vectors, session stickiness), eventual lowest.- Cost: Strong incurs higher compute/network (quorum traffic, cross-dc latency), causal medium, eventual cheapest.

Guidance for inventory:- Quantity reads/writes that affect purchase (checkout, reservation, decrement): use strong consistency or a hybrid with strong counters/serializability for critical paths. Example: use a per-SKU linearizable service or strongly-consistent distributed lock for commits to prevent oversell. Accept higher latency for correctness; cache carefully (read-through with validation).- High-throughput front-end reads (availability-focused) can use a read-through cache with short TTLs plus a synchronous strong write path for decrements—this hybrid reduces perceived latency while preserving correctness.- Background reporting, analytics, and dashboards: eventual consistency is acceptable. Use asynchronous replication/event streams (Kafka, CDC) to build denormalized reports. Causal consistency can be useful where session-level ordering matters (user view after their update).

Operational recommendations:- Partition by SKU to limit coordination scope.- Use quorum sizes tuned for region topology (local quorums where possible).- Provide compensating transactions and reconciliation for rare anomalies; instrument metrics and alarms for divergence.- Document SLA trade-offs for stakeholders so product can accept latency vs risk.

Summary: Strong (or hybrid strong for commits) for transactional inventory operations; eventual for reporting and analytics; consider causal for user-facing ordering guarantees where reordering would be harmful but full strong is too costly.

MediumSystem Design

25 practiced

You're splitting a large monolith into microservices. Propose an incremental reliability architecture during migration that preserves availability (including circuit-breakers, bulkheads, fallbacks and throttling), describes deployment strategy (strangler pattern), and explains how to validate each migration step before moving traffic completely.

Sample Answer

Requirements & constraints:- Preserve availability and user experience during incremental migration.- Add resilience primitives: circuit-breakers, bulkheads, fallbacks, throttling.- Use strangler pattern to incrementally extract functionality and shift traffic.- Validate each step with automated tests, canary and progressive traffic shifting, and observability.

High-level approach:1. Strangler gateway: Put an API Gateway / edge proxy that routes requests to either monolith or new microservice by route/rules. This is the single control point for progressive cutover.2. Service mesh + sidecars: Deploy sidecar proxies (Envoy) to provide circuit-breaker, retry, bulkhead (connection pool), timeouts, and throttling uniformly.3. Fallback & adapter layer: Implement graceful fallbacks in the gateway or client SDK—cache responses, degraded feature flags, or read-only fallback to monolith.4. Bulkheads: Use separate thread/work queues, connection pools, and node pools per service to isolate failures.5. Throttling & rate limits: Enforce global and per-caller quotas at the gateway and per-service ingress with token-bucket policies.6. Circuit breakers: Configure circuit thresholds (error %, latency) in sidecars; open to prevent cascading failures and trigger fallback paths.7. Observability & SLOs: Distributed tracing, metrics (latency, error rate), synthetic transactions, dashboards and alerts tied to SLOs.

Deployment strategy (Strangler + progressive):- Phase 0 (Prep): Add gateway & sidecars; instrument monolith; create feature flags.- Phase 1 (Shadow): Deploy microservice, route 0% production traffic but mirror (shadow) live requests to it; compare outputs (contract tests).- Phase 2 (Canary): Route small % (1-5%) to new service using gateway; monitor errors, latency, business metrics.- Phase 3 (Progressive Ramp): Move to 10/25/50% with rollback windows; run blue-green or rolling updates per service.- Phase 4 (Cutover): Switch routing rule to point 100% to microservice; keep fallback to monolith for time-window.- Phase 5 (Decommission): After stable, remove monolith endpoint.

Validation at each step:- Automated contract & schema checks for responses.- Golden comparison: shadow responses compared with monolith; tolerances defined.- Canary metrics: track request success, latency, user-facing KPIs; require pass gates before next ramp.- Chaos & failure injection: simulate downstream failures to validate circuit-breakers, fallbacks and bulkheads behave.- Load & soak tests at canary scale to ensure resource isolation (no noisy neighbor).- Rollback plan and automated health checks that instantly revert gateway routing and trigger alerts.

Trade-offs & rationale:- Sidecars/service mesh centralizes resilience policies—fast consistent control but operational complexity.- Gateway-level fallbacks are simpler but less granular than application-level degradations.- Shadowing ensures functional parity but doubles upstream load—limit with sampling.- Use feature flags and gradual traffic shifts to minimize blast radius.

This incremental architecture preserves availability by isolating failures (bulkheads), preventing cascade (circuit-breakers), protecting capacity (throttling), and keeping user experience acceptable (fallbacks) while enabling safe, observable migration via the strangler pattern.

Unlock Full Question Bank

Get access to hundreds of Reliability, High Availability, and Tradeoffs interview questions and detailed answers.

Join thousands of developers preparing for their dream job.