Fault Tolerance and System Resilience Questions

Designing systems to anticipate, tolerate, contain, and recover from component and network failures while minimizing customer impact and preserving correctness. Topics include identifying common failure modes and single points of failure, redundancy and isolation patterns at hardware, service, and geographic levels, and failover strategies including active active and active passive. Cover retry policies with exponential backoff, timeouts, circuit breaker and bulkhead patterns, graceful degradation, rate limiting, and backpressure techniques to protect systems during overload. Discuss orchestration of node rejoin and state rebuild, replication strategies and consistency trade offs, leader election and consensus implications, and techniques to avoid and mitigate split brain. Explain monitoring, health checks, alerting, and metrics such as mean time to recovery and mean time between failures to guide operational improvements. Include testing for resilience through chaos engineering and fault injection, handling flaky components in test environments, analysis of past failures and refactoring for resiliency, and operational practices that reduce blast radius and speed recovery.

HardTechnical

70 practiced

Evaluate etcd (Raft), Apache ZooKeeper (ZAB), and Consul (Serf/raft) for providing distributed coordination in a highly dynamic microservices environment. Compare latency, consistency guarantees, failure modes, operational complexity, and recommend which is best when service churn is high.

Sample Answer

Situation: You need a recommended coordination stack for a highly dynamic microservices environment where services frequently join/leave, require discovery, leader election, and occasional distributed locking/config.

Short answer / recommendation:- Use a hybrid: Consul (Serf + Raft) for membership and fast discovery; etcd (Raft) for strong-consistency primitives (critical locks, canonical config). If a single choice is required and churn is very high, choose Consul because its gossip-based membership gracefully handles churn while still providing Raft-backed strong primitives when needed.

Comparison

Latency- etcd (Raft): Low write latency for small clusters, but every strong-op goes through the leader — leader is a bottleneck; reads can be served locally if stale reads allowed.- ZooKeeper (ZAB): Similar leader-commit path; very low tail-latency for small clusters but also leader-bound.- Consul (Serf + Raft): Membership/events via Serf gossip are very low latency and scale with churn; KV/locks use Raft (higher latency) but only for operations that need strong consistency.

Consistency guarantees- etcd (Raft): Linearizability for reads/writes (if configured), strong consistency suitable for locks and critical config.- ZooKeeper (ZAB): Provides sequential consistency and linearizable writes w.r.t. the leader; good for coordination patterns (barriers, locks).- Consul: Dual model — gossip for eventually-consistent membership/catalog; Raft-backed KV for linearizable operations (opt-in).

Failure modes- etcd: Leader election pauses commits; long GC or disk issues can stall cluster; requires careful quorum sizing to avoid availability loss.- ZooKeeper: Leader election can be heavy on churn; client session handling (ephemeral nodes) can cascade under bursts of disconnects.- Consul: Gossip partitions can lead to inconsistent membership views (eventual reconciliation); Raft quorum loss affects KV/locks; but membership instability is contained to Serf subsystem.

Operational complexity- etcd: Operationally straightforward for small clusters; requires snapshotting, compaction tuning, monitoring (disk I/O, leader churn).- ZooKeeper: Mature tooling but more operational overhead (tick settings, JVM tuning, careful rolling upgrades), harder with very frequent ephemeral node churn.- Consul: More components (agent, Serf, server/leader) but designed for service discovery at scale; easier to tolerate churn; ACLs, WAN federation add ops work.

Why hybrid or Consul for high churn- High churn favors gossip-based membership (low per-change cost) and separate strong-path for critical state. Consul gives that split out of the box. etcd and ZooKeeper both provide strong consistency but are more sensitive to many ephemeral session changes (leader elections, client churn).

Practical recommendation (Solutions Architect view)- Deploy Consul cluster for service discovery and health; tune Serf settings (merge/rate-limits) to handle churn and avoid flooding.- Use a small etcd cluster (3-5 nodes) for global canonical config, leases and critical locks; expose an API layer to minimize direct client connections and limit churn impact on the etcd cluster.- Monitor: leader changes, commit latencies, Serf retransmits, Raft election counts, and ephemeral session churn. Automate graceful deregistration for services to reduce noise.

Trade-offs- Single-system simplicity vs resilience to churn. Consul-first reduces operational outages from membership storms; etcd-first gives strictly simpler consistency semantics. Hybrid gives best of both at modest added complexity.

EasyTechnical

68 practiced

Explain eventual consistency versus strong consistency. Provide one real-world example where eventual consistency is acceptable and one where it is not, and explain how a Solutions Architect should present these trade-offs to a customer in terms of user experience and operational cost.

EasyTechnical

71 practiced

What is the circuit breaker pattern? As a solutions architect, outline how you would integrate circuit breakers at the API gateway, service client libraries, and downstream services. Include configuration considerations (failure thresholds, reset windows, half-open behavior) and how to combine this with alerting.

MediumSystem Design

125 practiced

Design a monitoring and health-checking scheme for a microservice mesh. Specify liveness and readiness probes, synthetic transactions, key SLIs to track (e.g., p95 latency, error rate, saturation), alerting thresholds tied to SLOs, and how to avoid alert fatigue while keeping meaningful alerts.

Sample Answer

Requirements & goals:- Detect service death, degraded performance, and external user impact quickly.- Differentiate deployment/startup vs runtime readiness.- Tie alerts to SLOs and reduce noise.

Liveness & Readiness:- Liveness probe: simple TCP/connect or HTTP HEAD /health/live returning 200 if process and essential deps (process, threadpool) alive. Frequency: 10s, timeout 1s, failure threshold 3.- Readiness probe: deeper HTTP GET /health/ready that checks upstream dependency connectivity (DB connection pool, message broker reachability, config loaded). Frequency: 5s, timeout 1s, failure threshold 2. Readiness failures remove instance from load balancer for graceful draining.

Synthetic transactions:- Global and regional synthetic checks from external probes (every 30–60s) that perform representative flows: auth, a critical read, and a critical write. Measure end-to-end latency and correctness. Correlate with real traffic traces.

Key SLIs to track:- Latency: p50, p95, p99 for user-facing API endpoints (histograms).- Error rate: HTTP 5xx and domain errors per minute (rate).- Availability: successful requests / total requests (rolling 30d).- Saturation: CPU, memory, threadpool queue length, connection pool utilization.- Queue depth and consumer lag for async jobs.

Alerting & thresholds tied to SLOs:- Define SLOs (example): 99.9% availability monthly, p95 latency < 300ms.- Alert tiers: - P0 (urgent): Availability < SLO by >1% over 1h OR error rate > 5% absolute for 5m. - P1: p95 latency > 1.5x target for 10m, or synthetic transaction failures across region for 5m. - P2: resource saturation > 85% for 10m, readiness probe flapping on 2+ instances.- Use burn-rate alerts: if error budget is consumed at >4x rate, page on-call.

Avoiding alert fatigue:- Alert on symptoms that require action, not every metric: prefer composite alerts (error+latency) and anomaly detection (baseline + deviation).- Use rate-limited, grouped notifications and escalation policies. Suppress alerts during deploy windows (auto-silence) unless critical.- Implement automatic deduplication/correlation (same root cause) using traces and tags (deployment id).- Provide actionable runbooks linked in alert with immediate next steps and owner.- Tune thresholds iteratively using historical data; convert noisy alerts to dashboard-only or lower-priority tickets.

Observability & tooling:- Distributed tracing (trace IDs in logs), metrics (Prometheus/Grafana or managed), alertmanager for routing, RUM + synthetic for UX.- Correlate traces, logs, and metrics in incident timelines to reduce MTTI/MTTR.

This scheme balances fast detection, operational actionability, SLO alignment, and noise reduction.

MediumTechnical

75 practiced

How would you implement global per-customer API rate limiting across distributed gateways without a single coordination point? Discuss token-bucket approximations, local caches, periodic synchronization, and fallback behaviors when quotas are temporarily inconsistent.

Sample Answer

Requirements & constraints:- Enforce a global per-customer RPS/quota across many gateways- No single coordination point (must remain available with network partitions)- Accept some approximation/latency in enforcement but bound error- Provide predictable fallback behavior

High-level approach1. Local token-buckets on each gateway (fast, local decision). Each gateway maintains for each customer a local bucket with capacity C_local and refill rate r_local derived from global rate R_global and an expected sync/coordination interval T_sync: - r_local = R_global * (capacity share) - C_local = R_global * T_sync * safety_factor (safety_factor < 1 to reduce over-issue)

2. Capacity distribution (decentralized): each gateway computes its capacity share deterministically (e.g., consistent hashing of customer → set of responsible gateways, or static config of gateway weight) so we avoid a single leader. For example, hash(customer_id) mod N to assign primary replica set of size k; only those k gateways are allowed to issue tokens immediately, others are conservative.

3. Periodic synchronization (eventual consistency): - Gateways periodically exchange compact deltas about tokens consumed for each customer via a pub/sub (Kafka) or gossip: "consumed X tokens for customer Y in last window." - Use CRDT counters (PN-Counters) or sum-only deltas to merge without coordination. - Or use a distributed datastore (sharded Redis) for reconciliation if acceptable.

4. Token-bucket approximation and reconciliation: - Gateways make local decisions using their bucket. On sync, if global usage > R_global * elapsed_time, gateways apply corrective measures: reduce C_local, temporarily reject requests, or request tokens from the replica set.

5. Fallback behaviors when quotas inconsistent: - Conservative mode (preferred for strict quotas): if a gateway is isolated or sync shows overspend, deny excess requests with HTTP 429, logging for reconciliation. - Permissive mode (for availability): allow requests but mark them as "speculative" and meter them. On reconciliation, if overspend occurred, throttle future requests or bill adjustments. - Borrow-and-repay: gateways may borrow a bounded number of tokens from peer replicas (using gossip to record the borrow). Borrowing limit = borrow_factor * C_local to bound risk.

Operational details & trade-offs- Choose T_sync to trade accuracy vs message volume. Short T_sync → tighter enforcement, more bandwidth. Typical T_sync = 1s–10s; safety_factor ~0.6–0.8.- Use efficient data encoding for deltas and retention windows to limit memory.- Monitor overspend rate, 429 rate, reconciliation drift; alert when drift grows.

Example numbers- R_global = 1000 rps, N=10 gateways, T_sync=5s, safety_factor=0.7 - C_local ≈ 1000 * 5 / 10 * 0.7 = 350 tokens per gateway between syncs - r_local refills at 100 rps per gateway

Why this works- Local token-buckets give low-latency decisions and availability under partition.- Deterministic assignment and bounded local capacity cap global over-issue.- Periodic aggregation reconciles state without a single leader; CRDTs/gossip handle merges.- Explicit fallback modes let product teams choose stricter or more available behavior depending on SLA.

Monitoring & testing- Simulate partition and burst scenarios; measure global drift and 429s.- Tune safety_factor, T_sync, replica set size k, and borrow limits.- Add audit logs and metrics per-customer for billing and troubleshooting.

RecommendationStart with deterministic replica assignment + local buckets + conservative safety_factor and short T_sync; iterate based on observed overspend and operational cost.

Unlock Full Question Bank

Get access to hundreds of Fault Tolerance and System Resilience interview questions and detailed answers.

Join thousands of developers preparing for their dream job.