Reliability Patterns and Fault Tolerance Questions

Applying design patterns and engineering practices that make services resilient to faults. Core patterns include redundancy and replication, circuit breakers, bulkheads, graceful degradation, timeouts and retries with exponential backoff, throttling and rate limiting, bulkhead isolation, and retry idempotency strategies. Also includes measuring and improving reliability through metrics such as mean time to repair and mean time between failures, defining Service Level Indicator and Service Level Objective targets, designing error budgets, and choosing trade offs between consistency and availability. Candidates should explain when to apply each pattern and how patterns interact in distributed systems.

HardSystem Design

64 practiced

Design leader election and failover for a stateful proxy service that must avoid split-brain and provide fast failover. Evaluate choices such as Raft, ZooKeeper, and cloud-managed leader services, and explain trade-offs in latency, complexity, and recoverability.

Sample Answer

Requirements & constraints:- Strong single leader guarantee (no split-brain) for a stateful proxy holding TCP connections and routing state.- Fast failover (<1s preferred) to minimize connection loss, but safe—must not allow dual leaders.- Recoverability: ability to restore leader after partition without data corruption.- Operability: SRE must be able to observe, test, and roll back.

High-level approach:Use a quorum-based consensus system (Raft or ZooKeeper) or a cloud-managed leader service that provides linearizability. Leader election must be tied to leases plus graceful handoff and connection draining.

Options & trade-offs:1) Raft (e.g., embedded etcd/consensus):- Pros: Strong leader lease semantics, easy to reason about linearizability, deterministic failover with a quorum. Good for self-hosted clusters.- Cons: Operational complexity (cluster size, snapshotting), elected leader may change during partitions causing slightly longer election latency. Latency: election time = heartbeat timeout + election timeout (~200-500ms typical) but tunable.- Recoverability: strong—log replication ensures state can be reconstructed.

2) ZooKeeper:- Pros: Mature, ephemeral znodes give clear leader leases and watchers for fast notifications. Proven in many systems.- Cons: JVM ops complexity, requires careful znode/ACL management. Leader election latency similar to Raft; can be fast with tuned session timeouts.- Recoverability: good; but restoring large state requires external storage.

3) Cloud-managed leader services (e.g., Cloud Spanner/Coordination, AWS DynamoDB + leader election pattern, or managed etcd):- Pros: Offloads ops; often low-latency, SLA-backed, sometimes global coordination.- Cons: Potential vendor lock-in, less control over tuning. Latency depends on provider; cross-region elections slower.

Design details for fast safe failover:- Use quorum-based leader election (Raft/ZK/managed). Bind leader role to a short-term lease (e.g., 500–1000ms) with renewals; choose timeouts > average network RTT but small for fast failover.- On leader step-down detection: perform graceful connection handoff—new leader coordinates with previous (if reachable) to drain connections or transfer sticky-state; if impossible, accept some reconnections but preserve routing consistency.- Use in-proxy local persistence of ephemeral session state replicated to followers or a shared backend so new leader can continue routing with minimal interruption.- Implement fencing tokens: leader writes an epoch token to a replicated store; older leaders honor token checks so split-brain cannot accept writes.- Health checks & monitoring: expose metrics for election, lease expiry, connection drain times. Alert on repeated elections or quorum loss.- Testing: chaos experiments (network partitions, slow disks), recovery drills, and playbooks.

Recommendation for SRE:- Prefer Raft (managed etcd) if self-hosting for predictable linearizability and easier reasoning. Use 3 or 5 node odd quorum; tune heartbeat/election for your RTTs.- If you want lower ops, choose cloud-managed coordination with SLA but validate election latency and failure modes.- Always implement fencing tokens + graceful drain + state replication to minimize client impact. Automate failover playbooks, run regular chaos tests, and monitor election rates and lease expirations closely.

MediumSystem Design

61 practiced

A cache layer is returning stale data causing business errors. Design a cache invalidation strategy that balances availability and staleness: include cache TTLs, write-through vs write-behind, pub/sub invalidation, and considerations for partial failures and network partitions.

Sample Answer

Requirements:- Functional: avoid serving stale data that causes business errors (strong freshness for critical writes), keep high read availability and low latency.- Non-functional: tolerate partial failures, network partitions; aim for high availability with bounded staleness; reasonable operational complexity.

High-level approach:- Use hybrid strategy: short conservative TTLs for critical data + longer TTLs for non-critical; write-through for critical paths, write-behind or async invalidation for non-critical; pub/sub for cross-node invalidation.

Architecture & components:1. Cache nodes (distributed, e.g., Redis cluster) fronting primary datastore.2. Coordinator service / cache client library embedded in services to implement policies.3. Pub/Sub (Redis channels / Kafka) to broadcast invalidation events.4. Persistent change-log (optional) for reconciliation.

Data flow & policies:- Reads: check cache; on miss read DB and populate cache with TTL.- Writes (critical): - Write-through: on write, first update DB (or use transaction), then synchronously update cache (same consistent value) and publish invalidation event. This ensures cache reflects DB before acknowledging if required. - Use compare-and-swap or versioning (object version/timestamp) to avoid races.- Writes (non-critical): - Write-behind/async: acknowledge write after DB commit, enqueue cache update; meanwhile publish an invalidation event so readers know to refetch.- Pub/Sub invalidation: - Services subscribe to channels per key-prefix. On receiving invalidation, evict or refresh cache entry. - Use lightweight invalidation messages containing key + version/timestamp.- TTL strategy: - Critical keys: very short TTL (seconds) + jitter; or no TTL combined with explicit invalidation. - Less-critical: longer TTL (minutes–hours). - Add randomized TTL jitter to avoid stampedes.

Handling partial failures & partitions:- If Pub/Sub unavailable: fall back to TTL-based eviction; clients treat cache as potentially stale and can optionally read-through with validation for critical reads (read from DB if cache age > threshold).- If cache write fails after DB commit: write is already persisted; publish invalidation and rely on readers to reload; monitor with alerts and metrics (cache write errors).- During network partition, prefer availability: serve from cache but tag data with version/timestamp; for critical operations require read-after-write using primary DB or strong-consistency read path.- Reconciliation: a background scanner compares DB versions to cache and repairs divergences using change-log or periodic full-sync for hot keys.

Scalability & safety:- Shard channels by key-hash to scale Pub/Sub.- Rate-limit invalidation storms; use coalescing windows (batch invalidations per key).- Monitor metrics: cache hit ratio, invalidation lag, pub/sub latencies, stale-read incidents; set SLOs and alerts.

Trade-offs:- Write-through increases write latency but reduces staleness.- Write-behind improves write latency but risks temporary staleness and data loss if failures occur.- Pub/Sub reduces TTL reliance but adds operational complexity and dependency on messaging layer.

Summary:Use TTLs + pub/sub invalidations as primary mechanism, route critical writes through write-through (or DB-first + sync cache update), non-critical via write-behind with invalidation. Build robust fallbacks for pub/sub/cache failures, use versioning and read validation for critical reads, and monitor heavily to detect stale-data incidents.

HardSystem Design

60 practiced

Design a reconciliation process for eventually-consistent caches that must reconcile millions of keys nightly. Ensure the process is idempotent, rate-limited to avoid overloading origin services, and resilient to worker failures. Describe batching, backoff, checkpointing, and how to prevent thundering-herd during reconciliation.

Sample Answer

Requirements & constraints:- Reconcile millions of keys nightly to bring caches toward eventual consistency- Idempotent, rate-limited (per-origin), resilient to worker failures, avoid thundering herd- Must persist progress and support retries without duplicate costly origin hits

High-level design:- Source of truth lists keys to validate (change-log or snapshot) written to durable storage (S3 / DB) and split into logical shards.- Reconciliation workers consume tasks from a durable work queue (Kafka / SQS). Tasks are small batches of keys (e.g., 100–1000 keys).- Use a coordinator that assigns shard ownership via leases (ZK/etcd/consul or Kafka consumer groups) to make work distributed and avoid overlap.

Batching:- Group keys into batches sized to amortize overhead vs origin cost (start ~500 keys; tune). Each batch contains key ids and the expected cache version/checksum.- Worker validates batch against cache; only keys that differ are enqueued for origin read/update.

Idempotency:- Every reconciliation operation uses idempotency keys: <key-id>:<reconcile-timestamp> or the cache-version checksum. Updates to origin or cache are conditional (compare-and-set) so repeated attempts are safe.- Use PUT with conditional headers or DB optimistic concurrency (version/token) so repeated writes don’t create inconsistent state.

Rate-limiting & throttling:- Per-origin token-bucket rate-limiters embedded in workers (global rate-limiter service or sidecar) to enforce QPS/throughput budgets.- Respect origin SLAs: dynamic quotas adjustable via control plane.- Global circuit breaker: if origin errors exceed threshold, reduce rate or pause reconciliation for that origin.

Exponential backoff & jitter:- On transient failures, use exponential backoff with full jitter for retries per-batch: wait = random_between(0, base * 2^attempt).- Backoff state stored with task metadata so retries across restarts keep the schedule.

Checkpointing & durability:- Use the queue offset (Kafka) or message delete/visibility (SQS) as checkpointing. Only commit offset / delete message after batch processed (idempotent updates applied).- For long-running shards, persist progress (last-processed key index) to a durable store periodically (every N batches) so a new owner resumes near previous point.- Visibility timeouts / lease TTLs ensure in-progress batches reappear if worker dies.

Resilience to worker failures:- Lease-based shard ownership with TTL. When worker fails, lease expires and another worker picks up the shard and resumes from last checkpoint.- Use small batches so rework is bounded.- Health checks + autoscaling to maintain worker pool.

Preventing thundering-herd:- Stagger shard start times and add per-batch jitter when scheduling origin hits.- Use consistent hashing to spread hot keys across shards so no single origin sees concentrated load.- Implement cache-warming window: soft TTL extensions for recently reconciled keys to avoid re-reconciling immediately.- Honor origin-side hints (Retry-After, backpressure headers) and use negative caching for keys known absent.

Observability & control:- Metrics: keys processed/sec, origin QPS, errors, retry counts, lag per shard, checkpoint offsets.- Alerts on elevated origin error rates or reconciliation lag.- Control plane to pause/resume, throttle by origin, adjust batch sizes.

Trade-offs:- Larger batches reduce overhead but increase blast radius on failure and latency for checkpointing.- Using Kafka provides strong durability and consumer group semantics; SQS simpler but requires careful visibility timeout tuning.

This design ensures idempotent, rate-limited reconciliation with durable progress tracking, exponential backoff, worker fault tolerance, and anti-thundering-herd measures.

EasyTechnical

56 practiced

Explain the differences between rate limiting, throttling, and load shedding. For a public REST API, give an example policy that includes short-term rate limiting and longer-term throttling, and explain how clients should be notified.

HardTechnical

71 practiced

Evaluate active-active versus active-passive multi-region deployment models for a payment authorization service. For each model, list failure modes, consistency and latency implications, traffic routing approaches, and operational complexity. Conclude with a recommendation justified by risk and business needs.

Sample Answer

Active-active vs active-passive for a payment authorization service

Active-active- Failure modes: - Partial region/network partition causing split-brain and duplicate authorizations - Cross-region replication lag leading to conflicting state - Global rate limit exhaustion from duplicated retries- Consistency & latency: - Typically eventual consistency for user-account or transaction state unless using synchronized consensus (e.g., Paxos/Raft) which increases latency - Low read latency when routed to nearest region; writes may incur cross-region commit cost or risk stale reads- Traffic routing: - Geo-DNS, Anycast, or global load balancer with health checks and weighted routing; sticky routing for session affinity - Circuit-breakers to redirect traffic away from degraded regions- Operational complexity: - High: distributed consensus or conflict resolution logic, transactional idempotency, coordinated schema changes, complex observability to correlate cross-region traces

Active-passive- Failure modes: - Passive region latency before takeover leads to longer failover windows - Failover automation bugs causing split traffic or prolonged downtime - Data loss if replication lag not bounded- Consistency & latency: - Stronger consistency in active region; passive is read-only or cold - Better write latency for normal ops (single-region writes) - During failover, latency spikes and possibly increased error rates- Traffic routing: - DNS failover, health-check-driven global LB failover, or BGP reroutes; require TTL tuning and automated promotion- Operational complexity: - Moderate: simpler day-to-day but require tested failover playbooks, recovery drills, and replication monitoring

RecommendationFor a payment authorization service where correctness and idempotency are critical and SLA requires low latency: adopt active-passive unless you can afford the engineering and runtime cost to guarantee strong cross-region consistency (e.g., synchronous consensus with sufficient replicas). Active-passive minimizes risk of double-charges and eases regulatory auditability. If business requires <100ms global write latency and zero DR window, invest in active-active with strongly coordinated consensus, rigorous conflict resolution, and exhaustive chaos testing. Prioritize SLOs, error budgets, and runbook automation in either choice.

Unlock Full Question Bank

Get access to hundreds of Reliability Patterns and Fault Tolerance interview questions and detailed answers.

Join thousands of developers preparing for their dream job.