Fault Tolerance and System Resilience Questions

Designing systems to anticipate, tolerate, contain, and recover from component and network failures while minimizing customer impact and preserving correctness. Topics include identifying common failure modes and single points of failure, redundancy and isolation patterns at hardware, service, and geographic levels, and failover strategies including active active and active passive. Cover retry policies with exponential backoff, timeouts, circuit breaker and bulkhead patterns, graceful degradation, rate limiting, and backpressure techniques to protect systems during overload. Discuss orchestration of node rejoin and state rebuild, replication strategies and consistency trade offs, leader election and consensus implications, and techniques to avoid and mitigate split brain. Explain monitoring, health checks, alerting, and metrics such as mean time to recovery and mean time between failures to guide operational improvements. Include testing for resilience through chaos engineering and fault injection, handling flaky components in test environments, analysis of past failures and refactoring for resiliency, and operational practices that reduce blast radius and speed recovery.

MediumTechnical

68 practiced

You're designing failover between two data centers. Explain strategies to avoid split-brain across the clusters: quorum-based failover, fencing tokens, leader leases, and use of external consensus services. For each approach, describe pros and cons and operational implications.

Sample Answer

Quorum-based failover- What: Require a majority of nodes (or data centers) to agree before accepting writes or electing a leader.- Pros: Simple, mathematically prevents two partitions from both thinking they’re primary; well-understood (Paxos/Raft rely on this).- Cons: Availability can drop if quorum lost (network partition or multiple failures). Requires careful cluster size planning.- Operational implications: Monitor quorum health, automate alerts and safe shutdown when quorum lost, design placement of nodes across AZs/DCs to minimize correlated failures.

Fencing tokens (STONITH / boot/fence)- What: Before allowing a takeover, the new leader issues a fence token or physically isolates the old node (power-cycle, network block) so it cannot serve.- Pros: Strong protection against split-brain even when stale nodes remain reachable; supports heterogeneous systems.- Cons: Requires reliable fencing mechanism (power switches, managed BMCs, network ACLs); can be disruptive if fencing fails or is misconfigured.- Operational implications: Maintain inventory of fence targets, test fencing regularly, integrate with orchestration and runbooks; ensure fencing has strict RBAC.

Leader leases- What: Leaders obtain time-limited leases; a follower will not accept a new leader until lease expires or is revoked.- Pros: Low coordination cost, reduces chance of concurrent leaders if clocks/leases well-managed; good for systems with short failover windows.- Cons: Sensitive to clock skew and varying latencies; incorrect lease duration causes either unsafe overlap (too long) or unnecessary downtime (too short).- Operational implications: Use NTP/chrony with monitoring, choose conservative lease durations, implement lease renewal and reliable expiry handling.

External consensus services (ZooKeeper/etcd/Consul)- What: Use a dedicated, replicated consensus service to store locks/leader info and perform membership/quorum checks.- Pros: Battle-tested primitives (leader election, leases, locks), decouples logic from application; simplifies multi-dc coordination.- Cons: Adds dependency and operational complexity; consensus service itself must be highly available and correctly configured for cross-dc use.- Operational implications: Deploy the service with proper replication and locality rules, back up state, monitor latency and quorum, avoid exposing single-region clusters across high-latency links without tuning.

Trade-offs summary:- Quorum and external consensus prioritize correctness but can reduce availability.- Fencing gives strong safety but requires reliable side-channel control.- Leases are lightweight but demand precise time/latency assumptions.Choose a combination: e.g., Raft for intra-DC, fencing + external consensus for cross-DC failover, and robust monitoring/runbooks to handle edge cases.

EasyTechnical

76 practiced

Explain idempotency in distributed systems and why it's important for retries. Describe practical server-side strategies for making write operations idempotent in a REST API: idempotency keys, unique request IDs, storing operation results with TTL, and handling non-idempotent operations.

Sample Answer

Idempotency means performing the same operation multiple times has the same effect as performing it once. In distributed systems, network failures and client retries make idempotency crucial: without it, retries can cause duplicate side effects (double charges, duplicate orders).

Why important for retries:- Makes retry safe: clients can retry on timeouts without fear of creating duplicates.- Simplifies client logic and improves reliability across unreliable networks.

Practical server-side strategies for REST write operations:

1. Idempotency keys- Client generates a unique key (UUID) per logical operation and sends it in a header (e.g., Idempotency-Key).- Server checks a persistent store for that key: if present, return the stored result; if absent, perform the operation, store outcome + response, then return it.- Use strong consistency for the key check to avoid race conditions.

2. Unique request IDs- Similar to idempotency keys but often generated by client libraries or API gateways. Useful for tracing and deduplication across services.- Server logs and de-duplicates based on the ID.

3. Storing operation results with TTL- Persist the operation outcome and response metadata keyed by idempotency key/request ID.- Attach a TTL to limit storage size and balance correctness window vs. storage cost (e.g., keep for 24–72 hours).- Include status (in-progress, success, failure) to handle concurrent requests and resume semantics.

4. Handling non-idempotent operations- Design APIs so writes are idempotent when possible (e.g., PUT versus POST).- For inherently non-idempotent actions (refunds, external charge), use compensating actions, locks, or require client-provided idempotency keys and server-side dedupe logic.- For long-running operations, return an operation resource (202 Accepted) with an operation ID the client can poll; make the operation resource idempotent.

Trade-offs and best practices:- Ensure atomic check-and-create for idempotency-store (use transactions or compare-and-set).- Be explicit in API docs about idempotency semantics and key lifetime.- Log and monitor duplicate attempts to detect client or network issues.

EasyTechnical

56 practiced

Describe the difference between liveness and readiness probes in Kubernetes. For a stateful backend service, provide example probe endpoints, their semantics, and explain how these probes affect rolling updates, autoscaling decisions, and pod failover behavior.

Sample Answer

Liveness vs Readiness (short):- Liveness probe: "Is the container alive or dead?" If it fails, kubelet restarts the container. Use to detect deadlocks, unrecoverable states.- Readiness probe: "Is the pod ready to accept traffic?" If it fails, kube-controller removes the pod from Service endpoints (no restarts). Use for warm-up, dependency checks.

Example endpoints and semantics:- Liveness: GET /healthz/live → returns 200 only when process/event loop is responsive. Fast check; avoid expensive I/O. Semantics: if non-200 for configured failureThreshold → container restart.- Readiness: GET /healthz/ready → returns 200 when app finished initialization, has DB connection, caches warmed. Semantics: non-200 removes pod from load balancer; when 200 again it’s re-added.

How they affect rolling updates:- Readiness prevents new pods from receiving traffic until ready; during a rolling update, kube-rollout waits for new pod to become Ready (based on readinessProbe) before proceeding, ensuring no traffic is sent to not-yet-initialized pods.- Liveness can cause a failing new pod to be restarted during rollout; excessive liveness failures may cause rollout to retry/fail.

Autoscaling decisions:- HPA typically uses metrics (CPU, custom metrics) from Ready pods; readiness ensures only eligible pods receive traffic and contribute correct metrics. If many pods are NotReady, adaptive scaling may be triggered incorrectly if metrics aren't filtered—ensure metrics exporter only emits when Ready or use custom metrics.

Pod failover behavior:- Readiness controls traffic failover: when a pod becomes NotReady, endpoints removed immediately so traffic shifts to other Ready pods. Liveness causes restarts; kubelet restarts unhealthy containers which can speed recovery but may cause brief unavailability if readiness remains false until recovery completes.

Best practices:- Keep liveness simple and strict for process health.- Make readiness include external dependency checks and warm-up.- Use startupProbe for long initializations to avoid premature liveness restarts.- Tune initialDelay, periodSeconds, failureThreshold to avoid flapping.

MediumSystem Design

74 practiced

Design an active-active multi-region architecture for serving a read-heavy product catalog with eventual consistency. Specify replication approach, conflict resolution, caching strategy, read routing, and how you would keep user-facing latency low while handling writes across regions.

Sample Answer

Requirements & constraints:- Read-heavy catalog, global low read latency, eventual consistency acceptable for inventory/metadata.- Active-active: accept writes in any region.

High-level architecture:- Each region: app servers → regional cache (Redis/Memcached) → regional DB shard.- Global replication layer: async change-stream (CDC) / log-based replication (e.g., Kafka MirrorMaker or DB native logical replication) to propagate updates between regions.

Replication approach:- Asynchronous, append-only change log per region. Each write is assigned a monotonically increasing region sequence + logical timestamp (hybrid logical clock).- Replicate log to other regions; apply updates in causal order where possible.

Conflict resolution:- Use a hybrid strategy: - For most fields (title, description, price): last-write-wins using logical timestamp with region bias and client-provided version for strong updates. - For mergeable fields (tags, attributes): use CRDT sets/maps to merge without loss. - For inventory/stock: use domain-specific approach (e.g., central inventory service or per-region stock and periodic reconciliation) to avoid overselling.- Maintain version vector metadata per item to detect concurrent writes and surface conflicts to business logic if automatic resolution is unsafe.

Caching & invalidation:- Edge CDN for static assets (images).- Regional in-memory caches for catalog reads with short TTLs for freshness and write-through/invalidate-on-write: - On local write: update DB, push change to local cache, publish change to replication log and to a cache invalidation channel (pub/sub) so other regions invalidate/refresh cached entries. - Use stale-while-revalidate to serve reads while refreshing in background.

Read routing & latency:- Use geo-DNS / regional ingress to route reads to nearest region.- Reads served from regional cache; fall back to regional DB if miss.- Warm hot items proactively (pre-warming/popular lists) and use adaptive TTLs per item popularity.

Writes across regions & keeping latency low:- Accept writes locally (fast local commit). Synchronously ack to client once local DB + local cache updated.- Replicate asynchronously to other regions (no cross-region sync on write path).- For operations requiring stronger consistency (price-critical or admin updates), provide an option to use synchronous global coordination (leader election or consensus) or require writes to go to a designated region.- To reduce anomalies: include causal metadata in reads so clients can request “read-your-writes” by pinning to region or passing last-seen version; session stickiness for users who recently wrote.

Operational concerns & trade-offs:- Pros: low read latency, high availability, regional independence.- Cons: eventual consistency anomalies, complex conflict handling (mitigated via CRDTs and version vectors).- Monitor convergence lag, conflict rates, and set alerting for divergent items; provide manual reconciliation tools and audit logs.

This design balances low-latency global reads with practical eventual consistency for a read-heavy product catalog while keeping writes fast and conflicts manageable.

MediumTechnical

58 practiced

Design a runbook automation for emergency rollback that must execute within 10 minutes of detecting severe degradation. Which steps should be fully automated (traffic cutover, feature-flag toggle), which require human approval (DB schema changes), and how will you test the runbook without risking production stability?

Sample Answer

Situation: We must be able to rollback an incident causing severe degradation within 10 minutes of detection while protecting data integrity and stability.

Task: Design a runbook automation that maximizes speed for safe operations, defines which actions are automated vs human-approved, and includes a testing plan that doesn't risk production.

Action:- Detection & decision: - Automatic: automated alerting from SLO/monitoring triggers the runbook and starts an orchestration pipeline (timestamped runbook execution). - Human: require on-call engineer acknowledgement within 2 minutes to prevent false positives; if no ack, automated escalation follows.

- Fully automated steps (safe, reversible, stateless): - Traffic cutover: automated global/local load-balancer routine to divert traffic to previous stable version or a known-good cluster (Canary rollback -> 100% within pre-defined steps). - Feature-flag toggle: automated toggle of feature flags to disable risky features (stored in config service) with immediate verification. - Service restart or replica scaling: automated where no DB schema dependency exists. - Health checks: continuous automated probes that gate each step; rollback aborts if health doesn't improve.

- Require human approval (high-risk, irreversible): - DB schema changes or migrations: human approval and runbook pause; include pre-checks, backup snapshot, and migration plan. If urgent, trigger a war-room with DB SME and exec approval. - Data-backfill operations, irreversible state changes.

- Safety and observability: - Pre- and post-step verification (synthetic transactions, error rate, latency). - Immutable audit trail, automated notifications to stakeholders and incident channel. - Rate-limited automated actions with fail-open/closed policies depending on service criticality.

Testing the runbook without risking production:- Staging and pre-prod: run full automation against staging with production-like data and chaos-injection to validate timing and health gates.- Canary rehearsals in production-safe slices: schedule low-risk windows where automation runs on a small percentage of traffic with immediate manual abort capability.- Dry-run mode: orchestration simulates changes, logs steps and outputs without applying them.- Game days / tabletop drills: run the playbook regularly with engineers and on-call staff to validate human approval flows and communications.- Automated smoke tests and rollback drills that flip feature flags and perform cutover in a sandbox, measuring end-to-end timing to ensure sub-10-minute execution.

Result: This design balances speed and safety—automating fast, reversible actions (traffic cutover, feature flags) while gating high-risk, irreversible operations (DB schema) with human judgment. Regular testing (dry-run, staging, canary, game days) ensures the runbook executes within the 10-minute target without endangering production.

Unlock Full Question Bank

Get access to hundreds of Fault Tolerance and System Resilience interview questions and detailed answers.

Join thousands of developers preparing for their dream job.