Fault Tolerance and System Resilience Questions

Designing systems to anticipate, tolerate, contain, and recover from component and network failures while minimizing customer impact and preserving correctness. Topics include identifying common failure modes and single points of failure, redundancy and isolation patterns at hardware, service, and geographic levels, and failover strategies including active active and active passive. Cover retry policies with exponential backoff, timeouts, circuit breaker and bulkhead patterns, graceful degradation, rate limiting, and backpressure techniques to protect systems during overload. Discuss orchestration of node rejoin and state rebuild, replication strategies and consistency trade offs, leader election and consensus implications, and techniques to avoid and mitigate split brain. Explain monitoring, health checks, alerting, and metrics such as mean time to recovery and mean time between failures to guide operational improvements. Include testing for resilience through chaos engineering and fault injection, handling flaky components in test environments, analysis of past failures and refactoring for resiliency, and operational practices that reduce blast radius and speed recovery.

EasyTechnical

76 practiced

What does 'blast radius' mean in system reliability? Provide three operational and three architectural changes you would recommend to reduce blast radius in a polyglot microservices environment.

MediumSystem Design

62 practiced

Design monitoring, alerting, and on-call playbook for an active-active global service. Specify which alerts should page an engineer immediately (and why), thresholds tied to SLOs, and dashboards or runbook links that should be visible to the on-call responder.

Sample Answer

Requirements & SLOs:- Global active-active service, 99.95% availability per region, 99.99% global latency SLO (p95 < 200ms), error budget = 0.05% per region per month.- Alerts must balance noise vs actionable paging.

Immediate-page (P1) alerts — page on-call because they indicate user-facing outage or risk to SLO/error budget:- Regional request success rate < 99.9% over 5m (affects availability SLO). Why: immediate user impact and rapid burn of error budget.- Global p95 latency > 300ms for 5m OR sustained p95 > 200ms for 15m during peak (crosses latency SLO). Why: degraded UX and potential cascading failures.- All regional load balancers unhealthy or >30% backend unhealthy (5m). Why: traffic loss and failover risk.- Leader-election or cluster quorum loss in core control plane (immediate). Why: loss of global coordination.- Deployment rollback/failed canary where error rate increases >2x baseline within canary window. Why: automated deployments must be stopped.

Non-paging (alerts to ticket/Slack) — visible but not immediate:- CPU/memory node saturation >80% (30m) unless causing errors.- Single host disk >85% or pod restarts <3/hr aggregated.- Gradual SLO burn: projected to exhaust monthly error budget in >48h (warn) vs <6h (page).

Thresholds tied to SLOs:- Success-rate thresholds set at SLO - 0.05% margin for early warning, and SLO - 0.1% for paging.- Latency: warning at p95 = SLO + 25%, page at SLO + 50% or absolute limit.

Dashboards & Runbook links for on-call:- Top-level “Service Health” dashboard: global/regional success rate, p95/p99 latency, traffic, error budget remaining (link to runbook).- “Infrastructure” dashboard: LB health, instance counts, autoscaler status, region-to-region traffic.- “Deployments & Canary” dashboard: current deploys, canary metrics.- Each alert links to runbook sections: - Triage steps (how to confirm, quick checks) - Mitigation (failover steps, scale-up, rollback command snippets) - Escalation matrix (SRE -> Service Owner -> Platform Eng) - Post-incident checklist & links to logs, traces (Tracing UI), and recent config changes.

Playbook example (for regional success-rate P1):1. Acknowledge, open incident.2. Check Service Health + region LB health + error budget panel.3. If backend unhealthy: drain and replace instances, check autoscaler, rollback recent deploys.4. If traffic surge: engage rate-limiting, scale pools, failover region if needed.5. Notify stakeholders, document actions, and follow postmortem template.

Additional practices:- Auto-remediation for common transient errors (circuit-breaker, autoscale).- Alert fatigue controls: dedupe similar alerts per incident, use dynamic baselining during canaries/maintenance windows.- Regular alert review: monthly triage to adjust thresholds with SLO and traffic changes.

EasyTechnical

63 practiced

Explain MTTR (Mean Time To Recovery) and MTBF (Mean Time Between Failures). For a service with frequent transient errors, describe operational steps and engineering changes you would propose to reduce MTTR and increase MTBF. Include monitoring changes, runbooks, and automation examples.

Sample Answer

MTTR (Mean Time To Recovery) is the average time it takes to restore a service after a failure — detection → diagnosis → remediation → verification. MTBF (Mean Time Between Failures) is the average operational time between consecutive failures (uptime duration). SREs reduce MTTR and increase MTBF by improving detection, diagnosis, automated remediation, and root-cause fixes.

Operational steps and engineering changes:

1) Monitoring & alerts- Add fast, meaningful alerts (SLO-based): alert on symptom (error rate, latency p95) not raw logs.- Use multi-level alerts: paged for service-impacting, quieter for degradations.- Add rich telemetry: request traces, error tags, host/container metrics, deployment metadata.- Deduplicate/alert-suppression for transient spikes to avoid alert fatigue.

2) Runbooks / playbooks- Create concise runbooks per alert: symptoms, quick checks (health endpoints, circuit-breaker status), mitigation commands, rollback steps, post-mortem checklist.- Include runbook run-times and ownership escalation.

3) Automation examples- Automated emergency rollbacks on elevated error-rate + deployment tag:

bash

# example: simple CI/CD rollback trigger
if [ "$(curl -s http://svc/health | jq .error_rate)" -gt 5 ]; then
  curl -X POST -H "Authorization: token" https://ci.example/release/rollback -d '{"service":"svc"}'
fi

- Self-healing: restart failed containers using liveness probes, auto-scaler adjustments, circuit breakers and bulkhead patterns.- Automated mitigation: rate-limiting or feature-flags to turn off non-critical paths during incidents.

4) Reliability engineering to increase MTBF- Fix root causes: prioritize fixes for top-error classes from alert/trace data.- Harden transient areas: implement retries with exponential backoff and idempotency, connection pooling, bulkheads.- Improve testing: chaos testing, canary deployments, integration tests for flaky dependencies.- Capacity and dependency management: circuit breakers to prevent cascading failures; graceful degradation.

5) Post-incident- Blameless postmortems that produce action items mapped to owners and deadlines.- Track MTTR and MTBF as SLO/SLI metrics; iterate on runbooks, automation, and design until targets met.

This combination shortens detection and remediation (lower MTTR) and reduces recurrence by removing root causes and adding isolation (higher MTBF).

MediumSystem Design

74 practiced

Design an active-active multi-region deployment for a stateless HTTP microservice that needs <100ms P95 latency for users worldwide and must survive a single-region outage. Describe load balancing, data locality, health checks, DNS/anycast strategies, and how you avoid consistency pitfalls for session affinity.

HardSystem Design

73 practiced

Design a globally-distributed cache for user session data where eventual consistency is acceptable but stale reads must not cause monetary inconsistencies. Explain strategies for TTLs, versioned values, causal guarantees if needed, and how to invalidate or reconcile stale entries safely.

Sample Answer

Requirements:- Global, highly available read-mostly cache for user session data.- Eventual consistency allowed, but stale reads must never cause monetary inconsistencies (e.g., double-charging, wrong balance).- Low read latency; writes originate from a single authoritative service (payments, billing).- Tolerate regional partitions; minimize cross-region latency.

High-level design:- Multi-region cache tier (local read replicas) backed by a single authoritative datastore per user (sharded master or strongly-consistent DB).- Writes go to authoritative service which updates DB and then asynchronously publishes events to invalidate/update caches (pub/sub or CDC).- Cache stores versioned values and metadata: {value, version, commit_ts, safe_until, idempotency_token}.

Core strategies:1) TTL and safe windows- Short TTLs for monetary fields (e.g., 1–5s) or use per-key safe_until set to commit_ts + safety_margin. Use adaptive TTLs: longer for non-monetary session data.- Serve stale if within safe_until only for non-monetary ops; for monetary-affecting ops require revalidation.

2) Versioned values & optimistic checks- Every session change increments monotonically (version counter or commit timestamp from authoritative DB). Cache returns value+version.- Clients performing monetary operations must include the cached version; authoritative service rejects operations if client-supplied version != current DB version (force read-before-write or return conflict to retry). This prevents double-charge from stale reads.

3) Causal guarantees where needed- For sequences of operations within a user session, maintain session-affinity to a regional write-proxy that ensures causal ordering for that user (per-user sequencer or sticky connections). This avoids reordering anomalies across regions.

4) Invalidation & reconciliation- Synchronously write-through or write-around for critical ops: critical monetary updates bypass cache or update DB then synchronously invalidate/refresh local cache before acknowledging client.- Asynchronous invalidation via ordered pub/sub (Kafka with partition by user_id) to ensure consumers process updates in order.- On cache miss or conflict (version mismatch), clients should fetch authoritative state (read-through) and retry idempotent operations using idempotency tokens.

Operational considerations:- Monitor stale-hit ratio, invalidation lag, conflict rate; alert on increases.- Chaos test partitions and delayed invalidations.- Capacity: partition cache by user_id; expire tombstones for deletes.- Rate-limit retries to avoid thundering herd.

Trade-offs:- Strong protection for money ops increases latency (synchronous validation) but keeps user experience fast for non-critical reads via local caches.- Using version checks and idempotency keeps the system safe without requiring global strong consistency.

Unlock Full Question Bank

Get access to hundreds of Fault Tolerance and System Resilience interview questions and detailed answers.

Join thousands of developers preparing for their dream job.