Fault Tolerance and Failure Scenarios Questions

Designing systems resilient to component failures: timeouts, retries with exponential backoff, circuit breakers, bulkheads. Discuss cascading failure prevention and graceful degradation. At Staff level, demonstrate thinking about multi-layer failures (service failures, database failures, network partitions) and how to detect and recover from them.

EasyTechnical

90 practiced

Define the bulkhead isolation pattern in distributed systems. For an e-commerce microservices platform, explain process-level, thread-pool, and network-level bulkheads, how to size isolation boundaries conservatively, and trade-offs between containment and resource utilization.

Sample Answer

Bulkhead isolation is a resilience pattern that partitions resources so failures or overloads in one component don’t cascade across the system—like watertight compartments on a ship. For an e-commerce microservices platform it reduces blast radius during spikes, failures, or DoS.

Process-level bulkheads:- Run critical services (checkout, payments) in separate processes/containers or dedicated VMs. Example: payment service on its own Kubernetes Deployment with resource limits and Pod disruption budgets so a crash or OOM there won’t take down catalog or search.

Thread-pool bulkheads:- Limit concurrent work per component using dedicated thread pools or executor queues. Example: the inventory service uses a small, bounded thread pool for external supplier calls; when suppliers slow, only that pool backs up and threads queue, preserving other functionality.

Network-level bulkheads:- Partition network capacity and circuit-breaker rules per downstream dependency or tenant. Example: rate-limit external recommendation API calls and use separate egress routes or throttling policies so heavy traffic to recommendations doesn’t saturate egress or upstream proxies.

Sizing isolation boundaries conservatively:- Start with small, clearly separated quotas based on SLAs (e.g., reserve X% CPU/memory for checkout), use historical traffic percentiles (P95/P99) plus safety margin, and tune with gradual load tests. Prefer conservative defaults: reserve enough capacity to meet critical-path P95 latency under degraded conditions, then relax with monitoring.

Trade-offs:- Containment improves availability and predictability but can reduce overall utilization (idle reserved capacity). Finer-grained isolation reduces blast radius but increases operational complexity (more configs, tuning). Conservative sizing prevents outages but raises cost; aggressive bin-packing saves cost but risks cross-service interference. Mitigate by autoscaling isolated units, adaptive quotas, and robust observability so you can adjust isolation vs utilization dynamically.

MediumSystem Design

93 practiced

For an order processing system with high-volume payment and inventory services, propose bulkhead strategies at container, thread-pool, and network levels. Explain how to allocate resources between payment and inventory to avoid one noisy service starving the other and how to handle per-tenant vs per-service isolation.

Sample Answer

Requirements & constraints:- High throughput order processing with independent payment and inventory paths- Prevent cascading failure or noisy neighbor causing starvation- Multi-tenant support (some customers require stronger isolation)- Low added latency, operationally observable

High-level approach:- Apply bulkheads at three layers: container/runtime, thread-pool, and network (connection/queue) with capacity controls and fallbacks.

Container-level bulkheads:- Deploy payment and inventory services in separate Kubernetes Deployments/Namespaces; use resource requests/limits and Pod Disruption Budgets.- Use Vertical Pod Autoscaler + Horizontal Pod Autoscaler with custom metrics (e.g., payment TPS, inventory lock latency).- For noisy neighbor protection, use node affinity/taints + PodTopologySpread to avoid co-locating critical replicas; optional dedicated node pools for high-SLA tenants or payment service.

Thread-pool / runtime bulkheads:- Inside each service, implement bounded executor pools per upstream type (e.g., order-processing → payment pool, → inventory pool). Configure: - Fixed-size worker pools with queue size limits. - Rejection policy: fail-fast or enqueue to durable queue depending on criticality. - Circuit breakers and per-method timeouts to prevent thread starvation.- Allocate pool sizes by capacity planning: reserve >= baseline RPS headroom for payment (e.g., 40%) and inventory (40%), keep 20% shared or for bursts. Tune with load tests.

Network-level bulkheads:- Use service mesh (Istio/Linkerd) or API gateway rate limits and connection pools: - Per-route connection limits, concurrent request caps. - Per-service and per-tenant rate limiting using token bucket. - Backpressure via 503 with Retry-After or enqueue to Kafka for eventual processing.

Resource allocation strategy:- Prioritize based on business importance: payments often stricter (retries, user-facing), inventory eventual consistency tolerable.- Start with static split (e.g., payment 60% CPU/memory, inventory 30%, shared 10%), then move to dynamic using admission controllers/HPA with SLO-driven autoscaling.- Implement weighted queuing at gateway: high-priority payment requests admitted ahead of inventory during contention.

Per-tenant vs per-service isolation:- Per-service isolation: default—separate deployments/pools per service.- Per-tenant isolation: for high-value tenants provide dedicated namespace, node-pool, or even dedicated instances; for multi-tenant lower-tier use logical isolation: per-tenant rate limits, per-tenant thread-pool quotas, and tenant-aware prioritization.- Hybrid: tenant-aware resource controller that enforces quotas (CPU, concurrent requests) and escalates noisy tenants to throttling or dedicated resources.

Observability & operational controls:- Metrics: per-service and per-tenant throughput, latency, thread-pool saturation, queue depth, error rates, circuit state.- Alerts on saturation thresholds; auto-scale policies tied to SLO breaches.- Chaos experiments to validate bulkheads and fallbacks.

Trade-offs:- Strong isolation increases cost and complexity; prefer logical isolation and escalation paths.- Queuing can increase latency; use async processing for non-blocking inventory adjustments.

This design ensures one noisy service or tenant cannot starve others, provides clear escalation to dedicated resources, and keeps operational visibility for tuning.

HardTechnical

84 practiced

Design a strategy to guarantee safe retries across multiple services without a global transaction manager. Include patterns such as SAGA (choreography vs orchestration), compensating transactions, idempotency tokens, and how to observe and reconcile partial commits. Discuss when each option is appropriate.

Sample Answer

Requirements & constraints:- No global transaction manager, multi-service consistency needed, retries must be safe, low-latency where possible, observability and eventual correctness acceptable.

Strategy overview:1. Prefer a Saga-based approach (local transactions + compensations) combined with idempotency and an out-of-band reconciliation process.

Sagas: choreography vs orchestration- Choreography: Each service emits domain events and reacts. Lightweight, low coupling, good for simple linear flows (order -> payment -> inventory). Use when teams own bounded contexts and failure handling logic is simple.- Orchestration: A central coordinator (workflow engine) issues commands and records state. Better for complex flows, conditional branches, long-running transactions, retries, and clear audit trail.

Compensating transactions- Implement explicit compensating actions for each forward step (e.g., refund payment, restock inventory).- Compensations must be idempotent and designed to run even if forward action partially failed.

Idempotency tokens & deduplication- Every external command includes a client-generated idempotency key persisted by the receiving service (with TTL). Use at-service-level to make retries safe and to detect duplicate requests.- Store outcome (success/failure/in-progress) and operation metadata.

Observability & reconciliation- Emit structured events and persist saga state (if orchestrator) or produce audit logs (if choreographed) to a durable stream (Kafka).- Reconciliation process: periodic consumer scans for inconsistent states (e.g., reserved inventory but no payment) and triggers compensations or alerts.- Implement dead-lettering for failed compensations and human-in-loop workflows.

Implementation patterns & tooling- Use a workflow engine (Temporal, Cadence, or Step Functions) for orchestration to manage retries, timeouts, and persistence.- For choreography, leverage event bus + durable outbox pattern to guarantee event delivery.- Use circuit-breakers, exponential backoff, and backoff-with-jitter for retries.

When to choose:- Small/simple flows, many autonomous teams: choreography + outbox + reconciliation.- Complex flows, many branches, strong observability & retry semantics: orchestration with workflow engine.- Regulatory/auditing needs: orchestration (single source of truth for saga state).

Key operational controls- Metrics: sagas started/completed/compensated, compensation failures, stale sagas.- Alerts on reconciliation hits and DLQ size.- Runbook for manual compensation.

Example sketch (orchestrator): orchestrator issues ReserveInventory(idempotencyKey), receives success, then ChargePayment(...). On payment failure, orchestrator calls CancelReserveInventory(compensation). All service APIs check idempotencyKey and persist operation result; events are written via outbox.

Trade-offs- Choreography: less central control, harder to reason about global state; easier to scale and decouple.- Orchestration: simpler reasoning and retries, central dependency and operational complexity.

This combination (sagas + idempotency + durable events + reconciliation) guarantees safe retries and eventual consistency without a global transaction manager.

MediumSystem Design

74 practiced

Design a retry policy for a synchronous API that calls a flaky downstream service at 500 requests per second. The downstream has 99.95% availability and occasional short failures; your policy should minimize tail latency and avoid load amplification. Specify max attempts, backoff formula, jitter, and rules for when to short-circuit or drop requests. Include monitoring signals and adaptive behavior.

Sample Answer

Requirements & constraints:- Client-facing synchronous API at 500 RPS to a flaky downstream (99.95% availability). Minimize tail latency and avoid amplifying load on downstream.

Policy summary (high-level):- Max attempts: 1 retry (total attempts = 2) for idempotent requests only. For non‑idempotent requests, zero retries.- Backoff: single fixed short delay = 50ms + uniform jitter in [-25ms, +25ms] (i.e., 25–75ms). Rationale: one quick retry recovers transient spikes while keeping tail latency bounded.- Retry budget & amplification control: global retry token-bucket sized to allow at most 10% additional RPS (50 retry tokens/s). Each retry consumes a token; if none available, return failure immediately.- Circuit breaker: per-downstream & per-host CB that opens if error rate > 5% over a 1-minute sliding window AND at least 50 errors observed. Open period = 30s, then half‑open with a controlled probe (1 request/second). While open, short-circuit to fail-fast with 503 to clients (or use cached fallback).- Request eligibility rules: - Only idempotent methods (GET, PUT for idempotent semantics) or requests marked safe by caller. - Do not retry on 4xx client errors (except 429 where a limited retry may be considered with backoff). - Retry on network timeouts, connection resets, 5xx and 429 (respect rules above).- Drop/short-circuit: - If circuit breaker open OR retry token unavailable OR queue backlog above threshold (e.g., >100 pending), respond fast with 503/429 and include Retry-After header when appropriate.

Adaptive behavior & health-aware tuning:- If downstream healthy (error rate < 0.1% and p99 latency < baseline+20%), temporarily allow a second retry for severe errors and expand retry token-bucket to 15% for short period.- If downstream shows increased latency/p99 spike or increasing error rate, reduce retry budget to 0% and shorten CB probe interval to stop amplification.

Monitoring signals and alerts:- Metrics: incoming RPS, downstream requests RPS, retries RPS, retry-token utilization, error rate (1m/5m), latency p50/p95/p99 for first attempt and end-to-end, pending request queue length, circuit-breaker state.- Alerts: - Error rate > 1% sustained for 5m (page ops). - Retry-token utilization > 80% for 2m (investigate amplification risk). - p99 latency > SLO threshold (e.g., 500ms) for 5m. - Circuit breaker flapping.- Dashboards: split metrics by client, endpoint, and downstream host for root-cause.

Operational notes & trade-offs:- Why 1 retry? Keeps tail latency low and limits amplification; because availability is high (99.95%), most failures are transient and recoverable with one attempt.- Fixed short backoff + jitter simplifies timing and reduces synchronized retry storms.- Retry token-bucket prevents retries from pushing traffic > safe margin.- Circuit breaker prevents continued load during downstream outages and gives fast-failure semantics to clients.- For higher durability needs, implement async retry/queued background processing or client-side retry guidance rather than increasing server-side synchronous retries.

Example configuration (starter):- max_attempts = 2 (initial + 1 retry)- retry_delay = 50ms, jitter = uniform ±25ms- retry_budget = 50 tokens/sec- cb_error_threshold = 5% over 60s & min_errors = 50- cb_open_duration = 30s, half_open_probe = 1/s

This policy balances availability and user latency while preventing load amplification; tune thresholds based on observed traffic and downstream behavior.

HardTechnical

67 practiced

A prospective client wants 99.999% availability for an API. As a Solutions Architect, prepare an executive-level explanation covering the architectural changes required (multi-region active-active, redundancy, failover), operational requirements (runbooks, on-call staffing, drills), expected cost uplift, and residual risks that cannot be fully eliminated.

Sample Answer

Situation: The client requires 99.999% (five-nines) availability for an API—equates to ~5.25 minutes downtime/year. Achieving this requires both architecture and ops changes plus acceptance of residual, low-probability risks.

Architectural changes (executive summary):- Multi-region active‑active deployment: run identical stateless API frontends and regional load balancers in at least three availability zones across two cloud regions. Use global traffic manager with health‑aware routing and consistent hashing for sticky sessions if needed.- Data redundancy and consistency: replicate data across regions with a primary-secondary or multi-master approach depending on write latency tolerance. Prefer conflict-free multi-master or strongly consistent leaderless patterns for critical state; use regional caches with write‑through to durable store.- Network and dependency redundancy: dual transit providers, cross-region peering, redundant DNS, and infrastructure-as-code for automated reprovisioning.- Automated failover and chaos testing hooks: deploy canary/blue‑green with automatic rollback.

Operational requirements:- Detailed runbooks for every failure mode (region down, DB partition, rolling upgrade, DNS TTL issues).- 24/7 on‑call rota with escalation matrices and SLO/SLA dashboards; defined RTO/RPO per component.- Regular drills: quarterly simulated region failovers, monthly game days, and postmortems with RCA timelines.- Continuous observability: distributed tracing, synthetic transactions, error budgets and automated alerts.

Expected cost uplift:- Typical uplift is 2–4x baseline costs: multi-region compute, cross-region replication egress, standby capacity, higher licensing and network transit. Example: a $100k/year single-region SaaS may become $250–400k/year for five-nines; exact depends on data egress and replication choices.

Residual risks (cannot be fully eliminated):- Wide‑area cloud provider outages impacting control plane, or simultaneous region failures beyond design assumptions.- Software bugs triggered only in cross-region paths and unknown third‑party dependencies.- Network-level BGP or internet backbone issues outside provider control.- Human error during complex failovers despite runbooks.

Recommendation:- Agree measurable SLOs, run a cost/benefit tradeoff workshop to pick replication strategy, pilot a two-region active-active with automated failover and then evolve to three-region if SLA and budget require. Build a contractual SLA reflecting residual risks and shared responsibilities.

Unlock Full Question Bank

Get access to hundreds of Fault Tolerance and Failure Scenarios interview questions and detailed answers.

Join thousands of developers preparing for their dream job.