Fault Tolerance and Failure Scenarios Questions
Designing systems resilient to component failures: timeouts, retries with exponential backoff, circuit breakers, bulkheads. Discuss cascading failure prevention and graceful degradation. At Staff level, demonstrate thinking about multi-layer failures (service failures, database failures, network partitions) and how to detect and recover from them.
EasyTechnical
78 practiced
Implement a function in Python that returns the next retry delay using exponential backoff with full jitter. Signature: def next_delay(attempt: int, base: float = 0.1, max_delay: float = 10.0, seed: int = None) -> float. The function must be deterministic when seed is provided and ensure delay in [0, min(max_delay, base * 2**attempt)]. Include a brief comment explaining your approach.
Sample Answer
Approach: compute the exponential cap = min(max_delay, base * 2**attempt) and return a random value uniformly in [0, cap] (full jitter). If seed is provided, use a local Random instance seeded deterministically (I add attempt to the seed so different attempts produce different but reproducible values). Validate inputs and ensure non-negative bounds.Key points:- O(1) time and space.- Full jitter prevents synchronized retries.- Deterministic when seed provided; local RNG avoids changing global random state.- Edge cases handled: negative inputs, cap can be zero.
python
import random
from typing import Optional
def next_delay(attempt: int, base: float = 0.1, max_delay: float = 10.0, seed: Optional[int] = None) -> float:
"""
Return next retry delay using exponential backoff with full jitter.
Delay is in [0, min(max_delay, base * 2**attempt)].
If seed is given, result is deterministic; uses seed + attempt to vary per attempt reproducibly.
"""
if attempt < 0:
raise ValueError("attempt must be non-negative")
if base < 0 or max_delay < 0:
raise ValueError("base and max_delay must be non-negative")
cap = min(max_delay, base * (2 ** attempt))
if seed is None:
return random.random() * cap
# deterministic per-attempt: use a local RNG so global state unaffected
rng = random.Random(seed + attempt)
return rng.random() * capMediumSystem Design
64 practiced
Design a fallback where a service runs in cache-only read mode when its primary datastore is unavailable. Requirements: serve cached reads where possible, avoid serving data older than configured TTL, reject writes safely, and minimize inconsistent reads. Describe components, cache priming/invalidation, and failure modes.
Sample Answer
Requirements clarification:- Serve cached reads while primary DB is down- Never return data older than configured TTL- Reject writes safely (no silent data loss)- Minimize inconsistent reads (stale/incomplete data)High-level architecture:- Clients -> API gateway / service instances -> Read path checks Cache (Redis/Memcached) -> Primary DB- Health & Failover controller monitors DB (heartbeat, error rates) and flips service into cache-only-read mode when DB is unhealthy- Write requests in cache-only mode are rejected with a clear error code and retry guidance (HTTP 503 + Retry-After)Components and responsibilities:1. Cache (Redis cluster with persistence disabled for speed; use replication) - Stores recent read responses with per-key TTL - Each value includes metadata: lastUpdated, version/ETag, origin-timestamp2. Service instances - Normal mode: read-through cache (check cache; on miss, read DB, populate cache) - Failover mode: serve from cache only if entry exists and age <= configured TTL; otherwise return 503/NOT_FOUND - Write path: in normal mode write-through/transaction to DB and update cache; in failover mode reject writes3. Health & Failover controller - Multi-signal decisions: DB connectivity, sustained error rate, replication lag - Grace period and quorum to avoid flapping; expose state via config/feature flag4. Monitoring & Alerts - SLOs, cache hit ratio, stale-read rate, rejected-writes rate, failover events loggedCache priming and invalidation:- Priming: warm cache by read-through operations and background async prefetch for hot keys. Use sidecar prefetcher populating popular keys periodically.- Invalidation: writes update cache atomically (write-through or publish/subscribe invalidation). Use versioned keys or ETag to prevent race conditions. Set conservative TTLs aligned with staleness tolerance.- On DB recovery: controller triggers soft-reconciliation: read-back sync for keys updated during outage if writes were queued (if using write-queue approach) or simply resume normal write-through while gradually invalidating TTL-expired entries.Minimizing inconsistent reads:- Use version/ETag per object; clients can optionally request If-None-Match semantics.- Prefer read-after-write consistency for same instance by updating cache immediately after successful DB write.- For multi-region setups, ensure cache coherence with pub/sub invalidation across regions.Write rejection strategy:- Synchronous writes rejected with a clear status and idempotency token guidance.- Optionally offer a durable write-queue (append-only) as a degraded feature: accept writes to a durable queue with client-visible mode flag (eventual application once DB recovers) — but only if application semantics tolerate eventual consistency.Failure modes and mitigations:- Cold cache misses during outage -> increased 503s: mitigate via aggressive priming, longer TTL for very hot keys, and circuit-breaker to limit downstream load.- Cache corruption or partition -> fail closed (reject reads rather than serve stale) if integrity checks fail. Monitor and auto-replace nodes.- Flapping DB health -> hysteresis and quorum-based gating to avoid toggling modes.- Long outage with TTL expiry -> accept increased errors or enable degraded write-queue if acceptable.- Split-brain in multi-region caches -> use region-local caches and route clients to nearest healthy region; global invalidation via consistent pub/sub.Operational considerations:- Document behavior in API contract and runbook.- Provide metrics: %reads served from cache, age distribution, rejected-writes, failover duration.- Test with chaos engineering: simulate DB outage, cache eviction, and recovery to validate behavior and SLOs.This design prioritizes safety (reject writes), bounded staleness (TTL + metadata), and predictable client behavior during primary datastore failures.
HardSystem Design
84 practiced
Design capacity planning and failover strategies to keep a service operating during an entire availability zone (AZ) outage. Include provisioning strategies (over-provision vs warm standby), autoscaling policies, cross-AZ DNS/load-balancer changes, and detection/rebalancing timelines to restore target capacity within minutes.
Sample Answer
Requirements & constraints:- RTO: <5 minutes to restore target capacity after AZ failure; RPO: none (stateless preferred) or synchronous replication for stateful.- Target: survive entire AZ loss without >SLO violations (e.g., 99.95% monthly).- Cost vs risk: balance between over‑provisioning and warm standby.High-level approach:- Multi-AZ active-active where feasible; otherwise active-passive with warm standby in secondary AZs.- Combine a small over-provision buffer + warm standby fleet + aggressive autoscaling driven by fast detection and predictive signals.Provisioning strategies:- Active-active (preferred): distribute traffic evenly across at least 3 AZs. Each AZ runs ~40% of needed capacity so any single AZ loss still leaves >60% capacity, then autoscale remaining AZs to 100% of demand.- Active-warm-standby: run warm fleet in standby AZ sized to 30–50% of peak. Warm instances are kept up-to-date (AMI/container images, config), behind LB but with 0 weight until failure.- Over-provisioning: keep a headroom buffer (e.g., 20–30% spare capacity) across AZs for rapid absorb. Higher cost but fastest.Autoscaling policies:- Two-tier autoscaling: 1. Reactive: scale on CPU/RPS/queue depth with aggressive cooldowns (cooldown 30–60s), short evaluation periods (1 min), step-scaling: add large steps (20–50% capacity) initially. 2. Predictive: use recent traffic trends and schedule-based scaling for known peaks; incorporate anomaly signals from monitoring to pre-scale if AZ health degrades.- Warm start optimizations: use pre-warmed AMIs/containers and warm caches; target instance warm time <90s (image caching, container cold-start optimizations).- Minimum instances per AZ to maintain capacity for quick failover.Cross-AZ DNS / Load-balancer changes:- Keep global DNS TTL low (30–60s) but rely primarily on LBs with health-based weighting rather than DNS for immediate failover.- Use load balancer (ALB/NLB) with cross-AZ capability and per-AZ target group weights. On AZ loss: - LB detects unhealthy targets via health checks (30s, 2 failures) and stops sending traffic immediately. - If LB is regional and supports cross-zone routing, shift traffic instantly across remaining AZs.- For DNS-based global failover (multi-region scenarios), use health-checked Route53 failover or latency-based weighted routing; keep TTL low and pre-warm endpoints.Detection & timelines:- Multi-source detection: LB health checks, instance-level metrics, hypervisor/host alerts, network path monitoring.- Detection target: <30s for AZ outage using aggressive health-check freq and synthetic canaries from clients.- Immediate LB reweighting: 0–30s after detection (LB stops routing to failed AZ).- Traffic rebalancing: instantaneous at LB; DNS changes propagate within TTL (30–60s).- Autoscaling to restore capacity: use warm standby + step scaling to reach target capacity within 2–5 minutes: - Warm standby provides immediate ~30–50% capacity. - Reactive autoscaling triggers add more instances; with pre-warmed images expected scale-up to target in ~3 minutes.- Total RTO target: 3–5 minutes.Stateful considerations:- Use cross-AZ synchronous replication for critical state (RDS Multi-AZ, distributed caches with replication).- Prefer stateless services + externalize state to replicated datastores (multi-AZ writes where supported).Operational controls & hardening:- Health-check tuning: short intervals, conservative failure thresholds to avoid false positives.- Circuit breakers and rate limiters when remaining capacity absorbs extra load.- Session handling: avoid sticky sessions; use shared session store or JWT tokens so traffic can move between AZs.- Chaos engineering: regular AZ-failure drills and game days to verify detection, LB behavior, autoscaling timings, DB failovers.- Observability: dashboards, alerts, SLIs (latency/error rate), runbooks, automated remediation playbooks.Trade-offs:- Over-provision = fast recovery, high cost.- Warm standby = lower cost, slightly longer recovery.- Active-active = best availability but more complex (consistency, cross-AZ latency).- DNS-based failover is slower than LB reweighting; use LB-first strategy.Example quantitative plan:- Peak throughput requires 100 instances. Distribute 40/40/20 across AZs (active-active with uneven distribution) + 20% buffer -> loss of one AZ (40 instances) leaves 60 active + 20 buffer = 80 active => autoscale to 100 within 3 minutes using step-scaling (add 20 instances at t+30s, add 20 more at t+90s) with warm images to meet RTO <5m.This combination (active-active or active-warm standby + fast detection + aggressive autoscaling + LB-driven traffic shift + rehearsed runbooks) achieves minute-scale recovery while balancing cost and operational complexity.
MediumTechnical
80 practiced
Design liveness and readiness probes for a microservice that depends on cache, auth service, and message queue. Explain how you would detect partial degradation versus full service failure, avoid false positives, and design probe endpoints, latency thresholds and retry behavior suitable for Kubernetes or similar orchestrators.
Sample Answer
Start by separating readiness (can I receive traffic?) from liveness (am I healthy enough to keep running?). Design two endpoints: /healthz/live for liveness and /healthz/ready for readiness.Readiness (/healthz/ready)- Purpose: gate traffic during partial degradations.- Checks: shallow checks for critical dependencies required to serve requests: - Cache: ability to GET a small key (or local TTL check) within 50ms. - Auth service: validate ability to reach auth token introspection endpoint or local token cache warm state (no full token validation on each probe) within 100ms. - Message queue: confirm ability to publish to a lightweight health-topic (or check broker connection/consumer group state) within 100ms.- Behavior: aggregate as soft-fail. If any non-critical dependency (e.g., metrics exporter, optional analytics) fails, still ready. If critical dependency fails => NotReady.- Retries: orchestrator probe should use initialDelaySeconds=10, periodSeconds=10, failureThreshold=3 to avoid flapping.Liveness (/healthz/live)- Purpose: detect hard failures needing restart.- Checks: very narrow — process alive, event loop responsive, critical background worker threads running, ability to access local disk/config. Do NOT call remote dependencies to avoid false positives from transient network issues.- Behavior: return fail only if internal self-checks fail (deadlock, memory corruption).- Probe config: initialDelaySeconds=30, periodSeconds=10, failureThreshold=6.Detecting partial vs full failure- Partial degradation: readiness false but liveness true — Kubernetes will stop sending traffic but won’t restart. Use metrics/alerts when readiness toggles to false for >2 probe cycles.- Full failure: liveness false => pod restart.Avoid false positives- Keep liveness strictly internal.- Use quick, cached or synthetic lightweight checks for remote systems rather than heavy synchronous ops.- Use conservative probe timing and failureThresholds; implement exponential backoff and jitter in retries.- Circuit-breaker pattern in app-level checks: mark downstream as degraded after n failures and avoid probe-triggered cascading calls.Latency thresholds & retry behavior- Set probe timeouts smaller than periodSeconds (e.g., timeoutSeconds=2).- Choose thresholds based on p95 latency and SLA; typical values above are conservative for low-latency services.- Use health-check endpoints that return HTTP 200 with JSON {status: "ok", details: {cache: "ok", auth: "degraded"}} for observability.Observability & automation- Emit events/metrics when readiness changes (tag which dependency failed).- Auto-remediation: escalate to restart only after repeated readiness failures and correlated liveness anomalies.- Document which dependencies are critical vs optional and test probes in chaos exercises.
EasyTechnical
81 practiced
A popular cache key expires simultaneously and causes a sudden DB spike (thundering herd). Describe multiple approaches to prevent this as an SRE: jittered TTLs, request coalescing (single flight), mutexing fetches, stale-while-revalidate, proactively refreshing hot keys, and how to choose among them based on service constraints.
Sample Answer
Approach framework: identify risk (hot-key TTL alignment → DB spike), then choose mitigations by cost, complexity, and service constraints (read/write ratio, cache TTL control, tolerance for stale data, DB capacity).Mitigations (what & when):- Jittered TTLs - Add randomized ±x% to TTLs so keys expire spread over time. - Low complexity, zero runtime coordination. Good default for many services. - Trade-off: small variance in staleness; ineffective if keys are refreshed deterministically.- Request coalescing (single-flight) - Ensure only one upstream fetch for a missing key; others wait for result. - Good when reads spike and latency can tolerate short waits; implemented in-process or via library. - Trade-off: waiting clients see slightly higher latency; requires app-level change.- Mutexing fetches (distributed lock) - Acquire a lock (e.g., Redis SETNX) to let one worker refresh while others serve fallback or wait. - Useful in distributed systems where single-flight per process is insufficient. - Trade-off: complexity, deadlock handling, lock expiry tuning.- Stale-While-Revalidate - Serve stale cached value immediately while triggering background refresh. - Ideal when slightly stale data is acceptable and latency must stay low. - Trade-off: clients may see outdated data briefly.- Proactive refresh (warming) - Periodically refresh hot keys before expiry (cron or reactive refresh when hotness detected). - Best when hot keys are identifiable and limited in number. - Trade-off: additional load on DB spread over time; needs tracking of hot keys.How to choose:- If low effort and many keys → jittered TTLs.- If strict freshness required but DB can handle single refresh → single-flight or distributed lock.- If low latency critical and slight staleness OK → stale-while-revalidate.- If small set of hot keys → proactive refresh/warming.- Combine: jittered TTLs + stale-while-revalidate + single-flight gives robust protection.Operational notes:- Monitor cache miss rate, DB QPS, and latency; add alerts when surge patterns appear.- Test under load and simulate expirations; iterate on TTL variance, lock timeouts, and refresh schedules.
Unlock Full Question Bank
Get access to hundreds of Fault Tolerance and Failure Scenarios interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.