Reliability Patterns and Fault Tolerance Questions
Applying design patterns and engineering practices that make services resilient to faults. Core patterns include redundancy and replication, circuit breakers, bulkheads, graceful degradation, timeouts and retries with exponential backoff, throttling and rate limiting, bulkhead isolation, and retry idempotency strategies. Also includes measuring and improving reliability through metrics such as mean time to repair and mean time between failures, defining Service Level Indicator and Service Level Objective targets, designing error budgets, and choosing trade offs between consistency and availability. Candidates should explain when to apply each pattern and how patterns interact in distributed systems.
HardTechnical
54 practiced
Your idempotency store is experiencing high write volumes causing latency spikes during traffic peaks. Propose architectural strategies to scale the idempotency mechanism (e.g., sharding, TTL, local caches, probabilistic filters) while preserving correctness and preventing unbounded memory growth.
Sample Answer
Requirements & constraints (clarify): clients require at-least-once network delivery but logical exactly-once processing per idempotency key within a bounded window (e.g., 24h). System must absorb high write QPS during peaks, avoid unbounded memory, and preserve correctness (no false accepts).High-level approach: combine sharded persistent store (durable ground truth) + bounded in-memory caches per shard + probabilistic front-line filter + write coordination for correctness.Architecture:- Ingress layer: lightweight API gateway that extracts idempotency key and routes to shard via consistent hashing (key-based sharding ensures single-writer/owner per key).- Sharded idempotency service: N-stateful nodes each owning shards; each node maintains an in-memory LRU cache (size-limited) + persistent backing store (e.g., distributed KV like Dynamo/Cassandra or single-partition RDBMS per shard) with TTL.- Front cache & filter: per-instance local cache (LRU) for hot keys; optionally a Bloom filter to quickly reject definitely-seen keys (tunable false-positive).- Durable store with TTL + compaction: each idempotency record has creation timestamp and TTL; background compaction removes expired keys to bound storage; use time-partitioned tables for efficient expiry.- Correctness on writes: perform a conditional write (compare-and-set / lightweight transaction) at shard owner: if key absent, insert with response state; if present, return stored result. For performance, optimize single-hop writes to the shard. Use versioned results or idempotency status (processing/completed) to handle in-flight requests.- Asynchronous fast-path: on cache miss, optimistic local acceptance with an async confirm to shard only if operation is non-idempotent—only if business can tolerate eventual reconciliation. Otherwise always sync to shard for strict correctness.- Backpressure & batching: if persistent store becomes a bottleneck, batch writes per shard and use write-behind with ordered logs ensuring at-least-once persistence; ensure visibility only after persistence for strict guarantees.- Probabilistic filters trade-offs: Bloom filters reduce DB checks but can cause false positives (leading to rejecting fresh requests); acceptable only if false-positive rate is very low and business can retry. Never rely on Bloom filter alone for correctness.- Scaling: add shards and rebalance via consistent hashing with virtual nodes; autoscale shard owners and separate read replicas for result reads.- Monitoring & ops: track cache hit rate, latency, DB write queue length, false-positive rate, and storage growth. Provide dashboards and alerts; run regular compaction and re-sharding maintenance windows.Example flow (strict correctness):1. Client sends request with idempotency key.2. Gateway routes to shard owner.3. Shard owner checks local cache → if hit return stored response.4. If miss, perform conditional insert in backing store (CAS): - If insert succeeds: process request, persist result, update cache. - If insert fails (already present): read stored result and return.5. TTL ensures eventual cleanup.Trade-offs:- Strong correctness (sync CAS to durable store) costs higher latency; local cache + Bloom filters reduce latency but add complexity and risk.- Batching/write-behind improves throughput but requires careful ordering to avoid duplicate processing.- Sharding limits cross-node coordination but requires rebalancing when topology changes.This design balances correctness (single-writer per key + conditional durable writes), throughput (sharding + local caches + batching), and bounded storage (TTL + compaction), and provides operational levers to tune latency vs cost.
MediumTechnical
56 practiced
What telemetry and logs should a team collect to evaluate whether a circuit breaker is functioning correctly? Include state transition events, counts of failures/successes, latency buckets, and suggested alert rules to detect misconfiguration or thrashing.
Sample Answer
Situation: As a Solutions Architect you must specify observability for circuit-breakers so teams can prove they work, spot misconfiguration, and detect thrashing.Telemetry & logs to collect (with labels: service, instance, endpoint, region, environment, cb_policy_id):- State transition events (event log): timestamped records of Closed → Open, Open → Half-Open, Half-Open → Closed, with reason (threshold type) and current counts. Keep raw event log and aggregated counts.- Current state gauge: numeric enum (0=Closed,1=Open,2=Half-Open) and last_transition_ts.- Transition counters: counter metric for each transition type (cb.transition.open, cb.transition.close, cb.transition.half_open).- Failure/success counters: cb.request.total, cb.request.success, cb.request.failure, cb.request.timeout, cb.request.short_circuit.- Latency histograms/buckets: cb.request.latency_ms histogram (fine-grained buckets: <50,50–100,100–300,300–1000,>1000).- Half-open trial outcomes: cb.halfopen.trials, cb.halfopen.success, cb.halfopen.failure.- Duration metrics: cb.open.duration_seconds (how long circuit stayed open).- Saturation/backpressure signals: queued_requests, retry_attempts.- Error classification: exception_type, HTTP status buckets (4xx,5xx), upstream service id.Key dashboards / aggregates:- Transition rate per minute, median/95/99 latency, success rate over sliding windows (1m/5m/1h), open-duration distribution.- Heatmap of transitions by endpoint/instance to find noisy nodes.Alert rules (examples):- Misconfiguration / dead circuit: - No state transitions for X days while failures are reported: alert if cb.transition.* == 0 AND cb.request.failure > threshold over 1h → possible miswired CB. - Circuit always open: gauge state == Open for > 10m and upstream success rate > 90% → misconfiguration. - Circuit always closed despite errors: state == Closed AND failure_rate > 25% for 5m → CB thresholds ineffective.- Thrashing detection: - High transition frequency: sum(cb.transition.open + cb.transition.close) > 10 within 1m → alert (possible thrash). - Frequent half-open failures: cb.halfopen.failure / cb.halfopen.trials > 0.7 over 5m → abort retries, escalate.- Correctness checks: - Short-circuit spike: cb.request.short_circuit / cb.request.total > 0.5 over 1m → investigate legitimate failures vs false positives. - Latency-rise-before-open: rising 95th percentile latency followed by open: use anomaly detection to correlate.- SLA-impact: - Increase in user-facing errors: 5xx rate increase > X% and cb.state != Open → upstream issue leaking through.Reasoning: track both events (discrete transitions) and continuous counters/histograms so you can answer “did it open when it should?” (transitions vs failures), “is it oscillating?” (transition frequency + open duration), and “is behavior expected during half-open?” (trial success ratio). Labels let you narrow to endpoints/instances for remediation. These metrics support runbooks: when thrashing, widen thresholds or add jitter/backoff; when misconfigured, revert to safe defaults.
HardSystem Design
60 practiced
Design a multi-tenant resource isolation and throttling architecture so noisy tenants cannot impact others. Requirements: support 1M tenants, dynamic quotas per tenant, burst allowances, monitoring, and ability to enforce different SLAs per tenant. Discuss enforcement points (edge, proxy, application), data models, and cost trade-offs.
Sample Answer
Requirements clarification:- 1M tenants, per-tenant dynamic quotas and SLA tiers, burst allowances, observability, strict isolation so noisy tenants can’t degrade others, low-latency enforcement.High-level architecture:- Global API Edge (CDN + WAF) → Ingress Proxy Tier (regional fleet) → Throttling/Enforcement Tier (sidecar or dedicated proxies) → Application services → Backend resources (DB, compute, shared caches).- Control plane: Quota policy manager, tenant metadata store, metrics pipeline, and enforcement config distributor.Enforcement points & trade-offs:- Edge (CDN/WAF + rate-limit rules): best for coarse, cheap protection (block floods, global rate-limits). Low cost, high scale, limited per-tenant state (useful for very large tenant counts).- Regional Proxy (Envoy/NGINX with xDS): fine-grained, per-tenant dynamic quotas, burst windows, SLA-based prioritization. Good latency; needs horizontal scale and state sharding.- Application/Sidecar: strongest isolation (request shaping, token buckets, circuit breakers per tenant). Highest cost (compute overhead) but enforces resource-specific limits (DB connections, CPU).Quota model and data model:- Tenant table: tenant_id, SLA_tier, base_qps, burst_capacity, burst_window_seconds, priority_weight, limit_version, last_update.- Policy store: time-based rules, scheduled overrides, emergency limits.- Runtime state: per-tenant token-bucket counters sharded by tenant_id hash, stored in distributed in-memory stores (Redis Cluster with key prefix tenant:{id} or local in-proxy counters with sync).- Metrics: per-tenant time-series (ingress_qps, rejected, latency, resource_usage) into scalable TSDB (Prometheus remote write → Cortex/Mimir).Scalability & consistency:- For 1M tenants, avoid per-tenant active objects at edge. Store compact policy in CDN headers; use proxies to fetch and cache policy (LRU cache). Use probabilistic data structures (leaky bucket approximations) for low-memory enforcement; move "hot" tenants to dedicated enforcement pods.- Shard runtime counters by tenant hash across Redis shards; set TTLs to expire idle tenants to save memory.Burst handling & fairness:- Token-bucket with burst_capacity + decay. SLA-aware weighted queuing: when overloaded, serve high-tier tenants first (weighted fair queuing or priority tokens).- Emergency backpressure: circuit-break per tenant and global shedding policies.Monitoring & control plane:- Telemetry pipeline aggregates per-tenant metrics, anomaly detection triggers policy changes. Expose dashboards, alerting, and on-demand throttle adjustments via admin APIs. Audit trails for enforcement actions.Cost trade-offs:- Push enforcement closer to edge reduces backend cost but increases CDN/proxy complexity and per-request compute cost.- Centralized Redis counters are simpler but expensive at 1M tenants; use TTLs, cold-tenant eviction, and move heavy hitters to dedicated pods to optimize cost.- Sidecars provide best isolation but multiply compute (per-service). Use adaptive deployment: sidecars for critical services/tenants; shared proxies for the rest.Security & operational:- Authenticated policy changes, staged rollouts, canary enforcement, and blameless playbooks. Test with synthetic noisy-tenant simulations.Result: hybrid approach — coarse global edge limits, smart regional proxies with cached per-tenant policies, token-bucket counters sharded in-memory for hot tenants and probabilistic/cached enforcement for cold tenants — meets 1M scale with dynamic quotas, burst allowances, SLA differentiation and controllable costs.
MediumSystem Design
56 practiced
Design a per-user rate limiter for a public REST API that must support 500k active users and 10k requests/s at peak. Requirement: limit to 100 req/min per user, low-latency enforcement, global clients across regions. Describe architecture (edge vs central), algorithm choice, consistency trade-offs, and cross-region sync.
Sample Answer
Requirements clarification:- 100 req/min per user (1.67 r/s), 500k active users, 10k req/s peak, global clients with low latency and enforcement at edge.High-level architecture:- API Gateway at each region/edge (CloudFront/ALB + Envoy) receives requests.- Local rate-limiting tier per region using fast in-memory store (Redis or local in-process token bucket with small Redis backing).- Global coordination only when needed.Algorithm choice:- Token Bucket per user (allows short bursts, smooth refill at 100/min). Implement per-user buckets keyed by user-id.- Primary enforcement at regional Redis instances for low latency (single GET/INCR+Lua script to atomically check and decrement tokens).- Use a local in-process cache for hot users to avoid network hop; synchronize with Redis TTL.Scaling:- Shard Redis by user-id (consistent hashing) across clusters in each region. Each region holds subset of user shards; users mapped deterministically to a home region to keep counters mostly local.- For users hitting other regions, use a fallback local bucket with conservative limits (e.g., 80% of quota) to prevent exceeding global quota.Consistency & trade-offs:- Strong global consistency (single central counter) achieves exact enforcement but adds high latency and single-point bottleneck — not acceptable.- We prefer eventual/weak consistency: enforce per-region counters with occasional sync to approximate global usage. Accept small temporary overshoot across regions (bounded by number of regions × conservative local quota).- Use Lua scripts for atomic ops inside Redis to ensure per-region atomicity.Cross-region sync:- Periodic async reconciliation (every 5–30s) using CRDT-like merge: maintain per-region sub-counters for each user; global usage = sum(sub-counters). When user maps to a home region, home region aggregates and can enforce stricter limits.- Use change-stream or Redis Streams to publish counter deltas to other regions; apply anti-entropy jobs for correctness.- For strict cases (fraud, billing), run centralized batch checks and temporarily block offenders.Operational considerations:- Metrics/alerts for overshoot, hot keys, latency.- Rate-limit headers to clients and 429 responses.- TTLs to auto-expire inactive-user counters.- Use capacity planning: Redis memory for active users ~ 500k * size per key; shard and autoscale.This design gives low-latency enforcement at edge, horizontally scalable counters, bounded cross-region inconsistency, and operational controls to tune strictness vs latency.
HardTechnical
71 practiced
How would you design a distributed tracing and instrumentation strategy that helps detect fault propagation and root cause in a complex microservice system? Cover sampling strategies, header propagation, integration with logs and metrics, and how to use traces to diagnose tail latency incidents.
Sample Answer
Requirements and constraints:- End-to-end causal visibility across services, low overhead, support for high QPS, ability to detect fault propagation and find root cause, integrate with existing logging/metrics, and actionable for tail-latency incidents.High-level approach:- Standardize on OpenTelemetry for instrumentation and exporters.- Enforce W3C Trace-Context (traceparent/ tracestate) header propagation across HTTP/gRPC/async transports and messaging (inject/extract in message headers). Ensure libraries/middleware automatically propagate contexts; add SDK hooks for custom transports.Sampling strategy:- Use hybrid sampling: - Low-overhead probabilistic sampling (e.g., 0.1–1%) for global coverage. - Deterministic keep rules for errors, HTTP 5xx, exceptions, and high-resource events. - Tail-based sampling: buffer trace summaries for recent requests and preferentially keep traces from the slowest X% (e.g., top 1% latency) and traces that exhibit errors. Implement at collector/ingest layer to avoid losing crucial slow/error traces. - Adaptive sampling: adjust rates per-service during load spikes to bound ingestion cost.- Record aggregate metrics for all requests (counts, p50/p95/p99 latencies) even when detailed traces are not kept.Instrumentation and telemetry linkage:- Instrument key spans (ingress, auth, DB, cache, downstream calls). Use semantic conventions (OpenTelemetry) for span names and attributes (http.status_code, db.statement, peer.service).- Correlate logs: inject trace_id and span_id into structured logs at logging layer (e.g., via MDC/logging context). Ensure logs emitted without trace still include request identifiers where possible.- Correlate metrics: emit metrics with dimensions for service, endpoint, and HTTP status; use trace sampling decisions to tag metrics that came from traced requests.- Centralize collectors (OTel Collector) to enrich, apply tail sampling, redact sensitive fields, and forward to APM/Tracing backend + long-term metrics store and logging system.Integration with logs & metrics:- Use a single observability pipeline that links traces → logs → metrics. In UI, enable “jump to logs” from a selected trace/span (via trace_id) and surface relevant logs automatically.- Create dashboards that combine metrics (p99 latency) with trace sampling rate and error rates to monitor instrumentation health.Diagnosing tail-latency incidents:- Trigger: alert on p99/p995 latency or increase in error budget.- Use tail-based-sampled traces to get full waterfall of slow requests. Steps: 1. Identify slow traces (top latency) and group by service/endpoint, user-agent, region using trace attributes. 2. Analyze span duration distributions to find the bottleneck span(s) and conditional probability that a particular span > threshold when end-to-end is slow. 3. Use trace flamegraphs/waterfalls to spot retry storms, blocking dependencies, queueing (long client-to-service or service-to-service gaps), threadpool exhaustion, or downstream cascading failures. 4. Correlate with logs (same trace_id) to see exceptions, GC pauses, or circuit-breaker trips; correlate with metrics (CPU, latency, queue lengths) to confirm resource saturation. 5. If propagation of faults is suspected, follow causal edges: a pattern of increased latency in service A followed by errors/latency in B indicates propagation; instrument and visualize service map with edge latency percentiles and error rates.- Use span attributes to run root-cause queries: filter traces where db.query latency > X or where cache miss rate high.Operational considerations & trade-offs:- Privacy: redact PII at collector; store sampled traces securely and for a bounded retention.- Cost control: tune sampling and aggregation; pre-aggregate metrics and logs to reduce storage.- Reliability: make tracing non-blocking and asynchronous; don't let exporter failures affect request paths.- Evolution: start with auto-instrumentation then add manual spans for complex flows; enforce naming/attribute guidelines.Outcome:- This strategy yields efficient global coverage, guaranteed capture of errors and tail events for postmortem, fast root-cause via linked traces/logs/metrics, and controllable costs with adaptive/tail sampling.
Unlock Full Question Bank
Get access to hundreds of Reliability Patterns and Fault Tolerance interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.