Load Balancing, Failover, and Fault Tolerance Questions

Understand load balancing strategies (round-robin, least connections, consistent hashing, weighted load balancing). At Staff Level, understand the trade-offs between different strategies and when each is appropriate. Master failover mechanisms, service discovery, and circuit breakers. Understand concepts like graceful degradation, bulkheads (service isolation), and how to design systems that remain operational when components fail. Be comfortable discussing health checks, monitoring, and alerting strategies to detect failures and trigger failover.

EasyTechnical

46 practiced

Define graceful degradation and the bulkhead pattern. For each, provide concrete examples of how you'd apply them in a microservices architecture to keep core user-facing functionality available during partial failures. Explain trade-offs such as reduced feature surface vs operational complexity.

MediumTechnical

50 practiced

Observability question: List and justify the key metrics and alerting rules you would create to detect unhealthy backends behind a load balancer before users see errors. Include thresholds, alert severity differentiation to avoid fatigue, and how to correlate metrics (latency, error rate, connection errors, health-check failures, queue depth) for confident notifications.

Sample Answer

Situation: We need to detect unhealthy backends behind a load balancer before users see errors. The goal is high signal-to-noise: catch real problems early, avoid alert fatigue, and provide actionable context.

Key metrics and why:- Backend latency (p50/p95/p99): early indicator of degradation. Track per-backend and per-cluster.- Backend error rate (4xx/5xx by backend): direct user-impact signal.- Connection errors (TCP RST, connect failures): indicates network or process-level failures.- Health-check failures (LB probe failures per backend): deterministic unhealthy signal.- Queue depth / pending requests per backend: indicates overload/backpressure.- Success rate and request throughput for overall service: correlate system-wide impact.

Alerting rules (examples — PromQL-style):

promql

# Severe: Backend failing health checks (immediate)
expr: sum by (backend)(probe_failures[1m]) > 0
labels: severity=critical

# High latency (actionable warning -> escalate if persistent)
expr: histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[2m])) by (le, backend)) > 1.5
for: 5m
labels: severity=warning

# Rising error rate (5xx) per backend
expr: increase(http_requests_total{code=~"5.."}[5m]) / increase(http_requests_total[5m]) > 0.05
for: 3m
labels: severity=warning

# Queue depth high -> capacity issue
expr: avg_over_time(backend_queue_depth[2m]) > 100
for: 2m
labels: severity=warning

Severity differentiation & fatigue control:- Critical: deterministic signals (health-check failures, connect failures, node down). Fire immediately with short for: 0–1m.- Warning: probabilistic signals (latency, error-rate, queue depth). Require sustained window (3–10m) and higher thresholds.- Use Alertmanager inhibition: critical alerts suppress related warning alerts (e.g., health-check failure suppresses latency/error warnings for same backend).- Group alerts by backend/cluster and route to on-call rotation; use rate-limiting and deduplication.

Correlation strategy for confident notifications:- Require at least two orthogonal signals for automated escalation: e.g., health-check_failure OR (latency_p95 > threshold AND error_rate > threshold) OR (connection_errors > threshold AND queue_depth > threshold).- Include context in alert payload: recent trends (last 15m), per-backend p95, error rate, health-check count, active connections.- Use enrichment: link to topology (which load balancer, instance IDs), recent deploys, and circuit-breaker state.- Use anomaly detection / baselining for services with variable traffic: alert on relative deviation (e.g., p95 > 3x baseline).

Runbook pointers (short in alert): check LB health, tail app logs, check GC/threads, restart backend, or scale out. This design balances early detection with low noise and provides correlated evidence for faster remediation.

HardSystem Design

47 practiced

Design load balancing and health-check strategy for a service using a primary writable database and geographically distributed read replicas. Some users require read-after-write consistency. Explain routing rules for reads and writes, how you detect replica lag, how to handle read-after-write requests, and failover considerations if the primary becomes unreachable.

Sample Answer

Requirements & constraints:- Primary single-writer (low-latency writes). Many geographically distributed read replicas for scaling reads and locality.- Some users require read-after-write (RAW) consistency; most reads can be eventually consistent.- SRE goals: minimal user-visible latency, availability, correct failover, observable replication lag.

High-level routing- All writes → routed to primary via a write-proxy (HAProxy/Envoy/ProxySQL or cloud TCP LB) that performs health checks for primary role.- Reads → geo-routing layer (CDN/LB/proxy) sends reads to nearest replica if replica is healthy and lag ≤ acceptable threshold; otherwise fall back to primary or a healthier replica.

Health checks & replica-lag detection- Multi-dimensional checks per instance: - Process/port liveness - Application-level synthetic query (SELECT 1; and a small read query) - Replication metrics: for MySQL use seconds_behind_master and master_log_pos/relay_log_pos; for Postgres use pg_current_wal_lsn vs pg_last_wal_replay_lsn to compute byte/time lag. - Heartbeat table: writer inserts monotonic heartbeat row with timestamp or LSN/GTID; replicas report last-seen heartbeat timestamp.- Export metrics to Prometheus: replication_lag_seconds, replication_lag_bytes, last_heartbeat_timestamp. Alert when lag > soft threshold (e.g., 500ms–2s) and critical threshold (e.g., >30s).

Read-after-write handling- Two supported strategies (choose per-client or per-route): 1. Strong RAW via sticky write affinity: after a write, client session is “pinned” to primary for a short TTL (e.g., user session or transaction) so subsequent reads go to primary. 2. LSN/heartbeat-based read routing: on write, primary returns the commit LSN/GTID/heartbeat timestamp to client (or app server). Subsequent read requests present that token; the read-router chooses a replica whose last_replay_lsn >= token (or last_heartbeat_time >= token). If no replica is caught up within timeout, read either falls back to primary or waits (based on SLA).- Implementation: proxy (Envoy with metadata) or application middleware adds write_token and routing logic. This avoids long-term pinning and allows locality when replicas have caught up.

Load balancing & routing specifics- Use two logical pools: primary-write pool (single) and read-replica pool (many). Read pool entries carry metadata: region, lag_seconds, capacity.- Geo-aware LB prefers same-region replicas; fallback ranking = healthy & lag < soft threshold → other-region with lag OK → primary.- Employ rate-based balancing to avoid overloading replicas (circuit-breakers, max-connections per replica, slow-start).

Failover considerations- Detect primary failure via consensus of: failing healthchecks, no WAL activity, replicas promotion lag, and control-plane signals.- Use an orchestrated promotion process (Patroni/etcd for Postgres, MHA/Orchestrator for MySQL) with fencing: - Mark old primary as failed and block writes (IP/route revoke). - Elect a candidate replica that is most up-to-date (lowest lag, recent LSN), promote to primary. - Reconfigure proxies/LBs atomically (API/config push) to route writes to new primary; update DNS with low TTL only if necessary. - Ensure replicas point to new primary and resume replication.- Prevent split-brain: require quorum consensus before promoting; use external lease (consul/etcd/zookeeper) for leader lock. Use automated rollback if partition heals and two primaries would exist.

Operational and SRE practices- SLOs: define acceptable read staleness (e.g., 95th percentile < 200ms lag) and availability for writes/reads.- Runbook: automated scripts to promote, manual override options, post-failover checks.- Chaos testing: periodic failover drills and replication-lag injection tests.- Observability: dashboards for per-region lag, replica health, promotion events, and alerts for anomalous patterns.

Trade-offs- Sticky-to-primary is simplest but increases load and latency for geo users.- LSN-based routing gives strong RAW without fully pinning but requires reliable LSN propagation and proxy logic.- Automated failover reduces MTTR but needs robust fencing and testing to avoid split-brain.

This design balances locality and scalability while giving deterministic options for read-after-write guarantees and safe, observable failover.

MediumSystem Design

57 practiced

System-design: You operate an Envoy-based edge fleet serving 50k RPS for a region backed by 500 instances of a service. Design a health-checking and failover strategy so Envoy and the central control plane avoid large flapping events during rolling upgrades. Describe active/passive checks, backoff/jitter for health probes, how to handle transient failures, and how to alert on degraded capacity without false positives.

Sample Answer

Requirements:- Maintain 50k RPS across region with 500 instances.- Avoid mass flapping during rolling upgrades.- Fast failover for truly unhealthy hosts, tolerant to transient errors.- Accurate alerts for degraded capacity without false positives.

High-level approach:- Combine active (Envoy health checks) and passive (circuit-breaker / outlier detection + control-plane health signals).- Add server-side readiness markers pushed to control-plane during deploys.

Active checks (Envoy side):- HTTP/GRPC liveness & readiness probes to a lightweight /health endpoint.- Probe cadence: base interval 1s, timeout 500ms, consecutive failures before marking unhealthy = 3 (gives ~3s detection).- Use exponential backoff with jitter on re-probes after failure: backoff = min(1s * 2^n, 30s) with ±20% jitter to avoid synchronized retries.

Passive checks:- Envoy passive health via consecutive stream failure counts, 5 failures within 10s triggers ejection.- Outlier detection: consecutive 5xx or connect-failures => ejection for 60s; ejection time growth with factor 2 on repeated ejections up to 30m.- Use success rate tracking (sliding window) to detect degraded performance vs binary down.

Control-plane integration:- Hosts report readiness to control-plane (e.g., envoy/external health API). During rolling upgrades, orchestrator sets readiness=false before draining.- Control plane only removes hosts from config when both (a) active checks fail AND (b) control-plane mark absent or passive ejection observed — prevents mass removal due to transient probe spikes.

Handling transient failures:- Require multiple independent signals to evict: active probe failures + passive failures OR health=false from host.- Stagger rolling upgrades using throttles (e.g., 5% of instances at time) and use gradual ramp with health gating.

Alerting and degraded capacity detection:- Alert on capacity-based signals not raw host count: compute effective capacity = sum(weight_i * healthy_fraction_i).- Trigger warning when effective capacity < SLO threshold (e.g., 110% of target headroom) for >2min.- Use multi-window smoothing (1m and 5m) and require correlated metrics: increased latency, error-rate, and reduced healthy capacity to fire P1.- Suppress alerts during known deploy windows or when orchestrator signals rolling update in progress.

Trade-offs:- Slower detection vs fewer false positives: chosen parameters favor stability; you can tune for faster failover if risk tolerance lower.- Complexity increased by requiring multi-signal decisions, but reduces large flaps and control-plane churn.

Monitoring:- Dashboards: healthy host count, ejection events, probe failure histogram, deploy window flag, effective capacity, request latency and error rates.- Add automated rollback if effective capacity drops below emergency threshold (e.g., <70%) during upgrade.

This blend of active/passive checks, backoff+jitter, multi-signal eviction, rollout throttling and capacity-aware alerting minimizes flapping while ensuring timely failover.

HardTechnical

58 practiced

Outage detection: Propose an algorithm and specific monitoring queries (e.g., PromQL or Elasticsearch DSL) that detect a partial outage affecting only a subset of users (for example, clients from a single ISP or region) using request traces, metrics, and error budgets. Include example PromQL queries and how you reduce false positives from noise.

Sample Answer

Algorithm (high level)1. Group requests by client dimension (client_region, client_asn, client_ip_prefix).2. Compute short-term and baseline (rolling) metrics per group: error rate, p95 latency, request volume, unique users.3. Detect anomalies when short-term deviates from baseline beyond thresholds AND impact meets minimum size (traffic and unique users).4. Correlate with traces: verify increased error traces or failed downstream calls for the same group.5. Require sustained signal (e.g., 5m continuous) and cross-metric agreement before firing alert; integrate with error budget burn rules to escalate.

Example PromQL queries (assumes request labels: job="frontend", client_region, client_asn, code, le for histograms)

1) Error rate by region (1m rate, grouped)

sum by (client_region) (rate(http_requests_total{job="frontend"}[1m]) and on(client_region) sum by(client_region) (rate(http_requests_total{job="frontend", code=~"5.."}[1m])))
/
sum by (client_region) (rate(http_requests_total{job="frontend"}[1m]))

Simpler:

sum by(client_region) (rate(http_requests_total{job="frontend", code=~"5.."}[1m]))
/
sum by(client_region) (rate(http_requests_total{job="frontend"}[1m]))

2) p95 latency by region (histogram_quantile on 5m window)

histogram_quantile(0.95, sum by (le,client_region) (rate(http_request_duration_seconds_bucket{job="frontend"}[5m])))

3) Short-term vs baseline z-score (example: compare 5m error rate to 1h baseline mean/std)Short-term error rate:

err_5m = sum by(client_region) (rate(http_requests_total{job="frontend", code=~"5.."}[5m])) / sum by(client_region) (rate(http_requests_total{job="frontend"}[5m]))

Baseline mean and std (using recording rules or Prometheus functions via external math; approximate with Prometheus avg_over_time):

err_1h_mean = avg_over_time((
  sum by(client_region)(rate(http_requests_total{job="frontend", code=~"5.."}[5m])) /
  sum by(client_region)(rate(http_requests_total{job="frontend"}[5m]))
)[1h:5m])
err_1h_std = stddev_over_time((...)[1h:5m])  # same expression

Then compute z-score and threshold outside PromQL by alertmanager or use alert rule with simple ratio increase:

(alert when err_5m > err_1h_mean + 3 * err_1h_std) AND (sum by(client_region)(rate(http_requests_total{job="frontend"}[5m])) > 10)

4) Unique users affected (using a cardinality metric from instrumentation or approximate via labels)If you export user_id as label (careful with cardinality), then:

count_values("user", http_requests_total{job="frontend", code=~"5.."}[5m]) by (client_region)

Or use a sketch/bitmap exported (e.g., Prometheus quantile/TDigest not ideal — recommend using Redis/Druid or export approximate cardinality).

Trace correlation- Use tail-based sampling for traces matching client_region with elevated errors. Query tracer backend (Jaeger/Elastic APM) for traces where client_region=X and span error=true in last 10m. Verify common failing downstream service or error type to reduce false positive.

Alerting rule pattern (pseudo)- Condition A: err_rate_region > baseline + delta for >= 5m- Condition B: p95_latency_region > latency_baseline * 2 for >= 5m OR trace error count for region increased- Condition C: traffic_volume_region >= min_requests (avoid noise)- Fire alert when A AND (B OR trace-confirmation) AND C.- Auto-escalation if error budget burn rate over window exceeds SLO thresholds.

Reducing false positives- Require sustained windows (>= 5-10m) and majority of sub-metrics agreeing.- Minimum impact thresholds: min request/sec and min unique users.- Exclude known maintenance windows and deploy windows (annotate metrics).- Use rolling baseline (hour/day) and z-score to detect true deviations vs diurnal patterns.- Correlate independent signals: metrics + traces + logs (e.g., increase in error logs matching client_region).- Debounce/flapping suppression in Alertmanager; include automatic re-check query before paging.- Use aggregated group keys (region/asn/prefix) to avoid single noisy client label spikes.- Periodically validate rules with synthetic traffic to ensure sensitivity and tune thresholds.

Why this works- Grouping by client dimensions isolates partial outages.- Comparing to rolling baseline and requiring multiple corroborating signals reduces noise-driven false positives.- Trace correlation provides causal evidence (error types, failing downstream) to prioritize incidents and reduce MTTR.

Unlock Full Question Bank

Get access to hundreds of Load Balancing, Failover, and Fault Tolerance interview questions and detailed answers.

Join thousands of developers preparing for their dream job.