Load Balancing, Failover, and Fault Tolerance Questions

Understand load balancing strategies (round-robin, least connections, consistent hashing, weighted load balancing). At Staff Level, understand the trade-offs between different strategies and when each is appropriate. Master failover mechanisms, service discovery, and circuit breakers. Understand concepts like graceful degradation, bulkheads (service isolation), and how to design systems that remain operational when components fail. Be comfortable discussing health checks, monitoring, and alerting strategies to detect failures and trigger failover.

EasyTechnical

48 practiced

Explain consistent hashing and how it minimizes key remapping when nodes are added or removed. Describe why virtual nodes are used and provide a short example of how consistent hashing benefits a distributed cache or CDN when a node fails or is added.

HardTechnical

52 practiced

Root cause analysis exercise: after a partial outage your system experienced a retry storm and cascading failures. Lay out a postmortem plan: what logs, metrics, and traces to collect, how to determine the source of retries, and which short-term and long-term controls to add to prevent recurrence.

Sample Answer

Plan overview: perform a structured postmortem to reconstruct timeline, identify root cause of the retry storm, and implement short- and long-term controls to prevent recurrence.

1) Data collection (what to gather)- Logs: application logs (error, warn, info) with request IDs/correlation IDs, client-side logs (if available), load balancer / API gateway logs, upstream/downstream service logs, database and cache logs, autoscaler/infra logs, and deployment/change logs (CI/CD).- Metrics: request rate, error rate (4xx/5xx), latency percentiles (P50/P95/P99), retry counts per client/service, queue depths, CPU/memory, connection errors, throughput, circuit-breaker/open-circuit metrics, and rate-limit counters.- Traces: distributed traces (spans with service names, timings, parent-child) to follow failed requests end-to-end.- Alerts & incidents: timestamps of alerts, operator actions, and any throttling rules triggered.

2) Reconstruct timeline & scope- Build a timeline: mark when partial outage started, first error spike, retry increase, autoscaling/degarde events, cascading failures, and recovery.- Identify affected customers/endpoints, geographic scope, and services involved.

3) Determine source of retries- Use traces to find the first service that issued repeated retries (look for repeated outgoing spans with identical parent trace and increasing retry count).- Correlate logs with request IDs to see originating client IP/service.- Inspect client libraries configurations (default retry policies) and load balancer behavior (timeouts that may cause clients to retry).- Check for misconfigured timeouts: if client timeout < server processing time, clients may retry prematurely.- Verify backoff behavior: are retries exponential with jitter or immediate flood?- Look for a recent deploy/change that modified error handling, retry logic, or timeout values.

4) Short-term mitigations (stopgap, within hours)- Throttle or disable aggressive retries at ingress (API gateway) and apply per-client rate limits.- Enable circuit breakers or open circuits for overloaded downstream services to fail-fast.- Increase server timeouts temporarily if safe to avoid client-side premature retries.- Add ejection of unhealthy hosts from load balancer and reduce connection reuse if stuck.- Roll back recent deploys suspected as cause.- Put a temporary global or service-level retry cap (max attempts per minute per client).

5) Long-term controls (prevent recurrence)- Enforce standardized retry libraries with exponential backoff + jitter and idempotency tokens.- Align timeout hierarchies: ensure client timeouts > server worst-case processing time and consider server-side hedging carefully.- Add observability: track per-client retry rates, retry budget metrics, and alerts for retry spikes.- Implement and tune circuit breakers, bulkheads (isolate service pools), and rate limiting per tenant.- Add integration tests for retry/timeouts behavior and chaos tests that simulate partial failures to verify backoff and circuit-breakers.- Add automatic mitigation playbooks: automated throttles or dynamic backpressure when retry thresholds exceed safe limits.- Postmortem action items: assign owners, deadlines, acceptance criteria, and schedule a follow-up to verify fixes (runbook updates, monitoring dashboard additions).

6) Validation & learning- Run a replay or synthetic traffic tests to validate fixes.- Hold a blameless postmortem meeting, publish findings, and update runbooks and on-call procedures to include steps to detect and mitigate retry storms quickly.

This approach ensures evidence-driven root-cause identification and layered defenses (observability, behavioral controls, and architectural isolation) to prevent future cascading failures.

MediumSystem Design

47 practiced

Design a service mesh-based approach to provide load balancing, observability, and fault tolerance across microservices. Explain the roles of sidecar proxies, the control plane, policy enforcement for retries/circuit-breakers, and how the mesh should interact with edge load balancers.

Sample Answer

Requirements:- Provide transparent per-service load balancing, observability (metrics/tracing/logs), and fault tolerance (retries, timeouts, circuit breakers, outlier detection) across microservices with minimal app changes and secure inter-service traffic.

High-level architecture:- Sidecar proxies (Envoy) deployed alongside each service pod- Control plane (e.g., Istio/Consul/Linkerd) to manage config, policies, and certificate lifecycle- Ingress gateway (edge Envoy) behind external load balancer- Telemetry backends: Prometheus, Jaeger/OTel, ELK

Core components & responsibilities:1. Sidecar proxy (Envoy)- Handles L7 routing, client-side load balancing, retries, circuit-breakers, timeouts, mTLS termination- Emits metrics, traces, and access logs; collects health/outlier signals- Implements traffic shifting, locality-aware LB, and connection pooling

2. Control plane- Centralizes intent: routing rules, retry/circuit-breaker policies, rate limits, auth- Distributes configs via xDS APIs to sidecars- Issues mTLS certs and rotates them (SPIRE/Cert-Manager)

3. Policy enforcement- Define SLO-driven retry policy (max attempts, per-try timeout, backoff)- Circuit-breaker per upstream: max connections, pending requests, consecutive errors threshold -> open state- Outlier detection (eject hosts with high error rates)- Rate limiting at sidecar + ingress for burst control

Data flow:External LB -> Ingress Envoy (edge) which enforces edge auth, rate limits, TLS termination -> routes to service-side sidecars -> intra-mesh mTLS and routing -> upstream sidecar -> application.

Observability:- Sidecars emit Prometheus metrics (request latencies, error rates, retries), structured access logs, and traces (context propagation via W3C/OT).- Control plane aggregates config-change events and can produce topology views.- Use dashboards/alerts for latency/error thresholds to drive circuit-breaker tuning.

Scalability & resilience:- Keep control plane stateless where possible; scale horizontally.- Push config via xDS so sidecars operate even if control plane is down.- Use locality-aware LB and consistent hashing for sticky sessions where needed.- Edge LB performs coarse balancing (distribute ingress across gateway replicas); per-service LB logic lives in the mesh for fine-grained decisions.

Trade-offs:- Added complexity and resource overhead per pod (sidecar CPU/memory).- Control plane centralization introduces blast radius—mitigate via RBAC, canary config rollout.- For low-latency extreme workloads, tune sidecar and retry settings carefully to avoid amplification.

Example policy snippet (conceptual):- Retry: max_attempts=3, per_try_timeout=200ms, backoff=exponential- Circuit-breaker: max_connections=100, max_pending=50, failure_threshold=5 in 1m -> eject 30s

Why this works:- Sidecars localize network intelligence making apps unchanged.- Control plane centralizes intent and security (mTLS), while sidecars provide runtime enforcement and telemetry.- Edge LB + ingress gateway separates perimeter concerns (DDoS, TLS, WAF) from service-to-service concerns (retries, LB, observability).

MediumTechnical

40 practiced

Explain the circuit breaker pattern. Describe the circuit states (closed, open, half-open), the metrics and thresholds used to transition between states, and how to tune those thresholds in production. Give a concrete example for protecting calls to an external payments API.

Sample Answer

The circuit breaker pattern prevents cascading failures by failing fast when an external dependency is unhealthy. It has three states:

- Closed: calls are allowed; the breaker monitors success/failure metrics.- Open: calls are rejected immediately (or return a fallback); a timer starts.- Half‑open: after the timeout, a limited number of trial requests are allowed to check recovery; success moves to Closed, failures back to Open.

Metrics and thresholds:- Failure rate (%) over a rolling window (e.g., errors / total requests in last N seconds). Transition Closed→Open when failure rate > threshold (e.g., 50%) and minimum request volume reached (e.g., ≥ 20 requests).- Consecutive failures: simpler approach—open after K consecutive failures.- Open duration: how long to stay Open before Half‑open trial (e.g., 30s–2m).- Half‑open probe limits: max concurrent trial requests (e.g., 1–5) and required success ratio to close (e.g., 80% over M trials).

Tuning in production:- Start conservative: lower sensitivity (higher failure threshold, larger min volume) to avoid flapping.- Use metrics: monitor latency, error types (timeouts vs 5xx), and traffic volume. If most errors are transient timeouts, shorter open duration and fewer trials help; if errors indicate load issues, increase open duration.- Canary and gradual rollouts: enable on a subset of traffic; use feature flags.- Automatic vs manual overrides: provide dashboards and manual “force open/close”.- Observe downstream impact: tune to minimize user-visible errors while protecting system capacity.

Concrete payments API example:- Rolling window: 60s, min volume 30 requests.- Threshold: open if failure rate ≥ 40% or 10 consecutive timeouts.- Open duration: 60s.- Half‑open: allow 3 probe requests; require ≥2 successes to close.Behavior: If the payments gateway returns many 502/timeout errors, the breaker opens quickly and your service returns a cached decline or an async retry/fallback payment flow — protecting your workers and preventing retries that would further overload the gateway. Monitor payment success rates and customer impact to iteratively adjust thresholds.

MediumTechnical

54 practiced

Implement a weighted round-robin scheduler in Python that distributes requests proportionally to server weights and supports dynamic weight updates without restarting the scheduler. Describe your algorithm and its time and space complexity.

Sample Answer

Approach:Use the Smooth Weighted Round Robin algorithm (like Nginx). Each server has static weight and a runtime current_weight. On each pick, increment each server's current_weight by its weight, pick the server with max current_weight, then subtract total_weight from that server's current_weight. This yields proportional distribution and is O(n) per pick but produces smooth proportionality. To support dynamic updates, maintain a dict of servers and update weights/adding/removing at runtime; adjust total_weight accordingly and keep current_weight for continuity.

python

from typing import Dict, Optional

class Server:
    def __init__(self, name: str, weight: int):
        self.name = name
        self.weight = max(0, int(weight))
        self.current = 0  # runtime current_weight

class SmoothWeightedRoundRobin:
    def __init__(self, servers: Dict[str, int] = None):
        self.servers: Dict[str, Server] = {}
        self.total_weight = 0
        if servers:
            for name, w in servers.items():
                self.add_or_update(name, w)

    def add_or_update(self, name: str, weight: int):
        # Add new server or update weight dynamically
        if name in self.servers:
            old = self.servers[name].weight
            self.servers[name].weight = max(0, int(weight))
            self.total_weight += (self.servers[name].weight - old)
        else:
            s = Server(name, weight)
            self.servers[name] = s
            self.total_weight += s.weight

    def remove(self, name: str):
        if name in self.servers:
            self.total_weight -= self.servers[name].weight
            del self.servers[name]

    def pick(self) -> Optional[str]:
        if not self.servers or self.total_weight == 0:
            return None
        # Increase current by weight for all, find max
        best = None
        for s in self.servers.values():
            s.current += s.weight
            if best is None or s.current > best.current:
                best = s
        # Subtract total_weight from chosen server's current
        best.current -= self.total_weight
        return best.name

Key points:- Smooth WRR gives proportional and smooth distribution (less bursty).- Dynamic weight updates supported via add_or_update/remove; total_weight maintained.- If weights change drastically, current values preserve relative state for continuity.

Time & Space Complexity:- pick(): O(n) time where n = number of servers (must inspect each server); O(n) space for server state.- add_or_update/remove: O(1) average (dict ops), adjusting total_weight.

Edge cases:- All weights zero -> pick() returns None.- Negative weights -> normalized to 0.- Frequent large n and very high request rate -> consider heap or sharding to reduce per-pick cost, or use consistent hashing + weighted token buckets for O(log n) or O(1) alternatives.

Alternatives:- Weighted random with alias method (O(1) pick, O(n) rebuild on weight changes).- Token bucket or consistent hashing for sticky sessions and large-scale distributed routing.

Unlock Full Question Bank

Get access to hundreds of Load Balancing, Failover, and Fault Tolerance interview questions and detailed answers.

Join thousands of developers preparing for their dream job.