Load Balancing and Traffic Management Questions
Distributing requests across capacity: load-balancing algorithms (round-robin, least-connections, consistent hashing), L4 versus L7 balancing, health checks, and traffic shaping. Covers sticky sessions, canary and blue-green routing, rate limiting, and graceful draining. The traffic-distribution layer that keeps a scaled system balanced and available.
Design an automated control loop that shifts a percentage of production traffic to a canary over a fixed window and rolls back automatically on an SLO breach. Cover how you would smooth the weight changes, what guardrails you'd set (minimum observation windows, maximum error thresholds), and how the rollback itself executes quickly and safely.
Sample Answer
Direct answer
An automated canary control loop is a periodic process that increases a canary's traffic weight in small, guarded steps, checks health metrics against the baseline after every step using a minimum-observation window, and holds or immediately rolls back to zero on an SLO breach (a violation of the service-level objective, the reliability target you've committed to, such as 99.9% success rate). The design problem is really about the guardrails and the rollback path, not the ramp itself.
Structured elaboration
Components:
- A traffic router (Envoy, an ingress controller, or a cloud LB) that supports weighted routing and an instant weight-update API.
- A telemetry pipeline producing short and long rolling windows of error rate, latency, and request volume, split by canary vs. baseline.
- A stateless controller that runs the step/check/decide loop and calls the router's weight API.
- A rollback path that can zero out canary weight immediately, independent of the step loop.
Guardrails (the part that actually matters):
| Guardrail | Purpose | Example setting |
|---|---|---|
| Minimum observation window | Prevents decisions on too little data | require >= 100 canary requests in the last 60s before advancing |
| Short window (fast signal) | Catches sharp regressions quickly | 60s rolling window |
| Long window (stability) | Filters transient noise, confirms sustained breach | 300s rolling window |
| Max error delta | Hard stop on quality regression | canary error rate > baseline + 0.5 pp |
| Max latency ratio | Hard stop on tail-latency regression | canary p99 > baseline p99 x 1.2 |
| Monotonic ramp | Never sneak weight back up after a hold | only increase, decrease only via rollback |
| Cooldown after rollback | Prevents a flapping loop from re-triggering immediately | 30 minute freeze + incident ticket |
Smoothing the weight changes. Compute a fixed step size from the target ramp, then apply the step in small sub-increments at the router level (e.g. over a few seconds) so a single jump doesn't itself cause a connection-pool or cache-warming spike.
Worked example
Target: shift 10% of production traffic to canary over a 10 minute (600s) window, checking every 30s.
steps=30s600s=20 step size=2010%=0.5% per stepSo the loop runs 20 times, adding 0.5 percentage points of canary weight each pass, provided the guardrails above all pass. If step 12 (weight = 6%) shows canary p99 = 340ms against a baseline p99 of 260ms:
260340≈1.31>1.2That exceeds the 1.2x latency-ratio guardrail, so the controller does not advance to step 13. It instead sets weight back to 0% immediately (not a gradual decrease), opens an incident, and freezes further attempts for the cooldown period.
# runs every 30s
current = get_current_weight()
target = min(10, current + 0.5)
m = fetch_metrics(short=60, long=300)
if m.canary_requests_short < 100:
return # not enough signal yet, hold at current weight
if m.error_rate_canary > m.error_rate_baseline + 0.005 or \
m.p99_canary_long > m.p99_baseline_long * 1.2:
set_weight(0) # immediate rollback, not gradual
open_incident(m)
start_cooldown(minutes=30)
else:
set_weight(target) # router ramps this smoothly over a few seconds
Extensions: downstream capacity and CI/CD integration
Downstream-capacity-constrained ramp. The guardrails above are all canary-side (error rate, latency). A canary can look perfectly healthy on its own metrics while the traffic it forwards saturates a downstream dependency, a connection pool, a queue, or a rate-limited third-party API, before that dependency's own health signal even reflects the problem. Add a downstream capacity ceiling as its own guardrail: before each step, check the downstream dependency's current utilization (connection-pool saturation, queue depth, or a published capacity headroom number) and cap the canary weight so its incremental load stays under that ceiling, independent of what the canary's own error and latency metrics say. In practice the step size becomes the smaller of the scheduled step and the downstream headroom divided by requests-per-weight-point, and downstream saturation becomes its own rollback trigger alongside the error and latency guardrails, since the canary's own SLOs will not catch a downstream problem until it is already failing.
CI/CD pipeline integration. The loop is usually not a standalone daemon, it is a stage in the deploy pipeline. A build that passes its test stage triggers the deploy, which starts the canary at 0% weight and hands control to this loop; the pipeline blocks promotion to the next stage (wider rollout, or the next environment) until the loop reports a verdict, not a fixed timer. On success the loop reports pass and the pipeline proceeds to full rollout. On a guardrail breach, the loop's immediate zero-weight rollback from the worked example above is reported back as a failed pipeline stage: the deploy is marked failed, the previous version stays serving at 100%, and the pipeline stops rather than continuing to the next stage. That makes the human's only manual step reviewing the failure, not remembering to check on the canary.
Trade-offs and pitfalls
- Short windows are noisy, long windows are slow. A short-only window trips on transient blips; a long-only window lets real regressions run for minutes before anyone notices. Using both, and requiring the long window to confirm, is the standard resolution.
- Rollback must never itself be a slow ramp. The instinct to "gradually" bring weight back down defeats the purpose; on breach, cut to zero immediately and investigate after.
- Minimum-request guardrails matter more for low-traffic services. A service doing 50 RPS total needs a longer observation window or a lower canary weight cap just to get statistically meaningful samples per step.
- Business metrics can lag system metrics. A checkout canary can look perfectly healthy on latency and error rate while conversion silently drops; if the metric that matters is business-level, it needs its own window and threshold, not just infra SLOs.
- This is inherently single-service. Running the same loop concurrently across many dependent services without coordination risks compounding partial failures across a call graph; large orgs typically centralize this into a shared canary-analysis service rather than one loop per team.
Compare consistent hashing and rendezvous (highest-random-weight) hashing. How does each handle node addition and removal, what is the cost of rebalancing, and which would you use for cache routing versus partitioned storage with heterogeneous node weights?
Sample Answer
Direct Answer
Consistent hashing (CH) places nodes and keys on a hash ring and routes each key to the next node clockwise; rendezvous hashing (also called highest-random-weight, HRW) scores every node for a given key with an independent hash and routes to the node with the highest score. Both move only a small, bounded fraction of keys when the node set changes, but they differ in what they need to operate: CH needs a maintained ring data structure (and virtual nodes to balance load), while HRW needs nothing shared at all, it recomputes from scratch on every lookup. That operational difference, not raw performance, usually decides which one you pick.
How Each Handles Node Changes
Consistent hashing. Nodes (or their virtual tokens) occupy positions on a ring. A key is owned by the first node clockwise of its hash position. Adding a node inserts a new position and reassigns only the keys that fall in the arc between the new node and its counterclockwise neighbor. Removing a node deletes its position, and in a plain single-token ring, every key that node owned is reassigned entirely to its clockwise successor. That successor absorbs the whole load, it is not spread across the rest of the cluster. This is why production consistent-hash rings almost always use virtual nodes: giving each physical node many tokens scattered around the ring means a removal's load lands on many different successors instead of one.
Rendezvous hashing. For a key, each node's score is an independent draw from a hash function, and the node with the maximum score wins. Adding a node adds one more independent score into the comparison; only keys where the new score happens to be the maximum move, and they move only to the new node. Removing a node drops one score out of the comparison; for every key that node used to win, the new winner is whichever of the remaining nodes had the second-highest score, which, because all scores are independently drawn from the same distribution, is uniformly distributed across the survivors. Rendezvous hashing gets even redistribution on removal for free, without virtual nodes.
Cost of Rebalancing
| Operation | Consistent hashing (single-token) | Consistent hashing (with V virtual nodes/node) | Rendezvous hashing |
|---|---|---|---|
| Node join | Only the new node's arc moves | Only the new node's arc(s) move | Only keys where the new score wins move |
| Node leave | All of the removed node's keys land on one successor | Removed node's keys spread across roughly V nearby successors | Removed node's keys spread uniformly across all survivors |
| Lookup cost | O(log T) with T ring tokens (binary search) | O(log T), T grows with V | O(N) hash evaluations per key, N = node count |
| Metadata to maintain | Sorted ring of N positions | Sorted ring of N*V positions | None beyond the current node list |
Worked Example
Take a cluster of N=10 nodes holding keys uniformly on the ring, so each node owns an expected 101 of the keyspace.
Join (both schemes). Adding one node makes it 1 of 11 exchangeable ring owners (CH) or 1 of 11 exchangeable score-holders (HRW). By symmetry, the new node's expected share is:
N+11=111≈9.1%and every other node's share is unaffected in expectation, for both schemes.
Leave, plain single-token consistent hashing. The removed node held N1=101=10% of keys. All of it moves to one clockwise successor, whose share becomes:
101+101=102=20%That successor now carries double its prior load, a hotspot, while the other 8 surviving nodes are untouched.
Leave, rendezvous hashing (or CH with enough virtual nodes to approximate it). The removed node's 10% share spreads uniformly across the 9 survivors. Each survivor's new share:
101+10×91=909+1=91≈11.1%which is exactly the uniform N−11 share you would expect if the whole keyspace were freshly and evenly split across 9 nodes. This algebra generalizes: N1+N(N−1)1=N−11, confirming that rendezvous hashing (and a well-provisioned virtual-node ring) reach the ideal uniform post-removal distribution automatically, while a plain single-token ring does not.
Which to Use
For cache routing (many roughly similar-capacity nodes, request-level lookups, simplicity matters, weights may differ per node's memory size), rendezvous hashing is usually the simpler choice: no ring to maintain, exact per-node weighting by scaling scores, and even load redistribution on failure without tuning a virtual-node count. The O(N) lookup cost is rarely the bottleneck unless N reaches the high hundreds or thousands of cache nodes.
For partitioned storage with a large number of partitions (sharded databases, distributed key-value stores), consistent hashing with virtual nodes is usually preferred: ownership maps to contiguous ranges, which supports range scans and simple "move this token's range" rebalancing tooling, and lookup stays O(log T) even as the ring grows large. Heterogeneous weights are handled by giving higher-capacity nodes proportionally more virtual nodes, though that is a discrete approximation rather than an exact ratio.
Trade-offs and Pitfalls
- The single-token consistent-hashing hotspot on removal (shown above) is a frequent interview trap: candidates who describe CH without mentioning virtual nodes are implicitly describing a scheme with an uneven-failover problem.
- Rendezvous hashing's O(N) per-lookup cost is a real scaling limit for very large node counts; production systems mitigate it with a hierarchical or bounded-candidate variant rather than switching to CH purely for that reason.
- Weighting is exact and continuous with rendezvous hashing (scale each node's score by its weight) but only approximate with CH virtual nodes (weight is quantized by how many tokens you assign).
- Neither scheme is free of the "every key touches every node during full re-sharding" cost if you ever need to change the hash function itself (e.g., migrating hash algorithms); that requires a coordinated dual-write or shadow-ring migration regardless of which scheme you use.
You manage a pool of stateless web servers behind a single load balancer that sees bursty traffic and heterogeneous server capacity. Compare round robin, least connections, and weighted load balancing for this situation: how does each pick a server, and which would you choose?
Sample Answer
Direct answer
Round robin, least connections, and weighted load balancing differ in what information they use to pick a server. Round robin uses none, it just cycles through the pool. Least connections uses current load (active connection count) but assumes every server has equal capacity. Weighted load balancing (typically combined with one of the other two, giving weighted round robin or weighted least connections) uses declared server capacity. For a pool with bursty traffic and heterogeneous capacity, weighted least connections is the right default, because it is the only one of the three that reacts to both current load and known capacity differences.
How each one picks
| Algorithm | Selection rule | Reacts to load? | Reacts to capacity? |
|---|---|---|---|
| Round robin | Next server in a fixed rotation | No | No |
| Least connections | Server with fewest active connections | Yes | No (treats all servers as equal) |
| Weighted (least connections) | Server with the lowest active-connections-to-weight ratio | Yes | Yes |
Why plain round robin and plain least connections both fall short here
Round robin sends the same share of traffic to every server regardless of what it can handle, so a small server gets hit exactly as often as a large one and becomes the bottleneck under burst. Least connections is an improvement, since it avoids piling more work onto an already-busy server, but it still treats a 2-vCPU box and an 8-vCPU box as equally capable: an equal number of active connections is not an equal amount of work when the servers are different sizes.
Worked example: weighted least connections in action
Say the pool has three servers with declared capacity weights 4, 2, and 1 (roughly matching, say, an 8-vCPU, 4-vCPU, and 2-vCPU box). Weighted least connections picks, for each new request, whichever server minimizes:
scorei=wiciwhere ci is server i's current active connection count and wi is its weight. Ties are broken toward the higher-weight server. Tracing 7 back-to-back requests (a burst, so nothing completes mid-trace):
| Request | Scores (S1, S2, S3) | Picked | Active after |
|---|---|---|---|
| 1 | 0, 0, 0 (tie) | S1 | S1=1, S2=0, S3=0 |
| 2 | 0.25, 0, 0 (tie) | S2 | S1=1, S2=1, S3=0 |
| 3 | 0.25, 0.5, 0 | S3 | S1=1, S2=1, S3=1 |
| 4 | 0.25, 0.5, 1.0 | S1 | S1=2, S2=1, S3=1 |
| 5 | 0.5, 0.5 (tie), 1.0 | S1 | S1=3, S2=1, S3=1 |
| 6 | 0.75, 0.5, 1.0 | S2 | S1=3, S2=2, S3=1 |
| 7 | 0.75, 1.0, 1.0 | S1 | S1=4, S2=2, S3=1 |
After 7 requests the split is S1=4, S2=2, S3=1, exactly proportional to the declared weights of 4:2:1, even though every request arrived back-to-back with no completions in between. Plain round robin would have sent requests 1-7 as S1,S2,S3,S1,S2,S3,S1 (a 3:2:2 split), overloading S3 relative to its actual capacity.
Trade-offs and pitfalls
Weighted least connections needs two things plain least connections does not: accurate weights (usually derived from instance size or a load test, not guessed) and a mechanism to keep them current as capacity changes (autoscaling, degraded instances). It is also more stateful, the balancer has to track active connections per server accurately, which matters less for round robin. If request cost varies wildly and isn't reflected by connection count (a handful of expensive long-running requests versus many cheap ones), even weighted least connections can misjudge load, at which point request-cost-aware balancing or queuing becomes worth considering.
Implement a thread-safe least-connections scheduler. Provide addBackend(id), removeBackend(id), incConn(id), decConn(id), and selectBackend(), where selectBackend() returns the backend with the fewest active connections and ties are broken deterministically. Describe your concurrency strategy, its complexity, and how you would handle backends with very different capacities so the busiest small backend isn't starved.
Sample Answer
Direct answer
Keep a min-heap keyed not by raw connection count but by connection count divided by declared capacity (a load ratio), with a deterministic tie-break. Deletions and count updates are handled with lazy invalidation (a version token per backend) rather than an expensive heap-fix-and-remove, so every operation stays logarithmic and selectBackend never returns a stale or removed backend.
Approach
Raw least-connections (ranking purely by active connection count) starves small, low-capacity backends when capacities differ: a backend with capacity 1 and 1 active connection is fully loaded, but a raw min-heap would still prefer it over a capacity-4 backend already holding 2 connections, because 1 < 2. Dividing by capacity turns the heap key into a load ratio, so the comparison becomes "who has more headroom," which is what you actually want when capacities are heterogeneous.
For concurrency, a single lock protects the heap, the connection-count map, and a per-backend version counter. incConn/decConn bump the version and re-push a fresh heap entry rather than trying to fix the existing entry's position in place; the old entry is left in the heap but tagged with a now-stale version. selectBackend pops entries off the top and discards any whose version doesn't match the backend's current version, until it finds a live one. This is the standard lazy-deletion trick for heaps that need cheap updates: amortized cost stays O(log n) per operation, and stale entries are simply cheap to skip past rather than expensive to keep synchronized in place.
Code (Python)
import heapq
import itertools
import threading
class LeastConnScheduler:
def __init__(self):
self._lock = threading.Lock()
self._heap = [] # (load_ratio, conns, tie, id, version) tuples
self._conns = {} # id -> current connection count
self._capacity = {} # id -> declared capacity (default 1)
self._version = {} # id -> token that invalidates stale heap entries
self._tie = itertools.count() # deterministic, insertion-order tiebreak
def add_backend(self, backend_id, capacity=1):
with self._lock:
if backend_id in self._conns:
return
self._conns[backend_id] = 0
self._capacity[backend_id] = capacity
self._version[backend_id] = 0
self._push(backend_id)
def remove_backend(self, backend_id):
with self._lock:
if backend_id not in self._conns:
return
self._version[backend_id] += 1 # invalidates any stale heap entries for this id
del self._conns[backend_id]
del self._capacity[backend_id]
def inc_conn(self, backend_id):
with self._lock:
if backend_id not in self._conns:
return
self._conns[backend_id] += 1
self._version[backend_id] += 1
self._push(backend_id)
def dec_conn(self, backend_id):
with self._lock:
if backend_id not in self._conns:
return
if self._conns[backend_id] > 0:
self._conns[backend_id] -= 1
self._version[backend_id] += 1
self._push(backend_id)
def select_backend(self):
with self._lock:
while self._heap:
ratio, conns, tie, backend_id, ver = self._heap[0]
if backend_id not in self._conns or ver != self._version[backend_id]:
heapq.heappop(self._heap) # stale entry, discard and keep looking
continue
return backend_id
return None
def _push(self, backend_id):
conns = self._conns[backend_id]
capacity = self._capacity[backend_id]
ratio = conns / capacity
heapq.heappush(
self._heap,
(ratio, conns, next(self._tie), backend_id, self._version[backend_id]),
)
Key points
- Capacity-weighted ratio, not raw count, is the fix for the starvation case the question asks about. A backend with capacity 4 and 2 connections (ratio 0.5) is correctly preferred over a backend with capacity 1 and 1 connection (ratio 1.0), even though the second one has fewer raw connections.
- Deterministic tie-break comes from the monotonic insertion counter, not from backend id string comparison, so ties resolve consistently regardless of what ids happen to be in play.
- Lazy deletion avoids the classic heap problem:
heapqhas no efficient "update the priority of an existing item" operation, so instead of searching for and fixing an entry (which would be O(n)), a new entry is pushed and the old one is left to be skipped later, at the cost of some extra memory for stale entries between updates.
Complexity
add_backend,remove_backend,inc_conn,dec_conn: O(log n) for the heap push (remove is O(1) plus lazy cleanup deferred to laterselect_backendcalls).select_backend: amortized O(log n); each stale entry it skips was already paid for by the update that created it, so total work across all operations stays bounded.- Space: O(n) live entries plus O(u) stale entries, where u is the number of updates since the last full cleanup; stale entries are bounded by the number of
inc_conn/dec_conncalls, not unbounded.
Edge cases
dec_connnever takes a count below zero.inc_conn/dec_connon an unknown id are no-ops (verified below along with everything else).select_backendon an empty scheduler returnsNone.- A backend removed while it was at the top of the heap is correctly skipped on the next
select_backendcall, verified by the concurrency test.
Running a capacity-skew check (a small backend fully loaded versus a big backend partially loaded), a tie-break check, and a concurrency stress test with 8 threads each doing 3000 select/inc/dec cycles against 4 backends:
pick after small=1/1, big=2/4: big
tie-break pick (both idle): a
concurrency errors: []
final connection counts: {'n0': 0, 'n1': 0, 'n2': 0, 'n3': 0}
The first line confirms the capacity-skew fix works as intended: big (2 of 4 capacity used, ratio 0.5) is preferred over small (1 of 1 capacity used, ratio 1.0) even though small has fewer raw connections. The stress test completed with zero errors and connection counts correctly balanced back to zero.
Trade-offs and pitfalls
- Lazy deletion trades memory for update speed. In a system with very high inc/dec churn and rare removals, stale entries can accumulate; a periodic heap rebuild (or a max-stale-entry threshold that triggers one) bounds this in production.
- A single lock is the simplest correct answer but caps throughput under extreme contention. Sharding backends across multiple scheduler instances (consistent-hash the backend id to a shard) removes the single-lock bottleneck at the cost of only having a locally, not globally, least-loaded view per shard.
- Capacity must come from somewhere trustworthy. If
capacityis self-reported by a backend rather than measured or configured centrally, a misconfigured or malicious backend can claim a huge capacity and starve itself unfairly of protection, or claim a tiny one and hoard traffic away from itself; treat capacity as operator-configured or telemetry-derived, not backend-asserted. - Connection count is a proxy for load, not load itself. For workloads where connections vary wildly in cost (such as a backend pool handling a mix of 50ms lookups and multi-second batch requests, where two backends can show the same active-connection count while carrying very different amounts of real work), least-connections by itself, even capacity-weighted, is a weaker signal than actual measured latency or queue depth.
Why are load balancers used in distributed systems? Describe at least four distinct problems they solve, and how they contribute to horizontal scaling, availability, and fault isolation. Where would you typically place load balancers in a three-tier architecture (edge, internal service, service-to-service)?
Sample Answer
Direct answer
Load balancers exist to decouple how much traffic arrives from which single machine has to answer it. They solve at least four distinct problems: spreading load across available capacity so no single server bottlenecks, removing single points of failure through health-based failover, enabling safe rollouts such as canary (releasing a new version to a small slice of traffic first) and blue-green deployment (running old and new versions side by side, then cutting all traffic over at once) with graceful draining, and centralizing cross-cutting concerns like TLS termination and rate limiting. In a typical three-tier architecture, load balancers show up at the edge (public-facing traffic), between internal services (service to service), and inside a service mesh (sidecar to sidecar).
Structured elaboration
Named problems load balancers solve:
- Single-server bottleneck / horizontal scaling. Spreading requests across many instances lets the system add capacity by adding instances instead of a bigger single machine.
- Availability and automatic failover. Health checks remove failing instances from rotation so one bad node does not take down the service.
- Fault isolation and safe rollout. Canary and blue-green routing, plus connection draining during deploys, contain a bad release to a small slice of traffic instead of every user.
- TLS termination and centralized security policy. Certificates and security rules (WAF, Web Application Firewall: a layer that inspects and blocks malicious HTTP traffic before it reaches a backend, and rate limiting) live in one place instead of being duplicated on every backend.
- Rate limiting and overload protection. The balancer can shed or throttle excess traffic before it reaches backend capacity, protecting downstream services.
Placement in a three-tier architecture:
graph LR
Client[Client] --> EdgeLB["Edge LB (L7, TLS termination)"]
EdgeLB --> GwA[Gateway instance A]
EdgeLB --> GwB[Gateway instance B]
GwA --> InternalLB[Internal service LB]
GwB --> InternalLB
InternalLB --> SvcA[Service instance A]
InternalLB --> SvcB[Service instance B]
SvcA --> MeshLB["Sidecar LB (service-to-service)"]
SvcB --> MeshLB
MeshLB --> Downstream[Downstream service]
The edge tier handles public traffic and usually terminates TLS. The internal service tier balances across replicas of a given service, often behind a service discovery layer. The service-to-service tier (frequently a sidecar, a small proxy process deployed alongside each service instance, in a mesh) balances outbound calls one internal service makes to another, applying the same health-check and retry logic without a separate hop through a centralized balancer.
A fast way to spot gaps in an unfamiliar system's traffic layer: check whether all three tiers actually exist (a single edge LB fanning straight out to every backend is a common shortcut that skips the internal tier), whether health checks are active at each tier, and whether TLS termination and rate limiting are centralized or scattered across services.
Worked example
Three backend instances, each rated for 1,000 requests per second, pooled behind a load balancer: combined capacity is 3×1000=3000 requests per second. If health checks detect one instance failing and pull it out of rotation, capacity becomes 2×1000=2000 requests per second, so the system keeps serving at 2000/3000≈66.7% of its original capacity instead of dropping to zero, which is what a single unpooled instance would do on failure.
Trade-offs & pitfalls
- The load balancer itself becomes critical infrastructure. A single-instance balancer just moves the single point of failure up one layer; it needs its own high availability (active-active pair, DNS or anycast failover).
- Every additional tier (edge, internal, mesh) adds a proxy hop's worth of processing and another thing to operate; do not add a tier a system does not need.
- A misconfigured health check at any tier can silently remove that tier's benefit (failover, safe rollout) even though the balancer process is technically running.
Unlock Full Question Bank
Get access to all Load Balancing and Traffic Management interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.