Load Balancing and Traffic Management Questions

Direct answer

When the load balancer's p99 rises but the backends' p95 and CPU stay flat, the divergence is itself the clue: the extra tail latency is being added somewhere the backend can't see, the network path, the LB's own accept/TLS/queueing layer, or client behavior, not inside request processing. Work through the request path in layers (network, LB proxy internals, TLS, kernel accept/backlog, client) and instrument each layer's own latency contribution separately, so the tail latency is attributed to a stage, not guessed at.

Layered checklist

• Network path (client to LB, LB to backend): interface errors, drops, retransmits (ip -s link, netstat -s), VPC flow logs for SYN/retransmit spikes, targeted traceroutes from affected client geographies. A tail-latency-only symptom with no backend involvement often starts here.
• LB/proxy internals: connection counts and churn per backend, event-loop stalls or worker saturation in the proxy process itself (profiling, strace on accept), time-to-first-byte versus time-to-last-byte split per backend. A proxy that's CPU-saturated or GC-pausing (if it's a managed-runtime proxy) shows up here, invisible to backend CPU metrics.
• TLS handshakes: split full-handshake latency from resumed/session-ticket handshake latency, and track the resumption hit rate; a drop in session cache hit rate (cache eviction, a cache not shared across proxy instances, client churn) turns cheap resumed handshakes into expensive full ones for a subset of requests, exactly the shape of a tail-latency-only regression.
• Accept queue / kernel backlog: listen backlog occupancy (ss -ltn), SYN_RECV counts (connections stuck mid-handshake, waiting on the final ACK), somaxconn and tcp_max_syn_backlog limits, ephemeral port and TIME_WAIT counts. If the accept queue is intermittently near its limit, new connections queue briefly even though every connection that does get through is processed at normal speed, invisible to backend request-processing metrics by construction.
• Client behavior: slow or high-RTT clients, retry storms, or a small subset of very chatty clients; correlate p99 offenders by client IP, geography, or user agent rather than assuming they're uniform across all traffic.

Worked example: why queueing produces a p99 problem with a flat p95

This is a general queueing-theory property, illustrated with a simple M/M/1 model (queueing-theory shorthand: Markovian/memoryless request arrivals, Markovian/memoryless service times, 1 server) and pinned parameters, not a claim about any specific measured system. For a queue with service rate $\mu$ and utilization $\rho = \lambda/\mu$ (arrival rate $\lambda$), the expected wait time in queue is:

Fix $\mu = 1000$ (an arbitrary service-rate unit, just for the shape of the curve) and evaluate at three utilizations:

A move from $\rho=0.80$ to $\rho=0.995$ (a 24% increase in load) inflates queueing wait by roughly 50x. The requests that land during the brief windows where the accept queue or a proxy worker pool is momentarily near saturation (micro-bursts, GC pauses, a slow client holding a worker) are exactly the ones that generate the p99 tail, while the bulk of requests, arriving when the system is comfortably under its knee, still finish fast and keep p95 (and backend CPU, which averages over time) looking healthy. This is why p99 and p95 can diverge sharply even though nothing about the backend's steady-state processing changed: the tail is a queueing phenomenon, not a processing-time phenomenon.

Instrumentation to add

• Break end-to-end latency into stages, TCP connect, TLS handshake, proxy accept/queue wait, backend processing, response write, and emit each as its own histogram (not just the total), tagged by backend and client region.
• Accept-queue depth and backlog saturation as a time series, not just a point-in-time check, so a transient near-saturation event that self-resolves in under a second is still visible.
• TLS session-resumption rate as its own metric, separate from handshake latency, since a resumption-rate drop is a leading indicator of the handshake-latency problem.
• Kernel-level counters (SYN_RECV: mid-handshake connections, TIME_WAIT: recently closed connections still held by the kernel, retransmits) exported alongside application metrics on the same dashboard and timeline, so a kernel-layer cause doesn't require manually correlating two separate tools during an incident.
• Sampled packet captures triggered automatically when p99 crosses a threshold, so there's raw evidence from the actual bad window instead of trying to reproduce it after the fact.

Trade-offs and pitfalls

• Chasing this in backend-only dashboards is the classic dead end, since by construction the backend's own view (CPU, p95, request-processing time) is healthy; the cause lives in a layer that doesn't report through the backend's own telemetry.
• Averages and even p95 hide exactly this kind of problem by design, since it only affects a small fraction of requests, that's what makes it a p99 problem and not a p50 problem; don't let a "p95 looks fine" dashboard close the investigation.
• Fixing the wrong layer (e.g. scaling backend CPU when the problem is TLS resumption cache eviction) burns real money and doesn't move the metric; stage-by-stage instrumentation exists specifically to prevent this kind of misdiagnosis.
• Once queueing is confirmed as the mechanism, the fix is capacity or admission control (a bigger backlog, more proxy workers, load shedding, or reducing $\rho$ by scaling out), not code-level micro-optimization of request handling, since request handling was never where the time went.

Direct Answer

All three deployment strategies need the same underlying primitive when sticky sessions are involved: keep serving already-affinitized sessions from their old destination while steering new sessions to the new version, and validate with real traffic before fully committing. What differs is blast radius and rollback speed. Canary gives the smallest blast radius and fastest safe rollback since only a slice of traffic ever touches the new version. Blue-green gives the fastest full rollback, a single traffic flip back to the untouched old fleet. Rolling update is the middle ground: it uses less infrastructure than blue-green but exposes both versions to production traffic for the longest window.

Comparing the Three Approaches

Worked Example

If a canary receives 5% of instance capacity and that version has a latent bug, the fraction of live traffic that can be affected during the validation window is bounded above by that same 5%, by construction, since the routing weight is what determines exposure. A rolling update, by contrast, exposes a growing share of the fleet as each batch completes; if you roll in 10 batches of 10% each and something is only caught at the fourth batch, roughly 40% of the fleet has already been exposed by the time you halt, an order of magnitude more exposure than the canary case for the same underlying bug. This is the concrete reason canary is preferred for higher-risk changes even though it takes more total upfront tooling to run the automated gates.

Trade-offs and Pitfalls

• Blue-green doubles infrastructure cost for the duration of the cutover window, since two full fleets run simultaneously; this is the price of the fastest possible full rollback.
• Rolling update's extended window with two versions live simultaneously creates a real risk of version-skew bugs: if a sticky client reconnects mid-rollout and lands on a different version than its previous request, and the two versions disagree on session schema or API contract, that's a bug class that blue-green and canary largely avoid by keeping version boundaries sharper.
• Canary's small sample size is a double-edged sword: it bounds blast radius but also means a rare bug (one that only manifests on 1 in 1000 requests) may not surface at all before the canary is judged "clean" and promoted.
• Whichever strategy is used, connection draining timeouts need to be sized for the actual protocol involved; a WebSocket-heavy service needs a much longer drain window than a typical request-response API, and a drain window that's too short converts a graceful strategy into a de facto hard cut for anything still in flight.

Direct answer

Keep an immutable snapshot (tuple) of backends and a single lightweight lock that only protects a counter increment. get_next_backend reads the snapshot reference (a single atomic pointer read in Python, and cheap in most languages), takes the lock only long enough to advance an integer counter, and computes the index modulo the snapshot length outside the lock. add_backend/remove_backend build a new snapshot under a separate lock. This keeps the hot path (get_next_backend) to one tiny critical section instead of locking the whole backend list on every call.

Approach

Two separate locks serve two different purposes:

• A write lock around mutating the backend list, so add_backend/remove_backend never race with each other, and each mutation publishes a brand-new tuple (copy-on-write) rather than mutating a shared list in place.
• A counter lock around advancing the round-robin index, so concurrent get_next_backend calls never read the same index twice; this is the only lock the hot path pays for.

Reading the current backend tuple needs no lock at all: in CPython, and in most managed runtimes, reassigning a reference (self._backends = new_tuple) is atomic with respect to readers, a reader either sees the old tuple or the new one, never a partially-built one. If you needed this guarantee in a language without that property, you'd wrap the read in an atomic pointer/reference type instead of relying on the runtime.

Code (Python)

import threading
from itertools import count


class RoundRobinLoadBalancer:
    def __init__(self):
        self._backends = ()          # immutable snapshot, swapped under _write_lock
        self._write_lock = threading.Lock()
        self._counter = count()      # advanced only while holding _counter_lock
        self._counter_lock = threading.Lock()

    def add_backend(self, backend_id):
        with self._write_lock:
            if backend_id in self._backends:
                return
            self._backends = self._backends + (backend_id,)

    def remove_backend(self, backend_id):
        with self._write_lock:
            if backend_id not in self._backends:
                return
            self._backends = tuple(b for b in self._backends if b != backend_id)

    def get_next_backend(self):
        backends = self._backends       # single reference read, no lock needed
        n = len(backends)
        if n == 0:
            return None
        with self._counter_lock:
            i = next(self._counter)
        return backends[i % n]

Key points

• get_next_backend is the hot path and is designed to hold a lock for the shortest possible time: only the counter increment, not the modulo or the indexing.
• add_backend/remove_backend are copy-on-write: each call allocates a new tuple rather than mutating the existing one in place, which is what lets readers avoid locking. This is a deliberate trade of O(n) mutation cost for lock-free reads, correct because backend membership changes are far rarer than get_next_backend calls in a load balancer.
• The counter grows without bound; since it is only ever used modulo the current backend count, this is safe in Python (unbounded integers) and in any language with a 64-bit counter, the counter would need billions of requests per second sustained for years before wraparound became a practical concern.

Complexity

• get_next_backend: O(1) time, holds the counter lock for O(1) work only.
• add_backend/remove_backend: O(n) time to rebuild the tuple, under the write lock; this does not block concurrent readers.
• Space: O(n) for the backend snapshot.

Edge cases

• Empty backend list: get_next_backend returns None rather than raising, so callers must check for that.
• Removing a backend that the counter currently "points to" mid-rotation: correct by construction here, because the index is always taken modulo the snapshot's length at read time, so a shrink just means the sequence continues over the smaller set without needing to adjust the counter.
• Duplicate add_backend calls: no-op if the id already exists, verified in the sanity run below.
• Concurrent add_backend/remove_backend happening while many threads call get_next_backend: verified in the run below to complete without error and to only ever return currently-valid backend ids.

Running the implementation with a single-threaded rotation check, then 8 threads each making 2000 calls, then a concurrent add/remove stress test against 5000 reads:

single-thread sequence: ['a', 'b', 'c', 'a', 'b', 'c', 'a', 'b', 'c']
total picks: 16000
distribution: {'a': 5334, 'b': 5333, 'c': 5333}
concurrent add/remove during reads: OK, final backends = ('a', 'b', 'c')

16000 picks across 3 backends split 5334/5333/5333, confirming the rotation stays even under concurrent load, and the concurrent add/remove stress test completed with no exceptions and no invalid backend ever returned.

Trade-offs and pitfalls

• Copy-on-write is wrong if backends churn constantly (e.g., hundreds of adds/removes per second from a very chatty autoscaler): the O(n) rebuild cost on every mutation would dominate. In that regime, a version-counter-plus-fixed-array design or a lock-free ring buffer would fit better.
• A single global lock around everything is a common but weaker first instinct. It's simpler to reason about, but every get_next_backend call then contends with every other call and with every mutation, which is exactly the contention this design avoids.
• This is not linearizable across add/remove and reads happening at the "same instant." A reader can legitimately see the backend list either just before or just after a concurrent mutation; if a caller needs strict linearizability (every operation appears to happen at a single global point in time), that requires a stronger primitive (a single mutex around all operations) at the cost of read throughput.

Direct answer

Active checks are probes the load balancer or service discovery system initiates on a schedule; passive checks are inferences drawn from real request outcomes, such as a 5xx response, a timeout, or a connection reset, as traffic naturally flows. Active checks can catch a problem before a real user hits it, but can false-positive on transient blips; passive checks add no extra probe traffic and reflect real user impact directly, but by definition only detect a failure after at least one real request has already suffered it. Production systems combine both.

Structured elaboration

Worked example

A production strategy that combines both without over-reacting to a single blip: mark an instance "suspect" after 2 consecutive active probe failures, but do not remove it from rotation yet. Corroborate with a passive signal over a rolling window of the last 20 real requests: if 50% or more of those requests failed, that is a threshold of $20 \times 0.5 = 10$ failed requests out of 20, remove the instance from rotation. Requiring both signals means a single flaky probe or a single unlucky real request cannot remove a healthy instance on its own, while a genuinely failing instance still gets caught quickly because the two signals reinforce each other.

Trade-offs & pitfalls

• Active-only checks can pass while real traffic fails: a backend can answer a generic TCP or HTTP probe correctly while returning errors to requests carrying real auth headers or payloads the probe never sends.
• Passive-only checks mean, by definition, that some real users experience the failure before it is detected; that is not acceptable on its own for canary or rollout gating, where the goal is to catch a bad release before it reaches most users.
• Avoid routing probe traffic through the same accounting a load-balancing algorithm like least connections uses for real capacity; counting probes as connections can skew the balance away from actual user traffic.

Direct answer

Round robin, weighted round robin, least connections, and consistent hashing differ mainly in what signal they use to pick a backend: round robin uses none (pure sequence), weighted round robin uses a static capacity ratio, least connections uses live connection counts, and consistent hashing uses a deterministic key (session id, user id) rather than load at all. That difference in signal is exactly why they behave so differently under variable request latency, long-lived connections, and mixed backend capacity.

Structured elaboration

When health checks and session affinity change the pick: if backends must hold session state without an external store, consistent hashing (or cookie-based affinity layered on any of the other three) becomes close to mandatory regardless of its load-balancing weaknesses. If health checks are slow to detect a failing node, least connections is more forgiving than round robin because it naturally routes fewer new requests to a node that is already accumulating unanswered connections, buying time until the health check catches up.

Worked example

Two backends, A and B, where A has twice the capacity of B (weight ratio 2:1). A weighted round robin scheduler realizing that ratio with a repeating pattern of period 3, "A, A, B", sent over 9 requests: A, A, B, A, A, B, A, A, B. Counting assignments: A receives requests 1, 2, 4, 5, 7, 8 (6 total), B receives requests 3, 6, 9 (3 total). That is 6:3, which simplifies to 2:1, exactly matching the configured weight, with no need to observe live load at all.

Consistent hashing, traced the same way: using a simplified illustrative hash (real systems use SHA-256 or similar, not this), sum a key's ASCII codes and take the result mod 100 to place it on a 0-99 ring. Three servers sit at fixed ring positions: X = 25, Y = 60, Z = 90 (a key is served by the first server whose position is at or after the key's hash, walking clockwise and wrapping to X if the hash is past the last position).

Now remove X from the ring (a node failure or a planned decommission): "session-42" (hash 19) has no server at or after position 19 until Y at 60, so it remaps from X to Y. "session-77" and "session-3" are untouched, since Y and Z were never X's successor for their hashes. This is the traced version of the bounded-remap claim in the table above: removing one of three nodes only disturbs the keys that node actually owned, not the whole ring.

Trade-offs & pitfalls

• Least connections needs an accurate, ideally shared, view of active connections per backend. Behind multiple load balancer replicas each keeping its own local count, the counts are only approximations, and the algorithm's advantage over round robin shrinks.
• Consistent hashing balances key placement, not load. A single popular key still lands entirely on one node no matter how good the hash function is; that is a cache-hot-key problem, not something the algorithm fixes.
• Common wrong turn: reaching for consistent hashing "for scalability" when the actual requirement is even load distribution. It solves affinity and cache locality, not general load balancing; without capacity-weighted virtual nodes it can be less balanced than plain least connections.