Load Balancing and Traffic Management Questions

Direct answer

When the load balancer's p99 rises but the backends' p95 and CPU stay flat, the divergence is itself the clue: the extra tail latency is being added somewhere the backend can't see, the network path, the LB's own accept/TLS/queueing layer, or client behavior, not inside request processing. Work through the request path in layers (network, LB proxy internals, TLS, kernel accept/backlog, client) and instrument each layer's own latency contribution separately, so the tail latency is attributed to a stage, not guessed at.

Layered checklist

• Network path (client to LB, LB to backend): interface errors, drops, retransmits (ip -s link, netstat -s), VPC flow logs for SYN/retransmit spikes, targeted traceroutes from affected client geographies. A tail-latency-only symptom with no backend involvement often starts here.
• LB/proxy internals: connection counts and churn per backend, event-loop stalls or worker saturation in the proxy process itself (profiling, strace on accept), time-to-first-byte versus time-to-last-byte split per backend. A proxy that's CPU-saturated or GC-pausing (if it's a managed-runtime proxy) shows up here, invisible to backend CPU metrics.
• TLS handshakes: split full-handshake latency from resumed/session-ticket handshake latency, and track the resumption hit rate; a drop in session cache hit rate (cache eviction, a cache not shared across proxy instances, client churn) turns cheap resumed handshakes into expensive full ones for a subset of requests, exactly the shape of a tail-latency-only regression.
• Accept queue / kernel backlog: listen backlog occupancy (ss -ltn), SYN_RECV counts (connections stuck mid-handshake, waiting on the final ACK), somaxconn and tcp_max_syn_backlog limits, ephemeral port and TIME_WAIT counts. If the accept queue is intermittently near its limit, new connections queue briefly even though every connection that does get through is processed at normal speed, invisible to backend request-processing metrics by construction.
• Client behavior: slow or high-RTT clients, retry storms, or a small subset of very chatty clients; correlate p99 offenders by client IP, geography, or user agent rather than assuming they're uniform across all traffic.

Worked example: why queueing produces a p99 problem with a flat p95

This is a general queueing-theory property, illustrated with a simple M/M/1 model (queueing-theory shorthand: Markovian/memoryless request arrivals, Markovian/memoryless service times, 1 server) and pinned parameters, not a claim about any specific measured system. For a queue with service rate $\mu$ and utilization $\rho = \lambda/\mu$ (arrival rate $\lambda$), the expected wait time in queue is:

$$W_q(\rho) = \frac{\rho}{\mu(1-\rho)}$$

Fix $\mu = 1000$ (an arbitrary service-rate unit, just for the shape of the curve) and evaluate at three utilizations:

$$\begin{aligned} \rho = 0.80 &: W_q = \frac{0.80}{1000\times0.20} = 0.0040 \Rightarrow 4.00\text{ ms} \\ \rho = 0.98 &: W_q = \frac{0.98}{1000\times0.02} = 0.0490 \Rightarrow 49.00\text{ ms} \\ \rho = 0.995 &: W_q = \frac{0.995}{1000\times0.005} = 0.1990 \Rightarrow 199.00\text{ ms} \end{aligned}$$

A move from $\rho=0.80$ to $\rho=0.995$ (a 24% increase in load) inflates queueing wait by roughly 50x. The requests that land during the brief windows where the accept queue or a proxy worker pool is momentarily near saturation (micro-bursts, GC pauses, a slow client holding a worker) are exactly the ones that generate the p99 tail, while the bulk of requests, arriving when the system is comfortably under its knee, still finish fast and keep p95 (and backend CPU, which averages over time) looking healthy. This is why p99 and p95 can diverge sharply even though nothing about the backend's steady-state processing changed: the tail is a queueing phenomenon, not a processing-time phenomenon.

Instrumentation to add

• Break end-to-end latency into stages, TCP connect, TLS handshake, proxy accept/queue wait, backend processing, response write, and emit each as its own histogram (not just the total), tagged by backend and client region.
• Accept-queue depth and backlog saturation as a time series, not just a point-in-time check, so a transient near-saturation event that self-resolves in under a second is still visible.
• TLS session-resumption rate as its own metric, separate from handshake latency, since a resumption-rate drop is a leading indicator of the handshake-latency problem.
• Kernel-level counters (SYN_RECV: mid-handshake connections, TIME_WAIT: recently closed connections still held by the kernel, retransmits) exported alongside application metrics on the same dashboard and timeline, so a kernel-layer cause doesn't require manually correlating two separate tools during an incident.
• Sampled packet captures triggered automatically when p99 crosses a threshold, so there's raw evidence from the actual bad window instead of trying to reproduce it after the fact.

Trade-offs and pitfalls

• Chasing this in backend-only dashboards is the classic dead end, since by construction the backend's own view (CPU, p95, request-processing time) is healthy; the cause lives in a layer that doesn't report through the backend's own telemetry.
• Averages and even p95 hide exactly this kind of problem by design, since it only affects a small fraction of requests, that's what makes it a p99 problem and not a p50 problem; don't let a "p95 looks fine" dashboard close the investigation.
• Fixing the wrong layer (e.g. scaling backend CPU when the problem is TLS resumption cache eviction) burns real money and doesn't move the metric; stage-by-stage instrumentation exists specifically to prevent this kind of misdiagnosis.
• Once queueing is confirmed as the mechanism, the fix is capacity or admission control (a bigger backlog, more proxy workers, load shedding, or reducing $\rho$ by scaling out), not code-level micro-optimization of request handling, since request handling was never where the time went.

Direct answer

Both are ways to make an algorithm capacity-aware, but they optimize different things. Weighted round robin (WRR) distributes a fixed pattern of requests proportional to weight, without looking at current load: it is a scheduling problem, solved up front. Weighted least connections picks, per request, whichever server has the lowest connections-to-weight ratio: it is a live load-balancing decision that reacts to what's actually happening on each server right now. WRR is cheaper and predictable; weighted least connections is more adaptive when request duration varies.

Weighted round robin mechanics

A naive WRR just repeats each server in the rotation list a number of times equal to its weight (weight 5, 1, 1 becomes the list [A,A,A,A,A,B,C]), which is simple but bursty, all of A's requests land in a clump. The standard fix is smooth weighted round robin, used by nginx and others: each server keeps a running counter, and at every request:

$$c_i^{(t+1)} = c_i^{(t)} + w_i \quad \text{for every server } i$$

then the server with the highest counter is selected, and only that server's counter is reduced by the total weight:

$$c_{i^*}^{(t+1)} = c_i^{(t+1)} - W, \qquad W = \sum_{j} w_j$$

Worked trace: weights A=5, B=1, C=1 (W=7)

Sequence: A, A, B, A, C, A, A. Final counts: A=5, B=1, C=1, exactly matching the declared weights, and after 7 steps every counter returns to 0, so the pattern repeats cleanly. Compare this to the naive list [A,A,A,A,A,B,C]: smooth WRR interleaves B and C between A's turns instead of clumping them at the end.

Weighted least connections mechanics

Instead of a precomputed pattern, each request goes to whichever server minimizes:

$$score_i = \frac{c_i}{w_i}$$

where $c_i$ is current active connections. This needs live state (the balancer must track connection counts accurately) but it self-corrects when request durations vary: if one of A's requests is unusually slow and its connection count stays elevated, weighted least connections routes around it immediately, while WRR would keep sending A its scheduled 5-out-of-7 share regardless.

When to adjust weights dynamically

• Autoscaling: when an instance joins or leaves the pool, recompute weights from the group's new capacity (commonly proportional to vCPU count or a load-tested throughput figure), otherwise new instances sit idle while old ones stay pinned to stale weights.
• Degraded instance: on failed or borderline health checks, reduce the instance's weight toward zero (a soft drain) rather than hard-removing it, so in-flight requests finish before it stops receiving new ones.
• Heterogeneous hardware: base weights on measured throughput under load, not just instance-type labels, since two "same size" instances can have different real capacity due to noisy neighbors or different hardware generations.
• Oscillation risk: reacting to every latency blip by changing weights can cause a feedback loop (a server's weight drops, it gets less traffic, its metrics improve, weight goes back up, traffic returns, metrics degrade again). Apply a decay or minimum-dwell-time before a weight change takes effect.

Trade-offs and pitfalls

WRR is stateless and cheap to run at very high request rates because it never inspects live connection counts, but it is blind to reality: if a server silently starts responding slowly, WRR keeps sending it its full scheduled share. Weighted least connections fixes that but needs accurate, low-latency visibility into connection counts across the fleet, which is harder in a distributed balancer with multiple LB instances that don't share state. A common middle ground is weighted least connections with a smoothing factor so weight changes and connection counts don't cause the oscillation described above.

Direct Answer

Pick the probe interval and consecutive-failure threshold together, since detection latency is just their product, then push the false-positive rate down by requiring multiple consecutive failures rather than reacting to one, and add jitter so thousands of instances don't probe (or fail) in lockstep during a rolling deploy. Keep per-instance removal decisions fast and deterministic, but gate any cluster-level alerting or autoscaling reaction on an aggregated, percentage-based signal, so the normal wave of transient blips a rolling deploy produces never triggers a page or a scaling event.

Recommended Parameters

Jitter and Staggering

Two different problems need two different fixes:

• Probe timing jitter: randomize each instance's probe interval by roughly ±10-25% (interval = base * (1 + rand(-0.15, 0.15))) so thousands of probes don't all fire in the same tick.
• Deploy-wave staggering: stagger each instance's initial probe phase deterministically, e.g. start_offset = hash(instance_id) mod base_interval, so a rolling deploy that restarts instances in batches doesn't produce synchronized probe failures across the whole batch at once.

Startup vs Steady-State Probing

A common misconfiguration is using one probe, tuned for steady state, for both traffic-removal and process-restart decisions. A slow-starting instance under a strict steady-state threshold gets killed and restarted before it ever becomes ready, which can cascade into a thundering-restart loop under load (each restart is slower than the last because the fleet is smaller and more loaded). The fix is to separate concerns the way most orchestrators do: a startup-phase probe with a longer grace period and higher failure tolerance that only governs the initial ready-or-not decision, a readiness probe (steady state) that governs load-balancer inclusion, and a liveness probe, decoupled from readiness, that governs whether the process gets restarted at all. Traffic removal should never be driven by the same signal that triggers a restart.

Per-Instance vs Aggregated Logic

• Per-instance (LB inclusion): strict, deterministic, uses the consecutive-failure rule above. One bad instance should not need cluster-wide context to get pulled.
• Cluster-level (alerting, autoscaling): rate-based, not count-based. Example policy: only alert if more than 5% of instances are unhealthy for more than 2 minutes, or if the absolute unhealthy count exceeds a fixed floor and is still rising. This absorbs the expected noise of a rolling deploy (where some fraction of the fleet is always transiently cycling) without needing per-deploy tuning.

Worked Example

Detection latency. With interval $= 10\text{s}$ and a consecutive-failure threshold $k = 3$:

$$\text{detection latency} = \text{interval} \times k = 10\text{s} \times 3 = 30\text{s}$$

False-positive suppression from raising the threshold. Suppose a transient, independent failure (a coincidental GC pause or a single dropped packet) affects any given probe with probability $p = 0.02$ (2%). Requiring $k$ consecutive failures before declaring unhealthy, assuming failures are independent across probes, makes the false-trip probability:

$$P(\text{false trip}) = p^k$$

For $k = 1$ (react on the first failure): $P = 0.02 = 2\%$ per check.
For $k = 3$: $P = 0.02^3 = 8 \times 10^{-6} = 0.0008\%$.

That is a reduction factor of:

$$\frac{p}{p^3} = \frac{1}{p^2} = \frac{1}{0.02^2} = 2500\times$$

fewer false trips, at the cost of 3x the detection latency (10s to 30s). This independence assumption is the important caveat: a sustained GC pause or a real network partition produces correlated consecutive failures, where raising $k$ buys much less protection, because the failures aren't independent draws anymore. The threshold you pick is really a statement about how much you trust that your transient-failure sources are actually transient (single-probe-length) rather than sustained.

Validation and Safe Rollout

1. Bench test in staging: simulate probe latency and intermittent failures against a synthetic fleet to measure the observed false-positive rate and detection latency before touching production defaults.
2. Canary the setting itself: apply the new thresholds to a small slice of instances (roughly 1-2%) first, watching LB churn rate, 4xx/5xx trends, and alert volume against the rest of the fleet as a baseline.
3. Ramp: expand to larger slices (10%, 25%, 50%) only if churn and alert-noise metrics stay flat relative to baseline at each stage.
4. Perturbation testing: inject latency, packet loss, or CPU pressure in staging to confirm the new thresholds don't cause mass removal under conditions you consider acceptable to tolerate.
5. Rollback path: keep the previous thresholds one config change away, and define the specific churn or alert-noise threshold that triggers an automatic revert rather than waiting for a human to notice.

Trade-offs and Pitfalls

• Lower interval and lower threshold means faster detection but more false positives; there is no setting that improves both simultaneously, only a documented trade-off appropriate to the service's error budget.
• Conflating liveness and readiness (see above) is one of the most common causes of deploy-induced outages: it turns a slow-starting instance into a restart loop instead of just a delayed rotation entry.
• Aggregated, percentage-based cluster alerts are necessary specifically because per-instance thresholds, tuned tightly enough to catch real failures fast, will always produce some background noise during any large rolling deploy; treating that noise as page-worthy trains the on-call team to ignore alerts.
• The independence assumption behind the $p^k$ false-trip math is an approximation; validate it against real observed failure correlation in staging (step 1) rather than trusting the formula on defaults alone.

Direct answer

Client-side load balancing puts backend selection logic in the calling process, using a locally cached view of the service registry; server-side balancing puts selection in a centralized proxy or LB that every client talks to. The core trade-off is where you pay the cost: client-side removes a network hop and lets each caller apply its own resilience policy, at the price of pushing service-discovery integration and consistent LB logic into every client; server-side centralizes control, observability, and policy enforcement, at the price of an extra hop and a fleet of LB infrastructure to run. If you implement client-side selection yourself, the hard part beyond basic routing is handling membership changes (scale-up, scale-down, deploys) without needlessly disrupting connections that didn't need to move.

Comparison

A sidecar/service-mesh pattern (e.g. a local proxy per instance) is the common middle ground: it gets client-side benefits (one hop after the sidecar, per-caller circuit breaking) while keeping policy and telemetry centrally managed, at the cost of running a sidecar per instance.

Building client-side selection yourself

Beyond basic round-robin over a discovered list, three things need explicit design:

• Dynamic discovery integration: decide push vs pull (long-poll/streaming updates vs periodic refresh), how staleness is bounded (max age before a cached endpoint list is considered untrustworthy), and what happens on registry unavailability (fail open with the last-known-good list, generally, rather than fail closed).
• Weighted/health-aware routing: incorporate backend health and load signals (not just "is it in the registry") so the client doesn't keep sending traffic to an instance that's registered but degraded; this usually means combining discovery with lightweight client-observed health (recent error rate, latency) rather than trusting the registry alone.
• Minimizing connection churn on rebalance: this is the part naive implementations get wrong. If the client picks a backend by hashing the request key modulo the current endpoint count, every membership change (an instance added or removed) reshuffles almost the entire mapping, forcing most existing connections to move even though only one instance actually changed.

Worked example

Compare naive modulo hashing to consistent hashing when the backend pool grows from 100 to 101 instances. With consistent hashing on a ring, only the requests that fall in the ring segment newly owned by the added instance need to move; since 101 nodes partition the ring into (in expectation) equal arcs, that segment is 1/101 of the total:

$$\begin{aligned} p_{consistent} &= \frac{1}{N+1} = \frac{1}{101} \approx 0.0099\ (0.99\%) \\ p_{modulo} &= \frac{N}{N+1} = \frac{100}{101} \approx 0.990\ (99.0\%) \end{aligned}$$

With naive modulo hashing, key mod 100 and key mod 101 agree for almost no keys, so essentially the entire mapping (about 99%) reshuffles for a single instance added to a 100-instance pool, meaning nearly every existing connection gets torn down and re-established even though only 1 backend changed. Consistent hashing bounds that churn to roughly 1%, which is the actual justification for using it in client-side selection, not just an abstract preference.

Trade-offs and pitfalls

• Client-side LB trades a network hop for distributed complexity: every client language/runtime needs the same discovery + selection + circuit-breaking logic kept in sync, which is why organizations that adopt it usually invest in a shared client library or move the logic into a sidecar instead of reimplementing it per service.
• A common pitfall in building this yourself is optimizing routing quality (best-of-N latency-aware selection, say) while ignoring rebalance churn; a routing policy that's marginally better under steady state but reshuffles most connections on every scale event can net out worse for tail latency during autoscaling events, which happen far more often than people plan for.
• Server-side LB is usually the safer default for heterogeneous or external clients (mobile, third-party) where you cannot control client behavior; client-side (or sidecar) LB earns its complexity budget mainly inside a trusted internal service mesh where the latency and infrastructure savings compound across many services.

Direct answer

Round robin, weighted round robin, least connections, and consistent hashing differ mainly in what signal they use to pick a backend: round robin uses none (pure sequence), weighted round robin uses a static capacity ratio, least connections uses live connection counts, and consistent hashing uses a deterministic key (session id, user id) rather than load at all. That difference in signal is exactly why they behave so differently under variable request latency, long-lived connections, and mixed backend capacity.

Structured elaboration

When health checks and session affinity change the pick: if backends must hold session state without an external store, consistent hashing (or cookie-based affinity layered on any of the other three) becomes close to mandatory regardless of its load-balancing weaknesses. If health checks are slow to detect a failing node, least connections is more forgiving than round robin because it naturally routes fewer new requests to a node that is already accumulating unanswered connections, buying time until the health check catches up.

Worked example

Two backends, A and B, where A has twice the capacity of B (weight ratio 2:1). A weighted round robin scheduler realizing that ratio with a repeating pattern of period 3, "A, A, B", sent over 9 requests: A, A, B, A, A, B, A, A, B. Counting assignments: A receives requests 1, 2, 4, 5, 7, 8 (6 total), B receives requests 3, 6, 9 (3 total). That is 6:3, which simplifies to 2:1, exactly matching the configured weight, with no need to observe live load at all.

Consistent hashing, traced the same way: using a simplified illustrative hash (real systems use SHA-256 or similar, not this), sum a key's ASCII codes and take the result mod 100 to place it on a 0-99 ring. Three servers sit at fixed ring positions: X = 25, Y = 60, Z = 90 (a key is served by the first server whose position is at or after the key's hash, walking clockwise and wrapping to X if the hash is past the last position).

Now remove X from the ring (a node failure or a planned decommission): "session-42" (hash 19) has no server at or after position 19 until Y at 60, so it remaps from X to Y. "session-77" and "session-3" are untouched, since Y and Z were never X's successor for their hashes. This is the traced version of the bounded-remap claim in the table above: removing one of three nodes only disturbs the keys that node actually owned, not the whole ring.

Trade-offs & pitfalls

• Least connections needs an accurate, ideally shared, view of active connections per backend. Behind multiple load balancer replicas each keeping its own local count, the counts are only approximations, and the algorithm's advantage over round robin shrinks.
• Consistent hashing balances key placement, not load. A single popular key still lands entirely on one node no matter how good the hash function is; that is a cache-hot-key problem, not something the algorithm fixes.
• Common wrong turn: reaching for consistent hashing "for scalability" when the actual requirement is even load distribution. It solves affinity and cache locality, not general load balancing; without capacity-weighted virtual nodes it can be less balanced than plain least connections.