Load Balancing and Traffic Management Questions

Direct answer

Draining safely with a mix of short HTTP and long-lived WebSocket connections means stopping admission of new work immediately (fail readiness, deregister), letting in-flight HTTP finish naturally since it is already short, and giving WebSocket connections an explicit grace period bounded by a measured drain timeout, after which any still-open connection is force-closed. The timeout itself should not be a guess: measure the real duration distribution of long-lived connections in staging, pick a percentile with a safety margin, and validate that choice by re-running the drain and checking how many connections it actually force-closes.

Structured elaboration

• Stop new traffic first. Flip readiness to failing (or deregister from the load balancer's target pool) before doing anything else, so no new HTTP request or WebSocket upgrade attempt lands on the draining instance. New upgrade attempts should get a fast rejection (503), not a connection that immediately gets drained.
• Let short HTTP finish on its own. In-flight HTTP requests are typically done within seconds; rely on the load balancer's own connection-draining or deregistration-delay setting to keep the instance reachable for just those existing connections, not new ones.
• Give WebSockets a bounded grace period. Send a close frame with a reason and, where the client supports it, a suggested reconnect delay. Track how many WebSocket connections remain open and how long the drain has been running; when the timeout is reached, force-close whatever is left rather than draining indefinitely.
• Measuring the timeout in staging. Generate a realistic connection-duration distribution under staged load (not just connection count, the actual spread of session lengths), trigger a controlled drain, record how long each connection was open, and compute a high percentile (for example p90 or p99) of that distribution as the timeout floor before adding a safety margin.

stateDiagram-v2
    [*] --> InRotation
    InRotation --> Draining: marked for removal, readiness fails
    Draining --> Draining: HTTP finishes naturally; WS gets close frame
    Draining --> SafeToTerminate: HTTP count = 0 and WS count = 0
    Draining --> ForceClose: drain timeout reached
    ForceClose --> SafeToTerminate
    SafeToTerminate --> [*]

Worked example

A staging drain test records 10 WebSocket session durations in seconds (fully specified for this example): 12, 15, 20, 22, 30, 35, 40, 55, 90, 240. Already sorted ascending, with $n = 10$.

Using the nearest-rank method, the 90th percentile index is:

The 9th value in the sorted list is 90 seconds, so $p_{90} = 90\text{ s}$.

Add a 20% safety margin and round to a configuration-friendly value:

Validate by re-running the same drain with the 110-second timeout: only the 240-second session exceeds it, so exactly 1 of 10 sessions would be force-closed, a $1/10 = 10\%$ forced-close rate in this sample. If the team's tolerance for forced WebSocket closures during a deploy is, say, under 5%, this result fails validation and the timeout needs to go higher, or the long tail needs a real fix (session handoff and client-side reconnect) instead of a longer wait.

Autoscaling context. The same drain path fires on scale-in, not just deploys. If sticky routing pins long-lived sessions to specific pods, a scale-in event has to drain those pinned sessions the same way, and frequent autoscaling churn means paying this drain cost far more often than a deploy cadence would; that is a real argument for moving session state out of the pod (a shared store) rather than tuning the drain timeout ever higher to compensate.

Trade-offs & pitfalls

• Too-short a timeout forces out legitimate long sessions and shows up as user-visible disconnects; too-long a timeout slows every deploy and scale-in and holds resources the orchestrator thinks it already reclaimed.
• A percentile chosen from a staging dataset is only as good as how representative that dataset's traffic mix and connection-duration shape are; validate against real production duration distributions periodically, not once at design time.
• Common wrong turn: reusing the same drain timeout for HTTP and WebSocket paths. HTTP's tail is usually seconds; WebSocket's tail can be minutes to hours, and forcing HTTP to wait for the WebSocket-sized timeout just slows every deploy for no benefit.
• Session handoff or reconnect logic on the client is what actually solves the extreme tail (a connection open far longer than any reasonable timeout); a timeout alone only bounds how long you wait before giving up on it.

Direct answer

When the load balancer's p99 rises but the backends' p95 and CPU stay flat, the divergence is itself the clue: the extra tail latency is being added somewhere the backend can't see, the network path, the LB's own accept/TLS/queueing layer, or client behavior, not inside request processing. Work through the request path in layers (network, LB proxy internals, TLS, kernel accept/backlog, client) and instrument each layer's own latency contribution separately, so the tail latency is attributed to a stage, not guessed at.

Layered checklist

• Network path (client to LB, LB to backend): interface errors, drops, retransmits (ip -s link, netstat -s), VPC flow logs for SYN/retransmit spikes, targeted traceroutes from affected client geographies. A tail-latency-only symptom with no backend involvement often starts here.
• LB/proxy internals: connection counts and churn per backend, event-loop stalls or worker saturation in the proxy process itself (profiling, strace on accept), time-to-first-byte versus time-to-last-byte split per backend. A proxy that's CPU-saturated or GC-pausing (if it's a managed-runtime proxy) shows up here, invisible to backend CPU metrics.
• TLS handshakes: split full-handshake latency from resumed/session-ticket handshake latency, and track the resumption hit rate; a drop in session cache hit rate (cache eviction, a cache not shared across proxy instances, client churn) turns cheap resumed handshakes into expensive full ones for a subset of requests, exactly the shape of a tail-latency-only regression.
• Accept queue / kernel backlog: listen backlog occupancy (ss -ltn), SYN_RECV counts (connections stuck mid-handshake, waiting on the final ACK), somaxconn and tcp_max_syn_backlog limits, ephemeral port and TIME_WAIT counts. If the accept queue is intermittently near its limit, new connections queue briefly even though every connection that does get through is processed at normal speed, invisible to backend request-processing metrics by construction.
• Client behavior: slow or high-RTT clients, retry storms, or a small subset of very chatty clients; correlate p99 offenders by client IP, geography, or user agent rather than assuming they're uniform across all traffic.

Worked example: why queueing produces a p99 problem with a flat p95

This is a general queueing-theory property, illustrated with a simple M/M/1 model (queueing-theory shorthand: Markovian/memoryless request arrivals, Markovian/memoryless service times, 1 server) and pinned parameters, not a claim about any specific measured system. For a queue with service rate $\mu$ and utilization $\rho = \lambda/\mu$ (arrival rate $\lambda$), the expected wait time in queue is:

Fix $\mu = 1000$ (an arbitrary service-rate unit, just for the shape of the curve) and evaluate at three utilizations:

A move from $\rho=0.80$ to $\rho=0.995$ (a 24% increase in load) inflates queueing wait by roughly 50x. The requests that land during the brief windows where the accept queue or a proxy worker pool is momentarily near saturation (micro-bursts, GC pauses, a slow client holding a worker) are exactly the ones that generate the p99 tail, while the bulk of requests, arriving when the system is comfortably under its knee, still finish fast and keep p95 (and backend CPU, which averages over time) looking healthy. This is why p99 and p95 can diverge sharply even though nothing about the backend's steady-state processing changed: the tail is a queueing phenomenon, not a processing-time phenomenon.

Instrumentation to add

• Break end-to-end latency into stages, TCP connect, TLS handshake, proxy accept/queue wait, backend processing, response write, and emit each as its own histogram (not just the total), tagged by backend and client region.
• Accept-queue depth and backlog saturation as a time series, not just a point-in-time check, so a transient near-saturation event that self-resolves in under a second is still visible.
• TLS session-resumption rate as its own metric, separate from handshake latency, since a resumption-rate drop is a leading indicator of the handshake-latency problem.
• Kernel-level counters (SYN_RECV: mid-handshake connections, TIME_WAIT: recently closed connections still held by the kernel, retransmits) exported alongside application metrics on the same dashboard and timeline, so a kernel-layer cause doesn't require manually correlating two separate tools during an incident.
• Sampled packet captures triggered automatically when p99 crosses a threshold, so there's raw evidence from the actual bad window instead of trying to reproduce it after the fact.

Trade-offs and pitfalls

• Chasing this in backend-only dashboards is the classic dead end, since by construction the backend's own view (CPU, p95, request-processing time) is healthy; the cause lives in a layer that doesn't report through the backend's own telemetry.
• Averages and even p95 hide exactly this kind of problem by design, since it only affects a small fraction of requests, that's what makes it a p99 problem and not a p50 problem; don't let a "p95 looks fine" dashboard close the investigation.
• Fixing the wrong layer (e.g. scaling backend CPU when the problem is TLS resumption cache eviction) burns real money and doesn't move the metric; stage-by-stage instrumentation exists specifically to prevent this kind of misdiagnosis.
• Once queueing is confirmed as the mechanism, the fix is capacity or admission control (a bigger backlog, more proxy workers, load shedding, or reducing $\rho$ by scaling out), not code-level micro-optimization of request handling, since request handling was never where the time went.

Direct answer

HTTP/1.1 keep-alive needs roughly one TCP connection per concurrent in-flight request, so an L7 balancer scales by managing connection count. HTTP/2 and gRPC multiplex many concurrent streams over a small number of long-lived connections, so the balancer has to scale by managing stream concurrency within a connection instead, and load signals that were adequate for HTTP/1.1 (active connection count) become nearly useless. The practical consequence: fewer sockets and less TLS/TCP handshake overhead, but a new failure mode where one connection getting stalled or overloaded can degrade every stream multiplexed on it (head-of-line effects), which the balancer has to actively guard against.

Connection model comparison

Per-connection limits to tune

• HTTP/1.1: max connections per backend (bounds concurrency directly), idle keep-alive timeout (frees sockets from clients that went quiet), and pool size on the LB's upstream side.
• HTTP/2 / gRPC: max_concurrent_streams per connection (how many in-flight requests one connection may carry, commonly capped well below the protocol's theoretical maximum to bound blast radius), a small upstream connection pool per backend (a handful, not one) so a single stalled connection doesn't take out all traffic to that backend, per-stream flow-control window size (the amount of unacknowledged data a stream may have in flight before the sender must pause), and a keepalive ping interval to detect a half-open connection before streams queue behind a dead peer.

Load measurement and backpressure

Connection count stops being a useful load signal once multiplexing is in play: a backend with 4 connections and 400 streams looks identical to one with 4 connections and 4 streams if you only count sockets. Instead:

• Measure in-flight streams per backend (or max_concurrent_streams - current_streams as available capacity) and use that as the weighting signal for load-aware routing.
• Track per-stream and per-connection latency percentiles separately; a rising per-connection tail with stable per-stream counts points at connection-level contention (CPU, flow-control), not request volume.
• Apply admission control at the stream level: reject new streams past a concurrency threshold with a fast, typed backpressure signal (gRPC RESOURCE_EXHAUSTED, HTTP 429) rather than accepting and queuing, which just moves the head-of-line problem later.
• Detect stalled streams (no progress against their flow-control window) and reset them individually instead of tearing down the whole connection, so one bad stream doesn't punish every other stream sharing it.

Worked example

Suppose the LB maintains a pool of 4 HTTP/2 connections to a backend, each configured with max_concurrent_streams = 100:

That backend can serve 400 concurrent RPCs using 4 sockets. Reaching the same 400 concurrent in-flight requests under HTTP/1.1 keep-alive would require roughly 400 separate TCP+TLS-established connections (one per in-flight request), which is exactly the socket and handshake overhead multiplexing removes. The other side of that number: if one of those 4 connections stalls, up to 100 of the 400 in-flight requests (25%) can be head-of-line blocked simultaneously, which is why the pool size (not just 1 connection) and per-stream stall detection both matter.

Trade-offs and pitfalls

• Fewer, fatter connections are more efficient but concentrate risk: a single TCP-level packet loss stalls every stream on that connection at the transport layer, even though the streams are logically independent at the application layer. This TCP-level head-of-line blocking is a known limitation of running multiplexing over TCP (it's the reason HTTP/3/QUIC exists), but that's depth beyond what most interviews expect; the interview-relevant point is just that a small connection pool, not a single connection, bounds the blast radius.
• A common mistake is reusing HTTP/1.1-era LB health/load metrics (connection count, connections-per-second) unchanged for HTTP/2 backends; they will systematically under-detect overload because a backend can look "quiet" on connections while being saturated on streams.
• Setting max_concurrent_streams too high trades efficiency for blast radius; setting it too low defeats the purpose of multiplexing and pushes you back toward connection-count scaling.

Direct answer

Round robin, least connections, and weighted load balancing differ in what information they use to pick a server. Round robin uses none, it just cycles through the pool. Least connections uses current load (active connection count) but assumes every server has equal capacity. Weighted load balancing (typically combined with one of the other two, giving weighted round robin or weighted least connections) uses declared server capacity. For a pool with bursty traffic and heterogeneous capacity, weighted least connections is the right default, because it is the only one of the three that reacts to both current load and known capacity differences.

How each one picks

Why plain round robin and plain least connections both fall short here

Round robin sends the same share of traffic to every server regardless of what it can handle, so a small server gets hit exactly as often as a large one and becomes the bottleneck under burst. Least connections is an improvement, since it avoids piling more work onto an already-busy server, but it still treats a 2-vCPU box and an 8-vCPU box as equally capable: an equal number of active connections is not an equal amount of work when the servers are different sizes.

Worked example: weighted least connections in action

Say the pool has three servers with declared capacity weights $4$, $2$, and $1$ (roughly matching, say, an 8-vCPU, 4-vCPU, and 2-vCPU box). Weighted least connections picks, for each new request, whichever server minimizes:

where $c_i$ is server $i$'s current active connection count and $w_i$ is its weight. Ties are broken toward the higher-weight server. Tracing 7 back-to-back requests (a burst, so nothing completes mid-trace):

After 7 requests the split is S1=4, S2=2, S3=1, exactly proportional to the declared weights of 4:2:1, even though every request arrived back-to-back with no completions in between. Plain round robin would have sent requests 1-7 as S1,S2,S3,S1,S2,S3,S1 (a 3:2:2 split), overloading S3 relative to its actual capacity.

Trade-offs and pitfalls

Weighted least connections needs two things plain least connections does not: accurate weights (usually derived from instance size or a load test, not guessed) and a mechanism to keep them current as capacity changes (autoscaling, degraded instances). It is also more stateful, the balancer has to track active connections per server accurately, which matters less for round robin. If request cost varies wildly and isn't reflected by connection count (a handful of expensive long-running requests versus many cheap ones), even weighted least connections can misjudge load, at which point request-cost-aware balancing or queuing becomes worth considering.

Direct answer

Load balancers exist to decouple how much traffic arrives from which single machine has to answer it. They solve at least four distinct problems: spreading load across available capacity so no single server bottlenecks, removing single points of failure through health-based failover, enabling safe rollouts such as canary (releasing a new version to a small slice of traffic first) and blue-green deployment (running old and new versions side by side, then cutting all traffic over at once) with graceful draining, and centralizing cross-cutting concerns like TLS termination and rate limiting. In a typical three-tier architecture, load balancers show up at the edge (public-facing traffic), between internal services (service to service), and inside a service mesh (sidecar to sidecar).

Structured elaboration

Named problems load balancers solve:

1. Single-server bottleneck / horizontal scaling. Spreading requests across many instances lets the system add capacity by adding instances instead of a bigger single machine.
2. Availability and automatic failover. Health checks remove failing instances from rotation so one bad node does not take down the service.
3. Fault isolation and safe rollout. Canary and blue-green routing, plus connection draining during deploys, contain a bad release to a small slice of traffic instead of every user.
4. TLS termination and centralized security policy. Certificates and security rules (WAF, Web Application Firewall: a layer that inspects and blocks malicious HTTP traffic before it reaches a backend, and rate limiting) live in one place instead of being duplicated on every backend.
5. Rate limiting and overload protection. The balancer can shed or throttle excess traffic before it reaches backend capacity, protecting downstream services.

Placement in a three-tier architecture:

graph LR
  Client[Client] --> EdgeLB["Edge LB (L7, TLS termination)"]
  EdgeLB --> GwA[Gateway instance A]
  EdgeLB --> GwB[Gateway instance B]
  GwA --> InternalLB[Internal service LB]
  GwB --> InternalLB
  InternalLB --> SvcA[Service instance A]
  InternalLB --> SvcB[Service instance B]
  SvcA --> MeshLB["Sidecar LB (service-to-service)"]
  SvcB --> MeshLB
  MeshLB --> Downstream[Downstream service]

The edge tier handles public traffic and usually terminates TLS. The internal service tier balances across replicas of a given service, often behind a service discovery layer. The service-to-service tier (frequently a sidecar, a small proxy process deployed alongside each service instance, in a mesh) balances outbound calls one internal service makes to another, applying the same health-check and retry logic without a separate hop through a centralized balancer.

A fast way to spot gaps in an unfamiliar system's traffic layer: check whether all three tiers actually exist (a single edge LB fanning straight out to every backend is a common shortcut that skips the internal tier), whether health checks are active at each tier, and whether TLS termination and rate limiting are centralized or scattered across services.

Worked example

Three backend instances, each rated for 1,000 requests per second, pooled behind a load balancer: combined capacity is $3 \times 1000 = 3000$ requests per second. If health checks detect one instance failing and pull it out of rotation, capacity becomes $2 \times 1000 = 2000$ requests per second, so the system keeps serving at $2000 / 3000 \approx 66.7\%$ of its original capacity instead of dropping to zero, which is what a single unpooled instance would do on failure.

Trade-offs & pitfalls

• The load balancer itself becomes critical infrastructure. A single-instance balancer just moves the single point of failure up one layer; it needs its own high availability (active-active pair, DNS or anycast failover).
• Every additional tier (edge, internal, mesh) adds a proxy hop's worth of processing and another thing to operate; do not add a tier a system does not need.
• A misconfigured health check at any tier can silently remove that tier's benefit (failover, safe rollout) even though the balancer process is technically running.