Load Balancing and Traffic Management Questions

Direct answer

Round robin, weighted round robin, least connections, and consistent hashing differ mainly in what signal they use to pick a backend: round robin uses none (pure sequence), weighted round robin uses a static capacity ratio, least connections uses live connection counts, and consistent hashing uses a deterministic key (session id, user id) rather than load at all. That difference in signal is exactly why they behave so differently under variable request latency, long-lived connections, and mixed backend capacity.

Structured elaboration

When health checks and session affinity change the pick: if backends must hold session state without an external store, consistent hashing (or cookie-based affinity layered on any of the other three) becomes close to mandatory regardless of its load-balancing weaknesses. If health checks are slow to detect a failing node, least connections is more forgiving than round robin because it naturally routes fewer new requests to a node that is already accumulating unanswered connections, buying time until the health check catches up.

Worked example

Two backends, A and B, where A has twice the capacity of B (weight ratio 2:1). A weighted round robin scheduler realizing that ratio with a repeating pattern of period 3, "A, A, B", sent over 9 requests: A, A, B, A, A, B, A, A, B. Counting assignments: A receives requests 1, 2, 4, 5, 7, 8 (6 total), B receives requests 3, 6, 9 (3 total). That is 6:3, which simplifies to 2:1, exactly matching the configured weight, with no need to observe live load at all.

Consistent hashing, traced the same way: using a simplified illustrative hash (real systems use SHA-256 or similar, not this), sum a key's ASCII codes and take the result mod 100 to place it on a 0-99 ring. Three servers sit at fixed ring positions: X = 25, Y = 60, Z = 90 (a key is served by the first server whose position is at or after the key's hash, walking clockwise and wrapping to X if the hash is past the last position).

Now remove X from the ring (a node failure or a planned decommission): "session-42" (hash 19) has no server at or after position 19 until Y at 60, so it remaps from X to Y. "session-77" and "session-3" are untouched, since Y and Z were never X's successor for their hashes. This is the traced version of the bounded-remap claim in the table above: removing one of three nodes only disturbs the keys that node actually owned, not the whole ring.

Trade-offs & pitfalls

• Least connections needs an accurate, ideally shared, view of active connections per backend. Behind multiple load balancer replicas each keeping its own local count, the counts are only approximations, and the algorithm's advantage over round robin shrinks.
• Consistent hashing balances key placement, not load. A single popular key still lands entirely on one node no matter how good the hash function is; that is a cache-hot-key problem, not something the algorithm fixes.
• Common wrong turn: reaching for consistent hashing "for scalability" when the actual requirement is even load distribution. It solves affinity and cache locality, not general load balancing; without capacity-weighted virtual nodes it can be less balanced than plain least connections.

Direct answer

Catching unhealthy backends before users notice means monitoring the LB tier along four angles: traffic, latency, and errors as seen from the LB itself (the RED metrics), backend health signals independent of user-facing symptoms, saturation and capacity of the LB tier, and TLS/connection-layer health, then tying alert severity to how much of the user-facing error budget (the amount of failure your reliability target still allows before you've broken your promise to users) is actually being consumed rather than to a single static threshold, so a brief blip and a sustained outage don't page the same way.

The metric set

Alert severity: tie it to error-budget burn, not just a raw threshold

A raw threshold ("page if 5xx rate > 1%") treats a 10-minute blip the same as a slow multi-hour decline, a common cause of alert fatigue in either direction: it either pages on noise or is set so loose it misses real problems. The more robust approach expresses the threshold as a multiple of your error-budget burn rate: if you're consuming your monthly error budget faster than the rate at which "consume it evenly over 30 days" would, page proportionally to how much faster.

Worked example

Suppose the LB tier's SLO (service-level objective, the reliability target you've committed to, here 99.9% of requests succeeding) is 99.9% success rate over a 30-day window. The allowed error budget is $0.1\%$ of requests, equivalently, 0.1% of the 30-day window can be fully down and still meet the SLO:

• 30 days in minutes: $30 \times 24 \times 60 = 43{,}200$ minutes.
• Budget: $43{,}200 \times 0.001 = 43.2$ minutes of full-downtime-equivalent for the entire month.

Now suppose the current 5xx rate over the last hour is holding at 2%, against an objective of 0.1%. The burn rate is:

At 20 times the sustainable rate, the entire month's budget would be exhausted in:

A burn rate that would exhaust the monthly budget in 1.5 days is unambiguously page-worthy; a burn rate of, say, 1.2 times sustainable is not, it just needs tracking. That is the difference between a severity-1 page and a low-priority ticket, expressed in the same unit (budget consumption), not two arbitrarily different percentage cutoffs.

Avoiding alert fatigue

• Tier severity by projected budget-exhaustion time (minutes-to-exhaust pages immediately, weeks-to-exhaust becomes a ticket), rather than by a single static percentage.
• Deduplicate and correlate: a health-check-flapping alert and a 5xx-rate alert firing from the same underlying cause should collapse into one incident, not page twice.
• Route by audience: on-call needs the actionable, current-state view (which backend, since when); a weekly review needs the trend (is health-check-induced churn increasing over time).
• Require every page-level alert to link directly to a runbook step; an alert with no clear next action trains people to ignore it.

Trade-offs and pitfalls

Over-indexing on system-level metrics (CPU, connections) without a user-facing correlate risks paging on things users never notice; conversely, only alerting on aggregate error rate can hide a problem affecting one backend pool or one tenant while the aggregate stays healthy. Circuit-breaker and rate-limit rejection counts are easy to skip because they're not "errors" in the traditional sense, but they're often the earliest signal, by the time health checks start failing outright, the circuit breaker has usually already tripped.

Direct Answer

Pull an instance automatically only when a signal shows a sustained departure from its own steady-state baseline, smoothed over a short window, and prefer requiring at least two independent signals to agree before removing an instance, rather than trusting any single metric. The one exception is a hard, unambiguous signal, like a completely unresponsive health-check endpoint, which should short-circuit straight to removal since there's nothing left to corroborate.

Signals, Triggers, and False-Positive Modes

The table below spans all three signal categories the question names, metrics, logs, and traces, not just the metrics-heavy ones that are easiest to instrument first.

Smoothing and Windowing

Use a rolling window (not an instantaneous sample) for every signal except hard health-check failure, and require the trigger condition to hold across the whole window rather than at a single point. This is what separates "sustained degradation" from "one bad request." The window length is a direct trade-off against detection speed: shorter windows catch real problems faster but inherit more noise from small sample sizes, especially on low-traffic instances where a 2-minute window might contain only a handful of requests.

Combining Signals to Cut False Positives

Requiring two independent signals to agree before acting cuts the false-positive rate multiplicatively, at the cost of occasionally missing a real problem that only shows up in one signal. Suppose each of three independent signals (error rate, latency, saturation) has an independent 1% chance of a false trigger in any given window ($p = 0.01$). Requiring any 2 of 3 to agree:

Compared to acting on any single signal alone ($p = 0.01$), that's a reduction of:

fewer false removals, assuming the signals really are independent. In practice error rate and latency are often correlated (a struggling instance tends to show both), so the real-world reduction is smaller than this idealized number, but the direction and the general logic hold: corroboration is strictly cheaper in false positives than any single signal.

Trade-offs and Pitfalls

• A hard signal (health-check endpoint completely unreachable) should bypass the corroboration requirement entirely; waiting for a second signal to agree that a dead instance is dead only adds latency for no benefit.
• Combining signals reduces false positives but can delay detection of a real problem that manifests in only one signal strongly (say, a memory leak that spikes memory well before latency degrades); the corroboration requirement is a precision/recall trade (catching more real problems, higher recall, versus fewer false alarms, higher precision, pull in opposite directions), not a free lunch.
• Automated removal needs its own safety valve: a cap on how many instances can be removed in a short window, so a correlated event (a bad deploy, a shared dependency outage) that trips the same signal across many instances at once doesn't remove enough capacity to cause the outage it was meant to prevent.
• Removal should default to a graceful drain (stop new traffic, let in-flight requests finish) rather than an immediate hard kill, except when the health signal indicates the instance can no longer serve any traffic at all.

Direct answer

The load balancer must gate two moments in an instance's life: it should not receive traffic until its readiness check passes (not just a process-alive check), and on scale-in it must stop receiving new connections while letting in-flight requests finish, deregistration with connection draining, before the instance is terminated. Both are the LB's job, independent of whatever policy decided to scale in the first place.

Structured elaboration

Scale-out: readiness gates traffic, not liveness. A liveness check ("is the process up") and a readiness check ("is the app actually able to serve correctly") are different questions. The LB should register the instance in rotation, or the orchestrator should mark it Ready, only after readiness passes: connection pools opened, local caches warmed enough to avoid a cold-cache latency cliff, and any required config pulled. A liveness-only gate puts unwarmed instances into rotation and produces a latency or error spike on every scale-out event.

Scale-in: draining before termination, in order:

1. Instance is marked for termination (by the autoscaler or an operator).
2. The LB immediately stops sending new connections to it (deregister from rotation) but does not kill existing connections.
3. A deregistration delay (connection draining window) gives in-flight requests time to complete normally.
4. Only after the drain window elapses, or all in-flight requests finish, whichever is defined as the policy, does the instance actually terminate.

Sizing the drain window. It must exceed the longest realistic in-flight request duration for the service, with margin, or draining will forcibly cut live requests, defeating the point.

Health-check readiness during warm-up. The initial health-check grace period (the time before a failed check counts against a brand-new instance) must exceed real startup time, so a slow-booting instance is not falsely marked unhealthy and cycled before it ever gets a chance to serve.

Cooldowns, and where autoscaling policy stops and LB mechanics start. Which metric and threshold trigger a scale event is an autoscaling-policy decision outside this question's scope, but the cooldown itself belongs here since the question names it directly: a cooldown is the minimum time an autoscaler must wait after one scaling action before it is allowed to trigger another, and it exists because the fleet's aggregate metrics take time to reflect the previous action, an autoscaler that reacts to a still-stale signal will overreact, stacking a second scale-out on top of one still warming up, or yanking capacity out from under a scale-in that hasn't finished draining. The cooldown has to be sized against the LB mechanics above, not chosen independently of them: it should exceed both the drain window (a scale-in's capacity isn't actually gone until draining finishes) and the health-check grace period (a scale-out's capacity isn't actually serving, and won't show up in the fleet's metrics, until warm-up and readiness finish), see the worked example below for a concrete number. Whatever triggers a scale-out or scale-in event, the LB's readiness gate and drain sequence are what make that event safe for in-flight traffic; the cooldown is what keeps the autoscaler from firing faster than those mechanics can complete.

Worked example

Say p99 request duration for this service, measured over the last observation period, is 8 seconds, and cold-start warm-up (open DB pool + populate local cache) takes 15 seconds.

Deregistration delay, with margin over the slowest realistic request:

Health-check grace period, with margin over startup time:

So: a new instance is excluded from the health check's failure count for its first 30 seconds while it warms up, and a terminating instance gets up to 10 seconds to finish in-flight requests after being pulled from rotation, before the process is killed.

Cooldown, sized to exceed both of those numbers so the autoscaler never acts on a stale signal:

So the autoscaler waits at least 45 seconds after any scale-out or scale-in action before it will consider triggering another one, comfortably longer than the 30 second grace period a new instance needs before its readiness, and therefore its true added capacity, is even visible in the fleet's metrics.

Trade-offs and pitfalls

• Too-short drain windows silently drop real user requests mid-response; this is one of the most common causes of a small, hard-to-reproduce error-rate blip during routine scale-in.
• Too-long drain windows slow down scale-in, which matters when scale-in is reacting to a cost or capacity signal; there's a real tension between "never cut a request" and "shed capacity promptly."
• A liveness-only readiness gate causes a recurring latency spike on every scale-out, because unwarmed instances start serving immediately; teams often discover this only when scale-out frequency increases (e.g. spiky traffic) and the spikes become visible in aggregate latency graphs.
• Grace periods that are too short flap healthy-but-slow-booting instances, causing the autoscaler and LB to fight each other: instance boots, fails health check before finishing warm-up, gets cycled, repeats.
• Drain and readiness are load-balancer-owned; they are not a substitute for correct autoscaling trigger design. A perfectly tuned drain window cannot fix an autoscaler that scales in too aggressively in the first place, that is a separate, upstream problem.

Direct Answer

Hash the session cookie onto a consistent-hash ring of virtual nodes so a session's requests keep landing on the same backend under normal conditions, publish the ring from a small versioned control plane so every proxy agrees on current ownership, pull unhealthy nodes out via active health checks, and when a node is intentionally removed, keep it serving its existing sessions for a bounded drain window while new sessions route elsewhere, only decommissioning it once its active-session count reaches zero or the window expires. The two things that make this safe are versioning the ring (so proxies never disagree mid-transition) and treating removal as a drain, not a hard cut.

Architecture

flowchart LR
  Client -->|cookie + ring_version| Proxy[Edge Proxy]
  Proxy -->|lookup| RingSvc[Ring Metadata Service]
  HealthChecker[Health Checker] -->|mark unhealthy| RingSvc
  Proxy --> Backend1[Backend 1]
  Proxy --> Backend2[Backend 2]
  Proxy -.draining.-> Backend3[Backend 3]
  DrainOrch[Drain Orchestrator] -->|coordinate| RingSvc
  Backend3 -->|flush session state| Store[(Shared Session Store)]
  Backend1 --> Store
  Backend2 --> Store

The proxies are stateless: all ring state lives in the ring metadata service and is cached locally with a version number. Health checks and the drain orchestrator only ever mutate the ring through that service, never by having a proxy make a unilateral decision.

Hash Ring and Virtual Nodes

• Use a large hash space (64-bit) with each physical backend assigned many virtual nodes, commonly in the range of 100 to 500, so that any single node's removal spreads its load across many different successors instead of dumping it on one (see the worked example below for why this matters).
• Store the ring as a sorted array in the metadata service; proxies cache a copy and watch for version bumps rather than polling on every request.
• Every ring mutation (join, leave, health-driven removal) increments the ring version atomically, so a proxy can always tell whether its cached copy is current.

Cookie and Version Handling

The session cookie carries three fields: the session id (the hash key), the ring version the session was originally assigned under, and an HMAC (a keyed cryptographic checksum: proof the cookie's contents haven't been altered since a server signed them) to prevent tampering. On a new session, the proxy issues a fresh cookie against the current ring version. On a returning session, the proxy honors the session's recorded ring version for a bounded overlap window even after the ring has moved on, which is what lets an in-flight session keep reaching its original backend during a drain instead of being silently reassigned mid-conversation.

Health Checks

Active HTTP health checks run against each backend on a fixed interval. A backend that fails its threshold is marked unhealthy in the ring metadata service, which bumps the ring version; proxies pick up the change and stop routing new sessions there. Existing sessions already pinned to that backend via cookie should still be judged against the same health signal: if the backend is actually down, in-flight requests will fail regardless of stickiness, so unhealthy removal is immediate, not drained (draining is reserved for planned, intentional removal, covered next).

Graceful Drain and Session Migration

For a planned node removal (scale-down, deploy, decommission):

1. Mark the node "draining" in the ring metadata service.
2. Remove the node's virtual-node entries from the ring and bump the ring version.
3. Proxies pick up the new version for new sessions immediately; sessions whose cookie still carries the old ring version continue routing to the draining node for a bounded overlap window.
4. During the overlap window, the draining node's session state is either already externalized (shared store, no action needed) or actively migrated: the drain orchestrator streams in-memory session state to the new owning node or to the shared store.
5. Once the draining node's active-session count reaches zero, or the overlap window expires (whichever comes first), the node is decommissioned. Sessions that hadn't finished by then either see a session reset (acceptable for many applications) or, if externalized state was used, resume transparently on the new node.

For node addition, the reverse: add the virtual nodes, bump the version, and let new sessions start flowing to the new node immediately. Existing sessions are unaffected because their arcs on the ring did not move.

Session State Strategy

Prefer an external session store (a clustered cache such as Redis) over in-memory backend state whenever possible: it decouples session survival from any single backend's lifecycle entirely, removing the need for the migration step above. If in-memory sessions are unavoidable (e.g., a stateful protocol upgrade like a long-lived WebSocket), the migration path in step 4 is mandatory, not optional.

Worked Example: How Much Load a Removal Redistributes

Take 20 physical backends, each with $V = 150$ virtual nodes, so $T = 3000$ ring tokens total. Removing one physical backend removes its 150 tokens; each of those tokens' clockwise successors is, in a well-shuffled ring, effectively a random draw among the other 19 physical nodes. So the removed node's $\frac{1}{20} = 5\%$ share of keys spreads across roughly 19 recipients rather than one. Each surviving node's expected new share:

which is exactly the uniform share you'd expect from evenly splitting the whole keyspace across 19 nodes, the same identity that shows up whenever virtual-node count is high enough to approximate rendezvous hashing's (an alternative hashing scheme that scores every node directly against each key and always redistributes evenly on removal, without needing virtual nodes) exact uniform redistribution. Compare that to a single-token ring (no virtual nodes) removing one of 20 nodes: the removed node's entire 5% lands on one successor, whose share jumps from 5% to 10%, a 2x hotspot instead of a 0.26-point bump.

Trade-offs and Pitfalls

• The ring metadata service is a coordination dependency; if it uses a consensus protocol (Raft or similar) for consistency, that adds latency to ring mutations (acceptable, since mutations are rare) but also means the service itself needs its own HA story.
• Overlap window length is a direct trade-off between session continuity and routing complexity: a longer window keeps more in-flight sessions alive during churn but means proxies must correctly honor two ring versions simultaneously for longer.
• Signed ring-version cookies need HMAC key rotation handled carefully: rotating the signing key while old cookies are still in flight requires accepting both old and new keys for a transition period, or sessions get silently invalidated.
• If a CDN or edge cache sits in front of this layer, cache keys for session-specific content must not be shared across ring reassignment; either scope those responses as non-cacheable or ensure the cache key includes a stable shard identifier that survives rebalancing.
• A single very active session ("hot session") can skew load on whichever node currently owns it; virtual nodes fix cluster-wide statistical balance, not a single oversized key, so hot-key handling needs a separate mitigation (e.g., splitting that session's read traffic) if it becomes a real problem.