Load Balancing and Traffic Management Questions
Distributing requests across capacity: load-balancing algorithms (round-robin, least-connections, consistent hashing), L4 versus L7 balancing, health checks, and traffic shaping. Covers sticky sessions, canary and blue-green routing, rate limiting, and graceful draining. The traffic-distribution layer that keeps a scaled system balanced and available.
You are choosing a load balancing strategy for a model-serving fleet where some models need GPU warm-up (making certain instances effectively sticky) and others are sharded across instances. Compare round robin, least connections, and consistent hashing for this workload, and explain how the load balancer's health checks and autoscaling should interact with your choice.
Sample Answer
Direct answer
Load balancing is just the policy for deciding which backend instance handles the next request; for a mixed fleet of GPU-warm-up models and sharded models, the right choice is not one algorithm but two: session or model-key affinity (via consistent hashing) for anything that pays a warm-up cost or owns a shard, and least-connections for anything stateless where you just want to spread inference load evenly. Round robin is rarely the right default here because it ignores both state and load.
Structured elaboration
Comparing the three algorithms for this workload:
| Algorithm | Behavior | Fit for GPU warm-up stickiness | Fit for model sharding |
|---|---|---|---|
| Round robin | Cycles requests evenly across healthy backends, ignores load and state | Poor: a request can land on an instance that hasn't loaded/warmed the needed model, forcing a cold load | Poor: no guarantee a shard-owning instance gets the request for its shard |
| Least connections | Routes to the backend with fewest active connections | Weak: balances load well but has no concept of "this instance already has the model warm," so it can route away from a warm instance toward an idle-but-cold one | Weak: same reason, no shard awareness |
| Consistent hashing (on model id or shard key) | Deterministically maps a key to a backend, stable under scaling | Strong: repeated requests for the same model land on the same warmed instance, amortizing warm-up cost | Strong: a shard key maps to the instance owning that shard, correctness-critical for sharded serving |
Why this fleet needs both, not one: treat GPU-warmed models as sticky by hashing on model id (or model id plus a session/tenant id if you also want request affinity within one caller's session), and treat shard-routing the same way, hash on shard key so requests always land on the instance holding that shard. For any remaining stateless, non-sharded inference traffic where any warm instance can serve it equally well, least-connections is the better default over round robin because inference request duration varies (some prompts and batch sizes cost more GPU time than others), and least-connections adapts to that where round robin does not.
Health checks: the critical nuance for GPU-backed nodes is that "process is up" and "ready to serve" are different states. A model that is still loading onto the GPU is alive but must not receive traffic yet, so readiness must gate on model-load completion, not just process liveness. Liveness probes should stay lightweight (process/health endpoint responds), while readiness probes should reflect actual serving capability (model loaded, GPU memory allocated, a lightweight warm-up inference has succeeded).
Autoscaling interaction: scale-out and scale-in both interact with the routing choice:
- Scale-out: a newly added instance starts cold. If it immediately receives its full share of consistent-hash traffic before its model is warm, the first requests routed to it pay the warm-up penalty. Keep it out of the hash ring (or hold its readiness false) until warm-up completes, so consistent hashing only remaps traffic to instances that are actually ready.
- Scale-in: draining matters more here than in a stateless fleet. An instance being removed may be the sole owner of a shard's affinity; abruptly removing it forces every request for that shard/model to cold-start elsewhere at once. Drain gracefully (stop new routing, let in-flight inferences finish) and, if using consistent hashing, removing the node causes only its share of keys to remap (proportional to its ring weight), not the whole fleet, this is the property that makes consistent hashing preferable to a hash-mod-N scheme for a fleet that autoscales.
- GPU utilization as an autoscaling signal: request count alone is a weak autoscaling signal for GPU serving because request cost varies enormously by model and batch size; GPU utilization or queue depth is a better trigger than instance count or plain request rate.
Worked example
Use a simplified illustrative hash (real systems use SHA-256 or a similarly well-distributed hash, not this): sum the ASCII codes of a key's characters and take the result mod 100 to place it on a 0-99 ring.
Four GPU-backed instances sit on the ring at fixed positions: I0 = 5, I1 = 68, I2 = 80, I3 = 95 (a request is served by the first instance whose position is at or after the key's hash, walking clockwise and wrapping to I0 if the hash falls past the last position).
| Key | ASCII sum | mod 100 | Routed to |
|---|---|---|---|
| "resnet50-v2" | 971 | 71 | I2 (80) |
| "bert-base" | 885 | 85 | I3 (95) |
| "gpt2-small" | 963 | 63 | I1 (68) |
A second, later request for "resnet50-v2" hashes to the same value (971 mod 100 = 71) every time, so it lands on I2 again: this is the same-model-same-instance stickiness the comparison table above claims, made concrete with an actual routing decision instead of just asserted.
Now scale out: add a fifth instance, I4, at position 75, between I1 (68) and I2 (80). Only keys whose hash falls in the range (68, 75] now remap, because I4 becomes the new first successor for that slice of the ring:
- "resnet50-v2" (71) falls in (68, 75], so it remaps from I2 to I4. Any warm model state resnet50-v2 had on I2 is now cold on I4, a real cost of scaling out a sticky fleet, not a free capacity add.
- "bert-base" (85) and "gpt2-small" (63) both fall outside (68, 75], so they keep routing to their original instances (I3 and I1 respectively), completely undisturbed by the scale-out.
This is the bounded-remap property in action: adding one instance only disturbs the slice of keyspace between its new position and its clockwise predecessor's, not the whole ring, which is exactly why the comparison table above can claim consistent hashing amortizes warm-up cost even as the fleet autoscales.
Trade-offs and pitfalls
- Consistent hashing does not fix an imbalanced key distribution. If a small number of models or shards receive most of the traffic (a hot key), those specific instances will be overloaded no matter how evenly the hash ring is built; this needs a separate mitigation (replicate hot models across multiple instances and route among the replicas) rather than a change to the base algorithm.
- Sticky routing conflicts with fast autoscaling. The more you rely on affinity for warm-up amortization, the slower the fleet can usefully absorb a scale-out event, because new capacity does not help until traffic actually remaps to it. This is a real throughput-versus-elasticity trade, not a solved problem.
- Readiness gating during warm-up is easy to get backwards. If readiness returns true before the model is actually loaded (a common mistake when readiness is copy-pasted from a stateless service template), the load balancer will send live traffic to an instance that cannot serve it yet, causing exactly the cold-start failures the affinity strategy was meant to prevent.
- Least-connections alone does not protect a warm instance from being starved of the very traffic that keeps it warm. If a model's affinity key is dropped in favor of pure least-connections once traffic is "spread out," idle time can cause a warm instance to cool (evicted from GPU memory), reintroducing the warm-up cost on its next request.
That is every published Load Balancing and Traffic Management question for AI Engineer so far. Browse the other topics in this category, or practice this one interactively.