Load Balancing and Traffic Management Questions

Direct answer

An external load balancer is a single centralized hop that all traffic passes through; a sidecar-based service mesh (Envoy) puts a proxy next to every workload instance, decentralizing routing decisions to the edge of each service. The trade is centralized simplicity and lower per-call overhead against distributed per-call policy control and much richer observability, at the cost of running and operating a control plane.

Structured elaboration

graph LR
    subgraph ExternalLB [External Load Balancer Model]
        C1[Client] --> LB1[External LB]
        LB1 --> S1[Service A]
        LB1 --> S2[Service B]
    end
    subgraph Mesh [Sidecar Mesh Model]
        S3[Service A] --> P3[Envoy Sidecar A]
        P3 --> P4[Envoy Sidecar B]
        P4 --> S4[Service B]
    end

Control plane vs. data plane. A mesh formally separates the two: sidecars (data plane) enforce routing, retries, timeouts, and circuit-breaker policy on every call; a control plane (e.g. an xDS server, Envoy's discovery-protocol API for pushing routing and policy config out to proxies) pushes that configuration out to every sidecar. An external LB usually collapses both into one component. This separation is what makes per-route policy (canary weight, retry budget, header-based routing) something you can change centrally without touching a single service's code, but it's also the piece most likely to become a scale bottleneck.

How traffic distribution patterns change with a mesh. Without a mesh, load-balancing decisions concentrate at the LB, often coarse (round-robin or least-connections over a VIP). With a mesh, every calling service's sidecar makes its own load-balancing decision against the live, health-checked endpoint set for the callee, enabling patterns that are awkward at a centralized LB: zone-local routing to reduce cross-AZ cost, subset routing that only sends canary traffic to pods carrying a specific version label, and per-call hedging or retry budgets that a single shared LB cannot reasonably apply per-caller.

Worked example: a 200-service rollout

At 200 services, a single Envoy control plane pushing full configuration to every sidecar on every change is the concrete bottleneck to plan for. If each config push must reach, say, 4,000 total sidecar instances (200 services x an average 20 instances each) and the control plane naively recomputes and re-pushes full config on every endpoint change anywhere in the mesh:

At this scale, a mesh implementation without incremental (delta) config distribution, pushing only what changed to only the sidecars that care, turns a single pod restart anywhere in the fleet into a full-mesh config storm. This is why production mesh control planes (Istio's istiod, for example) implement scoped, incremental xDS updates rather than full-state pushes; it's the single most common cause of "the mesh worked fine in the pilot and fell over at scale" reports.

Trade-offs and pitfalls

• Don't treat this as all-or-nothing. A common, lower-risk pattern is to keep an external LB at the edge for north-south traffic (TLS termination, WAF, DDoS protection) and introduce the mesh only for east-west traffic, incrementally by namespace or team, to bound blast radius while the team builds operational maturity.
• The retry and circuit-breaker policy surface a mesh exposes is also a footgun surface. Misconfigured retries (no budget, no jitter) at the sidecar layer can amplify a downstream outage into a retry storm across the whole mesh; policy-as-code review and conservative defaults matter as much as the feature itself.
• mTLS (mutual TLS, where both client and server present certificates to authenticate each other, not just the server) by default is a real security upgrade, but certificate rotation and identity issuance become mesh-critical infrastructure; an outage in the certificate authority path is now an availability incident, not just a security one.
• Measure before generalizing about latency overhead. Sidecar proxy cost depends heavily on payload size, TLS handshake reuse, and filter-chain complexity; a number that's fine for a JSON API can be very different for a large-payload streaming workload.

Direct answer

When an L7 proxy tier like Envoy is CPU-saturated, the symptoms show up everywhere at once because CPU is the shared resource behind accept handling, filter processing, and control-plane config updates: rising accept-queue depth, growing request tail latency, and stale/delayed configuration from xDS (Envoy's config-discovery protocol, the channel that streams routing and policy updates out to the proxy) all point back to the same root cause. Confirming it means correlating kernel-level signals (accept backlog, run-queue length) with Envoy's own worker and listener stats, not just watching request latency in isolation. Mitigation is layered: immediate load-shedding and horizontal scale-out to stop the bleeding, then runtime tuning (worker count, filter chain cost, TLS offload) to fix the actual bottleneck.

Failure modes and how to confirm them

The confirming signal is a rising kernel run-queue length and accept-queue depth together with Envoy-reported worker CPU near 100%, not just high request latency alone, since request latency can rise for many unrelated reasons (slow upstreams, network issues).

Root causes worth checking

• Worker/thread count misconfigured relative to available vCPUs, or a container CPU limit lower than what Envoy's --concurrency assumes.
• CPU-heavy work on the hot path: TLS handshake/crypto without session reuse, verbose access logging, or a custom filter doing synchronous, expensive work per request.
• High-frequency xDS config churn competing with the request path for the same CPU.

Mitigation plan

Immediate (stop the bleeding):

1. Shed load explicitly (local or global rate limiting, or return 503 for non-critical routes) rather than letting the kernel accept queue silently grow.
2. Disable or bypass the most expensive optional filters (deep inspection, verbose logging) to buy back CPU.
3. Scale out horizontally (add replicas) so aggregate CPU capacity matches offered load; this is the correct lever specifically because the bottleneck is CPU, not memory or network.

Runtime tuning (fix the underlying cost):

• Right-size --concurrency to available vCPUs and confirm the container's CPU limit isn't throttling below that.
• Enable TLS session resumption and consider offloading TLS termination to a dedicated tier or hardware accelerator if crypto is the dominant cost.
• Reduce logging verbosity and stats cardinality on the hot path.
• Rate-limit or batch xDS updates so config churn doesn't compete with request processing during a load event.

Architectural (prevent recurrence):

• Set circuit breakers (max_connections, max_pending_requests, max_requests) so an overloaded proxy tier fails fast instead of queuing and amplifying retries.
• Maintain standing CPU headroom and autoscale on a leading indicator (accept-queue depth or worker CPU) rather than solely on request latency, which lags.

Worked example

Suppose load testing established that each Envoy worker (one per vCPU) safely sustains 3,000 RPS at around 70% CPU before tail latency degrades, and the current fleet is 20 instances of 8 vCPUs each:

If telemetry during the incident shows offered load of 620,000 RPS, that's 29% over the fleet's safe capacity, which is consistent with observed CPU saturation and explains the accept-queue growth. To restore headroom at the same per-instance ceiling, with a further 15% buffer against continued growth:

So scaling from 20 to 30 instances (10 additional replicas) restores safe headroom against the observed load, assuming the per-instance ceiling holds, i.e. that CPU (not some other resource) really is the binding constraint at the new scale too.

Trade-offs and pitfalls

• Aggressive load shedding protects the tier but trades away availability for non-critical traffic; it should be scoped (by route or priority) rather than uniform where possible.
• TLS offload reduces proxy CPU but adds a new component and a new trust boundary to operate and secure.
• Horizontal scaling only helps if CPU really is the bottleneck; if the real constraint turns out to be memory, NIC bandwidth, or a downstream dependency, adding replicas burns cost without fixing the incident, which is why confirming the failure mode (not just reacting to "latency is up") matters before choosing a mitigation.
• Scaling reactively off request latency alone is slower than scaling off accept-queue depth or worker CPU, since latency is a lagging signal.

Direct answer

Combine Anycast at the network layer (fastest first-hop routing to the nearest healthy point of presence, with free DDoS absorption) with GSLB (Global Server Load Balancing) at the DNS layer (coarser but policy-rich region-level steering: weighted, health-aware). Neither alone is enough: Anycast's BGP (Border Gateway Protocol, the protocol routers use to exchange and agree on internet routes) convergence is too slow and blunt for regional-outage failover, GSLB's DNS caching makes it too slow for first-hop latency decisions. Everything below has to be sized against the actual availability target, so start there.

Availability budget, and what it buys you

99.99% availability means:

That's roughly 4.4 minutes a month of total unavailability, budget for everything: deploys, incidents, and failovers. A regional failover that takes even a minute or two to fully propagate is a meaningful chunk of that budget on its own, which is why failover speed, not just failover correctness, is a first-class requirement here.

Architecture

flowchart TB
  Client --> Anycast[Anycast edge / POP]
  Anycast --> GSLB[GSLB / DNS steering]
  GSLB --> R1[Region A: regional LB + services]
  GSLB --> R2[Region B: regional LB + services]
  GSLB --> R3[Region C: regional LB + services]
  R1 --> HA[Health aggregator]
  R2 --> HA
  R3 --> HA
  HA -->|adjust weights| GSLB
  HA -->|withdraw route| Anycast

Routing options compared

The practical answer is hybrid: Anycast gets each client to a nearby POP quickly, GSLB (with short TTLs on the records that matter) makes the region-level policy decision behind that POP, and a hard BGP route withdrawal is the last-resort lever for a catastrophic regional failure that can't wait for DNS.

Failover without a traffic storm

When a region fails, the naive move (send its entire share to the remaining regions) means:

against a baseline of

per region, an increase of

on each survivor, delivered all at once if the shift isn't paced. Two regions absorbing a 50% step-increase in load at the instant of failover is exactly the traffic-storm failure mode. Mitigate it by ramping the weight shift (rate-limited, e.g. move 10% of the failed region's traffic every few seconds rather than all of it immediately), keeping standby capacity headroom in each region ahead of time so 500,000 RPS is inside its tested ceiling rather than a surprise, and shedding low-priority traffic first if headroom isn't enough.

Health propagation and session affinity

• Aggregate health from multiple vantage points, not just self-reported instance health, and require quorum before a region-wide unhealthy verdict propagates; a single flaky monitor should not be able to trigger a global failover.
• For the subset of requests needing session affinity, issue a signed token that encodes the assigned region and a fallback region; on failover, the token's fallback is honored, and the session state itself must already be replicated (or externalized to a shared store) or the fallback is meaningless.
• Stateless traffic ignores affinity entirely and should absorb the failover shift first, since it has no session to lose.

Trade-offs and pitfalls

• Anycast's biggest operational cost is BGP itself: route flapping, multi-homing complexity, and needing real network engineering expertise on call, not just application on-call.
• Short GSLB TTLs increase DNS query volume and infrastructure cost at this scale; there's a real trade-off between failover speed and DNS infrastructure load, not a free win.
• Spare capacity to absorb a region's worth of failover load is a standing cost paid every day whether or not a failover ever happens; budget and get sign-off for it explicitly rather than discovering the gap during an incident.
• A failover policy with no hysteresis (a strict threshold with no dwell time before reversing) will flap a borderline-healthy region in and out, which is often worse for users than staying degraded in one place.

Direct answer

For three active regions with a 5 minute RTO and eventual consistency accepted, DNS-based failover works if the failure-detection time, the DNS TTL, and resolver-propagation slack are budgeted explicitly against the 5 minute target, with the honest caveat that DNS-based failover has a hard, structural ceiling: some fraction of clients or resolvers will not honor TTL promptly, so this design meets a 5 minute RTO for the large majority of traffic, not a guarantee for every single client.

Structured elaboration

graph TD
    Client[Client] --> DNS[Authoritative DNS]
    DNS --> HC[Health Check Aggregator]
    HC -->|healthy: in rotation| RA[Region A LB]
    HC -->|healthy: in rotation| RB[Region B LB]
    HC -.->|FAIL: removed from DNS| RC[Region C LB]
    RA --> AppA[Region A App]
    RB --> AppB[Region B App]
    RC --> AppC[Region C App]

Health checks. Each region runs a synthetic health endpoint validating the app and its critical dependencies (DB reachability, cache reachability), probed from at least 3 independent locations. Require a majority of probes to agree before flipping a region's status, a single flaky probe should never trigger a regional failover.

DNS configuration. Weighted, health-checked records (e.g. Route 53 weighted routing with health checks, or an equivalent GSLB (Global Server Load Balancer, a DNS-based service that routes clients to different regions based on health and proximity)) for all three regions, roughly equal weights in steady state (34/33/33). On a confirmed FAIL, the provider's health check automatically removes that region's record from the answer set, no manual DNS edit required on the failure path.

Session handling. Prefer stateless sessions (signed JWTs) so any region can serve any request. If server-side session state is unavoidable, replicate it asynchronously across regions and design the app to tolerate a small window of stale or missing session data on failover, consistent with the "eventual consistency acceptable" requirement.

Worked example: the RTO budget

A 5 minute (300s) RTO has to cover detection, DNS propagation, and client-side retry, add these up explicitly rather than asserting the number:

That leaves a 150 second margin against the 300 second target, room to absorb a slower-than-expected health check aggregation step, a resolver that ignores TTL more aggressively than budgeted, or a partial rollout of the DNS change. If the measured margin were negative, the fix is to lower the TTL and/or tighten the detection interval, not to assume the RTO will be hit anyway.

Comparison: active-active vs. active-passive

Given eventual consistency is acceptable and three regions are already active, active-active is the natural fit here: the RTO budget above assumes surviving regions are already warm and serving, which is only true in active-active. An active-passive design would need to add standby promotion and cache warm-up time on top of the same DNS budget, likely pushing past the 5 minute target unless the standby is kept continuously warm (which erodes most of its cost advantage). A p99 < 200ms latency target reinforces the same conclusion from a different angle: it is a steady-state requirement, not just a failover one, and the same latency-based routing that gives active-active its fast RTO is also what keeps ordinary traffic on the nearest healthy region so the latency target is met even when nothing has failed. Active-passive would route every region's users through one primary, so a global p99 < 200ms target would require either a large, latency-sensitive edge/CDN layer in front of it or accepting that distant users miss the target entirely.

Failback

1. Confirm the recovered region is fully healthy (health checks green) and, if it holds any writable state, that data has caught up to an acceptable staleness bound.
2. Re-introduce it at a low weight (e.g. 5 to 10%) rather than immediately restoring equal weights, to reduce risk from a still-caching population and to let its caches warm under real traffic.
3. Ramp weight back to steady state over a defined window while watching error rate and latency.
4. Run a post-incident review, in particular checking for any data divergence accumulated during the outage that eventual consistency needs to reconcile.

Trade-offs and pitfalls

• DNS-based failover's fundamental limitation is that it is a caching system, not a control system. It can only ever bound resolver-level staleness by TTL; it cannot force an already-cached client to re-resolve, and it cannot see or control every resolver's actual caching behavior.
• A concrete shape of that limitation: deprecated IPs served from stale cache. After a region is pulled from DNS rotation, some resolvers and long-lived clients keep resolving to, or keep an open connection against, that now-decommissioned region's IP well past the configured TTL. Some public resolvers apply their own TTL floors, some corporate or OS-level resolvers cache longer than instructed, and a client that already holds an open connection or a cached resolved IP will not re-resolve until that connection breaks. Real incidents from this look like a small, persistent tail of traffic still hitting a region minutes after it was pulled everywhere else.
• Client-side mitigations bound this blast radius, they do not eliminate it. Retry-with-re-resolution on connection failure (treat a failed connect or a run of failed requests as a signal to force a fresh DNS lookup instead of reusing the cached IP), connection pools that honor the record's TTL and recycle idle connections rather than holding them indefinitely, and a happy-eyeballs-style fallback that races the primary resolved address against a secondary one and fails over at the connection layer if the first does not respond. None of these are under server-side control, they have to be built into the client or its SDK.
• Lower TTL is not free. Halving the TTL from 60s to 30s tightens the worst-case staleness bound to about 30s but roughly doubles steady-state DNS query volume against the authoritative servers, since twice as many cached entries expire and get re-resolved per unit time; the 60s TTL chosen above is a deliberate balance point between failover speed and DNS infrastructure load, not the lowest theoretically possible value.
• Anycast (announcing the same IP address from multiple locations and letting network routing send each client to the nearest one) or a client-side/edge load balancer removes the DNS-caching ceiling entirely by routing at the network layer instead of the naming layer, worth naming as the answer when a client asks "how do you get below what DNS can offer."
• Failback done too fast is a common cause of a second incident: dumping full traffic back onto a region whose caches are still cold reproduces the exact warm-up latency spike that a proper connection-draining and readiness gate is designed to prevent: stop routing new traffic to a recovering region, ramp its share up gradually instead of all at once, and only count it at full weight once its caches and connection pools have actually warmed under real traffic, the same low-weight reintroduction and ramp already described above in Failback steps 2-3.

Direct answer

Both are ways to make an algorithm capacity-aware, but they optimize different things. Weighted round robin (WRR) distributes a fixed pattern of requests proportional to weight, without looking at current load: it is a scheduling problem, solved up front. Weighted least connections picks, per request, whichever server has the lowest connections-to-weight ratio: it is a live load-balancing decision that reacts to what's actually happening on each server right now. WRR is cheaper and predictable; weighted least connections is more adaptive when request duration varies.

Weighted round robin mechanics

A naive WRR just repeats each server in the rotation list a number of times equal to its weight (weight 5, 1, 1 becomes the list [A,A,A,A,A,B,C]), which is simple but bursty, all of A's requests land in a clump. The standard fix is smooth weighted round robin, used by nginx and others: each server keeps a running counter, and at every request:

then the server with the highest counter is selected, and only that server's counter is reduced by the total weight:

Worked trace: weights A=5, B=1, C=1 (W=7)

Sequence: A, A, B, A, C, A, A. Final counts: A=5, B=1, C=1, exactly matching the declared weights, and after 7 steps every counter returns to 0, so the pattern repeats cleanly. Compare this to the naive list [A,A,A,A,A,B,C]: smooth WRR interleaves B and C between A's turns instead of clumping them at the end.

Weighted least connections mechanics

Instead of a precomputed pattern, each request goes to whichever server minimizes:

where $c_i$ is current active connections. This needs live state (the balancer must track connection counts accurately) but it self-corrects when request durations vary: if one of A's requests is unusually slow and its connection count stays elevated, weighted least connections routes around it immediately, while WRR would keep sending A its scheduled 5-out-of-7 share regardless.

When to adjust weights dynamically

• Autoscaling: when an instance joins or leaves the pool, recompute weights from the group's new capacity (commonly proportional to vCPU count or a load-tested throughput figure), otherwise new instances sit idle while old ones stay pinned to stale weights.
• Degraded instance: on failed or borderline health checks, reduce the instance's weight toward zero (a soft drain) rather than hard-removing it, so in-flight requests finish before it stops receiving new ones.
• Heterogeneous hardware: base weights on measured throughput under load, not just instance-type labels, since two "same size" instances can have different real capacity due to noisy neighbors or different hardware generations.
• Oscillation risk: reacting to every latency blip by changing weights can cause a feedback loop (a server's weight drops, it gets less traffic, its metrics improve, weight goes back up, traffic returns, metrics degrade again). Apply a decay or minimum-dwell-time before a weight change takes effect.

Trade-offs and pitfalls

WRR is stateless and cheap to run at very high request rates because it never inspects live connection counts, but it is blind to reality: if a server silently starts responding slowly, WRR keeps sending it its full scheduled share. Weighted least connections fixes that but needs accurate, low-latency visibility into connection counts across the fleet, which is harder in a distributed balancer with multiple LB instances that don't share state. A common middle ground is weighted least connections with a smoothing factor so weight changes and connection counts don't cause the oscillation described above.