Load Balancing and Traffic Management Questions

Direct answer

For three active regions with a 5 minute RTO and eventual consistency accepted, DNS-based failover works if the failure-detection time, the DNS TTL, and resolver-propagation slack are budgeted explicitly against the 5 minute target, with the honest caveat that DNS-based failover has a hard, structural ceiling: some fraction of clients or resolvers will not honor TTL promptly, so this design meets a 5 minute RTO for the large majority of traffic, not a guarantee for every single client.

Structured elaboration

graph TD
    Client[Client] --> DNS[Authoritative DNS]
    DNS --> HC[Health Check Aggregator]
    HC -->|healthy: in rotation| RA[Region A LB]
    HC -->|healthy: in rotation| RB[Region B LB]
    HC -.->|FAIL: removed from DNS| RC[Region C LB]
    RA --> AppA[Region A App]
    RB --> AppB[Region B App]
    RC --> AppC[Region C App]

Health checks. Each region runs a synthetic health endpoint validating the app and its critical dependencies (DB reachability, cache reachability), probed from at least 3 independent locations. Require a majority of probes to agree before flipping a region's status, a single flaky probe should never trigger a regional failover.

DNS configuration. Weighted, health-checked records (e.g. Route 53 weighted routing with health checks, or an equivalent GSLB (Global Server Load Balancer, a DNS-based service that routes clients to different regions based on health and proximity)) for all three regions, roughly equal weights in steady state (34/33/33). On a confirmed FAIL, the provider's health check automatically removes that region's record from the answer set, no manual DNS edit required on the failure path.

Session handling. Prefer stateless sessions (signed JWTs) so any region can serve any request. If server-side session state is unavoidable, replicate it asynchronously across regions and design the app to tolerate a small window of stale or missing session data on failover, consistent with the "eventual consistency acceptable" requirement.

Worked example: the RTO budget

A 5 minute (300s) RTO has to cover detection, DNS propagation, and client-side retry, add these up explicitly rather than asserting the number:

That leaves a 150 second margin against the 300 second target, room to absorb a slower-than-expected health check aggregation step, a resolver that ignores TTL more aggressively than budgeted, or a partial rollout of the DNS change. If the measured margin were negative, the fix is to lower the TTL and/or tighten the detection interval, not to assume the RTO will be hit anyway.

Comparison: active-active vs. active-passive

Given eventual consistency is acceptable and three regions are already active, active-active is the natural fit here: the RTO budget above assumes surviving regions are already warm and serving, which is only true in active-active. An active-passive design would need to add standby promotion and cache warm-up time on top of the same DNS budget, likely pushing past the 5 minute target unless the standby is kept continuously warm (which erodes most of its cost advantage). A p99 < 200ms latency target reinforces the same conclusion from a different angle: it is a steady-state requirement, not just a failover one, and the same latency-based routing that gives active-active its fast RTO is also what keeps ordinary traffic on the nearest healthy region so the latency target is met even when nothing has failed. Active-passive would route every region's users through one primary, so a global p99 < 200ms target would require either a large, latency-sensitive edge/CDN layer in front of it or accepting that distant users miss the target entirely.

Failback

1. Confirm the recovered region is fully healthy (health checks green) and, if it holds any writable state, that data has caught up to an acceptable staleness bound.
2. Re-introduce it at a low weight (e.g. 5 to 10%) rather than immediately restoring equal weights, to reduce risk from a still-caching population and to let its caches warm under real traffic.
3. Ramp weight back to steady state over a defined window while watching error rate and latency.
4. Run a post-incident review, in particular checking for any data divergence accumulated during the outage that eventual consistency needs to reconcile.

Trade-offs and pitfalls

• DNS-based failover's fundamental limitation is that it is a caching system, not a control system. It can only ever bound resolver-level staleness by TTL; it cannot force an already-cached client to re-resolve, and it cannot see or control every resolver's actual caching behavior.
• A concrete shape of that limitation: deprecated IPs served from stale cache. After a region is pulled from DNS rotation, some resolvers and long-lived clients keep resolving to, or keep an open connection against, that now-decommissioned region's IP well past the configured TTL. Some public resolvers apply their own TTL floors, some corporate or OS-level resolvers cache longer than instructed, and a client that already holds an open connection or a cached resolved IP will not re-resolve until that connection breaks. Real incidents from this look like a small, persistent tail of traffic still hitting a region minutes after it was pulled everywhere else.
• Client-side mitigations bound this blast radius, they do not eliminate it. Retry-with-re-resolution on connection failure (treat a failed connect or a run of failed requests as a signal to force a fresh DNS lookup instead of reusing the cached IP), connection pools that honor the record's TTL and recycle idle connections rather than holding them indefinitely, and a happy-eyeballs-style fallback that races the primary resolved address against a secondary one and fails over at the connection layer if the first does not respond. None of these are under server-side control, they have to be built into the client or its SDK.
• Lower TTL is not free. Halving the TTL from 60s to 30s tightens the worst-case staleness bound to about 30s but roughly doubles steady-state DNS query volume against the authoritative servers, since twice as many cached entries expire and get re-resolved per unit time; the 60s TTL chosen above is a deliberate balance point between failover speed and DNS infrastructure load, not the lowest theoretically possible value.
• Anycast (announcing the same IP address from multiple locations and letting network routing send each client to the nearest one) or a client-side/edge load balancer removes the DNS-caching ceiling entirely by routing at the network layer instead of the naming layer, worth naming as the answer when a client asks "how do you get below what DNS can offer."
• Failback done too fast is a common cause of a second incident: dumping full traffic back onto a region whose caches are still cold reproduces the exact warm-up latency spike that a proper connection-draining and readiness gate is designed to prevent: stop routing new traffic to a recovering region, ramp its share up gradually instead of all at once, and only count it at full weight once its caches and connection pools have actually warmed under real traffic, the same low-weight reintroduction and ramp already described above in Failback steps 2-3.

Direct answer

All four algorithms answer the same question, "has this client used too much capacity", but they differ in what they track (a count in a time bucket vs a bank of tokens vs a queue) and in how they treat bursts. Token bucket and leaky bucket are the two production workhorses, though the specific tool decides which: nginx's limit_req module implements leaky bucket by its own documentation, while Envoy's local rate limit filter and most API gateways default to token bucket; fixed window is the simplest and the one with the well-known boundary flaw; sliding window is the fix for that flaw at some extra cost.

How each algorithm works

Fairness in all four comes from your key, not the algorithm: per-client buckets/counters are fair, a shared global counter or queue lets one aggressive client crowd out the rest. Leaky bucket is the exception worth calling out, since a single shared queue is the one case where "fair" requires a deliberate per-client sub-queue design, not just a per-client key.

Worked example: the fixed-window boundary spike, bounded by token bucket

Take a limit of $L = 100$ requests per minute ($T = 60\text{s}$).

Fixed window: nothing stops a client from sending its full limit in the last instant of window $k$ and its full limit again in the first instant of window $k+1$, since the counter resets at the boundary regardless of timing. The worst-case burst that can land in an arbitrarily short span straddling a boundary is:

twice the nominal per-minute limit, with no time gap required between them.

Token bucket, configured with capacity $C = L = 100$ and refill rate

cannot exceed $C$ tokens instantaneously no matter when the burst lands, because tokens only exist if they were refilled or never spent. Its worst-case burst is bounded by the bucket's own capacity:

a strictly tighter and, crucially, tunable bound (shrink $C$ for stricter burst control, or grow it for more headroom) that fixed window cannot offer without a redesign.

Trade-offs and pitfalls

• Fixed window is cheap (one counter, one TTL) but the $2L$ boundary spike is a real production failure mode, not a theoretical one; it shows up as bursty traffic right at minute or hour boundaries.
• Sliding window counter (the common production approximation) trades exactness for cheap, fixed-size storage by blending the previous window's count, weighted by how much of it still overlaps the current interval; the sliding log is exact but its memory grows with request volume, which matters at high request rates per client.
• Token bucket's two knobs, capacity and refill rate, are independent: capacity controls burst tolerance, refill rate controls sustained average. Conflating them (e.g. setting capacity to one second's worth of the sustained rate) either kills legitimate bursts or defeats the limit.
• Leaky bucket is the right choice when you must protect a downstream that cannot absorb bursts at all (a fixed-rate consumer), but adding a queue trades burst-drop for burst-delay; a queue that's too deep just turns a rate-limit problem into a latency problem.
• Distributed enforcement (the same client hitting many nodes) is the pitfall every one of these hits eventually: none of the four is inherently safe for a multi-instance deployment without a shared store or an approximation scheme layered on top.

Direct answer

When an L7 proxy tier like Envoy is CPU-saturated, the symptoms show up everywhere at once because CPU is the shared resource behind accept handling, filter processing, and control-plane config updates: rising accept-queue depth, growing request tail latency, and stale/delayed configuration from xDS (Envoy's config-discovery protocol, the channel that streams routing and policy updates out to the proxy) all point back to the same root cause. Confirming it means correlating kernel-level signals (accept backlog, run-queue length) with Envoy's own worker and listener stats, not just watching request latency in isolation. Mitigation is layered: immediate load-shedding and horizontal scale-out to stop the bleeding, then runtime tuning (worker count, filter chain cost, TLS offload) to fix the actual bottleneck.

Failure modes and how to confirm them

The confirming signal is a rising kernel run-queue length and accept-queue depth together with Envoy-reported worker CPU near 100%, not just high request latency alone, since request latency can rise for many unrelated reasons (slow upstreams, network issues).

Root causes worth checking

• Worker/thread count misconfigured relative to available vCPUs, or a container CPU limit lower than what Envoy's --concurrency assumes.
• CPU-heavy work on the hot path: TLS handshake/crypto without session reuse, verbose access logging, or a custom filter doing synchronous, expensive work per request.
• High-frequency xDS config churn competing with the request path for the same CPU.

Mitigation plan

Immediate (stop the bleeding):

1. Shed load explicitly (local or global rate limiting, or return 503 for non-critical routes) rather than letting the kernel accept queue silently grow.
2. Disable or bypass the most expensive optional filters (deep inspection, verbose logging) to buy back CPU.
3. Scale out horizontally (add replicas) so aggregate CPU capacity matches offered load; this is the correct lever specifically because the bottleneck is CPU, not memory or network.

Runtime tuning (fix the underlying cost):

• Right-size --concurrency to available vCPUs and confirm the container's CPU limit isn't throttling below that.
• Enable TLS session resumption and consider offloading TLS termination to a dedicated tier or hardware accelerator if crypto is the dominant cost.
• Reduce logging verbosity and stats cardinality on the hot path.
• Rate-limit or batch xDS updates so config churn doesn't compete with request processing during a load event.

Architectural (prevent recurrence):

• Set circuit breakers (max_connections, max_pending_requests, max_requests) so an overloaded proxy tier fails fast instead of queuing and amplifying retries.
• Maintain standing CPU headroom and autoscale on a leading indicator (accept-queue depth or worker CPU) rather than solely on request latency, which lags.

Worked example

Suppose load testing established that each Envoy worker (one per vCPU) safely sustains 3,000 RPS at around 70% CPU before tail latency degrades, and the current fleet is 20 instances of 8 vCPUs each:

If telemetry during the incident shows offered load of 620,000 RPS, that's 29% over the fleet's safe capacity, which is consistent with observed CPU saturation and explains the accept-queue growth. To restore headroom at the same per-instance ceiling, with a further 15% buffer against continued growth:

So scaling from 20 to 30 instances (10 additional replicas) restores safe headroom against the observed load, assuming the per-instance ceiling holds, i.e. that CPU (not some other resource) really is the binding constraint at the new scale too.

Trade-offs and pitfalls

• Aggressive load shedding protects the tier but trades away availability for non-critical traffic; it should be scoped (by route or priority) rather than uniform where possible.
• TLS offload reduces proxy CPU but adds a new component and a new trust boundary to operate and secure.
• Horizontal scaling only helps if CPU really is the bottleneck; if the real constraint turns out to be memory, NIC bandwidth, or a downstream dependency, adding replicas burns cost without fixing the incident, which is why confirming the failure mode (not just reacting to "latency is up") matters before choosing a mitigation.
• Scaling reactively off request latency alone is slower than scaling off accept-queue depth or worker CPU, since latency is a lagging signal.

Direct answer

Keep an immutable snapshot (tuple) of backends and a single lightweight lock that only protects a counter increment. get_next_backend reads the snapshot reference (a single atomic pointer read in Python, and cheap in most languages), takes the lock only long enough to advance an integer counter, and computes the index modulo the snapshot length outside the lock. add_backend/remove_backend build a new snapshot under a separate lock. This keeps the hot path (get_next_backend) to one tiny critical section instead of locking the whole backend list on every call.

Approach

Two separate locks serve two different purposes:

• A write lock around mutating the backend list, so add_backend/remove_backend never race with each other, and each mutation publishes a brand-new tuple (copy-on-write) rather than mutating a shared list in place.
• A counter lock around advancing the round-robin index, so concurrent get_next_backend calls never read the same index twice; this is the only lock the hot path pays for.

Reading the current backend tuple needs no lock at all: in CPython, and in most managed runtimes, reassigning a reference (self._backends = new_tuple) is atomic with respect to readers, a reader either sees the old tuple or the new one, never a partially-built one. If you needed this guarantee in a language without that property, you'd wrap the read in an atomic pointer/reference type instead of relying on the runtime.

Code (Python)

import threading
from itertools import count


class RoundRobinLoadBalancer:
    def __init__(self):
        self._backends = ()          # immutable snapshot, swapped under _write_lock
        self._write_lock = threading.Lock()
        self._counter = count()      # advanced only while holding _counter_lock
        self._counter_lock = threading.Lock()

    def add_backend(self, backend_id):
        with self._write_lock:
            if backend_id in self._backends:
                return
            self._backends = self._backends + (backend_id,)

    def remove_backend(self, backend_id):
        with self._write_lock:
            if backend_id not in self._backends:
                return
            self._backends = tuple(b for b in self._backends if b != backend_id)

    def get_next_backend(self):
        backends = self._backends       # single reference read, no lock needed
        n = len(backends)
        if n == 0:
            return None
        with self._counter_lock:
            i = next(self._counter)
        return backends[i % n]

Key points

• get_next_backend is the hot path and is designed to hold a lock for the shortest possible time: only the counter increment, not the modulo or the indexing.
• add_backend/remove_backend are copy-on-write: each call allocates a new tuple rather than mutating the existing one in place, which is what lets readers avoid locking. This is a deliberate trade of O(n) mutation cost for lock-free reads, correct because backend membership changes are far rarer than get_next_backend calls in a load balancer.
• The counter grows without bound; since it is only ever used modulo the current backend count, this is safe in Python (unbounded integers) and in any language with a 64-bit counter, the counter would need billions of requests per second sustained for years before wraparound became a practical concern.

Complexity

• get_next_backend: O(1) time, holds the counter lock for O(1) work only.
• add_backend/remove_backend: O(n) time to rebuild the tuple, under the write lock; this does not block concurrent readers.
• Space: O(n) for the backend snapshot.

Edge cases

• Empty backend list: get_next_backend returns None rather than raising, so callers must check for that.
• Removing a backend that the counter currently "points to" mid-rotation: correct by construction here, because the index is always taken modulo the snapshot's length at read time, so a shrink just means the sequence continues over the smaller set without needing to adjust the counter.
• Duplicate add_backend calls: no-op if the id already exists, verified in the sanity run below.
• Concurrent add_backend/remove_backend happening while many threads call get_next_backend: verified in the run below to complete without error and to only ever return currently-valid backend ids.

Running the implementation with a single-threaded rotation check, then 8 threads each making 2000 calls, then a concurrent add/remove stress test against 5000 reads:

single-thread sequence: ['a', 'b', 'c', 'a', 'b', 'c', 'a', 'b', 'c']
total picks: 16000
distribution: {'a': 5334, 'b': 5333, 'c': 5333}
concurrent add/remove during reads: OK, final backends = ('a', 'b', 'c')

16000 picks across 3 backends split 5334/5333/5333, confirming the rotation stays even under concurrent load, and the concurrent add/remove stress test completed with no exceptions and no invalid backend ever returned.

Trade-offs and pitfalls

• Copy-on-write is wrong if backends churn constantly (e.g., hundreds of adds/removes per second from a very chatty autoscaler): the O(n) rebuild cost on every mutation would dominate. In that regime, a version-counter-plus-fixed-array design or a lock-free ring buffer would fit better.
• A single global lock around everything is a common but weaker first instinct. It's simpler to reason about, but every get_next_backend call then contends with every other call and with every mutation, which is exactly the contention this design avoids.
• This is not linearizable across add/remove and reads happening at the "same instant." A reader can legitimately see the backend list either just before or just after a concurrent mutation; if a caller needs strict linearizability (every operation appears to happen at a single global point in time), that requires a stronger primitive (a single mutex around all operations) at the cost of read throughput.

Direct answer

An external load balancer is a single centralized hop that all traffic passes through; a sidecar-based service mesh (Envoy) puts a proxy next to every workload instance, decentralizing routing decisions to the edge of each service. The trade is centralized simplicity and lower per-call overhead against distributed per-call policy control and much richer observability, at the cost of running and operating a control plane.

Structured elaboration

graph LR
    subgraph ExternalLB [External Load Balancer Model]
        C1[Client] --> LB1[External LB]
        LB1 --> S1[Service A]
        LB1 --> S2[Service B]
    end
    subgraph Mesh [Sidecar Mesh Model]
        S3[Service A] --> P3[Envoy Sidecar A]
        P3 --> P4[Envoy Sidecar B]
        P4 --> S4[Service B]
    end

Control plane vs. data plane. A mesh formally separates the two: sidecars (data plane) enforce routing, retries, timeouts, and circuit-breaker policy on every call; a control plane (e.g. an xDS server, Envoy's discovery-protocol API for pushing routing and policy config out to proxies) pushes that configuration out to every sidecar. An external LB usually collapses both into one component. This separation is what makes per-route policy (canary weight, retry budget, header-based routing) something you can change centrally without touching a single service's code, but it's also the piece most likely to become a scale bottleneck.

How traffic distribution patterns change with a mesh. Without a mesh, load-balancing decisions concentrate at the LB, often coarse (round-robin or least-connections over a VIP). With a mesh, every calling service's sidecar makes its own load-balancing decision against the live, health-checked endpoint set for the callee, enabling patterns that are awkward at a centralized LB: zone-local routing to reduce cross-AZ cost, subset routing that only sends canary traffic to pods carrying a specific version label, and per-call hedging or retry budgets that a single shared LB cannot reasonably apply per-caller.

Worked example: a 200-service rollout

At 200 services, a single Envoy control plane pushing full configuration to every sidecar on every change is the concrete bottleneck to plan for. If each config push must reach, say, 4,000 total sidecar instances (200 services x an average 20 instances each) and the control plane naively recomputes and re-pushes full config on every endpoint change anywhere in the mesh:

At this scale, a mesh implementation without incremental (delta) config distribution, pushing only what changed to only the sidecars that care, turns a single pod restart anywhere in the fleet into a full-mesh config storm. This is why production mesh control planes (Istio's istiod, for example) implement scoped, incremental xDS updates rather than full-state pushes; it's the single most common cause of "the mesh worked fine in the pilot and fell over at scale" reports.

Trade-offs and pitfalls

• Don't treat this as all-or-nothing. A common, lower-risk pattern is to keep an external LB at the edge for north-south traffic (TLS termination, WAF, DDoS protection) and introduce the mesh only for east-west traffic, incrementally by namespace or team, to bound blast radius while the team builds operational maturity.
• The retry and circuit-breaker policy surface a mesh exposes is also a footgun surface. Misconfigured retries (no budget, no jitter) at the sidecar layer can amplify a downstream outage into a retry storm across the whole mesh; policy-as-code review and conservative defaults matter as much as the feature itself.
• mTLS (mutual TLS, where both client and server present certificates to authenticate each other, not just the server) by default is a real security upgrade, but certificate rotation and identity issuance become mesh-critical infrastructure; an outage in the certificate authority path is now an availability incident, not just a security one.
• Measure before generalizing about latency overhead. Sidecar proxy cost depends heavily on payload size, TLS handshake reuse, and filter-chain complexity; a number that's fine for a JSON API can be very different for a large-payload streaming workload.