Load Balancing and Traffic Management Questions

Direct Answer

Enforce limits locally at the edge with a fast in-memory bucket, and reconcile that local state against a global store periodically rather than checking the global store on every request. Which algorithm backs the global side depends on how much overshoot is acceptable: a centralized token bucket (a counter that holds a capped number of request "tokens", drains one per allowed request, and refills at a steady rate, enforcing both a sustained rate and a bounded burst) gives exact enforcement but adds cross-region latency and a single dependency, distributed counters trade a bounded, quantifiable overshoot for availability, and probabilistic sketches trade a small one-directional error rate for handling very large key cardinalities cheaply. Almost every production global limiter ends up as a hybrid of local enforcement plus asynchronous global reconciliation, with a documented, bounded degradation path for when the global store is unreachable.

Architecture

flowchart LR
  Client --> EdgeA[Edge Gateway - Region A]
  Client --> EdgeB[Edge Gateway - Region B]
  EdgeA -->|async sync| GlobalStore[(Global Store, quorum)]
  EdgeB -->|async sync| GlobalStore
  GlobalStore --> Reconciler[Reconciliation Job]
  Reconciler -->|adjust local cap| EdgeA
  Reconciler -->|adjust local cap| EdgeB
  EdgeA -.fail open.-> LocalCap[Conservative Local Cap]
  EdgeB -.fail open.-> LocalCap

Each edge gateway enforces a local token bucket immediately (no cross-region round trip on the request path) and asynchronously syncs its counts to the global store. The reconciliation job periodically redistributes each user's remaining global budget across the regions currently seeing their traffic. If the global store is unreachable, each edge gateway falls back to a conservative local cap rather than allowing unlimited traffic.

Comparing the Three Approaches

Sharding User Keys

Route a given user's traffic to a small, bounded set of enforcement points via consistent hashing on the user key, rather than letting it fan out across the whole fleet. This does two things: it keeps the local-bucket state for one user concentrated on a few nodes (so local enforcement is actually meaningful), and it bounds the worst case for the fail-open fallback, shown concretely below.

Worked Example: Sizing the Fail-Open Cap Correctly

Say the intended global limit is 600 requests/minute per user. The naive fail-open formula is often stated as:

The critical detail is what $N_{\text{active nodes}}$ should mean. It is not the size of the whole fleet, it is the number of distinct enforcement points a single user's traffic can concurrently reach.

Correct sizing. Suppose user keys are consistent-hashed to exactly $R = 3$ regions (a primary plus two fallbacks). The fail-open cap per region:

If the global store is fully partitioned and the user simultaneously maxes out all 3 reachable regions, the worst-case total is:

exactly the intended global bound, with zero overshoot, even during a total global-store outage.

Incorrect sizing. If instead the fail-open cap is naively divided across the entire fleet of, say, $F = 50$ edge nodes:

A user who (via the consistent-hash routing above) only ever actually reaches 3 of those 50 nodes would be capped at $3 \times 12 = 36$ req/min during a partition, a needless 16.7x under-enforcement relative to their intended 600 req/min limit. The fail-open denominator has to match the real fan-out bound, not the fleet size, or the fallback cap is simply wrong in one direction or the other.

Burst Tolerance and Tiered Limits

Sizing burst capacity. A token bucket has two parameters: a steady refill rate and a burst capacity (the bucket size). The refill rate enforces the sustained limit; the burst capacity is what lets a legitimate spike (a page loading several resources at once, a client retrying after a transient error) through without being throttled, while still bounding how long an abusive client can sustain a rate above the steady limit.

Using the same numbers as the fail-open example above: the global sustained rate is 600 req/min, which is

and with $R = 3$ shards reachable per user, each shard's steady refill rate is

matching the $200$ req/min per-shard cap used earlier ($200 / 60 \approx 3.33$). Pick a burst window that covers a realistic legitimate spike, for example 5 seconds. Per-shard burst capacity is then

rounded down, not up, so the fallback stays conservative. A client can spend all 16 tokens instantly, then is limited back to the 3.33 tokens/sec refill. If a user's traffic hits all 3 reachable shards simultaneously, the worst-case instantaneous burst is $16 \times 3 = 48$ tokens, about 4.8 seconds of full-rate traffic, after which the steady 600 req/min bound reasserts itself. This is what separates burst tolerance from opening the door to sustained abuse: the extra capacity is bounded and one-time, not an increase to the refill rate.

Composing tiered limits. Real deployments usually layer more than one scope: per-API-key, per-tenant (an account holding several keys), and a fleet-wide global tier. The composition rule is that the most specific tier is evaluated first, since it is the cheapest single-bucket check and the one most likely to reject, so a request fails fast before spending work on broader tiers. A request is admitted only if it passes every applicable tier; failing any one of them throttles the request regardless of how much headroom it has in the others.

Concretely, extending the per-key figure used throughout: suppose a tenant owns 5 API keys, each individually capped at 600 req/min. The naive sum of those keys' allowances is $5 \times 600 = 3000$ req/min. A per-tenant tier deliberately caps below that naive sum, for example at 80%:

so no single tenant can claim the full linear sum of its keys, leaving fairness headroom for other tenants sharing the same shard. The global tier sits above both: it is the same fleet-wide store from the architecture above, sized to actual infrastructure capacity rather than to any per-tenant math, and it throttles across every tenant once aggregate traffic approaches that capacity, independent of whether any individual key or tenant is within its own limit. That is why the global tier is the backstop: it is checked last, and it is the only tier tied to real capacity rather than to a customer-facing contract.

Trade-offs and Pitfalls

• Reconciliation after a partition heals needs to avoid double-counting requests that local buckets already allowed independently in each region; the reconciliation job should read each region's already-consumed count and merge, not replay raw request logs.
• Fixed time windows are vulnerable to clock skew across regions; a sliding-window or log-based approach avoids the skew problem at the cost of more memory per key, and is worth it specifically for the regions where you can't guarantee tight NTP sync.
• Whether to fail open or fail closed when the global store is unreachable depends on what the limiter protects: availability-oriented read paths usually fail open with a conservative bound (as above), while abuse- or cost-critical paths (e.g., a paid external API call per request) more often fail closed, accepting some availability loss to avoid an unbounded cost blow-up.
• A probabilistic sketch's overcount-only error means it can occasionally throttle a fully compliant user due to a hash collision with a heavy user; that is a real user-facing failure mode, not just a theoretical one, and is the reason sketches are used for cheap high-cardinality abuse mitigation rather than as the record of truth for a customer's contracted quota.

Direct answer

Map both nodes (as multiple virtual replicas) and keys onto the same circular hash space, keep the virtual-node positions in a sorted array, and find a key's owner with binary search for the first position at or after the key's hash, wrapping to the start of the ring if the key hashes past the last node. Virtual nodes exist so that a single physical node isn't a single point on the ring, without them, adding or removing one node would create a wildly uneven split.

Approach

Three pieces: a stable hash function (any well-distributed hash works; SHA-256 truncated to an integer is simple and collision-safe enough for this purpose), a sorted list of ring positions for binary search, and a map from ring position back to node id. Each physical node is hashed multiple times (once per virtual replica index) so it occupies many scattered points on the ring rather than one; this is what smooths the distribution and is what limits key movement to roughly the fraction of the ring the added or removed node owned.

Code (Python)

import bisect
import hashlib


class ConsistentHash:
    def __init__(self, replicas=100):
        self.replicas = replicas
        self.ring = []            # sorted list of integer hash positions
        self.hash_to_node = {}    # position -> node_id

    def _hash(self, value: str) -> int:
        return int(hashlib.sha256(value.encode("utf-8")).hexdigest(), 16)

    def add_node(self, node_id: str):
        for i in range(self.replicas):
            h = self._hash(f"{node_id}#{i}")
            if h in self.hash_to_node:
                continue
            bisect.insort(self.ring, h)
            self.hash_to_node[h] = node_id

    def remove_node(self, node_id: str):
        for i in range(self.replicas):
            h = self._hash(f"{node_id}#{i}")
            idx = bisect.bisect_left(self.ring, h)
            if idx < len(self.ring) and self.ring[idx] == h:
                self.ring.pop(idx)
                self.hash_to_node.pop(h, None)

    def get_node_for_key(self, key: str):
        if not self.ring:
            return None
        h = self._hash(key)
        idx = bisect.bisect_right(self.ring, h)
        if idx == len(self.ring):
            idx = 0
        return self.hash_to_node[self.ring[idx]]

Key points

• Data structures: a sorted Python list (ring) used with bisect for binary search, plus a dict (hash_to_node) for the O(1) reverse lookup from a ring position to its owning node.
• Virtual nodes are what make consistent hashing actually consistent-ish in practice: with only one point per physical node, adding a node could take an arbitrarily large or small share of the ring depending on hash luck; with many replicas per node, each node's total share converges toward its fair proportion.
• Wraparound: bisect_right finds the first position strictly greater than the key's hash; if that search runs off the end of the list, the key belongs to the first node on the ring (index 0), since the ring is circular.

Complexity

• add_node/remove_node: O(replicas * n), not O(replicas * log n). Each of the replicas calls does an O(log n) binary search (the search phase of bisect.insort/bisect_left) to find the insertion or removal point, but the actual insertion or removal on a plain Python list then requires shifting every element after that point by one slot, which is O(n) and dominates the O(log n) search. So each replica costs O(log n) to locate plus O(n) to shift, giving O(replicas * n) total per add_node/remove_node call, consistent with the list-shift cost called out below in Trade-offs and pitfalls.
• get_node_for_key: O(log n) for the binary search, O(1) for the dict lookup.
• Space: O(n) for the ring and the reverse map, where n = number of nodes times replicas.

Edge cases

• Empty ring: get_node_for_key returns None rather than raising.
• Hash collisions between two different virtual-node keys: skipped defensively (if h in self.hash_to_node: continue), though with SHA-256 this is not a practical concern at any realistic node count.
• Removing a node that was never added: no-op, since none of its hashed replica positions exist in the ring.
• Wraparound key (hash greater than every ring position): correctly routed to the first node on the ring.

Running this with 3 nodes (A, B, C) and 100 replicas each, then adding a 4th node and removing one, against 10000 fixed keys (key-0 through key-9999):

distribution across A/B/C (10000 keys): {'A': 3257, 'B': 3536, 'C': 3207}
keys moved after adding D: 2380 / 10000 = 0.2380
expected fraction ~= 1/4 = 0.25
keys moved after removing B: 2677 / 10000 = 0.2677
keys still mapped to B after removal (should be 0): 0

With 4 equal-weight nodes, adding a node should move roughly 1 in 4 keys (the new node's fair share); the measured 23.80% is close to the 25% expectation, with the gap explained by finite-sample variance at 100 replicas per node (more replicas would tighten this further, at the cost of a bigger ring). After removing B, 0 keys are still mapped to it, confirming the removal is complete.

Trade-offs and pitfalls

• Replica count is a distribution-smoothness versus memory/lookup-cost trade. More replicas per node means a more even split (lower variance) but a bigger ring array, so slower inserts and more memory; 100 to a few hundred replicas is a common practical range.
• A plain Python list with bisect.insort/.pop is O(n) per underlying array shift, not O(log n), for the insertion/removal itself, even though the search to find the position is O(log n). At very large node counts, a balanced tree or skip list would be needed to make mutation itself sub-linear; call this out explicitly if asked to scale this past a modest node count.
• This design assumes a single process owns the ring. In a distributed setting, every client needs the same ring (same node set, same replica count, same hash function) to agree on key ownership; a stale or divergent ring on one client silently routes some keys to the wrong node.

Direct answer

The fix has to work at both layers: the load balancer should bring a recovered instance back into rotation gradually (slow-start / weight ramp) rather than at full weight immediately, and clients should retry with jittered exponential backoff rather than reconnecting in lockstep the moment they see the instance healthy again. Neither alone is sufficient: LB-side ramping protects against a coordinated flood, but internal callers that bypass client backoff will still hit the instance hard once its weight is nonzero, so admission control (a hard per-instance connection/request cap) is the backstop that holds regardless of what clients do.

Load balancer mitigations

• Slow-start / weighted ramp-up: on transition to healthy, start the instance's LB weight near zero and increase it on a schedule (linear or exponential) instead of jumping to full share immediately.
• Health-check hysteresis: require several consecutive successful probes, not one, before even starting the ramp, so a flapping instance doesn't repeatedly trigger fresh thundering-herd events.
• Hard admission caps: independent of weight, enforce a per-instance connection/request ceiling at the LB so the ramp schedule isn't the only thing standing between the instance and overload.
• Sticky-routing caution: if session affinity is in use, avoid re-establishing it fully during ramp-up, since affinity can concentrate a disproportionate share of long-lived clients onto a still-warming instance.

Application-level mitigations

• Jittered exponential backoff on clients: spread reconnect attempts in time instead of retrying immediately or on a fixed interval that resynchronizes across clients.
• Admission queueing with a bound: the instance itself accepts up to a limit and returns 429/503 with Retry-After beyond that, rather than accepting everything and falling over.
• Circuit breakers upstream of the instance: if error/latency crosses a threshold during ramp-up, upstream callers back off automatically instead of continuing to hammer it.

Worked example

Suppose the recovered instance's established safe steady-state capacity is $C = 5{,}000$ RPS, and the pent-up demand specifically targeting it (clients that were failing over away from it and are now retrying) is $D = 50{,}000$ RPS, 10x its safe capacity. The LB ramps its weight starting at $w_0 = 1\%$, doubling every 10 seconds:

By t=40s the instance is already running at its full safe capacity even though the LB weight is still far below 100%; from that point on, the hard admission cap (not the weight ramp) is what's actually protecting it, and the ramp continuing upward just determines when the instance starts absorbing a "fair" share once the herd's backlog (D) has drained through the rest of the pool. This is the concrete reason both mechanisms are needed: the ramp controls the early window when D is far above C, and the hard cap controls everything after, once weight alone would otherwise overshoot.

Trade-offs and pitfalls

• Slow-start delays how quickly the pool's overall capacity recovers, which matters if the rest of the pool is itself under strain; the ramp duration is a real trade-off between protecting the recovering instance and relieving the rest of the fleet faster.
• Implementing only client-side backoff misses internal service-to-service callers that don't go through the same retry library; implementing only LB-side ramping misses the fact that a fixed weight schedule can still be overwhelmed if pent-up demand is large enough relative to the ramp rate, which is exactly why the hard admission cap has to exist independent of the ramp.
• A common failure in postmortems is tuning the ramp curve without ever computing whether the ramp rate can plausibly outrun realistic pent-up demand (as in the worked example above); a ramp that's too slow relative to D just delays the herd, and one that's too fast relative to C doesn't protect the instance at all.

Direct answer

Round-robin, weighted, and GeoDNS are all ways an authoritative DNS server chooses which IP to hand back for the same hostname, they differ only in the selection rule (rotate, weighted-random, or client-location-based), and all three inherit the same fundamental limitation: once a resolver caches an answer, DNS has no way to recall it. Failover speed is bounded entirely by TTL and by how faithfully resolvers respect it.

Structured elaboration

None of the three is health-aware on its own. In practice all three are paired with active health checks (e.g. Route 53 health-checked records, a GSLB, Global Server Load Balancing: DNS that also factors in live health checks, so it can steer traffic away from an unhealthy region instead of rotating blindly) that add or remove a record from the pool based on liveness, the DNS strategy alone only decides selection among whatever is currently in the pool.

Worked example: TTL vs. authoritative query load

Assume roughly 1,000,000 independently-caching clients (accounting for resolver fan-out, this is "cache entries," not literal end users) issuing steady lookups for the hostname. Because each client re-queries only when its cached TTL expires, the sustained query rate the authoritative server sees is approximately:

Dropping TTL by 10x (300s to 30s) multiplies authoritative query load by 10x for the same client population. Designing TTL is therefore a direct trade between failover speed and nameserver load, not a free lunch: pick the lowest TTL that (a) your authoritative infrastructure (typically anycast-backed managed DNS) can sustain at expected client scale, and (b) actual public resolvers will honor, many ISP and public resolvers apply a practical floor around 30 to 60 seconds regardless of the record's stated TTL.

Trade-offs and pitfalls

• Negative caching is caching too. An NXDOMAIN or SERVFAIL response gets cached according to the zone's SOA minimum TTL, a botched record removal can produce a resolvable-but-wrong period followed by an unexpectedly long "not found" period.
• Very low TTLs are not guaranteed to help. Below the resolver's practical floor, requesting a 5s TTL buys nothing while still multiplying real query volume against the servers that do honor it.
• DNS is a request for a new answer, not a push. A client that already opened a connection or cached the resolution client-side (browser, HTTP client, container DNS cache) is unaffected by a DNS change until it does a fresh lookup, so DNS TTL bounds resolver-level staleness, not necessarily client-visible staleness.
• A common practical pattern: keep a stable, moderate TTL (e.g. 300s) in steady state to protect the authoritative servers, and only drop to a low TTL (e.g. 30 to 60s) proactively ahead of a planned migration or maintenance window, since a TTL change itself only takes effect after the previous TTL expires.

Direct answer

Layer 4 load balancers make routing decisions using only transport-layer metadata (source and destination IP, port, protocol) and forward or proxy TCP/UDP connections without looking at the payload. Layer 7 load balancers terminate the application protocol, usually HTTP or HTTPS, and route on request content: host header, URL path, cookies, or other headers. L4 is faster and protocol-agnostic because it never parses the payload; L7 costs more CPU per request but can make far smarter routing, security, and traffic-shaping decisions. Pick L4 when you need raw throughput or must preserve end-to-end encryption; pick L7 when routing needs to understand HTTP semantics.

Structured elaboration

Decision guidance: choose L4 when the balancer must not (or need not) understand the payload, or when throughput at minimal overhead is the priority. Choose L7 when the routing decision itself depends on request content. Many production systems run both: an L4 tier absorbing raw connections at the edge, with an L7 tier immediately behind it for content-aware routing.

Worked example

Consider two systems that need a load balancer. First, a Postgres connection pooler in front of a cluster: clients authenticate to the database itself over TLS, connections are long-lived, and there is no HTTP semantics to route on. An L4 balancer is the right fit here: it forwards TCP connections without touching the encrypted stream, so end-to-end TLS stays intact and per-connection overhead stays minimal.

Second, a public REST API that must send /v1/users and /v1/orders to two different backend services, apply a WAF rule to block known bad user agents, and terminate TLS once at the edge. None of that is possible without reading the request line and headers, so this requires an L7 balancer, even though it costs more CPU per request than the L4 case.

Trade-offs & pitfalls

• L7 termination breaks end-to-end encryption unless the balancer re-encrypts to the backend (TLS bridging); this is a common follow-up in security-conscious interviews.
• L4 cannot do content-based routing or cookie affinity. Bolting host/path logic onto an L4 balancer just pushes the work down a layer where it is harder to operate.
• Hybrid designs (L4 at the outer edge, L7 immediately behind it) are standard practice, not a compromise: the L4 tier absorbs raw connection volume and the L7 tier handles content-aware decisions.
• Common wrong turn: treating L7 as strictly better. It adds a per-request parsing and termination cost and a larger attack surface (header injection, request smuggling) that an L4 balancer never has to reason about.