Load Balancing and Traffic Management Questions

Direct answer

For a backend service's health checks, reach for the cheapest check that actually proves what you care about: a TCP connect check to confirm the port is open, a TLS handshake check when the service terminates TLS itself, an HTTP(S) status and body check for a general web service, and an application-level readiness endpoint, including gRPC's dedicated health-checking protocol for gRPC services, when you need to know the instance can correctly serve a request right now, not just that the process is running.

Structured elaboration

Frequency, timeout, and failure threshold interact to set failover sensitivity. In the worst case, a failure starts right after a successful check, so detection is bounded by how often the balancer checks and how many consecutive failures it demands before acting:

With a 5-second interval and a 3-failure threshold:

Tightening the threshold to 2 failures with the same interval:

The 10-second configuration fails over faster but reacts to two consecutive blips instead of three, so it is more exposed to a single transient network hiccup being read as a real outage. Timeout works the same way from the other side: a timeout shorter than the service's genuine worst-case response time will misclassify a slow-but-alive backend as failed.

Adapting for a service with a slow startup: point the load balancer's check at a readiness endpoint that returns unhealthy until initialization finishes, rather than reusing the liveness check. Where the platform supports a distinct startup probe, use it to give the service a longer grace period before liveness checks even begin, so a legitimately slow boot (schema checks, cache warm-up) is never mistaken for a crash.

Trade-offs & pitfalls

• Putting a heavy dependency check inside a liveness-style probe means a slow downstream can trigger unnecessary restarts of an otherwise-healthy process; keep checks that can restart a process local and cheap.
• Tightening thresholds "for safety" often produces more incidents from flapping and restart churn than the failure category it was meant to catch quickly; tune against the service's actual latency variance, not a default.
• Reusing one endpoint for both a shallow liveness-style check and a deep readiness check makes it impossible to tune the two independently; keep them as separate endpoints even if they share code internally.

Direct Answer

Hash the session cookie onto a consistent-hash ring of virtual nodes so a session's requests keep landing on the same backend under normal conditions, publish the ring from a small versioned control plane so every proxy agrees on current ownership, pull unhealthy nodes out via active health checks, and when a node is intentionally removed, keep it serving its existing sessions for a bounded drain window while new sessions route elsewhere, only decommissioning it once its active-session count reaches zero or the window expires. The two things that make this safe are versioning the ring (so proxies never disagree mid-transition) and treating removal as a drain, not a hard cut.

Architecture

flowchart LR
  Client -->|cookie + ring_version| Proxy[Edge Proxy]
  Proxy -->|lookup| RingSvc[Ring Metadata Service]
  HealthChecker[Health Checker] -->|mark unhealthy| RingSvc
  Proxy --> Backend1[Backend 1]
  Proxy --> Backend2[Backend 2]
  Proxy -.draining.-> Backend3[Backend 3]
  DrainOrch[Drain Orchestrator] -->|coordinate| RingSvc
  Backend3 -->|flush session state| Store[(Shared Session Store)]
  Backend1 --> Store
  Backend2 --> Store

The proxies are stateless: all ring state lives in the ring metadata service and is cached locally with a version number. Health checks and the drain orchestrator only ever mutate the ring through that service, never by having a proxy make a unilateral decision.

Hash Ring and Virtual Nodes

• Use a large hash space (64-bit) with each physical backend assigned many virtual nodes, commonly in the range of 100 to 500, so that any single node's removal spreads its load across many different successors instead of dumping it on one (see the worked example below for why this matters).
• Store the ring as a sorted array in the metadata service; proxies cache a copy and watch for version bumps rather than polling on every request.
• Every ring mutation (join, leave, health-driven removal) increments the ring version atomically, so a proxy can always tell whether its cached copy is current.

Cookie and Version Handling

The session cookie carries three fields: the session id (the hash key), the ring version the session was originally assigned under, and an HMAC (a keyed cryptographic checksum: proof the cookie's contents haven't been altered since a server signed them) to prevent tampering. On a new session, the proxy issues a fresh cookie against the current ring version. On a returning session, the proxy honors the session's recorded ring version for a bounded overlap window even after the ring has moved on, which is what lets an in-flight session keep reaching its original backend during a drain instead of being silently reassigned mid-conversation.

Health Checks

Active HTTP health checks run against each backend on a fixed interval. A backend that fails its threshold is marked unhealthy in the ring metadata service, which bumps the ring version; proxies pick up the change and stop routing new sessions there. Existing sessions already pinned to that backend via cookie should still be judged against the same health signal: if the backend is actually down, in-flight requests will fail regardless of stickiness, so unhealthy removal is immediate, not drained (draining is reserved for planned, intentional removal, covered next).

Graceful Drain and Session Migration

For a planned node removal (scale-down, deploy, decommission):

1. Mark the node "draining" in the ring metadata service.
2. Remove the node's virtual-node entries from the ring and bump the ring version.
3. Proxies pick up the new version for new sessions immediately; sessions whose cookie still carries the old ring version continue routing to the draining node for a bounded overlap window.
4. During the overlap window, the draining node's session state is either already externalized (shared store, no action needed) or actively migrated: the drain orchestrator streams in-memory session state to the new owning node or to the shared store.
5. Once the draining node's active-session count reaches zero, or the overlap window expires (whichever comes first), the node is decommissioned. Sessions that hadn't finished by then either see a session reset (acceptable for many applications) or, if externalized state was used, resume transparently on the new node.

For node addition, the reverse: add the virtual nodes, bump the version, and let new sessions start flowing to the new node immediately. Existing sessions are unaffected because their arcs on the ring did not move.

Session State Strategy

Prefer an external session store (a clustered cache such as Redis) over in-memory backend state whenever possible: it decouples session survival from any single backend's lifecycle entirely, removing the need for the migration step above. If in-memory sessions are unavoidable (e.g., a stateful protocol upgrade like a long-lived WebSocket), the migration path in step 4 is mandatory, not optional.

Worked Example: How Much Load a Removal Redistributes

Take 20 physical backends, each with $V = 150$ virtual nodes, so $T = 3000$ ring tokens total. Removing one physical backend removes its 150 tokens; each of those tokens' clockwise successors is, in a well-shuffled ring, effectively a random draw among the other 19 physical nodes. So the removed node's $\frac{1}{20} = 5\%$ share of keys spreads across roughly 19 recipients rather than one. Each surviving node's expected new share:

which is exactly the uniform share you'd expect from evenly splitting the whole keyspace across 19 nodes, the same identity that shows up whenever virtual-node count is high enough to approximate rendezvous hashing's (an alternative hashing scheme that scores every node directly against each key and always redistributes evenly on removal, without needing virtual nodes) exact uniform redistribution. Compare that to a single-token ring (no virtual nodes) removing one of 20 nodes: the removed node's entire 5% lands on one successor, whose share jumps from 5% to 10%, a 2x hotspot instead of a 0.26-point bump.

Trade-offs and Pitfalls

• The ring metadata service is a coordination dependency; if it uses a consensus protocol (Raft or similar) for consistency, that adds latency to ring mutations (acceptable, since mutations are rare) but also means the service itself needs its own HA story.
• Overlap window length is a direct trade-off between session continuity and routing complexity: a longer window keeps more in-flight sessions alive during churn but means proxies must correctly honor two ring versions simultaneously for longer.
• Signed ring-version cookies need HMAC key rotation handled carefully: rotating the signing key while old cookies are still in flight requires accepting both old and new keys for a transition period, or sessions get silently invalidated.
• If a CDN or edge cache sits in front of this layer, cache keys for session-specific content must not be shared across ring reassignment; either scope those responses as non-cacheable or ensure the cache key includes a stable shard identifier that survives rebalancing.
• A single very active session ("hot session") can skew load on whichever node currently owns it; virtual nodes fix cluster-wide statistical balance, not a single oversized key, so hot-key handling needs a separate mitigation (e.g., splitting that session's read traffic) if it becomes a real problem.

Direct answer

Store rules as a versioned, prioritized ruleset in a central control plane, compile them into a fast local matcher (a trie, a tree structure where each path from the root spells out a prefix so a lookup finds a match in steps proportional to the key's length rather than scanning every rule, or decision graph, not a naive if-chain) on each edge node, and resolve conflicts with an explicit priority number plus a deterministic tie-break, never "whichever rule happens to match first" in file order. Safe fallback means every rule set has a default action if nothing matches or if validation fails, and that default is always the stable pool, never an unrouted request.

Structured elaboration

• Rule storage and distribution. Rules live in a control plane as immutable, versioned sets: predicate (header match, presence, regex, hash-bucket range), action (route to pool, set header), priority, and a version ID. The control plane pushes signed, compiled deltas to edge nodes (streaming or fast poll); each node keeps the last-known-good version and rolls back to it automatically if a new version fails local validation.
• Evaluation performance. Compile the ruleset into a structure suited to the predicate shape: a trie or hash lookup for exact-match headers like tenant ID, an Aho-Corasick automaton (a data structure that matches many string patterns against input in a single pass, faster than checking each pattern one at a time) or precompiled regex engine for pattern rules, and a short-circuit check for header presence before touching anything more expensive. The goal is that adding rules does not turn evaluation into a linear scan of every predicate on every request.
• Conflict resolution. Every rule carries an explicit priority number (lower evaluates first); ties break on rule creation time, then rule ID, so the outcome is fully deterministic and reproducible from the ruleset alone, not from evaluation order that happened to exist in memory.
• Header trust and spoofing. Only trust routing headers set by a component inside the trust boundary (the edge itself, after authentication), never a client-supplied header with the same name; if a header like X-Tenant-ID must originate from the client, validate it against the authenticated identity before using it for routing, or the routing layer becomes a way to reach another tenant's pool.
• Safe fallback. If no rule matches, or the active ruleset fails validation (schema error, missing referenced pool), route to the stable pool by default. This default must never be "no rule, no route."

graph LR
  Author[Rule author: API/UI] --> RCP[Control plane: versioned ruleset]
  RCP -->|signed compiled delta| Node1[Edge node: compiled matcher]
  Req[Request] --> Node1
  Node1 --> Match{Highest-priority match}
  Match -->|tenant rule| TenantPool[Tenant-specific pool]
  Match -->|canary rule| CanaryPool[Canary pool]
  Match -->|no match / validation failed| Stable[Stable pool: default fallback]

Worked example

A simplified illustrative hash (real systems use SHA-256 or similar, not this): sum the ASCII codes of a key's characters and take the result mod 100. A canary rule routes to the canary pool when that value is below 10 (a 10% split).

The boundary case ("n", exactly 10) shows why the predicate needs an explicit, documented comparison (strictly less than, not less-than-or-equal): the threshold is a rule, not a fuzzy target, and the same key always lands on the same side of it, which is what lets a canary rollout stay stable across LB restarts without a shared session store.

Now layer conflict resolution: a request carries both a tenant header identifying "acme" (matches the tenant rule, priority 10) and a canary flag with a hash landing in the canary bucket (matches the canary rule, priority 20). Because lower priority numbers evaluate first, the tenant rule wins and the request goes to acme's dedicated pool, even though it would otherwise have qualified for canary.

Trade-offs & pitfalls

• Compiling to a trie or precompiled matcher costs build time on every rule update; for a control plane pushing frequent changes, measure compile time against your update frequency, not just steady-state lookup speed.
• Never let a client-controlled header carry routing authority without validating it against something the client cannot forge (an authenticated claim); an unvalidated tenant header is a direct tenant-isolation bypass, not just a routing bug.
• Rolling out a new ruleset version without a dry-run or shadow-evaluation stage means the first time you learn two rules conflict unexpectedly is in production; validate new versions against recent real traffic before activating them.
• Common wrong turn: treating "most specific rule wins" as if it were the same as explicit priority. Specificity is a heuristic humans use when writing rules; the engine needs an explicit, deterministic number, or two engineers' intuitions about "more specific" will eventually disagree.

Direct answer

For three active regions with a 5 minute RTO and eventual consistency accepted, DNS-based failover works if the failure-detection time, the DNS TTL, and resolver-propagation slack are budgeted explicitly against the 5 minute target, with the honest caveat that DNS-based failover has a hard, structural ceiling: some fraction of clients or resolvers will not honor TTL promptly, so this design meets a 5 minute RTO for the large majority of traffic, not a guarantee for every single client.

Structured elaboration

graph TD
    Client[Client] --> DNS[Authoritative DNS]
    DNS --> HC[Health Check Aggregator]
    HC -->|healthy: in rotation| RA[Region A LB]
    HC -->|healthy: in rotation| RB[Region B LB]
    HC -.->|FAIL: removed from DNS| RC[Region C LB]
    RA --> AppA[Region A App]
    RB --> AppB[Region B App]
    RC --> AppC[Region C App]

Health checks. Each region runs a synthetic health endpoint validating the app and its critical dependencies (DB reachability, cache reachability), probed from at least 3 independent locations. Require a majority of probes to agree before flipping a region's status, a single flaky probe should never trigger a regional failover.

DNS configuration. Weighted, health-checked records (e.g. Route 53 weighted routing with health checks, or an equivalent GSLB (Global Server Load Balancer, a DNS-based service that routes clients to different regions based on health and proximity)) for all three regions, roughly equal weights in steady state (34/33/33). On a confirmed FAIL, the provider's health check automatically removes that region's record from the answer set, no manual DNS edit required on the failure path.

Session handling. Prefer stateless sessions (signed JWTs) so any region can serve any request. If server-side session state is unavoidable, replicate it asynchronously across regions and design the app to tolerate a small window of stale or missing session data on failover, consistent with the "eventual consistency acceptable" requirement.

Worked example: the RTO budget

A 5 minute (300s) RTO has to cover detection, DNS propagation, and client-side retry, add these up explicitly rather than asserting the number:

That leaves a 150 second margin against the 300 second target, room to absorb a slower-than-expected health check aggregation step, a resolver that ignores TTL more aggressively than budgeted, or a partial rollout of the DNS change. If the measured margin were negative, the fix is to lower the TTL and/or tighten the detection interval, not to assume the RTO will be hit anyway.

Comparison: active-active vs. active-passive

Given eventual consistency is acceptable and three regions are already active, active-active is the natural fit here: the RTO budget above assumes surviving regions are already warm and serving, which is only true in active-active. An active-passive design would need to add standby promotion and cache warm-up time on top of the same DNS budget, likely pushing past the 5 minute target unless the standby is kept continuously warm (which erodes most of its cost advantage). A p99 < 200ms latency target reinforces the same conclusion from a different angle: it is a steady-state requirement, not just a failover one, and the same latency-based routing that gives active-active its fast RTO is also what keeps ordinary traffic on the nearest healthy region so the latency target is met even when nothing has failed. Active-passive would route every region's users through one primary, so a global p99 < 200ms target would require either a large, latency-sensitive edge/CDN layer in front of it or accepting that distant users miss the target entirely.

Failback

1. Confirm the recovered region is fully healthy (health checks green) and, if it holds any writable state, that data has caught up to an acceptable staleness bound.
2. Re-introduce it at a low weight (e.g. 5 to 10%) rather than immediately restoring equal weights, to reduce risk from a still-caching population and to let its caches warm under real traffic.
3. Ramp weight back to steady state over a defined window while watching error rate and latency.
4. Run a post-incident review, in particular checking for any data divergence accumulated during the outage that eventual consistency needs to reconcile.

Trade-offs and pitfalls

• DNS-based failover's fundamental limitation is that it is a caching system, not a control system. It can only ever bound resolver-level staleness by TTL; it cannot force an already-cached client to re-resolve, and it cannot see or control every resolver's actual caching behavior.
• A concrete shape of that limitation: deprecated IPs served from stale cache. After a region is pulled from DNS rotation, some resolvers and long-lived clients keep resolving to, or keep an open connection against, that now-decommissioned region's IP well past the configured TTL. Some public resolvers apply their own TTL floors, some corporate or OS-level resolvers cache longer than instructed, and a client that already holds an open connection or a cached resolved IP will not re-resolve until that connection breaks. Real incidents from this look like a small, persistent tail of traffic still hitting a region minutes after it was pulled everywhere else.
• Client-side mitigations bound this blast radius, they do not eliminate it. Retry-with-re-resolution on connection failure (treat a failed connect or a run of failed requests as a signal to force a fresh DNS lookup instead of reusing the cached IP), connection pools that honor the record's TTL and recycle idle connections rather than holding them indefinitely, and a happy-eyeballs-style fallback that races the primary resolved address against a secondary one and fails over at the connection layer if the first does not respond. None of these are under server-side control, they have to be built into the client or its SDK.
• Lower TTL is not free. Halving the TTL from 60s to 30s tightens the worst-case staleness bound to about 30s but roughly doubles steady-state DNS query volume against the authoritative servers, since twice as many cached entries expire and get re-resolved per unit time; the 60s TTL chosen above is a deliberate balance point between failover speed and DNS infrastructure load, not the lowest theoretically possible value.
• Anycast (announcing the same IP address from multiple locations and letting network routing send each client to the nearest one) or a client-side/edge load balancer removes the DNS-caching ceiling entirely by routing at the network layer instead of the naming layer, worth naming as the answer when a client asks "how do you get below what DNS can offer."
• Failback done too fast is a common cause of a second incident: dumping full traffic back onto a region whose caches are still cold reproduces the exact warm-up latency spike that a proper connection-draining and readiness gate is designed to prevent: stop routing new traffic to a recovering region, ramp its share up gradually instead of all at once, and only count it at full weight once its caches and connection pools have actually warmed under real traffic, the same low-weight reintroduction and ramp already described above in Failback steps 2-3.

Direct answer

Layer 4 load balancers make routing decisions using only transport-layer metadata (source and destination IP, port, protocol) and forward or proxy TCP/UDP connections without looking at the payload. Layer 7 load balancers terminate the application protocol, usually HTTP or HTTPS, and route on request content: host header, URL path, cookies, or other headers. L4 is faster and protocol-agnostic because it never parses the payload; L7 costs more CPU per request but can make far smarter routing, security, and traffic-shaping decisions. Pick L4 when you need raw throughput or must preserve end-to-end encryption; pick L7 when routing needs to understand HTTP semantics.

Structured elaboration

Decision guidance: choose L4 when the balancer must not (or need not) understand the payload, or when throughput at minimal overhead is the priority. Choose L7 when the routing decision itself depends on request content. Many production systems run both: an L4 tier absorbing raw connections at the edge, with an L7 tier immediately behind it for content-aware routing.

Worked example

Consider two systems that need a load balancer. First, a Postgres connection pooler in front of a cluster: clients authenticate to the database itself over TLS, connections are long-lived, and there is no HTTP semantics to route on. An L4 balancer is the right fit here: it forwards TCP connections without touching the encrypted stream, so end-to-end TLS stays intact and per-connection overhead stays minimal.

Second, a public REST API that must send /v1/users and /v1/orders to two different backend services, apply a WAF rule to block known bad user agents, and terminate TLS once at the edge. None of that is possible without reading the request line and headers, so this requires an L7 balancer, even though it costs more CPU per request than the L4 case.

Trade-offs & pitfalls

• L7 termination breaks end-to-end encryption unless the balancer re-encrypts to the backend (TLS bridging); this is a common follow-up in security-conscious interviews.
• L4 cannot do content-based routing or cookie affinity. Bolting host/path logic onto an L4 balancer just pushes the work down a layer where it is harder to operate.
• Hybrid designs (L4 at the outer edge, L7 immediately behind it) are standard practice, not a compromise: the L4 tier absorbs raw connection volume and the L7 tier handles content-aware decisions.
• Common wrong turn: treating L7 as strictly better. It adds a per-request parsing and termination cost and a larger attack surface (header injection, request smuggling) that an L4 balancer never has to reason about.