Load Balancing and Traffic Management Questions

Direct answer

For a backend service's health checks, reach for the cheapest check that actually proves what you care about: a TCP connect check to confirm the port is open, a TLS handshake check when the service terminates TLS itself, an HTTP(S) status and body check for a general web service, and an application-level readiness endpoint, including gRPC's dedicated health-checking protocol for gRPC services, when you need to know the instance can correctly serve a request right now, not just that the process is running.

Structured elaboration

Frequency, timeout, and failure threshold interact to set failover sensitivity. In the worst case, a failure starts right after a successful check, so detection is bounded by how often the balancer checks and how many consecutive failures it demands before acting:

With a 5-second interval and a 3-failure threshold:

Tightening the threshold to 2 failures with the same interval:

The 10-second configuration fails over faster but reacts to two consecutive blips instead of three, so it is more exposed to a single transient network hiccup being read as a real outage. Timeout works the same way from the other side: a timeout shorter than the service's genuine worst-case response time will misclassify a slow-but-alive backend as failed.

Adapting for a service with a slow startup: point the load balancer's check at a readiness endpoint that returns unhealthy until initialization finishes, rather than reusing the liveness check. Where the platform supports a distinct startup probe, use it to give the service a longer grace period before liveness checks even begin, so a legitimately slow boot (schema checks, cache warm-up) is never mistaken for a crash.

Trade-offs & pitfalls

• Putting a heavy dependency check inside a liveness-style probe means a slow downstream can trigger unnecessary restarts of an otherwise-healthy process; keep checks that can restart a process local and cheap.
• Tightening thresholds "for safety" often produces more incidents from flapping and restart churn than the failure category it was meant to catch quickly; tune against the service's actual latency variance, not a default.
• Reusing one endpoint for both a shallow liveness-style check and a deep readiness check makes it impossible to tune the two independently; keep them as separate endpoints even if they share code internally.

Direct answer

For three active regions with a 5 minute RTO and eventual consistency accepted, DNS-based failover works if the failure-detection time, the DNS TTL, and resolver-propagation slack are budgeted explicitly against the 5 minute target, with the honest caveat that DNS-based failover has a hard, structural ceiling: some fraction of clients or resolvers will not honor TTL promptly, so this design meets a 5 minute RTO for the large majority of traffic, not a guarantee for every single client.

Structured elaboration

graph TD
    Client[Client] --> DNS[Authoritative DNS]
    DNS --> HC[Health Check Aggregator]
    HC -->|healthy: in rotation| RA[Region A LB]
    HC -->|healthy: in rotation| RB[Region B LB]
    HC -.->|FAIL: removed from DNS| RC[Region C LB]
    RA --> AppA[Region A App]
    RB --> AppB[Region B App]
    RC --> AppC[Region C App]

Health checks. Each region runs a synthetic health endpoint validating the app and its critical dependencies (DB reachability, cache reachability), probed from at least 3 independent locations. Require a majority of probes to agree before flipping a region's status, a single flaky probe should never trigger a regional failover.

DNS configuration. Weighted, health-checked records (e.g. Route 53 weighted routing with health checks, or an equivalent GSLB (Global Server Load Balancer, a DNS-based service that routes clients to different regions based on health and proximity)) for all three regions, roughly equal weights in steady state (34/33/33). On a confirmed FAIL, the provider's health check automatically removes that region's record from the answer set, no manual DNS edit required on the failure path.

Session handling. Prefer stateless sessions (signed JWTs) so any region can serve any request. If server-side session state is unavoidable, replicate it asynchronously across regions and design the app to tolerate a small window of stale or missing session data on failover, consistent with the "eventual consistency acceptable" requirement.

Worked example: the RTO budget

A 5 minute (300s) RTO has to cover detection, DNS propagation, and client-side retry, add these up explicitly rather than asserting the number:

That leaves a 150 second margin against the 300 second target, room to absorb a slower-than-expected health check aggregation step, a resolver that ignores TTL more aggressively than budgeted, or a partial rollout of the DNS change. If the measured margin were negative, the fix is to lower the TTL and/or tighten the detection interval, not to assume the RTO will be hit anyway.

Comparison: active-active vs. active-passive

Given eventual consistency is acceptable and three regions are already active, active-active is the natural fit here: the RTO budget above assumes surviving regions are already warm and serving, which is only true in active-active. An active-passive design would need to add standby promotion and cache warm-up time on top of the same DNS budget, likely pushing past the 5 minute target unless the standby is kept continuously warm (which erodes most of its cost advantage). A p99 < 200ms latency target reinforces the same conclusion from a different angle: it is a steady-state requirement, not just a failover one, and the same latency-based routing that gives active-active its fast RTO is also what keeps ordinary traffic on the nearest healthy region so the latency target is met even when nothing has failed. Active-passive would route every region's users through one primary, so a global p99 < 200ms target would require either a large, latency-sensitive edge/CDN layer in front of it or accepting that distant users miss the target entirely.

Failback

1. Confirm the recovered region is fully healthy (health checks green) and, if it holds any writable state, that data has caught up to an acceptable staleness bound.
2. Re-introduce it at a low weight (e.g. 5 to 10%) rather than immediately restoring equal weights, to reduce risk from a still-caching population and to let its caches warm under real traffic.
3. Ramp weight back to steady state over a defined window while watching error rate and latency.
4. Run a post-incident review, in particular checking for any data divergence accumulated during the outage that eventual consistency needs to reconcile.

Trade-offs and pitfalls

• DNS-based failover's fundamental limitation is that it is a caching system, not a control system. It can only ever bound resolver-level staleness by TTL; it cannot force an already-cached client to re-resolve, and it cannot see or control every resolver's actual caching behavior.
• A concrete shape of that limitation: deprecated IPs served from stale cache. After a region is pulled from DNS rotation, some resolvers and long-lived clients keep resolving to, or keep an open connection against, that now-decommissioned region's IP well past the configured TTL. Some public resolvers apply their own TTL floors, some corporate or OS-level resolvers cache longer than instructed, and a client that already holds an open connection or a cached resolved IP will not re-resolve until that connection breaks. Real incidents from this look like a small, persistent tail of traffic still hitting a region minutes after it was pulled everywhere else.
• Client-side mitigations bound this blast radius, they do not eliminate it. Retry-with-re-resolution on connection failure (treat a failed connect or a run of failed requests as a signal to force a fresh DNS lookup instead of reusing the cached IP), connection pools that honor the record's TTL and recycle idle connections rather than holding them indefinitely, and a happy-eyeballs-style fallback that races the primary resolved address against a secondary one and fails over at the connection layer if the first does not respond. None of these are under server-side control, they have to be built into the client or its SDK.
• Lower TTL is not free. Halving the TTL from 60s to 30s tightens the worst-case staleness bound to about 30s but roughly doubles steady-state DNS query volume against the authoritative servers, since twice as many cached entries expire and get re-resolved per unit time; the 60s TTL chosen above is a deliberate balance point between failover speed and DNS infrastructure load, not the lowest theoretically possible value.
• Anycast (announcing the same IP address from multiple locations and letting network routing send each client to the nearest one) or a client-side/edge load balancer removes the DNS-caching ceiling entirely by routing at the network layer instead of the naming layer, worth naming as the answer when a client asks "how do you get below what DNS can offer."
• Failback done too fast is a common cause of a second incident: dumping full traffic back onto a region whose caches are still cold reproduces the exact warm-up latency spike that a proper connection-draining and readiness gate is designed to prevent: stop routing new traffic to a recovering region, ramp its share up gradually instead of all at once, and only count it at full weight once its caches and connection pools have actually warmed under real traffic, the same low-weight reintroduction and ramp already described above in Failback steps 2-3.

Direct answer

An automated canary control loop is a periodic process that increases a canary's traffic weight in small, guarded steps, checks health metrics against the baseline after every step using a minimum-observation window, and holds or immediately rolls back to zero on an SLO breach (a violation of the service-level objective, the reliability target you've committed to, such as 99.9% success rate). The design problem is really about the guardrails and the rollback path, not the ramp itself.

Structured elaboration

Components:

• A traffic router (Envoy, an ingress controller, or a cloud LB) that supports weighted routing and an instant weight-update API.
• A telemetry pipeline producing short and long rolling windows of error rate, latency, and request volume, split by canary vs. baseline.
• A stateless controller that runs the step/check/decide loop and calls the router's weight API.
• A rollback path that can zero out canary weight immediately, independent of the step loop.

Guardrails (the part that actually matters):

Smoothing the weight changes. Compute a fixed step size from the target ramp, then apply the step in small sub-increments at the router level (e.g. over a few seconds) so a single jump doesn't itself cause a connection-pool or cache-warming spike.

Worked example

Target: shift 10% of production traffic to canary over a 10 minute (600s) window, checking every 30s.

So the loop runs 20 times, adding 0.5 percentage points of canary weight each pass, provided the guardrails above all pass. If step 12 (weight = 6%) shows canary p99 = 340ms against a baseline p99 of 260ms:

That exceeds the 1.2x latency-ratio guardrail, so the controller does not advance to step 13. It instead sets weight back to 0% immediately (not a gradual decrease), opens an incident, and freezes further attempts for the cooldown period.

# runs every 30s
current = get_current_weight()
target = min(10, current + 0.5)
m = fetch_metrics(short=60, long=300)
if m.canary_requests_short < 100:
    return  # not enough signal yet, hold at current weight
if m.error_rate_canary > m.error_rate_baseline + 0.005 or \
   m.p99_canary_long > m.p99_baseline_long * 1.2:
    set_weight(0)          # immediate rollback, not gradual
    open_incident(m)
    start_cooldown(minutes=30)
else:
    set_weight(target)     # router ramps this smoothly over a few seconds

Extensions: downstream capacity and CI/CD integration

Downstream-capacity-constrained ramp. The guardrails above are all canary-side (error rate, latency). A canary can look perfectly healthy on its own metrics while the traffic it forwards saturates a downstream dependency, a connection pool, a queue, or a rate-limited third-party API, before that dependency's own health signal even reflects the problem. Add a downstream capacity ceiling as its own guardrail: before each step, check the downstream dependency's current utilization (connection-pool saturation, queue depth, or a published capacity headroom number) and cap the canary weight so its incremental load stays under that ceiling, independent of what the canary's own error and latency metrics say. In practice the step size becomes the smaller of the scheduled step and the downstream headroom divided by requests-per-weight-point, and downstream saturation becomes its own rollback trigger alongside the error and latency guardrails, since the canary's own SLOs will not catch a downstream problem until it is already failing.

CI/CD pipeline integration. The loop is usually not a standalone daemon, it is a stage in the deploy pipeline. A build that passes its test stage triggers the deploy, which starts the canary at 0% weight and hands control to this loop; the pipeline blocks promotion to the next stage (wider rollout, or the next environment) until the loop reports a verdict, not a fixed timer. On success the loop reports pass and the pipeline proceeds to full rollout. On a guardrail breach, the loop's immediate zero-weight rollback from the worked example above is reported back as a failed pipeline stage: the deploy is marked failed, the previous version stays serving at 100%, and the pipeline stops rather than continuing to the next stage. That makes the human's only manual step reviewing the failure, not remembering to check on the canary.

Trade-offs and pitfalls

• Short windows are noisy, long windows are slow. A short-only window trips on transient blips; a long-only window lets real regressions run for minutes before anyone notices. Using both, and requiring the long window to confirm, is the standard resolution.
• Rollback must never itself be a slow ramp. The instinct to "gradually" bring weight back down defeats the purpose; on breach, cut to zero immediately and investigate after.
• Minimum-request guardrails matter more for low-traffic services. A service doing 50 RPS total needs a longer observation window or a lower canary weight cap just to get statistically meaningful samples per step.
• Business metrics can lag system metrics. A checkout canary can look perfectly healthy on latency and error rate while conversion silently drops; if the metric that matters is business-level, it needs its own window and threshold, not just infra SLOs.
• This is inherently single-service. Running the same loop concurrently across many dependent services without coordination risks compounding partial failures across a call graph; large orgs typically centralize this into a shared canary-analysis service rather than one loop per team.

Direct answer

Map both nodes (as multiple virtual replicas) and keys onto the same circular hash space, keep the virtual-node positions in a sorted array, and find a key's owner with binary search for the first position at or after the key's hash, wrapping to the start of the ring if the key hashes past the last node. Virtual nodes exist so that a single physical node isn't a single point on the ring, without them, adding or removing one node would create a wildly uneven split.

Approach

Three pieces: a stable hash function (any well-distributed hash works; SHA-256 truncated to an integer is simple and collision-safe enough for this purpose), a sorted list of ring positions for binary search, and a map from ring position back to node id. Each physical node is hashed multiple times (once per virtual replica index) so it occupies many scattered points on the ring rather than one; this is what smooths the distribution and is what limits key movement to roughly the fraction of the ring the added or removed node owned.

Code (Python)

import bisect
import hashlib


class ConsistentHash:
    def __init__(self, replicas=100):
        self.replicas = replicas
        self.ring = []            # sorted list of integer hash positions
        self.hash_to_node = {}    # position -> node_id

    def _hash(self, value: str) -> int:
        return int(hashlib.sha256(value.encode("utf-8")).hexdigest(), 16)

    def add_node(self, node_id: str):
        for i in range(self.replicas):
            h = self._hash(f"{node_id}#{i}")
            if h in self.hash_to_node:
                continue
            bisect.insort(self.ring, h)
            self.hash_to_node[h] = node_id

    def remove_node(self, node_id: str):
        for i in range(self.replicas):
            h = self._hash(f"{node_id}#{i}")
            idx = bisect.bisect_left(self.ring, h)
            if idx < len(self.ring) and self.ring[idx] == h:
                self.ring.pop(idx)
                self.hash_to_node.pop(h, None)

    def get_node_for_key(self, key: str):
        if not self.ring:
            return None
        h = self._hash(key)
        idx = bisect.bisect_right(self.ring, h)
        if idx == len(self.ring):
            idx = 0
        return self.hash_to_node[self.ring[idx]]

Key points

• Data structures: a sorted Python list (ring) used with bisect for binary search, plus a dict (hash_to_node) for the O(1) reverse lookup from a ring position to its owning node.
• Virtual nodes are what make consistent hashing actually consistent-ish in practice: with only one point per physical node, adding a node could take an arbitrarily large or small share of the ring depending on hash luck; with many replicas per node, each node's total share converges toward its fair proportion.
• Wraparound: bisect_right finds the first position strictly greater than the key's hash; if that search runs off the end of the list, the key belongs to the first node on the ring (index 0), since the ring is circular.

Complexity

• add_node/remove_node: O(replicas * n), not O(replicas * log n). Each of the replicas calls does an O(log n) binary search (the search phase of bisect.insort/bisect_left) to find the insertion or removal point, but the actual insertion or removal on a plain Python list then requires shifting every element after that point by one slot, which is O(n) and dominates the O(log n) search. So each replica costs O(log n) to locate plus O(n) to shift, giving O(replicas * n) total per add_node/remove_node call, consistent with the list-shift cost called out below in Trade-offs and pitfalls.
• get_node_for_key: O(log n) for the binary search, O(1) for the dict lookup.
• Space: O(n) for the ring and the reverse map, where n = number of nodes times replicas.

Edge cases

• Empty ring: get_node_for_key returns None rather than raising.
• Hash collisions between two different virtual-node keys: skipped defensively (if h in self.hash_to_node: continue), though with SHA-256 this is not a practical concern at any realistic node count.
• Removing a node that was never added: no-op, since none of its hashed replica positions exist in the ring.
• Wraparound key (hash greater than every ring position): correctly routed to the first node on the ring.

Running this with 3 nodes (A, B, C) and 100 replicas each, then adding a 4th node and removing one, against 10000 fixed keys (key-0 through key-9999):

distribution across A/B/C (10000 keys): {'A': 3257, 'B': 3536, 'C': 3207}
keys moved after adding D: 2380 / 10000 = 0.2380
expected fraction ~= 1/4 = 0.25
keys moved after removing B: 2677 / 10000 = 0.2677
keys still mapped to B after removal (should be 0): 0

With 4 equal-weight nodes, adding a node should move roughly 1 in 4 keys (the new node's fair share); the measured 23.80% is close to the 25% expectation, with the gap explained by finite-sample variance at 100 replicas per node (more replicas would tighten this further, at the cost of a bigger ring). After removing B, 0 keys are still mapped to it, confirming the removal is complete.

Trade-offs and pitfalls

• Replica count is a distribution-smoothness versus memory/lookup-cost trade. More replicas per node means a more even split (lower variance) but a bigger ring array, so slower inserts and more memory; 100 to a few hundred replicas is a common practical range.
• A plain Python list with bisect.insort/.pop is O(n) per underlying array shift, not O(log n), for the insertion/removal itself, even though the search to find the position is O(log n). At very large node counts, a balanced tree or skip list would be needed to make mutation itself sub-linear; call this out explicitly if asked to scale this past a modest node count.
• This design assumes a single process owns the ring. In a distributed setting, every client needs the same ring (same node set, same replica count, same hash function) to agree on key ownership; a stale or divergent ring on one client silently routes some keys to the wrong node.

Direct answer

A retry storm postmortem has two distinct jobs: find exactly where in the call chain retries originated (which service, whether it had backoff/jitter, whether its timeout was shorter than the downstream's real processing time), and quantify how much the retries amplified load as they fanned out through the system, since that amplification is usually what turned a partial outage into a full cascade. Short-term you throttle or disable the offending retries and roll back any recent change that touched timeout or retry config; long-term you replace uncoordinated per-hop retries with a single, budgeted retry policy enforced centrally.

Data to collect

Finding the source of the retries

• Use traces to find the first hop issuing repeated retries: look for spans with the same parent trace and an increasing retry count.
• Correlate logs by request ID to identify the originating client or service.
• Check whether the retrying layer's timeout is shorter than the downstream's actual (degraded) processing time; a timeout misconfigured relative to real latency causes retries even when the downstream would have eventually succeeded.
• Check whether retries use exponential backoff with jitter or fire immediately/on a fixed interval, since immediate/fixed retries are what synchronize into a flood rather than spreading out.
• Check recent deploys or config changes that touched retry policy, timeouts, or error handling around the incident start time.

Worked example: why retries amplify

Consider a call path Client -> Gateway -> Service A -> Service B, where each hop independently retries up to $r$ times on failure, with no shared retry budget. One original client request can result in up to $r$ attempts from the client, each retried up to $r$ times by the gateway, each retried up to $r$ times by Service A, before reaching Service B:

With $r = 3$ retries at each of 3 retrying hops, a single client intent can turn into up to 27 actual attempts hitting Service B. This is why a moderate, transient error at Service B (say, a brief GC pause or a single bad deploy) can explode into a full retry storm purely from independent, uncoordinated retry layers, without Service B's underlying problem ever getting worse on its own.

The standard fix is a single retry budget enforced at one point (commonly the edge or gateway) instead of per-hop retries: if the budget allows retries up to 10% of normal traffic ($b = 0.10$), the worst-case amplification is capped at:

regardless of how many hops exist downstream, because only the budgeted layer is allowed to retry at all; downstream hops propagate failures instead of retrying independently.

Short-term controls

• Disable or cap retries at the identified source hop immediately, or apply rate limiting at ingress.
• Roll back the recent change if one is implicated.
• Open circuit breakers on the overloaded downstream to fail fast rather than let retries keep queuing against it.

Long-term controls

• Replace independent per-hop retries with a single retry budget enforced at one layer, sized like the example above.
• Standardize retry libraries across services to always use exponential backoff with jitter, never immediate or fixed-interval retries.
• Align timeout hierarchies so client timeouts exceed realistic downstream processing time, including under degraded conditions, not just the happy path.
• Add chaos/failure-injection tests that specifically simulate partial downstream failure and verify retry amplification stays bounded.
• Add alerting on retry-rate spikes as a leading indicator, not just on the resulting error rate.

Trade-offs and pitfalls

• A shared retry budget requires either centralized enforcement (a single layer allowed to retry) or shared state (a token bucket every hop checks), both of which add coordination cost compared to each service just retrying independently; the payoff is that amplification is bounded by construction instead of by hoping every team tunes their retry count conservatively.
• Circuit breakers can false-open under legitimate bursty traffic if thresholds are tuned too aggressively, trading availability for protection; the threshold should be validated against real traffic variance, not just incident replay.
• Testing a fix by replaying incident traffic in staging is valuable but easy to get wrong if the replay doesn't preserve the original timing/burstiness, since retry storms are fundamentally a timing phenomenon (synchronized floods), not just a volume phenomenon.