Load Balancing and Traffic Management Questions

Direct answer

An automated canary control loop is a periodic process that increases a canary's traffic weight in small, guarded steps, checks health metrics against the baseline after every step using a minimum-observation window, and holds or immediately rolls back to zero on an SLO breach (a violation of the service-level objective, the reliability target you've committed to, such as 99.9% success rate). The design problem is really about the guardrails and the rollback path, not the ramp itself.

Structured elaboration

Components:

• A traffic router (Envoy, an ingress controller, or a cloud LB) that supports weighted routing and an instant weight-update API.
• A telemetry pipeline producing short and long rolling windows of error rate, latency, and request volume, split by canary vs. baseline.
• A stateless controller that runs the step/check/decide loop and calls the router's weight API.
• A rollback path that can zero out canary weight immediately, independent of the step loop.

Guardrails (the part that actually matters):

Smoothing the weight changes. Compute a fixed step size from the target ramp, then apply the step in small sub-increments at the router level (e.g. over a few seconds) so a single jump doesn't itself cause a connection-pool or cache-warming spike.

Worked example

Target: shift 10% of production traffic to canary over a 10 minute (600s) window, checking every 30s.

So the loop runs 20 times, adding 0.5 percentage points of canary weight each pass, provided the guardrails above all pass. If step 12 (weight = 6%) shows canary p99 = 340ms against a baseline p99 of 260ms:

That exceeds the 1.2x latency-ratio guardrail, so the controller does not advance to step 13. It instead sets weight back to 0% immediately (not a gradual decrease), opens an incident, and freezes further attempts for the cooldown period.

# runs every 30s
current = get_current_weight()
target = min(10, current + 0.5)
m = fetch_metrics(short=60, long=300)
if m.canary_requests_short < 100:
    return  # not enough signal yet, hold at current weight
if m.error_rate_canary > m.error_rate_baseline + 0.005 or \
   m.p99_canary_long > m.p99_baseline_long * 1.2:
    set_weight(0)          # immediate rollback, not gradual
    open_incident(m)
    start_cooldown(minutes=30)
else:
    set_weight(target)     # router ramps this smoothly over a few seconds

Extensions: downstream capacity and CI/CD integration

Downstream-capacity-constrained ramp. The guardrails above are all canary-side (error rate, latency). A canary can look perfectly healthy on its own metrics while the traffic it forwards saturates a downstream dependency, a connection pool, a queue, or a rate-limited third-party API, before that dependency's own health signal even reflects the problem. Add a downstream capacity ceiling as its own guardrail: before each step, check the downstream dependency's current utilization (connection-pool saturation, queue depth, or a published capacity headroom number) and cap the canary weight so its incremental load stays under that ceiling, independent of what the canary's own error and latency metrics say. In practice the step size becomes the smaller of the scheduled step and the downstream headroom divided by requests-per-weight-point, and downstream saturation becomes its own rollback trigger alongside the error and latency guardrails, since the canary's own SLOs will not catch a downstream problem until it is already failing.

CI/CD pipeline integration. The loop is usually not a standalone daemon, it is a stage in the deploy pipeline. A build that passes its test stage triggers the deploy, which starts the canary at 0% weight and hands control to this loop; the pipeline blocks promotion to the next stage (wider rollout, or the next environment) until the loop reports a verdict, not a fixed timer. On success the loop reports pass and the pipeline proceeds to full rollout. On a guardrail breach, the loop's immediate zero-weight rollback from the worked example above is reported back as a failed pipeline stage: the deploy is marked failed, the previous version stays serving at 100%, and the pipeline stops rather than continuing to the next stage. That makes the human's only manual step reviewing the failure, not remembering to check on the canary.

Trade-offs and pitfalls

• Short windows are noisy, long windows are slow. A short-only window trips on transient blips; a long-only window lets real regressions run for minutes before anyone notices. Using both, and requiring the long window to confirm, is the standard resolution.
• Rollback must never itself be a slow ramp. The instinct to "gradually" bring weight back down defeats the purpose; on breach, cut to zero immediately and investigate after.
• Minimum-request guardrails matter more for low-traffic services. A service doing 50 RPS total needs a longer observation window or a lower canary weight cap just to get statistically meaningful samples per step.
• Business metrics can lag system metrics. A checkout canary can look perfectly healthy on latency and error rate while conversion silently drops; if the metric that matters is business-level, it needs its own window and threshold, not just infra SLOs.
• This is inherently single-service. Running the same loop concurrently across many dependent services without coordination risks compounding partial failures across a call graph; large orgs typically centralize this into a shared canary-analysis service rather than one loop per team.

Direct answer

Draining safely with a mix of short HTTP and long-lived WebSocket connections means stopping admission of new work immediately (fail readiness, deregister), letting in-flight HTTP finish naturally since it is already short, and giving WebSocket connections an explicit grace period bounded by a measured drain timeout, after which any still-open connection is force-closed. The timeout itself should not be a guess: measure the real duration distribution of long-lived connections in staging, pick a percentile with a safety margin, and validate that choice by re-running the drain and checking how many connections it actually force-closes.

Structured elaboration

• Stop new traffic first. Flip readiness to failing (or deregister from the load balancer's target pool) before doing anything else, so no new HTTP request or WebSocket upgrade attempt lands on the draining instance. New upgrade attempts should get a fast rejection (503), not a connection that immediately gets drained.
• Let short HTTP finish on its own. In-flight HTTP requests are typically done within seconds; rely on the load balancer's own connection-draining or deregistration-delay setting to keep the instance reachable for just those existing connections, not new ones.
• Give WebSockets a bounded grace period. Send a close frame with a reason and, where the client supports it, a suggested reconnect delay. Track how many WebSocket connections remain open and how long the drain has been running; when the timeout is reached, force-close whatever is left rather than draining indefinitely.
• Measuring the timeout in staging. Generate a realistic connection-duration distribution under staged load (not just connection count, the actual spread of session lengths), trigger a controlled drain, record how long each connection was open, and compute a high percentile (for example p90 or p99) of that distribution as the timeout floor before adding a safety margin.

stateDiagram-v2
    [*] --> InRotation
    InRotation --> Draining: marked for removal, readiness fails
    Draining --> Draining: HTTP finishes naturally; WS gets close frame
    Draining --> SafeToTerminate: HTTP count = 0 and WS count = 0
    Draining --> ForceClose: drain timeout reached
    ForceClose --> SafeToTerminate
    SafeToTerminate --> [*]

Worked example

A staging drain test records 10 WebSocket session durations in seconds (fully specified for this example): 12, 15, 20, 22, 30, 35, 40, 55, 90, 240. Already sorted ascending, with $n = 10$.

Using the nearest-rank method, the 90th percentile index is:

The 9th value in the sorted list is 90 seconds, so $p_{90} = 90\text{ s}$.

Add a 20% safety margin and round to a configuration-friendly value:

Validate by re-running the same drain with the 110-second timeout: only the 240-second session exceeds it, so exactly 1 of 10 sessions would be force-closed, a $1/10 = 10\%$ forced-close rate in this sample. If the team's tolerance for forced WebSocket closures during a deploy is, say, under 5%, this result fails validation and the timeout needs to go higher, or the long tail needs a real fix (session handoff and client-side reconnect) instead of a longer wait.

Autoscaling context. The same drain path fires on scale-in, not just deploys. If sticky routing pins long-lived sessions to specific pods, a scale-in event has to drain those pinned sessions the same way, and frequent autoscaling churn means paying this drain cost far more often than a deploy cadence would; that is a real argument for moving session state out of the pod (a shared store) rather than tuning the drain timeout ever higher to compensate.

Trade-offs & pitfalls

• Too-short a timeout forces out legitimate long sessions and shows up as user-visible disconnects; too-long a timeout slows every deploy and scale-in and holds resources the orchestrator thinks it already reclaimed.
• A percentile chosen from a staging dataset is only as good as how representative that dataset's traffic mix and connection-duration shape are; validate against real production duration distributions periodically, not once at design time.
• Common wrong turn: reusing the same drain timeout for HTTP and WebSocket paths. HTTP's tail is usually seconds; WebSocket's tail can be minutes to hours, and forcing HTTP to wait for the WebSocket-sized timeout just slows every deploy for no benefit.
• Session handoff or reconnect logic on the client is what actually solves the extreme tail (a connection open far longer than any reasonable timeout); a timeout alone only bounds how long you wait before giving up on it.

Direct answer

The fix has to work at both layers: the load balancer should bring a recovered instance back into rotation gradually (slow-start / weight ramp) rather than at full weight immediately, and clients should retry with jittered exponential backoff rather than reconnecting in lockstep the moment they see the instance healthy again. Neither alone is sufficient: LB-side ramping protects against a coordinated flood, but internal callers that bypass client backoff will still hit the instance hard once its weight is nonzero, so admission control (a hard per-instance connection/request cap) is the backstop that holds regardless of what clients do.

Load balancer mitigations

• Slow-start / weighted ramp-up: on transition to healthy, start the instance's LB weight near zero and increase it on a schedule (linear or exponential) instead of jumping to full share immediately.
• Health-check hysteresis: require several consecutive successful probes, not one, before even starting the ramp, so a flapping instance doesn't repeatedly trigger fresh thundering-herd events.
• Hard admission caps: independent of weight, enforce a per-instance connection/request ceiling at the LB so the ramp schedule isn't the only thing standing between the instance and overload.
• Sticky-routing caution: if session affinity is in use, avoid re-establishing it fully during ramp-up, since affinity can concentrate a disproportionate share of long-lived clients onto a still-warming instance.

Application-level mitigations

• Jittered exponential backoff on clients: spread reconnect attempts in time instead of retrying immediately or on a fixed interval that resynchronizes across clients.
• Admission queueing with a bound: the instance itself accepts up to a limit and returns 429/503 with Retry-After beyond that, rather than accepting everything and falling over.
• Circuit breakers upstream of the instance: if error/latency crosses a threshold during ramp-up, upstream callers back off automatically instead of continuing to hammer it.

Worked example

Suppose the recovered instance's established safe steady-state capacity is $C = 5{,}000$ RPS, and the pent-up demand specifically targeting it (clients that were failing over away from it and are now retrying) is $D = 50{,}000$ RPS, 10x its safe capacity. The LB ramps its weight starting at $w_0 = 1\%$, doubling every 10 seconds:

By t=40s the instance is already running at its full safe capacity even though the LB weight is still far below 100%; from that point on, the hard admission cap (not the weight ramp) is what's actually protecting it, and the ramp continuing upward just determines when the instance starts absorbing a "fair" share once the herd's backlog (D) has drained through the rest of the pool. This is the concrete reason both mechanisms are needed: the ramp controls the early window when D is far above C, and the hard cap controls everything after, once weight alone would otherwise overshoot.

Trade-offs and pitfalls

• Slow-start delays how quickly the pool's overall capacity recovers, which matters if the rest of the pool is itself under strain; the ramp duration is a real trade-off between protecting the recovering instance and relieving the rest of the fleet faster.
• Implementing only client-side backoff misses internal service-to-service callers that don't go through the same retry library; implementing only LB-side ramping misses the fact that a fixed weight schedule can still be overwhelmed if pent-up demand is large enough relative to the ramp rate, which is exactly why the hard admission cap has to exist independent of the ramp.
• A common failure in postmortems is tuning the ramp curve without ever computing whether the ramp rate can plausibly outrun realistic pent-up demand (as in the worked example above); a ramp that's too slow relative to D just delays the herd, and one that's too fast relative to C doesn't protect the instance at all.

Direct answer

Use STAR: set up the situation and your role in one or two sentences, describe concretely what you did during the incident (not what "the team" did), state the outcome and what changed afterward, and end with the lesson. The strongest signal an interviewer is listening for is not the outage itself, it is whether your actions were deliberate and whether the fix addressed the root cause rather than just the symptom.

Structured elaboration

Situation: Name the system, the trigger, and the specific way the load balancer or failover mechanism misbehaved (routed to a dead node, failed to detect an unhealthy backend, split-brained across regions, flapped between healthy/unhealthy). Vague ("a server went down") is weak; specific ("health checks marked degraded instances as healthy because the probe timeout was shorter than the instance's real response time under load") is strong.

Task: State your role and scope of ownership plainly, on-call engineer, service owner, incident commander, so the interviewer knows what you controlled versus what you coordinated.

Action: This is the section to spend the most time on. Structure it in the order things actually happened:

• Immediate containment: what you changed to stop user impact (pulled a node/region from rotation, rolled back a config, manually failed over).
• Diagnosis: how you found the actual cause, not just the symptom (log correlation, comparing healthy vs unhealthy instance behavior, checking recent changes).
• Communication: who you told, how often, and why that mattered for a coordinated response.

Result: State the outcome honestly and in terms you can actually defend if asked to go deeper: service impact was contained, the specific fix that resolved it, and what changed in the system or process afterward (a corrected health-check timeout, an added synthetic probe, a canary requirement for load-balancer config changes). If you do not have precise numbers from memory, describe the outcome qualitatively (restored to normal operation, contained to a subset of traffic) rather than inventing statistics. An interviewer who probes "how many users were affected" and gets a fabricated number will trust the rest of the story less, not more.

Worked example

A representative version of this story, deliberately written without invented precision so it illustrates structure rather than fake data:

Situation: A payment service ran active-active across two regions behind a global load balancer. During a regional network blip, the load balancer's health checks did not fail the degraded region out of rotation, so a meaningful share of traffic kept landing on instances that were timing out.

Task: As the on-call engineer, I owned containing user impact and driving the fix.

Action: I manually pulled the degraded region from the load balancer's pool through the control plane rather than waiting for automatic health-check convergence, which was clearly not going to trigger in time. In parallel, I checked what changed recently and found that a prior change had shortened the health-check probe timeout below the instance's real p99 response time under load, so probes were failing even on instances that were still serving most requests correctly, the health check itself was miscalibrated, not just unlucky timing. I posted incident updates on a fixed cadence so stakeholders were not pinging for status mid-fix.

Result: Traffic normalized once the region was pulled and the timeout was corrected; the underlying fix was reverting the probe timeout to its previous value and adding a synthetic cross-region probe that exercises the failover path continuously (rather than only during real incidents) so a miscalibrated health check would be caught before the next config change shipped, not during the next real outage.

What I learned: A load balancer's failover is only as good as its health check's ability to distinguish "genuinely unhealthy" from "temporarily slow"; changes to health-check parameters deserve the same review rigor as changes to the service itself, because they directly control blast radius during a real failure.

Trade-offs and pitfalls

• Do not narrate what "we" did when the question asks what you did. Interviewers are listening for your specific decisions and reasoning, not a team summary.
• Avoid ending at "it got fixed." The strongest answers name the systemic change that makes the same failure mode structurally harder to repeat.
• Do not manufacture precision you do not have. A specific, remembered detail (which config changed, what the health check was checking) is more credible than an invented percentage or timestamp, and it holds up better under a skeptical follow-up.
• Watch for blame framing. A root cause that reads as "someone else misconfigured it" without acknowledging what allowed that misconfiguration to ship (no canary, no review gate) reads as less senior than one that owns the systemic gap.

Direct answer

Round-robin, weighted, and GeoDNS are all ways an authoritative DNS server chooses which IP to hand back for the same hostname, they differ only in the selection rule (rotate, weighted-random, or client-location-based), and all three inherit the same fundamental limitation: once a resolver caches an answer, DNS has no way to recall it. Failover speed is bounded entirely by TTL and by how faithfully resolvers respect it.

Structured elaboration

None of the three is health-aware on its own. In practice all three are paired with active health checks (e.g. Route 53 health-checked records, a GSLB, Global Server Load Balancing: DNS that also factors in live health checks, so it can steer traffic away from an unhealthy region instead of rotating blindly) that add or remove a record from the pool based on liveness, the DNS strategy alone only decides selection among whatever is currently in the pool.

Worked example: TTL vs. authoritative query load

Assume roughly 1,000,000 independently-caching clients (accounting for resolver fan-out, this is "cache entries," not literal end users) issuing steady lookups for the hostname. Because each client re-queries only when its cached TTL expires, the sustained query rate the authoritative server sees is approximately:

Dropping TTL by 10x (300s to 30s) multiplies authoritative query load by 10x for the same client population. Designing TTL is therefore a direct trade between failover speed and nameserver load, not a free lunch: pick the lowest TTL that (a) your authoritative infrastructure (typically anycast-backed managed DNS) can sustain at expected client scale, and (b) actual public resolvers will honor, many ISP and public resolvers apply a practical floor around 30 to 60 seconds regardless of the record's stated TTL.

Trade-offs and pitfalls

• Negative caching is caching too. An NXDOMAIN or SERVFAIL response gets cached according to the zone's SOA minimum TTL, a botched record removal can produce a resolvable-but-wrong period followed by an unexpectedly long "not found" period.
• Very low TTLs are not guaranteed to help. Below the resolver's practical floor, requesting a 5s TTL buys nothing while still multiplying real query volume against the servers that do honor it.
• DNS is a request for a new answer, not a push. A client that already opened a connection or cached the resolution client-side (browser, HTTP client, container DNS cache) is unaffected by a DNS change until it does a fresh lookup, so DNS TTL bounds resolver-level staleness, not necessarily client-visible staleness.
• A common practical pattern: keep a stable, moderate TTL (e.g. 300s) in steady state to protect the authoritative servers, and only drop to a low TTL (e.g. 30 to 60s) proactively ahead of a planned migration or maintenance window, since a TTL change itself only takes effect after the previous TTL expires.