Load Balancing and Traffic Management Questions

Direct answer

Keep a min-heap keyed not by raw connection count but by connection count divided by declared capacity (a load ratio), with a deterministic tie-break. Deletions and count updates are handled with lazy invalidation (a version token per backend) rather than an expensive heap-fix-and-remove, so every operation stays logarithmic and selectBackend never returns a stale or removed backend.

Approach

Raw least-connections (ranking purely by active connection count) starves small, low-capacity backends when capacities differ: a backend with capacity 1 and 1 active connection is fully loaded, but a raw min-heap would still prefer it over a capacity-4 backend already holding 2 connections, because 1 < 2. Dividing by capacity turns the heap key into a load ratio, so the comparison becomes "who has more headroom," which is what you actually want when capacities are heterogeneous.

For concurrency, a single lock protects the heap, the connection-count map, and a per-backend version counter. incConn/decConn bump the version and re-push a fresh heap entry rather than trying to fix the existing entry's position in place; the old entry is left in the heap but tagged with a now-stale version. selectBackend pops entries off the top and discards any whose version doesn't match the backend's current version, until it finds a live one. This is the standard lazy-deletion trick for heaps that need cheap updates: amortized cost stays O(log n) per operation, and stale entries are simply cheap to skip past rather than expensive to keep synchronized in place.

Code (Python)

import heapq
import itertools
import threading


class LeastConnScheduler:
    def __init__(self):
        self._lock = threading.Lock()
        self._heap = []                     # (load_ratio, conns, tie, id, version) tuples
        self._conns = {}                    # id -> current connection count
        self._capacity = {}                 # id -> declared capacity (default 1)
        self._version = {}                  # id -> token that invalidates stale heap entries
        self._tie = itertools.count()       # deterministic, insertion-order tiebreak

    def add_backend(self, backend_id, capacity=1):
        with self._lock:
            if backend_id in self._conns:
                return
            self._conns[backend_id] = 0
            self._capacity[backend_id] = capacity
            self._version[backend_id] = 0
            self._push(backend_id)

    def remove_backend(self, backend_id):
        with self._lock:
            if backend_id not in self._conns:
                return
            self._version[backend_id] += 1   # invalidates any stale heap entries for this id
            del self._conns[backend_id]
            del self._capacity[backend_id]

    def inc_conn(self, backend_id):
        with self._lock:
            if backend_id not in self._conns:
                return
            self._conns[backend_id] += 1
            self._version[backend_id] += 1
            self._push(backend_id)

    def dec_conn(self, backend_id):
        with self._lock:
            if backend_id not in self._conns:
                return
            if self._conns[backend_id] > 0:
                self._conns[backend_id] -= 1
            self._version[backend_id] += 1
            self._push(backend_id)

    def select_backend(self):
        with self._lock:
            while self._heap:
                ratio, conns, tie, backend_id, ver = self._heap[0]
                if backend_id not in self._conns or ver != self._version[backend_id]:
                    heapq.heappop(self._heap)   # stale entry, discard and keep looking
                    continue
                return backend_id
            return None

    def _push(self, backend_id):
        conns = self._conns[backend_id]
        capacity = self._capacity[backend_id]
        ratio = conns / capacity
        heapq.heappush(
            self._heap,
            (ratio, conns, next(self._tie), backend_id, self._version[backend_id]),
        )

Key points

• Capacity-weighted ratio, not raw count, is the fix for the starvation case the question asks about. A backend with capacity 4 and 2 connections (ratio 0.5) is correctly preferred over a backend with capacity 1 and 1 connection (ratio 1.0), even though the second one has fewer raw connections.
• Deterministic tie-break comes from the monotonic insertion counter, not from backend id string comparison, so ties resolve consistently regardless of what ids happen to be in play.
• Lazy deletion avoids the classic heap problem: heapq has no efficient "update the priority of an existing item" operation, so instead of searching for and fixing an entry (which would be O(n)), a new entry is pushed and the old one is left to be skipped later, at the cost of some extra memory for stale entries between updates.

Complexity

• add_backend, remove_backend, inc_conn, dec_conn: O(log n) for the heap push (remove is O(1) plus lazy cleanup deferred to later select_backend calls).
• select_backend: amortized O(log n); each stale entry it skips was already paid for by the update that created it, so total work across all operations stays bounded.
• Space: O(n) live entries plus O(u) stale entries, where u is the number of updates since the last full cleanup; stale entries are bounded by the number of inc_conn/dec_conn calls, not unbounded.

Edge cases

• dec_conn never takes a count below zero.
• inc_conn/dec_conn on an unknown id are no-ops (verified below along with everything else).
• select_backend on an empty scheduler returns None.
• A backend removed while it was at the top of the heap is correctly skipped on the next select_backend call, verified by the concurrency test.

Running a capacity-skew check (a small backend fully loaded versus a big backend partially loaded), a tie-break check, and a concurrency stress test with 8 threads each doing 3000 select/inc/dec cycles against 4 backends:

pick after small=1/1, big=2/4: big
tie-break pick (both idle): a
concurrency errors: []
final connection counts: {'n0': 0, 'n1': 0, 'n2': 0, 'n3': 0}

The first line confirms the capacity-skew fix works as intended: big (2 of 4 capacity used, ratio 0.5) is preferred over small (1 of 1 capacity used, ratio 1.0) even though small has fewer raw connections. The stress test completed with zero errors and connection counts correctly balanced back to zero.

Trade-offs and pitfalls

• Lazy deletion trades memory for update speed. In a system with very high inc/dec churn and rare removals, stale entries can accumulate; a periodic heap rebuild (or a max-stale-entry threshold that triggers one) bounds this in production.
• A single lock is the simplest correct answer but caps throughput under extreme contention. Sharding backends across multiple scheduler instances (consistent-hash the backend id to a shard) removes the single-lock bottleneck at the cost of only having a locally, not globally, least-loaded view per shard.
• Capacity must come from somewhere trustworthy. If capacity is self-reported by a backend rather than measured or configured centrally, a misconfigured or malicious backend can claim a huge capacity and starve itself unfairly of protection, or claim a tiny one and hoard traffic away from itself; treat capacity as operator-configured or telemetry-derived, not backend-asserted.
• Connection count is a proxy for load, not load itself. For workloads where connections vary wildly in cost (such as a backend pool handling a mix of 50ms lookups and multi-second batch requests, where two backends can show the same active-connection count while carrying very different amounts of real work), least-connections by itself, even capacity-weighted, is a weaker signal than actual measured latency or queue depth.

Direct answer

HTTP/1.1 keep-alive needs roughly one TCP connection per concurrent in-flight request, so an L7 balancer scales by managing connection count. HTTP/2 and gRPC multiplex many concurrent streams over a small number of long-lived connections, so the balancer has to scale by managing stream concurrency within a connection instead, and load signals that were adequate for HTTP/1.1 (active connection count) become nearly useless. The practical consequence: fewer sockets and less TLS/TCP handshake overhead, but a new failure mode where one connection getting stalled or overloaded can degrade every stream multiplexed on it (head-of-line effects), which the balancer has to actively guard against.

Connection model comparison

Per-connection limits to tune

• HTTP/1.1: max connections per backend (bounds concurrency directly), idle keep-alive timeout (frees sockets from clients that went quiet), and pool size on the LB's upstream side.
• HTTP/2 / gRPC: max_concurrent_streams per connection (how many in-flight requests one connection may carry, commonly capped well below the protocol's theoretical maximum to bound blast radius), a small upstream connection pool per backend (a handful, not one) so a single stalled connection doesn't take out all traffic to that backend, per-stream flow-control window size (the amount of unacknowledged data a stream may have in flight before the sender must pause), and a keepalive ping interval to detect a half-open connection before streams queue behind a dead peer.

Load measurement and backpressure

Connection count stops being a useful load signal once multiplexing is in play: a backend with 4 connections and 400 streams looks identical to one with 4 connections and 4 streams if you only count sockets. Instead:

• Measure in-flight streams per backend (or max_concurrent_streams - current_streams as available capacity) and use that as the weighting signal for load-aware routing.
• Track per-stream and per-connection latency percentiles separately; a rising per-connection tail with stable per-stream counts points at connection-level contention (CPU, flow-control), not request volume.
• Apply admission control at the stream level: reject new streams past a concurrency threshold with a fast, typed backpressure signal (gRPC RESOURCE_EXHAUSTED, HTTP 429) rather than accepting and queuing, which just moves the head-of-line problem later.
• Detect stalled streams (no progress against their flow-control window) and reset them individually instead of tearing down the whole connection, so one bad stream doesn't punish every other stream sharing it.

Worked example

Suppose the LB maintains a pool of 4 HTTP/2 connections to a backend, each configured with max_concurrent_streams = 100:

That backend can serve 400 concurrent RPCs using 4 sockets. Reaching the same 400 concurrent in-flight requests under HTTP/1.1 keep-alive would require roughly 400 separate TCP+TLS-established connections (one per in-flight request), which is exactly the socket and handshake overhead multiplexing removes. The other side of that number: if one of those 4 connections stalls, up to 100 of the 400 in-flight requests (25%) can be head-of-line blocked simultaneously, which is why the pool size (not just 1 connection) and per-stream stall detection both matter.

Trade-offs and pitfalls

• Fewer, fatter connections are more efficient but concentrate risk: a single TCP-level packet loss stalls every stream on that connection at the transport layer, even though the streams are logically independent at the application layer. This TCP-level head-of-line blocking is a known limitation of running multiplexing over TCP (it's the reason HTTP/3/QUIC exists), but that's depth beyond what most interviews expect; the interview-relevant point is just that a small connection pool, not a single connection, bounds the blast radius.
• A common mistake is reusing HTTP/1.1-era LB health/load metrics (connection count, connections-per-second) unchanged for HTTP/2 backends; they will systematically under-detect overload because a backend can look "quiet" on connections while being saturated on streams.
• Setting max_concurrent_streams too high trades efficiency for blast radius; setting it too low defeats the purpose of multiplexing and pushes you back toward connection-count scaling.

Direct answer

This is a deterministic-cause outage: the fix is to restore traffic through a path that has a valid certificate (failover to a standby LB, hot-load a backup cert, or temporarily terminate TLS elsewhere) as fast as possible, communicate on a fixed, predictable cadence while that happens, and then treat the root cause as a monitoring and automation gap rather than a one-time human mistake, since a cert that reached expiry without anyone acting means the process that was supposed to catch it failed silently well before the outage.

Immediate triage (first ~15 minutes)

• Confirm scope and cause: check LB logs for TLS handshake failures and the certificate's expiry timestamp to distinguish a cert issue from an application or network issue reporting similar symptoms (global 503s can also come from a bad deploy or DNS problem).
• Freeze churn: pause any in-flight deploys or automated config pushes so the incident isn't complicated by unrelated changes landing mid-triage.
• Pull in the right people: notify infra/network/security on-call and declare an incident commander if this is genuinely global impact.

Short-term mitigation

• Fastest path: if a hot-standby LB or path with a valid certificate exists, shift traffic to it (DNS weight change or traffic steering) rather than trying to fix the primary path under pressure.
• If the LB supports multiple certs or hot-reload, load a previously-issued backup/rollover certificate if one exists.
• If neither is available, use automated ACME (Automatic Certificate Management Environment, the protocol that lets software request and renew certificates without a human) / CA (certificate authority, the trusted party that issues certificates) tooling to issue a short-lived emergency certificate and install it, prioritizing the automated path over manual cert generation to reduce the chance of a second mistake under time pressure.
• As a last resort only, and only if policy allows, terminate TLS at a CDN or edge proxy that already has a valid cert and forward over a private/trusted link to the backend.
• Validate before declaring resolved: run synthetic TLS handshakes and smoke tests against key endpoints, not just "the dashboard looks green."

Stakeholder and customer communication

• Acknowledge within the first ~10 minutes: post to the status page and internal channels with scope, known impact, and a time for the next update, even if the next update is "still investigating."
• Update on a fixed cadence (every 15-30 minutes) regardless of whether there's new information, since silence reads worse than "no change yet" during a global outage.
• Close the loop once restored with an accurate timeline and a plain-language root cause, then follow with a fuller postmortem summary to execs, product, and customer-facing teams so they can answer customer questions consistently.

Long-term remediation and process changes

• Automate the full certificate lifecycle (issuance and renewal) end to end so a human is never the trigger for routine renewal; the emergency manual path should exist only as a fallback, not the primary mechanism.
• Alert on the action, not just the deadline: a single "certificate expires in N days" alert is not enough, because it doesn't distinguish "renewal will happen automatically before then" from "renewal has already been silently failing." A stronger design pages a human if both conditions hold: fewer than a threshold of days remain, and there has been no successful renewal within the automation's normal cycle. For example, with a 90-day certificate and automation that renews at the 30-days-remaining mark, an escalation alert at 14 days remaining plus "no successful renewal event logged in the last 16 days" catches a silently-failing automation with real margin before it becomes a repeat of this incident, rather than firing on the same schedule the (broken) automation was supposed to act on.
• Rehearse the failure: run a game day where the cert automation is deliberately disabled and confirm the escalation alert actually fires and the manual fallback actually works, rather than assuming it does.
• Track the postmortem to closure with named owners and deadlines for the automation and alerting changes, not just the incident write-up.

Trade-offs and pitfalls

• A pure "days until expiry" alert is the common mistake: it looks reasonable until you realize it fires on the same cadence the automation was already supposed to satisfy, so if the automation silently breaks right after a renewal, the alert doesn't add meaningfully more warning time than the automation's own schedule did. Tying the alert to "no successful renewal observed" closes that gap.
• Emergency manual certificate issuance under time pressure is itself a risk (wrong domain, wrong chain, wrong key usage); preferring automated emergency issuance over a fully manual process reduces that risk even during an incident.
• Communicating on a fixed cadence even with "no update" costs credibility less than going silent, but it does require someone dedicated to comms so the person fixing the issue isn't also the one writing status updates.

Direct answer

Layer 4 load balancers make routing decisions using only transport-layer metadata (source and destination IP, port, protocol) and forward or proxy TCP/UDP connections without looking at the payload. Layer 7 load balancers terminate the application protocol, usually HTTP or HTTPS, and route on request content: host header, URL path, cookies, or other headers. L4 is faster and protocol-agnostic because it never parses the payload; L7 costs more CPU per request but can make far smarter routing, security, and traffic-shaping decisions. Pick L4 when you need raw throughput or must preserve end-to-end encryption; pick L7 when routing needs to understand HTTP semantics.

Structured elaboration

Decision guidance: choose L4 when the balancer must not (or need not) understand the payload, or when throughput at minimal overhead is the priority. Choose L7 when the routing decision itself depends on request content. Many production systems run both: an L4 tier absorbing raw connections at the edge, with an L7 tier immediately behind it for content-aware routing.

Worked example

Consider two systems that need a load balancer. First, a Postgres connection pooler in front of a cluster: clients authenticate to the database itself over TLS, connections are long-lived, and there is no HTTP semantics to route on. An L4 balancer is the right fit here: it forwards TCP connections without touching the encrypted stream, so end-to-end TLS stays intact and per-connection overhead stays minimal.

Second, a public REST API that must send /v1/users and /v1/orders to two different backend services, apply a WAF rule to block known bad user agents, and terminate TLS once at the edge. None of that is possible without reading the request line and headers, so this requires an L7 balancer, even though it costs more CPU per request than the L4 case.

Trade-offs & pitfalls

• L7 termination breaks end-to-end encryption unless the balancer re-encrypts to the backend (TLS bridging); this is a common follow-up in security-conscious interviews.
• L4 cannot do content-based routing or cookie affinity. Bolting host/path logic onto an L4 balancer just pushes the work down a layer where it is harder to operate.
• Hybrid designs (L4 at the outer edge, L7 immediately behind it) are standard practice, not a compromise: the L4 tier absorbs raw connection volume and the L7 tier handles content-aware decisions.
• Common wrong turn: treating L7 as strictly better. It adds a per-request parsing and termination cost and a larger attack surface (header injection, request smuggling) that an L4 balancer never has to reason about.

Direct answer

Round-robin, weighted, and GeoDNS are all ways an authoritative DNS server chooses which IP to hand back for the same hostname, they differ only in the selection rule (rotate, weighted-random, or client-location-based), and all three inherit the same fundamental limitation: once a resolver caches an answer, DNS has no way to recall it. Failover speed is bounded entirely by TTL and by how faithfully resolvers respect it.

Structured elaboration

None of the three is health-aware on its own. In practice all three are paired with active health checks (e.g. Route 53 health-checked records, a GSLB, Global Server Load Balancing: DNS that also factors in live health checks, so it can steer traffic away from an unhealthy region instead of rotating blindly) that add or remove a record from the pool based on liveness, the DNS strategy alone only decides selection among whatever is currently in the pool.

Worked example: TTL vs. authoritative query load

Assume roughly 1,000,000 independently-caching clients (accounting for resolver fan-out, this is "cache entries," not literal end users) issuing steady lookups for the hostname. Because each client re-queries only when its cached TTL expires, the sustained query rate the authoritative server sees is approximately:

Dropping TTL by 10x (300s to 30s) multiplies authoritative query load by 10x for the same client population. Designing TTL is therefore a direct trade between failover speed and nameserver load, not a free lunch: pick the lowest TTL that (a) your authoritative infrastructure (typically anycast-backed managed DNS) can sustain at expected client scale, and (b) actual public resolvers will honor, many ISP and public resolvers apply a practical floor around 30 to 60 seconds regardless of the record's stated TTL.

Trade-offs and pitfalls

• Negative caching is caching too. An NXDOMAIN or SERVFAIL response gets cached according to the zone's SOA minimum TTL, a botched record removal can produce a resolvable-but-wrong period followed by an unexpectedly long "not found" period.
• Very low TTLs are not guaranteed to help. Below the resolver's practical floor, requesting a 5s TTL buys nothing while still multiplying real query volume against the servers that do honor it.
• DNS is a request for a new answer, not a push. A client that already opened a connection or cached the resolution client-side (browser, HTTP client, container DNS cache) is unaffected by a DNS change until it does a fresh lookup, so DNS TTL bounds resolver-level staleness, not necessarily client-visible staleness.
• A common practical pattern: keep a stable, moderate TTL (e.g. 300s) in steady state to protect the authoritative servers, and only drop to a low TTL (e.g. 30 to 60s) proactively ahead of a planned migration or maintenance window, since a TTL change itself only takes effect after the previous TTL expires.