Real-Time and Streaming System Design Questions

Direct answer

Offers should be pushed to drivers who are online, available and have a healthy connection, because a ride offer is time-critical and polling fast enough to match push wastes battery and server capacity. Pull stays as a safety net: the app asks "anything for me?" when it comes to the foreground, after a reconnect, and on a slow timer only while its push channel looks unhealthy. The fairness risk is real but it comes from who the dispatcher chooses to push to, not from push itself: if the offer always goes to the top-ranked few drivers, the same drivers are pinged again and again while others nearby wait. So the hybrid needs an exposure policy (who is eligible for the next offer) alongside the transport policy (how the offer travels).

Terms used below

• Push (server-initiated): the server sends the offer down an open connection (a WebSocket or similar long-lived link) or via the phone's platform push service (APNs, Apple Push Notification service, on iOS; FCM, Firebase Cloud Messaging, on Android), which can wake a backgrounded app.
• Pull (polling): the app asks the server at intervals. Long polling is a variant where the server holds the request open until something arrives or a timeout passes.
• Offer: a time-limited proposal ("pickup 4 min away, accept within 15 s"). An offer is useless once it expires.
• Over-exposure: some drivers receiving a disproportionate share of offers, which also means they decline more (fatigue) and other drivers see too few.

Push vs pull on the three named axes

That last row is why pure push is not enough: mobile connections die silently in tunnels, on network handoffs and when the OS suspends the app.

The hybrid design

flowchart TB
  D[Dispatcher: ranks candidates] --> X[Exposure policy: fairness band, caps, cooldowns]
  X --> S{Driver connection healthy?}
  S -->|yes| WS[Push offer on live socket]
  S -->|app backgrounded| OS[Platform push to wake app]
  WS --> ACK{Delivery ack within 2 s?}
  ACK -->|no| OS
  OS --> P[App wakes and pulls pending offers]

When each mode is used:

1. Push over the live connection: the default for drivers who are online, available and whose last heartbeat is fresh. Every offer carries an ID and an expiry; the app sends a delivery ack (a short "received" message) immediately, separate from accept/decline.
2. Platform push (APNs/FCM): when the app is backgrounded or the socket is down. It is best-effort and can be delayed, so it carries only "you have an offer, open the app", not the offer itself.
3. Pull: on app foreground, immediately after any reconnect, and as a fallback timer (for example every 10 s) that runs only while the socket is unhealthy. The pull endpoint returns any unexpired offers held for this driver, so a push that was lost is recovered within one pull.
4. Server-side reconciliation: if no delivery ack arrives within 2 s, the dispatcher sends a platform push, and if the offer is still unacknowledged when half its window has passed, it withdraws the offer and moves on, so a rider is not kept waiting on a driver who never saw it.

Idempotency: accepts carry the offer ID, and the server accepts the first valid accept for a ride and rejects the rest, so a driver who receives the same offer by push and by pull cannot be double-booked.

The over-exposure problem, and the fix

A push-heavy dispatcher usually ranks drivers by pickup time and sends to the best one, or broadcasts to the best few. Drivers parked near a hotspot, or with consistently good connections (who ack fastest and so rank well on responsiveness), win almost every ranking. They get offers they decline, while drivers slightly further away sit idle. It also skews the data: the system learns from the few who are over-offered.

Mitigations, in the order I would add them:

• Fairness band: treat all drivers whose pickup time is within a small margin of the best (say 60 s) as equivalent, and choose among them by fewest recent offers, not by rank.
• Exposure caps and cooldowns: at most K offers per driver per rolling window; a driver who just declined gets a short cooldown before the next offer.
• Sequential rather than broadcast offers: one driver at a time with a short window, instead of pinging the top three at once, which triples exposure per ride.
• Monitor exposure distribution: share of offers going to the top 10% of drivers, offers per driver-hour, decline rate by exposure level.

A small simulation of the effect

This simulation isolates the ranking effect: 200 drivers fixed in a 10 km by 10 km city, 2,000 ride requests clustered around a hotspot, no acceptance modelled. These drivers are static points with no modelled speed, so the simulation approximates the fairness band by pickup distance instead of pickup time; a real system would calibrate the seconds-to-kilometres conversion from live ETAs, and here 1 km stands in for the 60 s margin described above, the same fairness-band idea in the units the simulation can compute directly. It compares broadcasting to the three nearest drivers against the exposure-cap mitigation in concrete form: one offer per request chosen from a 1 km fairness band, with the "K offers per driver per rolling window" cap set to K=5 over a window of the last 100 requests.

import random, math

def simulate(policy, seed=7, n_drivers=200, n_requests=2000, cap=None):
    rng = random.Random(seed)
    # drivers scattered over a 10 km x 10 km city; riders cluster near a hotspot at (2, 2)
    drivers = [(rng.uniform(0, 10), rng.uniform(0, 10)) for _ in range(n_drivers)]
    offers = [0] * n_drivers
    recent = []                      # driver ids offered in the last 100 requests
    for _ in range(n_requests):
        rx, ry = rng.gauss(2, 1), rng.gauss(2, 1)
        ranked = sorted(range(n_drivers),
                        key=lambda d: math.hypot(drivers[d][0] - rx, drivers[d][1] - ry))
        if policy == "push_top3":
            chosen = ranked[:3]      # broadcast to the 3 closest drivers
        else:
            best = math.hypot(drivers[ranked[0]][0] - rx, drivers[ranked[0]][1] - ry)
            # eligible: within 1 km of the best pickup distance and under the exposure cap
            band = [d for d in ranked[:15]
                    if math.hypot(drivers[d][0] - rx, drivers[d][1] - ry) <= best + 1.0
                    and recent.count(d) < cap]
            band = band or ranked[:1]
            chosen = [min(band, key=lambda d: (offers[d], rng.random()))]
        for d in chosen:
            offers[d] += 1
        recent.extend(chosen)
        recent = recent[-100:]
    total = sum(offers)
    top = sorted(offers, reverse=True)
    share10 = sum(top[:n_drivers // 10]) / total
    return total, share10, top[0], sum(1 for o in offers if o > 0)

for name, kw in [("push to 3 nearest", dict(policy="push_top3")),
                 ("1 offer, fairness band + cap 5 per 100 requests", dict(policy="band", cap=5))]:
    total, share10, mx, reached = simulate(**kw)
    print(f"{name}: offers={total} top-10%-share={share10:.0%} max-per-driver={mx} drivers-offered={reached}/200")

Output:

push to 3 nearest: offers=6000 top-10%-share=67% max-per-driver=309 drivers-offered=69/200
1 offer, fairness band + cap 5 per 100 requests: offers=2000 top-10%-share=38% max-per-driver=39 drivers-offered=81/200

Reading it: broadcasting to the three nearest sends 6,000 offers for 2,000 rides, gives two thirds of them to the top 20 drivers, and one driver is pinged 309 times. The banded, capped policy sends only 2,000 offers, a third as many, yet the top 10% share falls to 38%, the maximum per driver falls to 39, and more drivers get at least one offer (81 versus 69). That last comparison, 81 versus 69, is not like-for-like on its own, since the banded policy had a third as many total offers to spread around; normalised per offer sent, it reaches a distinct driver about 3.5 times as often as broadcasting does (81/2,000 versus 69/6,000), which is the real size of the fairness gain, not just the raw head count. The model is deliberately simple (drivers do not move, nobody accepts), so the numbers show the direction and size of the ranking effect, not a production forecast; in a real system I would measure the same three statistics from offer logs.

Trade-offs and pitfalls

• A wider fairness band improves equity but costs pickup time: 60 s of extra pickup across millions of rides is real rider cost. Tune the band against rider wait, not in isolation.
• Polling "just to be safe" at a fast rate on every driver erases push's battery and server savings; poll only when the socket is unhealthy.
• Platform push is not a delivery guarantee and may be throttled or delayed by the OS; never put the only copy of a time-critical offer there.
• Counting an offer as "sent" when it was pushed, rather than when the app acknowledged it, hides delivery failures and makes exposure metrics wrong.

What would change the design: very low driver density (few candidates, so fairness bands are nearly empty and nearest-first is fine), or markets with poor connectivity where the pull fallback timer should be shorter and offer windows longer.

Direct answer

Reconnect with capped exponential backoff with full jitter (wait a random time between zero and a limit that doubles each failure), because it spreads a crowd of clients out so they do not all hit the server at once. Always obey the server's rate-limit signal (Retry-After) as a floor. Retry aggressively only for the first couple of minutes, then decay progressively to long intervals, and stop active retrying while the app is backgrounded or the OS says there is no network, relying on platform push to wake it. Reset the backoff immediately on real events (network change, app foregrounded, push received). If the network works but WebSockets keep failing, fall back to polling and probe back periodically. Measure time-to-reconnect, attempts per successful connect, and fallback share so you know when to tell the user and when to page someone.

Key terms

• Exponential backoff: after each failure, wait longer (1 s, 2 s, 4 s, ...), capped at a maximum.
• Jitter: randomness added to the wait. Full jitter picks uniformly between zero and the backoff limit.
• Thundering herd: many clients retrying in lockstep, for example after a server deploy disconnects them all at once.
• Rate limit: the server accepts at most N new connections per second and rejects the rest (HTTP 429 during the handshake, often with a Retry-After header saying how long to wait).

1. Backoff with jitter

For the n-th consecutive failure (starting at 0), with base 1 s and cap 60 s:

• The server's Retry-After always wins if larger: the server knows its own load.
• A deploy that deliberately closes connections should send a close reason telling clients to spread their reconnect over a window (for example "reconnect within 30 s"), so even the first attempt is jittered.

2. Maximum retry windows and progressive decay

Why these particular cutoffs: most transient drops (a Wi-Fi to cellular handoff, a brief tunnel or elevator, a cell-tower handover) resolve in well under two minutes, so it is worth retrying hard during that window. A disconnection that outlasts 30 minutes is more likely a real outage, a backgrounded app, or the user out of range for a while, so retrying every 60 seconds and spending battery on mostly-failed guesses stops paying for itself; stretching the interval to 5 minutes still reconnects within a bounded time if no trigger ever fires, while costing far less battery over a long dormant stretch. These are starting points to tune from the metrics in section 4, not fixed constants.

Triggers that reset backoff to attempt 0: the OS reports a network change, the app comes to the foreground, a push notification arrives (APNs, the Apple Push Notification service, or FCM, Firebase Cloud Messaging), or the user taps retry. These are moments when success is actually likely, so they are worth a battery wake-up; a timer firing in a tunnel is not.

Battery reasoning: each attempt wakes the cellular radio, which then stays in a high-power state for a few seconds. The design minimises attempts that are likely to fail (offline, backgrounded, server shedding load) and concentrates attempts where success is likely.

3. Fallback to polling

• If attempts fail while ordinary HTTPS requests to the same host succeed (for example the handshake is rejected or the socket never delivers the first server message), the network is probably blocking WebSockets rather than being down.
• After 3 such failures, switch to long polling (repeatedly issuing a plain HTTP request that the server holds open until it has something to send, then the client immediately opens the next one) or SSE (Server-Sent Events, a one-way stream of events over a single long-lived HTTP response) for the session, and send client messages as POSTs. Carry over the same sequence-number resume logic used on WebSocket reconnect: every message the server would have pushed also carries a per-connection sequence number, so on reconnect, or on a switch to a fallback transport, the client reports the last number it saw and the server resumes from there instead of replaying everything or leaving a gap.
• Probe WebSocket again every 10 minutes in the background while in fallback, and remember the verdict per network so the next launch on that network does not pay the detection cost again.
• Do not fall back when the failure is a 429 or 503: that means the server is overloaded, and switching to polling adds more requests.

4. Metrics, and when to escalate or notify

Client-side metrics (sampled, sent when connected):

• Time from disconnect to reconnect (p50, p99: 50th and 99th percentile).
• Attempts per successful connection.
• Share of sessions in fallback mode, by network type and carrier.
• Rate-limit responses received.
• Radio wake-ups per hour attributable to reconnect logic.

Notify the user (in the foreground only):

• Show a quiet "Reconnecting..." indicator after 3 s disconnected, not immediately (most drops heal in a second or two).
• After 30 s, show "Offline. Messages will send when you're back online" and queue outgoing messages locally.
• Never show a modal error for connectivity.

Escalate to the on-call engineer (fleet-wide server alerts):

• Reconnect success rate across all clients drops below a threshold for 5 minutes.
• p99 time-to-reconnect exceeds its SLO (service-level objective).
• Fallback share jumps (a new network or proxy is blocking WebSockets).
• 429 volume stays elevated after a deploy has finished, meaning clients are not spreading out.

Worked example: why jitter matters

This runnable simulation models 100,000 clients disconnected at the same instant by a server deploy, with the server accepting at most 5,000 handshakes per second (so 20 s is the theoretical best).

import random, heapq
from collections import Counter

def simulate(strategy, clients=100_000, capacity_per_s=5_000, base=1.0, cap=60.0, seed=7):
    """All clients lose their connection at t=0 (e.g. a server deploy).
    The server accepts at most capacity_per_s handshakes in each 1-second window;
    extra attempts are rejected (HTTP 429) and the client schedules a retry."""
    rng = random.Random(seed)
    def delay(n):  # n = number of failed attempts so far
        if strategy == "fixed_1s":
            return 1.0
        exp = min(cap, base * 2 ** n)
        if strategy == "exp_no_jitter":
            return exp
        if strategy == "exp_full_jitter":
            return rng.uniform(0, exp)
    events = [(delay(0), i, 0) for i in range(clients)]  # first attempt
    heapq.heapify(events)
    per_second = Counter()
    accepted = Counter()
    attempts = 0
    connected = 0
    t_99 = None
    while events:
        t, cid, fails = heapq.heappop(events)
        sec = int(t)
        per_second[sec] += 1
        attempts += 1
        if accepted[sec] < capacity_per_s:
            accepted[sec] += 1
            connected += 1
            if t_99 is None and connected >= 0.99 * clients:
                t_99 = t
        else:
            heapq.heappush(events, (t + delay(fails + 1), cid, fails + 1))
    return max(per_second.values()), attempts, t_99

for s in ["fixed_1s", "exp_no_jitter", "exp_full_jitter"]:
    peak, attempts, t99 = simulate(s)
    print(f"{s:16s} rejected handshakes={attempts - 100_000:7d}  "
          f"attempts per client={attempts / 100_000:.2f}  99% connected at t={t99:.1f}s  "
          f"peak attempts in one second={peak}")

Output:

fixed_1s         rejected handshakes= 950000  attempts per client=10.50  99% connected at t=20.0s  peak attempts in one second=100000
exp_no_jitter    rejected handshakes= 950000  attempts per client=10.50  99% connected at t=903.0s  peak attempts in one second=100000
exp_full_jitter  rejected handshakes= 307225  attempts per client=4.07  99% connected at t=38.2s  peak attempts in one second=124571

How to read it:

• Fixed 1 s retries recover in the theoretical minimum time but make each client try 10.5 times on average: 950,000 wasted handshakes the server must reject, and 10 radio wake-ups per phone.
• Exponential backoff without jitter is the worst of both: clients stay in lockstep, so each retry round is again 95,000 clients hitting a server that admits 5,000, and the rounds drift further apart as the delay grows. It takes 903 s to reconnect 99%.
• Full jitter reconnects 99% of clients in about 38 s with about 4 attempts per client and a third of the rejected handshakes of the other two.

The printed peak column gives a sharper, slightly counter-intuitive finding: full jitter's busiest single second (124,571 attempts) is actually higher than either lockstep strategy's (100,000 each). Every strategy's entire first wave of 100,000 clients tries within about a second of the disconnect either way, spread across [0 s, 1 s) with full jitter, or bunched at exactly t = 1.0 s with no spread at all when there is no jitter, since that delay is then a fixed number, not a random one. Jitter does not shrink that unavoidable first wave. What it changes is whether the wave re-forms: the two lockstep strategies give every rejected client the exact same next delay, so the roughly 95,000 clients turned away in round one all land on the same later second and get rejected together again, rebuilding a wave close to 100,000 on every round; full jitter spreads that same 95,000 thinly enough across a widening window that no later round ever rebuilds a wave anywhere near that size, which is the real reason it reaches 99% connected in 38 s instead of 903 s. The practical conclusion still holds: the server should spread the disconnects itself during a deploy rather than rely on client backoff alone to smooth that first, unavoidable wave.

Pitfalls

• Backoff that never resets, so a user returning to good coverage waits minutes.
• Retrying while offline or backgrounded, burning battery for nothing.
• Falling back to polling when the server is overloaded, which increases load.
• Treating every close as a failure: a server-initiated close with a reason should drive behaviour (for example "go to another region" or "reconnect in 20 s").

Direct answer

All four transports answer the same question: how does the server tell the client something changed? Regular polling asks on a timer, long polling asks and lets the server hold the request open until there is news, Server-Sent Events (SSE) keeps one HTTP response open and streams events down it, and WebSockets upgrade one connection into a two-way message pipe. For (a) a live chat app I would pick WebSockets, because both sides send frequently. For (b) a server-to-many broadcast I would pick SSE, because traffic is one-way and SSE is plain HTTP with reconnection built in. Polling stays as the fallback tier, not the primary design.

The four transports in plain terms

• Regular (short) polling: the client sends GET /updates?since=... every N seconds. Most responses are empty. Simple, works everywhere, wasteful.
• Long polling: the client sends the same request, but the server does not answer until an event arrives or a timeout (say 25 seconds) passes. The client immediately re-requests after each response. Near-real-time over ordinary HTTP.
• SSE: the browser's EventSource API opens one HTTP request whose response never ends; the server writes data: ... lines as events happen. One direction only (server to client). The client sends anything upstream with normal HTTP requests.
• WebSocket: starts as an HTTP request with an Upgrade: websocket header; after the server answers 101 Switching Protocols, the same TCP connection carries framed messages in both directions at any time.

Two terms the comparison needs. Full-duplex means both sides can send at the same moment on one connection (WebSocket). Half-duplex means one direction at a time: each HTTP request/response exchange is effectively half-duplex, so polling and long polling are half-duplex, and SSE is a one-way (simplex) stream.

Comparison across the five asked axes

A keep-alive connection (in the Connection model row) is a single TCP connection reused across several sequential requests instead of opening a new one each time; it still carries one request at a time, so reusing it does not make polling any less half-duplex.

Push versus pull, and who pays for it

Polling is pull: the client decides when to ask, so cost scales with the number of clients times the poll frequency, whether or not anything happened. SSE and WebSockets are push: cost scales with connection-time and with the number of real events. Long polling sits between (push semantics, pull mechanics).

This matters for the bill and for the network:

• Per-request pricing. Many managed gateways and load balancers charge per request or per processed unit. Polling generates requests even when there is no news; push generates billable messages only for real events plus heartbeats.
• Per-connection pricing. Managed WebSocket services typically charge for connection-minutes as well as messages. For clients that are connected all day but rarely receive anything, a long poll interval can be cheaper than a held connection. Run both numbers for your own traffic shape.
• Mobile radio and battery. Every poll wakes the cellular radio, and the radio stays in a high-power state for a while after each transfer. Frequent polling from a phone drains the battery far more than the tiny payloads suggest.

Worked cost example (1 million connected clients)

Assume an empty poll costs about 800 bytes of request plus response headers (an estimate; measure your own), and a WebSocket heartbeat (a small ping message either side sends on a timer, answered with a pong, so both sides know the connection is still alive) ping/pong pair costs about 100 bytes on the wire including TCP/IP overhead (also an estimate).

• Polling every 5 s: 1,000,000 / 5 = 200,000 requests per second, mostly empty. 200,000 × 800 B = 160 MB/s, which is 160 MB/s × 86,400 s ≈ 13.8 TB per day of overhead.
• Polling every 30 s: 1,000,000 / 30 ≈ 33,333 requests per second, with an average staleness of 15 s. That is 33,333 × 800 B ≈ 26.7 MB/s, which is 26.7 MB/s × 86,400 s ≈ 2.3 TB per day.
• WebSocket with a 30 s heartbeat: 1,000,000 / 30 × 100 B ≈ 3.3 MB/s, about 288 GB per day, and events arrive within one network latency.

The push option carries roughly 48 times less idle overhead than 5 s polling (13.8 TB vs 0.288 TB), but that comparison mixes a 5 s interval against a 30 s heartbeat, so do not read 48x as the inherent push-versus-pull gap. At the same 30 s interval, push still wins by roughly 8 times (2.3 TB vs 0.288 TB), because a heartbeat (about 100 bytes) is far smaller than a full HTTP poll's headers (about 800 bytes); push's real extra advantage on top of that is not needing a short interval at all, a WebSocket does not trade staleness for cost the way polling does. The price of push is a million held connections, which you must be able to run.

Worked hybrid: an intermittently connected mobile fleet

Consider a delivery-driver app for a globally distributed, intermittently connected fleet: drivers work worldwide, often on weak cellular signal and often with the app in the background. No single transport fits, so use three tiers that all feed one sync mechanism.

1. OS push for battery-efficient wake. When the app is backgrounded, the OS suspends its sockets. Use the platform push service (Apple Push Notification service, APNs, or Firebase Cloud Messaging, FCM) to send a small "you have a new job offer" signal. The OS delivers it on a connection it already maintains, so the app pays no battery for its own connection.
2. One pooled WebSocket when foregrounded and connected. When the app is in the foreground with a usable network, open a single WebSocket shared by every feature (job offers, chat with dispatch, route changes). "Pooled" here means one multiplexed (many independent logical streams of messages sharing one physical connection) connection per app, not one per screen.
3. Adaptive-interval polling as the fallback. If the WebSocket cannot connect (a proxy strips the Upgrade, or the network flaps too fast to hold a connection), poll with an interval keyed to device and network state: for example 5 s on Wi-Fi in the foreground, 15 s on cellular, 60 s in battery-saver mode, always with random jitter (a small random offset added to each client's timer so they do not all fire at the same instant) so a fleet does not poll in lockstep.

The design move that makes this safe: every tier is only a trigger; the source of truth is one sync(since=cursor) endpoint. (A cursor is a marker, often a timestamp or sequence number, for the last update the client already has, so the server returns only what is newer.) A push notification, a WebSocket message and a poll response all cause the client to fetch everything after its last cursor. Missed or duplicated triggers then cannot lose or double-apply data, and switching tiers mid-shift is invisible to the driver.

The two recommendations

(a) Live chat: WebSockets. Users send and receive constantly (messages, typing indicators, read receipts). A full-duplex connection avoids a separate HTTP request per sent message and keeps per-message overhead to a few bytes of framing. Keep long polling or SSE-plus-POST as the fallback for networks that block the Upgrade.

(b) Server-to-many broadcasting: SSE. Traffic is one-way, so WebSocket's upstream channel buys nothing. SSE is plain HTTP, so it passes through proxies, HTTP-aware load balancers and authentication middleware unchanged; EventSource reconnects automatically and sends a Last-Event-ID header so the server can resume. Serve it over HTTP/2 so many streams share one connection. If the same product already runs a WebSocket fleet, reusing it is a reasonable choice; the fan-out architecture behind the transport (how one update is copied to many connections) matters more than the transport.

Trade-offs and pitfalls

• Choosing by latency alone. SSE and WebSockets have the same latency; the real differentiators are direction of traffic, intermediary support and operational cost.
• Forgetting that held connections are state. Push transports move cost from request handling to connection holding: file descriptors (the OS handle each open connection consumes; a machine can run out of these before it runs out of memory), memory per connection, and reconnect storms (when a server or node dies, every client it was holding tries to reconnect within the same few seconds, spiking load on whatever takes over). Plan for those.
• Polling in lockstep. A fleet that polls on the minute creates self-inflicted traffic spikes; always add jitter.
• Treating the transport as the delivery guarantee. Every transport drops messages across disconnects. Resumable cursors or sequence numbers, not the socket, are what make delivery reliable.

Direct answer

A persistent connection goes through five phases: connect and handshake, authenticate, steady state with heartbeats, failure detection, and close. The core idea is that TCP alone will not tell you a connection is dead, so both sides run an application-level heartbeat on an interval shorter than the shortest idle timer on the network path (often a 60-second load balancer timeout), and treat silence past a deadline as death. Mobile clients use longer intervals than desktop and hand off to OS push notifications when backgrounded instead of holding a socket.

Phase 1: handshake

sequenceDiagram
  participant C as Client
  participant G as Gateway
  C->>G: TCP connect, then TLS handshake
  C->>G: GET /ws with Upgrade websocket and a ticket
  G->>G: Validate ticket, check Origin
  G-->>C: 101 Switching Protocols
  C->>G: resume with last seen sequence
  G-->>C: missed messages, then live stream
  G-->>C: ping every 25 s
  C-->>G: pong
  G-->>C: close 1001 going away during drain
  C->>G: close reply, reconnect elsewhere with jitter

1. TCP connect, then TLS (Transport Layer Security, the encryption layer behind wss://).
2. HTTP Upgrade. The client sends a normal HTTP GET with Upgrade: websocket, Connection: Upgrade and a random Sec-WebSocket-Key. The server replies 101 Switching Protocols with Sec-WebSocket-Accept, a hash derived from that key, proving it speaks the protocol (RFC 6455, the WebSocket standard). From then on the same TCP connection carries frames in both directions.
3. Resume. The client's first message says where it left off (for example "last sequence 8841 in each conversation"), and the server replays what was missed before switching to live traffic. Without this step every reconnect loses messages.

Phase 2: authentication at establishment

The constraint that shapes this: the browser WebSocket API cannot set custom headers, so you cannot send Authorization: Bearer ... the way a normal fetch does. Three workable patterns:

Reject before the 101 when you can (an HTTP 401 or 403), because a rejected Upgrade costs far less than an accepted socket. Tokens also expire during a long-lived connection: either the client sends a refresh message with a new token before expiry, or the server closes with an application close code (the 4000 to 4999 range is reserved for application use) that tells the client to re-authenticate and reconnect.

Phase 3: keepalive and heartbeats

Why they are needed. Every stateful box on the path (the client's home router, a mobile carrier's network address translation, NAT, gateway, a corporate firewall, your load balancer) keeps a table entry for the connection and deletes it after a period of silence. After that, packets are dropped silently and neither end knows. Known timers:

• The NAT standard for TCP (RFC 5382) says the established-connection idle timeout must not be less than 2 hours 4 minutes, but real carrier and firewall equipment is frequently configured far shorter, in the range of minutes.
• AWS Application Load Balancer's default idle timeout is 60 seconds (configurable from 1 to 4,000 seconds). nginx's default proxy_read_timeout is also 60 seconds.
• Linux TCP keepalive starts probing only after 2 hours idle by default (net.ipv4.tcp_keepalive_time = 7200), which is far too slow to be the liveness mechanism.

So the heartbeat interval is set by the shortest idle timer on the path, with margin. With a 60 s load balancer, a 25 to 30 s heartbeat keeps the connection alive.

Server side. The protocol has control frames for this (small WebSocket frames reserved for protocol bookkeeping, separate from the message frames your application sends): ping and pong. The server sends a ping every 25 s; a compliant client answers with a pong automatically.

Client side. Browser JavaScript cannot send protocol ping frames, and it may not learn that the connection is dead until it tries to write. So the client also runs an application-level heartbeat: send a tiny {"t":"hb"} message every 25 to 30 s, and if no traffic of any kind arrives from the server within about two intervals, close and reconnect. Any real message counts as proof of life, so busy connections need no extra heartbeats.

Phase 4: detecting stale and half-open connections

A half-open connection is one where one side believes it is connected while the other side is gone (the phone lost signal, the NAT entry expired, the server crashed without sending anything). Nothing arrives to say it ended.

• Reading does not detect it. A socket waiting to read will wait forever.
• Writing detects it slowly. Unacknowledged data is retransmitted with backoff (the kernel resends the same unacknowledged segment, waiting progressively longer between each attempt); with Linux defaults this can take roughly 15 minutes before the write fails. The TCP_USER_TIMEOUT socket option (a per-socket setting that bounds how long the kernel will keep retrying before it gives up and reports the connection dead) can cap that.
• The reliable detector is the heartbeat deadline. The server records the time of the last frame received from each connection. A sweep closes any connection silent for longer than, say, 2 × interval + 10 s grace (70 s at a 30 s interval). This frees file descriptors (FDs, the limited per-process kernel handles that each open socket consumes) and memory, and tells the presence system (the subsystem that tracks who is online right now, for example the green dot next to a contact's name) the user left.

Phase 5: graceful shutdown and termination

Normal close. Either side sends a close frame with a status code (1000 means normal closure, 1001 means going away, for example a server shutting down or a page navigating off). The other side replies with its own close frame, then TCP closes. Code 1006 (abnormal closure) is never sent on the wire; it is what a client library reports locally when the connection vanished without a close frame, which is the signal to reconnect with backoff.

Draining a server for a deploy.

1. Mark the node not-ready (flip its health check so the load balancer's health probe reports it unavailable, while it keeps serving connections already open) so the load balancer sends it no new connections.
2. Close existing connections in batches spread across a drain window, not all at once. A node holding 50,000 connections drained over 5 minutes closes 50,000 / 300 s ≈ 167 per second.
3. Send each client close code 1001 (or an application "reconnect" message) so it reconnects immediately to another node, with random jitter, and resumes from its last sequence number.
4. Exit when the connection count reaches zero or the window ends.

Client reconnect policy. Exponential backoff with full jitter: wait a random time between 0 and min(cap, base × 2^attempt), for example base 1 s and cap 30 s. Traced: attempt 1 waits a random amount between 0 and 2 s (1 × 2^1); attempt 3, between 0 and 8 s (1 × 2^3); attempt 6, between 0 and 30 s, because 1 × 2^6 = 64 s already exceeds the 30 s cap. Without jitter, every client from a crashed node reconnects at the same instant.

Typical values: mobile versus desktop

These are common starting points, not standards; tune them with your own disconnect and battery telemetry.

Worked example: a phone enters a tunnel

A mobile client with a 30 s heartbeat and a 90 s dead deadline enters a tunnel at t = 0 s. At t = 12 s the carrier drops the radio link. The server's pings at t = 30 s and t = 60 s get no pong. At t = 90 s the server's sweep closes the socket and marks the user offline. At t = 100 s the train exits; the phone's network-change event fires, and the client reconnects immediately (no backoff for a network change), presents a fresh ticket and its last sequence (say 8841), receives the messages sent while it was underground, and resumes. The user sees a "reconnecting" banner and no missing messages.

Trade-offs and pitfalls

• Shorter heartbeats detect death faster but cost battery and server work: 1,000,000 connections at a 30 s interval is about 33,000 pings per second; at 10 s it is 100,000.
• Relying on TCP keepalive defaults means connections are declared dead hours late.
• Putting a long-lived token in the URL leaks it into logs.
• Closing everything at once during a deploy turns a routine release into a reconnect storm against the remaining nodes.

Direct answer

Each failed reconnect doubles the wait: delay = base_delay_ms * 2^(attempt-1), clipped at max_delay_ms. With the jitter flag on, multiply that delay by a random factor between 0.9 and 1.1, then clip again so jitter never pushes past the ceiling. Once attempt exceeds the maximum-attempts cap, return None so the caller stops and shows an error. Jitter matters because when a server dies, every client it held disconnects at the same instant; without randomness they all retry at the same instants too, and the reconnect wave (the "thundering herd") can knock the recovering server over again.

Approach

Three independent pieces, each simple:

1. Exponential growth. Attempt 1 waits base, attempt 2 waits 2 x base, attempt 3 waits 4 x base. Waiting longer after each failure gives a struggling server room to recover instead of hammering it at a constant rate.
2. Ceiling. Without a cap, attempt 20 at a 500 ms base would wait 500 x 2^19 ms, about 3 days. max_delay_ms keeps the worst case bounded (30 s is a common choice for a chat or live-updates client). The exponent is also clamped so 2 ** attempt never becomes a huge integer.
3. Jitter. Random spread added to each delay. The question asks for plus or minus 10%, so the delay becomes delay x uniform(0.9, 1.1).

Code

import random

def next_retry_delay_ms(attempt, base_delay_ms=500, max_delay_ms=30_000,
                        jitter=False, max_attempts=10, rng=random):
    """Delay before reconnect attempt number `attempt` (1-based).
    Returns None when the caller should stop retrying."""
    if attempt < 1:
        raise ValueError("attempt is 1-based")
    if attempt > max_attempts:
        return None                              # give up; surface an error to the user
    # base * 2^(attempt-1), capped. Cap the exponent too so 2**attempt never gets huge.
    exp = min(attempt - 1, 32)
    delay = min(max_delay_ms, base_delay_ms * (2 ** exp))
    if jitter:
        delay *= rng.uniform(0.9, 1.1)           # +/-10% jitter
        delay = min(delay, max_delay_ms)         # jitter never breaks the ceiling
    return int(delay)

rng = random.Random(42)                          # pinned seed so the output reproduces
print("attempt  no-jitter  jitter")
for a in range(1, 12):
    print(f"{a:>7}  {str(next_retry_delay_ms(a)):>9}  "
          f"{str(next_retry_delay_ms(a, jitter=True, rng=rng)):>6}")

Output (Python 3.14):

attempt  no-jitter  jitter
      1        500     513
      2       1000     905
      3       2000    1910
      4       4000    3778
      5       8000    8378
      6      16000   16565
      7      30000   30000
      8      30000   27521
      9      30000   29531
     10      30000   27178
     11       None    None

Attempt 7 would be 32,000 ms uncapped, so it clips to 30,000. Attempt 11 is past the cap of 10 attempts and returns None. The rng parameter exists so tests can inject a seeded generator; production code uses the default.

How a client uses it:

import time
# uses next_retry_delay_ms from the block above

def reconnect_loop(connect):
    attempt = 1
    while True:
        delay = next_retry_delay_ms(attempt, jitter=True)
        if delay is None:
            raise ConnectionError("gave up reconnecting")
        time.sleep(delay / 1000)
        if connect():                            # True once the WebSocket handshake succeeds
            return
        attempt += 1

How jitter prevents the thundering herd

Picture 100,000 clients connected to a server that crashes. All 100,000 see the close at the same moment, so all of them are on attempt 1 together, then attempt 2 together, and so on. Without jitter, every retry wave lands inside the same millisecond. Jitter spreads each wave across a window so the server sees a steady trickle instead of a spike. This simulation counts the worst single second at attempt 5 (nominal delay 8,000 ms):

import random
from collections import Counter

def peak_per_second(delays_ms):
    """Largest number of reconnects landing in any single 1-second window."""
    return max(Counter(d // 1000 for d in delays_ms).values())

rng = random.Random(7)
clients, attempt, base, cap = 100_000, 5, 500, 30_000
fixed = min(cap, base * 2 ** (attempt - 1))                       # 8000 ms for everyone
no_jitter   = [fixed] * clients
ten_percent = [int(fixed * rng.uniform(0.9, 1.1)) for _ in range(clients)]
full_jitter = [int(rng.uniform(0, fixed)) for _ in range(clients)]
print("peak reconnects in one second, 100k clients, attempt 5:")
print("  no jitter  :", peak_per_second(no_jitter))
print("  +/-10%     :", peak_per_second(ten_percent))
print("  full jitter:", peak_per_second(full_jitter))

Output:

peak reconnects in one second, 100k clients, attempt 5:
  no jitter  : 100000
  +/-10%     : 50039
  full jitter: 12632

The plus or minus 10% band spans 7,200 to 8,800 ms, only 1.6 seconds wide, so it halves the peak. "Full jitter" (pick uniformly between 0 and the nominal delay) spreads the same wave across 8 seconds and cuts the peak to about one eighth. That is the honest caveat on the spec as written: plus or minus 10% prevents exact lock-step, but on a large fleet the waves still arrive in dense bursts. If the server side of this fleet is large, I would ship full jitter (or a wider band) behind the same function signature.

Key points

• Delays grow geometrically, are bounded by a ceiling, and the loop is bounded by an attempt cap.
• Jitter is applied after the exponential step and re-clipped, so the ceiling is a hard guarantee.
• Randomness is injectable, which makes the function unit-testable with a seed.

Complexity

O(1) time and O(1) memory per call: one power, one comparison, one random draw. The exponent clamp keeps the integer small even for absurd attempt numbers.

Edge cases

• attempt < 1: rejected with ValueError instead of returning a fractional delay.
• attempt > max_attempts: returns None; the caller must treat that as "stop", not as zero.
• base_delay_ms > max_delay_ms: every delay equals the ceiling, which is a safe degenerate case.
• At the ceiling, jitter can only move the delay down (upward jitter is clipped), so capped delays land between 27,000 and 30,000 ms.
• max_attempts=0: the first call returns None, meaning "never retry".

Trade-offs and pitfalls

• Reset the counter only after a stable connection. If the socket connects and drops two seconds later (a server accepting then crashing), resetting attempt to 1 on every successful handshake recreates a tight loop. Reset after the connection has stayed up for a while, for example 30 to 60 seconds.
• Not every close deserves a retry. An authentication rejection (for example an HTTP 401 on the upgrade request, or a WebSocket close with code 1008 "policy violation") should refresh credentials or stop, not back off and retry forever.
• Let the server steer. If the server sends a close reason or a Retry-After value during overload or a deploy, honour it over the local schedule.
• Mobile clients should pause the loop when the OS reports no network and restart from attempt 1 when connectivity returns, instead of burning battery on attempts that cannot succeed.
• Giving up is a product decision. After the cap, show a "Reconnect" button rather than silently looping; a user staring at stale data they believe is live is worse than an explicit offline state.