Real-Time and Streaming System Design Questions

Direct answer

A persistent connection goes through five phases: connect and handshake, authenticate, steady state with heartbeats, failure detection, and close. The core idea is that TCP alone will not tell you a connection is dead, so both sides run an application-level heartbeat on an interval shorter than the shortest idle timer on the network path (often a 60-second load balancer timeout), and treat silence past a deadline as death. Mobile clients use longer intervals than desktop and hand off to OS push notifications when backgrounded instead of holding a socket.

Phase 1: handshake

sequenceDiagram
  participant C as Client
  participant G as Gateway
  C->>G: TCP connect, then TLS handshake
  C->>G: GET /ws with Upgrade websocket and a ticket
  G->>G: Validate ticket, check Origin
  G-->>C: 101 Switching Protocols
  C->>G: resume with last seen sequence
  G-->>C: missed messages, then live stream
  G-->>C: ping every 25 s
  C-->>G: pong
  G-->>C: close 1001 going away during drain
  C->>G: close reply, reconnect elsewhere with jitter

1. TCP connect, then TLS (Transport Layer Security, the encryption layer behind wss://).
2. HTTP Upgrade. The client sends a normal HTTP GET with Upgrade: websocket, Connection: Upgrade and a random Sec-WebSocket-Key. The server replies 101 Switching Protocols with Sec-WebSocket-Accept, a hash derived from that key, proving it speaks the protocol (RFC 6455, the WebSocket standard). From then on the same TCP connection carries frames in both directions.
3. Resume. The client's first message says where it left off (for example "last sequence 8841 in each conversation"), and the server replays what was missed before switching to live traffic. Without this step every reconnect loses messages.

Phase 2: authentication at establishment

The constraint that shapes this: the browser WebSocket API cannot set custom headers, so you cannot send Authorization: Bearer ... the way a normal fetch does. Three workable patterns:

Reject before the 101 when you can (an HTTP 401 or 403), because a rejected Upgrade costs far less than an accepted socket. Tokens also expire during a long-lived connection: either the client sends a refresh message with a new token before expiry, or the server closes with an application close code (the 4000 to 4999 range is reserved for application use) that tells the client to re-authenticate and reconnect.

Phase 3: keepalive and heartbeats

Why they are needed. Every stateful box on the path (the client's home router, a mobile carrier's network address translation, NAT, gateway, a corporate firewall, your load balancer) keeps a table entry for the connection and deletes it after a period of silence. After that, packets are dropped silently and neither end knows. Known timers:

• The NAT standard for TCP (RFC 5382) says the established-connection idle timeout must not be less than 2 hours 4 minutes, but real carrier and firewall equipment is frequently configured far shorter, in the range of minutes.
• AWS Application Load Balancer's default idle timeout is 60 seconds (configurable from 1 to 4,000 seconds). nginx's default proxy_read_timeout is also 60 seconds.
• Linux TCP keepalive starts probing only after 2 hours idle by default (net.ipv4.tcp_keepalive_time = 7200), which is far too slow to be the liveness mechanism.

So the heartbeat interval is set by the shortest idle timer on the path, with margin. With a 60 s load balancer, a 25 to 30 s heartbeat keeps the connection alive.

Server side. The protocol has control frames for this (small WebSocket frames reserved for protocol bookkeeping, separate from the message frames your application sends): ping and pong. The server sends a ping every 25 s; a compliant client answers with a pong automatically.

Client side. Browser JavaScript cannot send protocol ping frames, and it may not learn that the connection is dead until it tries to write. So the client also runs an application-level heartbeat: send a tiny {"t":"hb"} message every 25 to 30 s, and if no traffic of any kind arrives from the server within about two intervals, close and reconnect. Any real message counts as proof of life, so busy connections need no extra heartbeats.

Phase 4: detecting stale and half-open connections

A half-open connection is one where one side believes it is connected while the other side is gone (the phone lost signal, the NAT entry expired, the server crashed without sending anything). Nothing arrives to say it ended.

• Reading does not detect it. A socket waiting to read will wait forever.
• Writing detects it slowly. Unacknowledged data is retransmitted with backoff (the kernel resends the same unacknowledged segment, waiting progressively longer between each attempt); with Linux defaults this can take roughly 15 minutes before the write fails. The TCP_USER_TIMEOUT socket option (a per-socket setting that bounds how long the kernel will keep retrying before it gives up and reports the connection dead) can cap that.
• The reliable detector is the heartbeat deadline. The server records the time of the last frame received from each connection. A sweep closes any connection silent for longer than, say, 2 × interval + 10 s grace (70 s at a 30 s interval). This frees file descriptors (FDs, the limited per-process kernel handles that each open socket consumes) and memory, and tells the presence system (the subsystem that tracks who is online right now, for example the green dot next to a contact's name) the user left.

Phase 5: graceful shutdown and termination

Normal close. Either side sends a close frame with a status code (1000 means normal closure, 1001 means going away, for example a server shutting down or a page navigating off). The other side replies with its own close frame, then TCP closes. Code 1006 (abnormal closure) is never sent on the wire; it is what a client library reports locally when the connection vanished without a close frame, which is the signal to reconnect with backoff.

Draining a server for a deploy.

1. Mark the node not-ready (flip its health check so the load balancer's health probe reports it unavailable, while it keeps serving connections already open) so the load balancer sends it no new connections.
2. Close existing connections in batches spread across a drain window, not all at once. A node holding 50,000 connections drained over 5 minutes closes 50,000 / 300 s ≈ 167 per second.
3. Send each client close code 1001 (or an application "reconnect" message) so it reconnects immediately to another node, with random jitter, and resumes from its last sequence number.
4. Exit when the connection count reaches zero or the window ends.

Client reconnect policy. Exponential backoff with full jitter: wait a random time between 0 and min(cap, base × 2^attempt), for example base 1 s and cap 30 s. Traced: attempt 1 waits a random amount between 0 and 2 s (1 × 2^1); attempt 3, between 0 and 8 s (1 × 2^3); attempt 6, between 0 and 30 s, because 1 × 2^6 = 64 s already exceeds the 30 s cap. Without jitter, every client from a crashed node reconnects at the same instant.

Typical values: mobile versus desktop

These are common starting points, not standards; tune them with your own disconnect and battery telemetry.

Worked example: a phone enters a tunnel

A mobile client with a 30 s heartbeat and a 90 s dead deadline enters a tunnel at t = 0 s. At t = 12 s the carrier drops the radio link. The server's pings at t = 30 s and t = 60 s get no pong. At t = 90 s the server's sweep closes the socket and marks the user offline. At t = 100 s the train exits; the phone's network-change event fires, and the client reconnects immediately (no backoff for a network change), presents a fresh ticket and its last sequence (say 8841), receives the messages sent while it was underground, and resumes. The user sees a "reconnecting" banner and no missing messages.

Trade-offs and pitfalls

• Shorter heartbeats detect death faster but cost battery and server work: 1,000,000 connections at a 30 s interval is about 33,000 pings per second; at 10 s it is 100,000.
• Relying on TCP keepalive defaults means connections are declared dead hours late.
• Putting a long-lived token in the URL leaks it into logs.
• Closing everything at once during a deploy turns a routine release into a reconnect storm against the remaining nodes.

Direct answer

A WebSocket starts life as an ordinary HTTP/1.1 GET with an Upgrade: websocket header; if the server agrees it answers 101 Switching Protocols, and from then on the same TCP connection stops being HTTP and carries WebSocket frames in both directions for as long as it lives. A normal HTTP exchange is one request, one response, done. That upgrade is exactly what trips up intermediaries: a proxy that does not understand it strips the header, times out the "idle" connection, or buffers it. SSE avoids the problem by being a plain long HTTP response, and HTTP/2 server push is not an alternative for messaging at all. The main mitigations are always use wss:// on port 443, send heartbeats shorter than any idle timeout, configure your own proxies explicitly, and keep an SSE or long-polling fallback.

The handshake step by step

Client request:

GET /chat HTTP/1.1
Host: example.com
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==
Sec-WebSocket-Version: 13
Origin: https://example.com

Server response:

HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=

• Sec-WebSocket-Key is 16 random bytes, base64-encoded. It is not a security credential; it proves the server really speaks WebSocket (and did not just echo a cached HTTP response).
• The server computes Sec-WebSocket-Accept = base64(SHA-1(key + the fixed GUID 258EAFA5-E914-47DA-95CA-C5AB0DC85B11)). This runnable Python reproduces the value above, which is the example from RFC 6455, the WebSocket specification:

import base64, hashlib
key = "dGhlIHNhbXBsZSBub25jZQ=="
guid = "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"
print(base64.b64encode(hashlib.sha1((key + guid).encode()).digest()).decode())

s3pPLMBiTxaQ9kYGzzhZRbK+xOo=

• Origin lets the server reject pages from other sites (browsers send it; check it, because cookies are sent automatically on the handshake).
• After the 101, both sides exchange frames: small headers plus payload, with ping/pong control frames and a close handshake. Client-to-server frames are masked with a random 4-byte key, which exists to stop a malicious page from making the bytes on the wire look like a valid HTTP request to a confused cache: without masking, JavaScript on an attacker's page could choose a payload that, byte for byte, reads as a second HTTP request smuggled inside the stream, and a shared cache sitting in front of the server could mistake that smuggled request's response for real traffic and serve the attacker's chosen content to other users.

How it differs from a normal HTTP request/response: normal HTTP is request-driven and finished after each response; the connection may be reused but every exchange is independent and visible to intermediaries as requests. After a WebSocket upgrade there are no more requests, no status codes, no per-message headers; the server can send at any moment, and a proxy sees one very long opaque exchange.

Compatibility comparison

Practical mitigations

1. Use wss:// on port 443 always. Encrypted traffic on the standard HTTPS port is the most likely to pass untouched; plain ws:// on port 80 is the most likely to be mangled.
2. Heartbeats below every idle timeout. Many load balancers and proxies drop idle connections after about 60 s by default (nginx's proxy_read_timeout defaults to 60 s, for example). Send an application ping every 20 to 30 s, and raise the timeouts on infrastructure you control.
3. Configure your own proxies explicitly. In nginx, a WebSocket location needs proxy_http_version 1.1; plus proxy_set_header directives that pass the client's Upgrade header through and set Connection: upgrade, otherwise the upgrade headers are dropped. For SSE, disable response buffering (nginx honours an X-Accel-Buffering: no response header).
4. Detect and fall back. Treat the connection as working only after a first application message arrives; after repeated failures on a network, fall back to SSE plus POST, then long polling.
5. Authentication through the handshake. Browsers cannot set custom headers on the WebSocket constructor, so use a cookie (and check Origin to prevent cross-site WebSocket hijacking, where a malicious page on another site opens a socket to your server and the browser attaches the victim's cookies automatically, handing the attacker a live authenticated connection) or a short-lived single-use ticket.
6. Deploys and draining: close connections gradually with a proper close frame, and have clients reconnect with jittered backoff (wait a random, growing amount of time before retrying, instead of a fixed delay, so thousands of clients do not all retry in the same instant), so a deploy does not look like a proxy failure to thousands of clients at once.

Worked example

A customer reports the chat widget "connects then dies every minute" from their office. Server logs show connections closing with 1006 at about 60 s on that customer's IP range only. Their proxy has a 60 s idle timeout and your app pings every 90 s. Dropping the ping interval to 25 s fixes it without a fallback. A second customer's proxy strips Upgrade entirely: the handshake gets a plain 200 response, the client never receives a welcome message, and it falls back to SSE, which also turns out to be buffered, so it lands on long polling. Both are "WebSocket doesn't work" tickets with completely different fixes.

Direct answer

For a mostly one-way, high-fan-out feed I would choose Server-Sent Events (SSE), a one-way stream of text events sent over a single, long-lived, plain HTTP response, no protocol upgrade needed, for browser clients: it is plain HTTP, reconnects on its own with a resume cursor, and needs no upstream channel the use case does not use. The harder problem is not the transport but the fan-out tier: serialize each update once, push it to a small number of edge nodes, and let each node write the same bytes to its local connections with per-client backpressure. I would expose a WebSocket endpoint (a persistent, full-duplex connection upgraded from an HTTP request, so either side can send at any moment instead of only replying to requests) on the same edge nodes for native mobile apps, where WebSocket support is first-party and SSE is not.

Fan-out means delivering one published update to many subscribers. Backpressure means what the server does when a client reads more slowly than updates arrive.

Why SSE wins here, axis by axis

Two terms used below: a CDN (content delivery network) is a fleet of third-party edge servers that caches and relays HTTP traffic close to users, and TLS (Transport Layer Security) is the encryption layer behind HTTPS and wss://.

One condition to handle: on HTTP/1.1, browsers allow about 6 concurrent connections per origin, and each SSE stream holds one. Serve SSE over HTTP/2, where many streams share one connection, or users with several tabs open will see requests hang.

What would flip the choice to WebSocket everywhere: clients that send frequent messages (chat, votes, cursor positions), binary payloads, or a user base concentrated behind proxies that buffer streamed responses and cannot be fixed.

The fan-out architecture

flowchart LR
  P[Score or metrics publisher] --> B[Pub/sub bus]
  B --> E1[Edge node 1]
  B --> E2[Edge node 2]
  B --> E3[Edge node N]
  E1 --> C1[About 12.5k clients]
  E2 --> C2[About 12.5k clients]
  E3 --> C3[About 12.5k clients]
  LB[HTTP load balancer] --> E1
  LB --> E2
  LB --> E3

1. Publish once. The producer (score service, metrics aggregator) publishes each update to a pub/sub (publish/subscribe) bus, where senders publish to named topics and every current subscriber of a topic gets a copy. Redis Pub/Sub or NATS (both lightweight message brokers) work well; the topic here is something like match:1234.
2. Subscribe per node, not per client. Each edge node subscribes to a topic once, the first time any of its clients asks for it, and keeps a local in-memory list of the connections interested in that topic. The bus therefore fans out to 8 nodes, not 100,000 clients.
3. Serialize once per node. The node formats the SSE event bytes once and writes the same buffer to every local connection. Formatting per client is the most common reason a fan-out node runs out of CPU.
4. Coalesce and bound per client. Each connection gets a small bounded queue. For a score or dashboard, only the latest value matters, so when a slow client's queue is full, replace the queued update with the newest one instead of growing the queue. A client that stays behind for long (say 30 s) is disconnected and reconnects to a fresh snapshot.
5. Resume on reconnect. Every event carries an id: (a per-topic sequence number). On reconnect the browser sends Last-Event-ID; the node replays from a short in-memory ring buffer (a fixed-size buffer of the most recent events; once full, each new event overwrites the oldest) if it has those events, or sends a full snapshot if it does not.

Worked capacity example (100,000 concurrent clients)

Assumptions (state them in the room): one update per second per topic, 500 bytes per event after serialization, and each client on one topic.

• Egress: 100,000 × 500 B × 1/s = 50 MB/s = 400 Mbps total. Modest for a small fleet.
• Nodes: assume one node comfortably holds 20,000 connections (an ESTIMATE; confirm with a load test on your hardware). 100,000 / 20,000 = 5 nodes at full load. Run 8 so that losing one node leaves 7 nodes carrying 100,000 / 7 ≈ 14,300 connections each, well under the ceiling.
• Memory: if a connection costs about 30 KB (kernel socket buffers, TLS state, application buffer; an ESTIMATE to measure), 20,000 connections × 30 KB = 0.6 GB per node.
• File descriptors: the operating system's handle for each open socket; a process has a limit on how many it can hold at once. Each connection is one open file descriptor on the node. Default per-process limits are often 1,024, so raise ulimit -n (the shell and OS setting for that per-process limit) and the system-wide limit well above 20,000.
• Proxy-to-node ports: if a reverse proxy (a server sitting in front of your edge nodes that clients connect to, which opens its own separate connection onward to the real node) opens one upstream connection per client, it is limited by its ephemeral port range per destination address and port (every outbound TCP connection needs a locally unique source port for a given destination IP and port, and the OS only hands those out from a bounded range). Linux's default range (32768 to 60999) is 28,232 ports, so a proxy talking to a single node address cannot exceed that; spread across more node addresses or let clients hit nodes more directly.

Trade-offs and pitfalls

• Arguing about transports and ignoring fan-out. Both SSE and WebSocket fail the same way if each client triggers its own database query or its own serialization.
• Reconnect storms. A deploy or node crash disconnects 12,500 clients at once. Send retry: values with random jitter (for example 1 to 10 seconds) and drain nodes (stop routing new connections to a node while its existing ones keep running, instead of killing it outright) gradually during deploys.
• Unbounded per-client queues. One client on a train tunnel can grow a queue until the node runs out of memory. Always bound and coalesce.
• Silent proxy buffering. Events appear in bursts every few seconds; the cause is an intermediary buffering the stream, not your code.

Direct answer

Keep two pieces of state: the confirmed state (what the server has acknowledged) and an ordered list of pending operations (sent, not yet acknowledged). What the UI shows is always "confirmed state with every pending operation re-applied on top". Each operation gets a client-generated ID before it is sent, and every retry reuses that same ID, so the server can recognise a duplicate and not apply it twice. An ACK moves the operation into confirmed state; a NACK simply removes it from the pending list (which is the rollback); a conflict replaces confirmed state with the server's version and either re-applies the operation on top (rebase) or drops it and tells the user.

Approach

• Optimistic update means showing the result of a user action before the server confirms it, betting that the server will accept it.
• ACK / NACK: the server's positive and negative acknowledgements. A conflict is a third answer: "your change was based on an older version than mine" (a version here is a number the server increments every time it applies a change; a request carries the version the client last saw, and if that is older than the server's current version for the field being changed, someone else's change landed first).
• Idempotent means doing it twice has the same effect as doing it once. Networks lose replies, so the client must retry, and the server must make retries harmless.

Why "recompute from confirmed + pending" instead of undoing a change in place: with two or more ops in flight, undoing op 1 by hand can clobber op 2's effect. Recomputing makes rollback trivial and correct in any order: remove the op, recompute.

Code (JavaScript, runnable with Node)

// Optimistic updates over a realtime channel: a to-do list keyed by item id.
// State model: confirmed (last server-acknowledged state) + pending ops.
// The UI always shows: confirmed with every pending op re-applied on top.

let nextTemp = 0;
function newClientOpId() {
  // In a browser use crypto.randomUUID(); a counter keeps this demo deterministic.
  nextTemp += 1;
  return `op-${nextTemp}`;
}

function applyOp(state, op) {
  const s = new Map(state);
  if (op.type === 'add') s.set(op.itemId, { text: op.text, done: false });
  if (op.type === 'toggle' && s.has(op.itemId)) {
    s.set(op.itemId, { ...s.get(op.itemId), done: !s.get(op.itemId).done });
  }
  if (op.type === 'rename' && s.has(op.itemId)) {
    s.set(op.itemId, { ...s.get(op.itemId), text: op.text });
  }
  return s;
}

class OptimisticStore {
  constructor(send) {
    this.send = send;            // function(message) that writes to the channel
    this.confirmed = new Map();  // server-acknowledged state
    this.pending = [];           // ops sent (or queued) but not yet ACKed, in order
    this.view = new Map();       // what the UI renders
  }

  recompute() {
    this.view = this.pending.reduce(applyOp, this.confirmed);
  }

  // 1. apply locally  2. enqueue with a client id  3. send
  submit(op) {
    const clientOpId = newClientOpId();
    const entry = { ...op, clientOpId, attempts: 0, baseVersion: this.version ?? 0 };
    this.pending.push(entry);
    this.recompute();              // user sees the change immediately
    this.transmit(entry);
    return clientOpId;
  }

  transmit(entry) {
    entry.attempts += 1;
    // The SAME clientOpId is sent on every retry: that is what makes retries idempotent.
    this.send({ kind: 'op', clientOpId: entry.clientOpId, op: entry, baseVersion: entry.baseVersion });
  }

  // Called by the reconnect logic or a per-op timer: resend everything still unacknowledged.
  retryUnacked() {
    for (const entry of this.pending) this.transmit(entry);
  }

  // 4. handle server responses
  onMessage(msg) {
    const idx = this.pending.findIndex(p => p.clientOpId === msg.clientOpId);
    if (msg.kind === 'ack') {
      if (idx === -1) return;                       // duplicate ACK for an op already settled
      const [entry] = this.pending.splice(idx, 1);
      // Fold into confirmed. If the server rewrote the op (e.g. assigned a real id),
      // it sends the canonical op back and we apply that instead of our guess.
      this.confirmed = applyOp(this.confirmed, msg.canonicalOp ?? entry);
      this.version = msg.version;
    } else if (msg.kind === 'nack') {
      if (idx === -1) return;
      this.pending.splice(idx, 1);                  // drop it: rollback is just "stop re-applying it"
      this.lastError = `${msg.clientOpId} rejected: ${msg.reason}`;
    } else if (msg.kind === 'conflict') {
      if (idx === -1) return;
      // Server says our base was stale and sends its current authoritative state.
      this.confirmed = new Map(msg.serverState);
      this.version = msg.version;
      const [entry] = this.pending.splice(idx, 1);
      if (msg.retryable) {                          // rebase: re-apply on the fresh state and resend
        entry.baseVersion = msg.version;
        this.pending.splice(idx, 0, entry);
        this.transmit(entry);
      } else {
        this.lastError = `${msg.clientOpId} conflicted and was discarded`;
      }
    } else if (msg.kind === 'remote') {
      // Someone else's change: update confirmed, pending ops stay on top.
      this.confirmed = applyOp(this.confirmed, msg.op);
      this.version = msg.version;
    }
    this.recompute();
  }
}

// ---- A fake server, so the demo runs with no network ----
const server = { state: new Map(), version: 0, seen: new Map() };
function serverHandle(msg) {
  // Idempotency: a clientOpId already processed returns the stored reply, no second apply.
  if (server.seen.has(msg.clientOpId)) return server.seen.get(msg.clientOpId);
  let reply;
  if (msg.op.type === 'rename' && msg.op.text.length > 20) {
    reply = { kind: 'nack', clientOpId: msg.clientOpId, reason: 'text too long' };
  } else if (msg.op.type === 'rename' && msg.baseVersion < server.lastRenameVersion) {
    reply = { kind: 'conflict', clientOpId: msg.clientOpId, retryable: true,
              serverState: [...server.state], version: server.version };
    return reply;                                 // conflicts are not cached: a rebased resend must be evaluated
  } else {
    server.state = applyOp(server.state, msg.op);
    server.version += 1;
    if (msg.op.type === 'rename') server.lastRenameVersion = server.version;
    reply = { kind: 'ack', clientOpId: msg.clientOpId, version: server.version };
  }
  server.seen.set(msg.clientOpId, reply);
  return reply;
}

const outbox = [];
const store = new OptimisticStore(m => outbox.push(m));
const show = label => console.log(label.padEnd(34), JSON.stringify([...store.view]));

store.submit({ type: 'add', itemId: 'a', text: 'buy milk' });
store.submit({ type: 'toggle', itemId: 'a' });
show('after 2 local ops (no reply yet):');

// Network hiccup: the first send is delivered, its ACK is lost, and the client retries.
serverHandle(outbox[0]);                           // server applies op-1, reply lost
store.retryUnacked();                              // resends op-1 and op-2 with the same ids
console.log('outbox after retry:', outbox.length, 'sends: op-1 and op-2 each appear twice (original entry plus retry copy)');
for (const m of outbox.splice(0)) store.onMessage(serverHandle(m));
console.log('server applied op-1 once? version =', server.version, '(expected 2)');
show('after ACKs:');

store.submit({ type: 'rename', itemId: 'a', text: 'buy oat milk and bread, 2 loaves' });
show('optimistic long rename:');
for (const m of outbox.splice(0)) store.onMessage(serverHandle(m));
show('after NACK (rolled back):');
console.log('error shown to user:', store.lastError);

// Conflict: another user renames 'a' first; our rename was based on an older version.
store.submit({ type: 'rename', itemId: 'a', text: 'milk x2' });
serverHandle({ kind: 'op', clientOpId: 'other-user-1', op: { type: 'rename', itemId: 'a', text: 'milk (whole)' }, baseVersion: server.version });
store.onMessage({ kind: 'remote', op: { type: 'rename', itemId: 'a', text: 'milk (whole)' }, version: server.version });
show('remote change arrives, ours on top:');
for (const m of outbox.splice(0)) store.onMessage(serverHandle(m));   // conflict -> rebase -> resend
for (const m of outbox.splice(0)) store.onMessage(serverHandle(m));   // resend is accepted
show('after conflict + rebase + ACK:');
console.log('pending ops left:', store.pending.length, '| server:', JSON.stringify([...server.state]));

Output of node optimistic.js:

after 2 local ops (no reply yet):  [["a",{"text":"buy milk","done":true}]]
outbox after retry: 4 sends: op-1 and op-2 each appear twice (original entry plus retry copy)
server applied op-1 once? version = 2 (expected 2)
after ACKs:                        [["a",{"text":"buy milk","done":true}]]
optimistic long rename:            [["a",{"text":"buy oat milk and bread, 2 loaves","done":true}]]
after NACK (rolled back):          [["a",{"text":"buy milk","done":true}]]
error shown to user: op-3 rejected: text too long
remote change arrives, ours on top: [["a",{"text":"milk x2","done":true}]]
after conflict + rebase + ACK:     [["a",{"text":"milk x2","done":true}]]
pending ops left: 0 | server: [["a",{"text":"milk x2","done":true}]]

What the run shows:

• Retry without double-apply. op-1's message reaches serverHandle three times in total: once from the direct serverHandle(outbox[0]) call that simulates delivery with a lost reply, then twice more when the batch loop replays both the still-queued original entry and its retry copy (neither was ever removed from outbox). op-2 reaches it twice: its original and its retry, both replayed for the first time in that same batch loop. Yet server.version ends at 2: each op is applied exactly once (server.seen catches every later call for the same clientOpId and returns the cached reply instead of re-applying), and on the client side pending only lets the very first ack through for each op, ignoring the rest because the op is no longer pending by then.
• Rollback on NACK. The long rename displayed instantly, the server rejected it, and the view went back to "buy milk" without any hand-written undo code.
• Conflict and rebase. Another user's rename landed first. The client took the server's state, re-applied its own rename on top, re-sent it with the newer base version, and ended with nothing pending and client and server agreeing.

Key points

1. Apply locally: submit() pushes the op onto pending and recomputes the view before any network I/O.
2. Enqueue with a temporary ID: clientOpId is created on the client. In a browser use crypto.randomUUID() (the demo uses a counter only to make output deterministic). If the op creates an entity, the client can also mint a temporary entity id; the ACK then carries the server's canonical op (canonicalOp) with the real id, and the client applies that instead of its guess.
3. Send: transmit() always sends the same clientOpId for the same op.
4. Handle responses: ACK folds the op into confirmed; NACK drops it and surfaces an error; conflict resets confirmed to the server's state and rebases or discards; a remote message (someone else's change) updates confirmed while local pending ops stay on top.
5. Reconcile: after every message, recompute() rebuilds the view, so the screen is always consistent with what the client knows.

Idempotency considerations for retries

• The key is the op, not the attempt. Generate clientOpId once per user action and persist it with the pending op (for example in IndexedDB) so a page reload or reconnect retries with the same id instead of minting a new one.
• The server stores the outcome, not just "seen". The fake server caches the full reply per clientOpId and returns it for duplicates, so a retry after a lost ACK gets the same ACK (and the same assigned version) rather than an error.
• Do not cache conflicts. A conflict reply depends on the state at that moment; the rebased resend must be evaluated fresh. In the demo, conflicts return before the cache write.
• Bound the dedup memory. Keep ids for longer than the client's maximum retry window (for example 24 hours) and then expire them; key them per user or per document so one client cannot evict another's entries.
• Order matters for dependent ops. "Add item" then "toggle item" must reach the server in order. Resending the whole pending list in order on reconnect (as retryUnacked() does) preserves that; the server can additionally reject an op whose target does not exist yet, and the client retries it after the earlier op is ACKed.

Complexity

• submit and every response do one recompute(), which is O(P x cost of applyOp) where P is the number of pending ops. P is normally 0 to a handful, so this is cheap; if P could grow large (long offline editing), snapshot the view and replay only ops after the changed one.
• Finding the op for a reply is O(P) with findIndex; a Map from clientOpId to index makes it O(1) if P is large.

Edge cases

• Duplicate ACK or late ACK for an already-settled op: ignored (idx === -1).
• NACK for an op that later ops depend on (rename of an item whose add was rejected): recompute naturally drops the effect of the dependent op because applyOp ignores ops whose target is missing; the dependent op will itself be NACKed by the server.
• Reconnect: resend all pending ops in order with their original ids; the server's dedup makes this safe.
• Rebase is last-writer-wins for the field. In the demo the user's rename overwrote the other user's. For content where that loses work (document text), show the conflict to the user or use a merge strategy instead of silently rebasing.
• Never-answered ops: a per-op timer should trigger retryUnacked(), and after a few failures the UI should mark the item as "not saved" rather than showing it as saved forever.

Direct answer

Each failed reconnect doubles the wait: delay = base_delay_ms * 2^(attempt-1), clipped at max_delay_ms. With the jitter flag on, multiply that delay by a random factor between 0.9 and 1.1, then clip again so jitter never pushes past the ceiling. Once attempt exceeds the maximum-attempts cap, return None so the caller stops and shows an error. Jitter matters because when a server dies, every client it held disconnects at the same instant; without randomness they all retry at the same instants too, and the reconnect wave (the "thundering herd") can knock the recovering server over again.

Approach

Three independent pieces, each simple:

1. Exponential growth. Attempt 1 waits base, attempt 2 waits 2 x base, attempt 3 waits 4 x base. Waiting longer after each failure gives a struggling server room to recover instead of hammering it at a constant rate.
2. Ceiling. Without a cap, attempt 20 at a 500 ms base would wait 500 x 2^19 ms, about 3 days. max_delay_ms keeps the worst case bounded (30 s is a common choice for a chat or live-updates client). The exponent is also clamped so 2 ** attempt never becomes a huge integer.
3. Jitter. Random spread added to each delay. The question asks for plus or minus 10%, so the delay becomes delay x uniform(0.9, 1.1).

Code

import random

def next_retry_delay_ms(attempt, base_delay_ms=500, max_delay_ms=30_000,
                        jitter=False, max_attempts=10, rng=random):
    """Delay before reconnect attempt number `attempt` (1-based).
    Returns None when the caller should stop retrying."""
    if attempt < 1:
        raise ValueError("attempt is 1-based")
    if attempt > max_attempts:
        return None                              # give up; surface an error to the user
    # base * 2^(attempt-1), capped. Cap the exponent too so 2**attempt never gets huge.
    exp = min(attempt - 1, 32)
    delay = min(max_delay_ms, base_delay_ms * (2 ** exp))
    if jitter:
        delay *= rng.uniform(0.9, 1.1)           # +/-10% jitter
        delay = min(delay, max_delay_ms)         # jitter never breaks the ceiling
    return int(delay)

rng = random.Random(42)                          # pinned seed so the output reproduces
print("attempt  no-jitter  jitter")
for a in range(1, 12):
    print(f"{a:>7}  {str(next_retry_delay_ms(a)):>9}  "
          f"{str(next_retry_delay_ms(a, jitter=True, rng=rng)):>6}")

Output (Python 3.14):

attempt  no-jitter  jitter
      1        500     513
      2       1000     905
      3       2000    1910
      4       4000    3778
      5       8000    8378
      6      16000   16565
      7      30000   30000
      8      30000   27521
      9      30000   29531
     10      30000   27178
     11       None    None

Attempt 7 would be 32,000 ms uncapped, so it clips to 30,000. Attempt 11 is past the cap of 10 attempts and returns None. The rng parameter exists so tests can inject a seeded generator; production code uses the default.

How a client uses it:

import time
# uses next_retry_delay_ms from the block above

def reconnect_loop(connect):
    attempt = 1
    while True:
        delay = next_retry_delay_ms(attempt, jitter=True)
        if delay is None:
            raise ConnectionError("gave up reconnecting")
        time.sleep(delay / 1000)
        if connect():                            # True once the WebSocket handshake succeeds
            return
        attempt += 1

How jitter prevents the thundering herd

Picture 100,000 clients connected to a server that crashes. All 100,000 see the close at the same moment, so all of them are on attempt 1 together, then attempt 2 together, and so on. Without jitter, every retry wave lands inside the same millisecond. Jitter spreads each wave across a window so the server sees a steady trickle instead of a spike. This simulation counts the worst single second at attempt 5 (nominal delay 8,000 ms):

import random
from collections import Counter

def peak_per_second(delays_ms):
    """Largest number of reconnects landing in any single 1-second window."""
    return max(Counter(d // 1000 for d in delays_ms).values())

rng = random.Random(7)
clients, attempt, base, cap = 100_000, 5, 500, 30_000
fixed = min(cap, base * 2 ** (attempt - 1))                       # 8000 ms for everyone
no_jitter   = [fixed] * clients
ten_percent = [int(fixed * rng.uniform(0.9, 1.1)) for _ in range(clients)]
full_jitter = [int(rng.uniform(0, fixed)) for _ in range(clients)]
print("peak reconnects in one second, 100k clients, attempt 5:")
print("  no jitter  :", peak_per_second(no_jitter))
print("  +/-10%     :", peak_per_second(ten_percent))
print("  full jitter:", peak_per_second(full_jitter))

Output:

peak reconnects in one second, 100k clients, attempt 5:
  no jitter  : 100000
  +/-10%     : 50039
  full jitter: 12632

The plus or minus 10% band spans 7,200 to 8,800 ms, only 1.6 seconds wide, so it halves the peak. "Full jitter" (pick uniformly between 0 and the nominal delay) spreads the same wave across 8 seconds and cuts the peak to about one eighth. That is the honest caveat on the spec as written: plus or minus 10% prevents exact lock-step, but on a large fleet the waves still arrive in dense bursts. If the server side of this fleet is large, I would ship full jitter (or a wider band) behind the same function signature.

Key points

• Delays grow geometrically, are bounded by a ceiling, and the loop is bounded by an attempt cap.
• Jitter is applied after the exponential step and re-clipped, so the ceiling is a hard guarantee.
• Randomness is injectable, which makes the function unit-testable with a seed.

Complexity

O(1) time and O(1) memory per call: one power, one comparison, one random draw. The exponent clamp keeps the integer small even for absurd attempt numbers.

Edge cases

• attempt < 1: rejected with ValueError instead of returning a fractional delay.
• attempt > max_attempts: returns None; the caller must treat that as "stop", not as zero.
• base_delay_ms > max_delay_ms: every delay equals the ceiling, which is a safe degenerate case.
• At the ceiling, jitter can only move the delay down (upward jitter is clipped), so capped delays land between 27,000 and 30,000 ms.
• max_attempts=0: the first call returns None, meaning "never retry".

Trade-offs and pitfalls

• Reset the counter only after a stable connection. If the socket connects and drops two seconds later (a server accepting then crashing), resetting attempt to 1 on every successful handshake recreates a tight loop. Reset after the connection has stayed up for a while, for example 30 to 60 seconds.
• Not every close deserves a retry. An authentication rejection (for example an HTTP 401 on the upgrade request, or a WebSocket close with code 1008 "policy violation") should refresh credentials or stop, not back off and retry forever.
• Let the server steer. If the server sends a close reason or a Retry-After value during overload or a deploy, honour it over the local schedule.
• Mobile clients should pause the loop when the OS reports no network and restart from attempt 1 when connectivity returns, instead of burning battery on attempts that cannot succeed.
• Giving up is a product decision. After the cap, show a "Reconnect" button rather than silently looping; a user staring at stale data they believe is live is worse than an explicit offline state.