Real-Time and Streaming System Design Questions
Designing client-facing, always-on live delivery systems: real-time communication transports (WebSockets, server-sent events, long-polling, WebRTC, MQTT), connection lifecycle and scaling for millions of persistent connections, presence, pub/sub fan-out to online users, chat and notifications, live feeds and tickers, real-time collaboration (CRDT vs OT, offline sync and reconciliation), and live and on-demand video delivery (ingest, transcoding, adaptive bitrate, CDN delivery, low-latency protocols, playback entitlement and content protection). Covers latency budgets, per-client ordering, delivery guarantees across disconnect and reconnect, per-client backpressure, capacity estimation for connection fleets, authentication, authorization, and revocation for real-time channels and premium content, and operational readiness (SLOs and error budgets, observability and incident response, rate limiting, safe rollout, and load and chaos testing) for live-delivery platforms. Data-pipeline stream processing (Kafka or Flink jobs, windowed aggregation, exactly-once pipelines, real-time analytics ingestion) is out of scope.
Write pseudocode (or JavaScript) for a client-side optimistic UI update when sending an update over a realtime channel. The function should: apply change locally, enqueue the outbound operation with a temporary client-generated ID, send the update to the server, handle ACK/NACK/conflict responses, and roll back or reconcile the local state accordingly. Explain idempotency considerations for retries.
Sample Answer
Direct answer
Keep two pieces of state: the confirmed state (what the server has acknowledged) and an ordered list of pending operations (sent, not yet acknowledged). What the UI shows is always "confirmed state with every pending operation re-applied on top". Each operation gets a client-generated ID before it is sent, and every retry reuses that same ID, so the server can recognise a duplicate and not apply it twice. An ACK moves the operation into confirmed state; a NACK simply removes it from the pending list (which is the rollback); a conflict replaces confirmed state with the server's version and either re-applies the operation on top (rebase) or drops it and tells the user.
Approach
- Optimistic update means showing the result of a user action before the server confirms it, betting that the server will accept it.
- ACK / NACK: the server's positive and negative acknowledgements. A conflict is a third answer: "your change was based on an older version than mine" (a version here is a number the server increments every time it applies a change; a request carries the version the client last saw, and if that is older than the server's current version for the field being changed, someone else's change landed first).
- Idempotent means doing it twice has the same effect as doing it once. Networks lose replies, so the client must retry, and the server must make retries harmless.
Why "recompute from confirmed + pending" instead of undoing a change in place: with two or more ops in flight, undoing op 1 by hand can clobber op 2's effect. Recomputing makes rollback trivial and correct in any order: remove the op, recompute.
Code (JavaScript, runnable with Node)
// Optimistic updates over a realtime channel: a to-do list keyed by item id.
// State model: confirmed (last server-acknowledged state) + pending ops.
// The UI always shows: confirmed with every pending op re-applied on top.
let nextTemp = 0;
function newClientOpId() {
// In a browser use crypto.randomUUID(); a counter keeps this demo deterministic.
nextTemp += 1;
return `op-${nextTemp}`;
}
function applyOp(state, op) {
const s = new Map(state);
if (op.type === 'add') s.set(op.itemId, { text: op.text, done: false });
if (op.type === 'toggle' && s.has(op.itemId)) {
s.set(op.itemId, { ...s.get(op.itemId), done: !s.get(op.itemId).done });
}
if (op.type === 'rename' && s.has(op.itemId)) {
s.set(op.itemId, { ...s.get(op.itemId), text: op.text });
}
return s;
}
class OptimisticStore {
constructor(send) {
this.send = send; // function(message) that writes to the channel
this.confirmed = new Map(); // server-acknowledged state
this.pending = []; // ops sent (or queued) but not yet ACKed, in order
this.view = new Map(); // what the UI renders
}
recompute() {
this.view = this.pending.reduce(applyOp, this.confirmed);
}
// 1. apply locally 2. enqueue with a client id 3. send
submit(op) {
const clientOpId = newClientOpId();
const entry = { ...op, clientOpId, attempts: 0, baseVersion: this.version ?? 0 };
this.pending.push(entry);
this.recompute(); // user sees the change immediately
this.transmit(entry);
return clientOpId;
}
transmit(entry) {
entry.attempts += 1;
// The SAME clientOpId is sent on every retry: that is what makes retries idempotent.
this.send({ kind: 'op', clientOpId: entry.clientOpId, op: entry, baseVersion: entry.baseVersion });
}
// Called by the reconnect logic or a per-op timer: resend everything still unacknowledged.
retryUnacked() {
for (const entry of this.pending) this.transmit(entry);
}
// 4. handle server responses
onMessage(msg) {
const idx = this.pending.findIndex(p => p.clientOpId === msg.clientOpId);
if (msg.kind === 'ack') {
if (idx === -1) return; // duplicate ACK for an op already settled
const [entry] = this.pending.splice(idx, 1);
// Fold into confirmed. If the server rewrote the op (e.g. assigned a real id),
// it sends the canonical op back and we apply that instead of our guess.
this.confirmed = applyOp(this.confirmed, msg.canonicalOp ?? entry);
this.version = msg.version;
} else if (msg.kind === 'nack') {
if (idx === -1) return;
this.pending.splice(idx, 1); // drop it: rollback is just "stop re-applying it"
this.lastError = `${msg.clientOpId} rejected: ${msg.reason}`;
} else if (msg.kind === 'conflict') {
if (idx === -1) return;
// Server says our base was stale and sends its current authoritative state.
this.confirmed = new Map(msg.serverState);
this.version = msg.version;
const [entry] = this.pending.splice(idx, 1);
if (msg.retryable) { // rebase: re-apply on the fresh state and resend
entry.baseVersion = msg.version;
this.pending.splice(idx, 0, entry);
this.transmit(entry);
} else {
this.lastError = `${msg.clientOpId} conflicted and was discarded`;
}
} else if (msg.kind === 'remote') {
// Someone else's change: update confirmed, pending ops stay on top.
this.confirmed = applyOp(this.confirmed, msg.op);
this.version = msg.version;
}
this.recompute();
}
}
// ---- A fake server, so the demo runs with no network ----
const server = { state: new Map(), version: 0, seen: new Map() };
function serverHandle(msg) {
// Idempotency: a clientOpId already processed returns the stored reply, no second apply.
if (server.seen.has(msg.clientOpId)) return server.seen.get(msg.clientOpId);
let reply;
if (msg.op.type === 'rename' && msg.op.text.length > 20) {
reply = { kind: 'nack', clientOpId: msg.clientOpId, reason: 'text too long' };
} else if (msg.op.type === 'rename' && msg.baseVersion < server.lastRenameVersion) {
reply = { kind: 'conflict', clientOpId: msg.clientOpId, retryable: true,
serverState: [...server.state], version: server.version };
return reply; // conflicts are not cached: a rebased resend must be evaluated
} else {
server.state = applyOp(server.state, msg.op);
server.version += 1;
if (msg.op.type === 'rename') server.lastRenameVersion = server.version;
reply = { kind: 'ack', clientOpId: msg.clientOpId, version: server.version };
}
server.seen.set(msg.clientOpId, reply);
return reply;
}
const outbox = [];
const store = new OptimisticStore(m => outbox.push(m));
const show = label => console.log(label.padEnd(34), JSON.stringify([...store.view]));
store.submit({ type: 'add', itemId: 'a', text: 'buy milk' });
store.submit({ type: 'toggle', itemId: 'a' });
show('after 2 local ops (no reply yet):');
// Network hiccup: the first send is delivered, its ACK is lost, and the client retries.
serverHandle(outbox[0]); // server applies op-1, reply lost
store.retryUnacked(); // resends op-1 and op-2 with the same ids
console.log('outbox after retry:', outbox.length, 'sends: op-1 and op-2 each appear twice (original entry plus retry copy)');
for (const m of outbox.splice(0)) store.onMessage(serverHandle(m));
console.log('server applied op-1 once? version =', server.version, '(expected 2)');
show('after ACKs:');
store.submit({ type: 'rename', itemId: 'a', text: 'buy oat milk and bread, 2 loaves' });
show('optimistic long rename:');
for (const m of outbox.splice(0)) store.onMessage(serverHandle(m));
show('after NACK (rolled back):');
console.log('error shown to user:', store.lastError);
// Conflict: another user renames 'a' first; our rename was based on an older version.
store.submit({ type: 'rename', itemId: 'a', text: 'milk x2' });
serverHandle({ kind: 'op', clientOpId: 'other-user-1', op: { type: 'rename', itemId: 'a', text: 'milk (whole)' }, baseVersion: server.version });
store.onMessage({ kind: 'remote', op: { type: 'rename', itemId: 'a', text: 'milk (whole)' }, version: server.version });
show('remote change arrives, ours on top:');
for (const m of outbox.splice(0)) store.onMessage(serverHandle(m)); // conflict -> rebase -> resend
for (const m of outbox.splice(0)) store.onMessage(serverHandle(m)); // resend is accepted
show('after conflict + rebase + ACK:');
console.log('pending ops left:', store.pending.length, '| server:', JSON.stringify([...server.state]));
Output of node optimistic.js:
after 2 local ops (no reply yet): [["a",{"text":"buy milk","done":true}]]
outbox after retry: 4 sends: op-1 and op-2 each appear twice (original entry plus retry copy)
server applied op-1 once? version = 2 (expected 2)
after ACKs: [["a",{"text":"buy milk","done":true}]]
optimistic long rename: [["a",{"text":"buy oat milk and bread, 2 loaves","done":true}]]
after NACK (rolled back): [["a",{"text":"buy milk","done":true}]]
error shown to user: op-3 rejected: text too long
remote change arrives, ours on top: [["a",{"text":"milk x2","done":true}]]
after conflict + rebase + ACK: [["a",{"text":"milk x2","done":true}]]
pending ops left: 0 | server: [["a",{"text":"milk x2","done":true}]]
What the run shows:
- Retry without double-apply.
op-1's message reachesserverHandlethree times in total: once from the directserverHandle(outbox[0])call that simulates delivery with a lost reply, then twice more when the batch loop replays both the still-queued original entry and its retry copy (neither was ever removed fromoutbox).op-2reaches it twice: its original and its retry, both replayed for the first time in that same batch loop. Yetserver.versionends at 2: each op is applied exactly once (server.seencatches every later call for the sameclientOpIdand returns the cached reply instead of re-applying), and on the client sidependingonly lets the very first ack through for each op, ignoring the rest because the op is no longer pending by then. - Rollback on NACK. The long rename displayed instantly, the server rejected it, and the view went back to "buy milk" without any hand-written undo code.
- Conflict and rebase. Another user's rename landed first. The client took the server's state, re-applied its own rename on top, re-sent it with the newer base version, and ended with nothing pending and client and server agreeing.
Key points
- Apply locally:
submit()pushes the op ontopendingand recomputes the view before any network I/O. - Enqueue with a temporary ID:
clientOpIdis created on the client. In a browser usecrypto.randomUUID()(the demo uses a counter only to make output deterministic). If the op creates an entity, the client can also mint a temporary entity id; the ACK then carries the server's canonical op (canonicalOp) with the real id, and the client applies that instead of its guess. - Send:
transmit()always sends the sameclientOpIdfor the same op. - Handle responses: ACK folds the op into
confirmed; NACK drops it and surfaces an error; conflict resetsconfirmedto the server's state and rebases or discards; aremotemessage (someone else's change) updatesconfirmedwhile local pending ops stay on top. - Reconcile: after every message,
recompute()rebuilds the view, so the screen is always consistent with what the client knows.
Idempotency considerations for retries
- The key is the op, not the attempt. Generate
clientOpIdonce per user action and persist it with the pending op (for example in IndexedDB) so a page reload or reconnect retries with the same id instead of minting a new one. - The server stores the outcome, not just "seen". The fake server caches the full reply per
clientOpIdand returns it for duplicates, so a retry after a lost ACK gets the same ACK (and the same assigned version) rather than an error. - Do not cache conflicts. A conflict reply depends on the state at that moment; the rebased resend must be evaluated fresh. In the demo, conflicts return before the cache write.
- Bound the dedup memory. Keep ids for longer than the client's maximum retry window (for example 24 hours) and then expire them; key them per user or per document so one client cannot evict another's entries.
- Order matters for dependent ops. "Add item" then "toggle item" must reach the server in order. Resending the whole pending list in order on reconnect (as
retryUnacked()does) preserves that; the server can additionally reject an op whose target does not exist yet, and the client retries it after the earlier op is ACKed.
Complexity
submitand every response do onerecompute(), which is O(P x cost of applyOp) where P is the number of pending ops. P is normally 0 to a handful, so this is cheap; if P could grow large (long offline editing), snapshot the view and replay only ops after the changed one.- Finding the op for a reply is O(P) with
findIndex; aMapfromclientOpIdto index makes it O(1) if P is large.
Edge cases
- Duplicate ACK or late ACK for an already-settled op: ignored (
idx === -1). - NACK for an op that later ops depend on (rename of an item whose add was rejected): recompute naturally drops the effect of the dependent op because
applyOpignores ops whose target is missing; the dependent op will itself be NACKed by the server. - Reconnect: resend all pending ops in order with their original ids; the server's dedup makes this safe.
- Rebase is last-writer-wins for the field. In the demo the user's rename overwrote the other user's. For content where that loses work (document text), show the conflict to the user or use a merge strategy instead of silently rebasing.
- Never-answered ops: a per-op timer should trigger
retryUnacked(), and after a few failures the UI should mark the item as "not saved" rather than showing it as saved forever.
Design a comments feature for collaborative document editing (comment threads anchored to document ranges). Clarify requirements with product (real-time vs. non-real-time, access control, edit history), state your assumptions, propose two different approaches for keeping a comment thread in sync in real time plus one for offline-first comments, discuss the trade-offs in consistency, latency, and usability, and outline how you'd communicate the design to product and QA.
Sample Answer
Direct answer
The hard part of comments is not the thread, it is the anchor: a comment points at "these words", and those words move, change, or disappear as people edit. So I would first pin down requirements with product, then keep comment threads as ordinary server-owned records pushed in real time, and store each comment's anchor as a position inside the document's own collaborative model (a pointer to specific characters that moves with them) rather than as a character offset. For real-time sync I would compare (A) a server-authoritative comment service with pub/sub push (publish/subscribe: publishers send events to a channel and every subscriber to that channel receives them, without publisher and subscriber knowing about each other directly) and (B) comments stored inside the document's replicated data model, pick A for most products, and support offline-first comments with a local outbox and client-generated IDs.
Clarifying requirements with product
| Question | Why it changes the design |
|---|---|
| Must comments appear live for co-editors, or is "on refresh" acceptable? | Live needs a push channel; non-live is a plain REST feature |
| Who can see, add, resolve, delete comments? Can a viewer comment? | Access checks per operation; commenter role separate from editor |
| Do we need edit history of comments, and "resolved" threads? | Store comment versions; a resolved state separate from deletion |
| What happens when the commented text is deleted? | Orphaned-comment behaviour must be a product decision, not an accident |
| Mentions and notifications? | Integrates with the notification service |
| Scale: comments per document, concurrent viewers? | Whether to paginate threads and how to fan out updates |
Assumptions I would state
- Comments must be live for anyone with the document open (within about 1 second).
- Roles: owner, editor, commenter, viewer. Commenters and above can comment; authors can edit their own comments; editors can resolve.
- Edit history per comment is kept; deletion is a soft delete visible as "comment deleted".
- Typical document: up to 500 threads, up to 50 concurrent viewers.
- The document editor already syncs text in real time with a collaborative model (a sequence CRDT, conflict-free replicated data type, or OT, operational transformation, with a central server).
The anchor problem, concretely
User A comments on "quarterly revenue" at characters 120 to 137. User B inserts 30 characters at position 50. If the anchor is stored as offsets 120..137, it now points at the wrong words. The fix is to anchor to stable identities: a CRDT gives every character a permanent ID, and an OT system can transform stored ranges through each operation, just as it transforms cursors. Store {start_id, end_id} (or a range the server keeps transformed), plus a copy of the quoted text for display if the range is later deleted.
Approach A: server-authoritative comment service with pub/sub push
- Comments are rows in a comments table:
thread_id, comment_id, doc_id, anchor, author, body, version, created_at, resolved. - Create and edit go through a REST (or RPC, remote procedure call: calling a function on another server as if it were a local one) API that checks permissions, writes the row, then publishes
comment.createdon the document's channel. - Everyone with the document open is subscribed to that channel over the WebSocket (a persistent, two-way connection between browser and server that lets the server push events without the client asking again) they already have for editing, and receives the event within the push latency.
- Ordering within a thread is by server-assigned sequence per thread, so every client shows the same order.
- Concurrent edits to the same comment use optimistic concurrency (proceed as if there is no conflict, but detect and reject the write if one occurred instead of locking up front): the client sends the
versionit edited; if the server has moved on, it returns a conflict and the client shows "this comment changed, review before saving".
Approach B: comments inside the document's collaborative model
- Threads are stored as part of the replicated document state (for example a map of thread IDs to comment lists inside the same CRDT), and sync over exactly the same channel and merge logic as text.
- Anchors and comments share one consistency model: a thread and the text it points at can never disagree about ordering, and undo and version history of the document include comments for free.
- But: permission checks become harder (the document model usually lets any editor change anything, and here a commenter must be able to add comments without editing text), notifications and search need to extract comments from the model, and the document grows with every comment ever written.
Offline-first comments
- The client creates the comment with a client-generated UUID (universally unique ID) and stores it in a local outbox (IndexedDB, the browser's local database) along with the anchor as character IDs and the quoted text.
- On reconnect, it sends each outbox item; the server treats the UUID as an idempotency key (a request identifier that makes a retry safe, because a repeat of the same key is recognised and not applied twice).
- The server then re-resolves the anchor against the current document: if the anchored characters still exist, the comment attaches normally; if they were deleted while the user was offline, the comment is saved as detached and shown with its quoted text in the sidebar.
- If the user lost permission while offline, the server rejects with a clear error and the client keeps the text so the user can copy it.
stateDiagram-v2
[*] --> Draft
Draft --> Pending: saved offline
Draft --> Open: saved online
Pending --> Open: synced, anchor found
Pending --> Detached: synced, anchored text deleted
Pending --> Rejected: synced, no permission
Open --> Detached: anchored text deleted later
Open --> Resolved: editor resolves
Resolved --> Open: reopened
Trade-offs and my choice
| Dimension | A: comment service + push | B: inside the document model |
|---|---|---|
| Consistency | Server order per thread; comment and text are separate, so a comment can briefly point at text a user has not received yet | One model, always consistent with the text |
| Latency | One API write plus push, about 100 to 300 ms to other viewers (estimate) | Same as a text edit |
| Access control | Natural: checked per API call | Awkward: the document model must enforce commenter-only rights |
| Usability features (mentions, notifications, search, email replies) | Easy: comments are rows | Harder: must be extracted from the model |
| Offline | Outbox plus idempotent upsert (an insert-or-update write, safe to repeat because retrying it does not create a duplicate) | Native: merges like any offline edit |
| Document size | Unaffected | Grows with comment history |
I would choose A, with anchors stored as CRDT character IDs (or OT-transformed ranges) so they move with the text. It keeps access control and integrations simple, and comments are not keystroke-rate data that needs the document's merge machinery. I would switch to B if the product is offline-first as a core promise (for example a field app used mostly without signal), because then native merging outweighs the permission complexity.
Worked example: fan-out load
Fan-out is copying one event out to every subscriber who should receive it. A popular document with 50 viewers and a burst of 20 comments in a minute produces 20 x 50 = 1,000 push events in that minute, about 17 per second: trivial for a WebSocket gateway. Load becomes interesting only for large all-hands documents (for example 2,000 viewers), where 20 comments per minute is 40,000 pushes per minute. There I would send a small "thread X changed" event and let clients fetch the thread lazily only if it is on screen.
Communicating the design to product and QA
- To product: a one-page decision record: the requirements as understood, the assumptions (numbered, so they can be confirmed or corrected one by one), the chosen approach and the one it was chosen over, and the visible behaviours they must sign off, especially the detached comment and conflicting edit experiences, with mock-ups.
- To QA: the state diagram above plus a test matrix derived from it: comment on text another user deletes at the same time; comment offline, text deleted before reconnect; two users edit the same comment; permission revoked while offline; the same offline comment sent twice (retry); 500-thread document load time; reconnect after a server restart. Each row names the expected state and message.
- Agree on acceptance criteria as measurable statements: "a comment appears to other viewers within 1 second at p95" (the 95th percentile), "no duplicate comments after retries".
Pitfalls
- Offset anchors are the classic bug: comments drift onto the wrong text after any edit above them.
- Deleting comments whose text was deleted silently loses reviewer work; detach instead.
- Server-generated IDs for offline comments make retries create duplicates.
- Treating comments as document edits for permissions lets commenters change text, or blocks them from commenting.
Compare WebSockets, Server-Sent Events (SSE), long polling, and regular polling for real-time client-server communication. For each transport explain: connection model (half/full-duplex), typical latency characteristics, browser/proxy support, server resource implications, and common use-cases (e.g., collaborative editing, live notifications, stock tickers). Conclude with recommended transport(s) for (a) a live chat app and (b) a server-to-many broadcasting system.
Sample Answer
Direct answer
All four transports answer the same question: how does the server tell the client something changed? Regular polling asks on a timer, long polling asks and lets the server hold the request open until there is news, Server-Sent Events (SSE) keeps one HTTP response open and streams events down it, and WebSockets upgrade one connection into a two-way message pipe. For (a) a live chat app I would pick WebSockets, because both sides send frequently. For (b) a server-to-many broadcast I would pick SSE, because traffic is one-way and SSE is plain HTTP with reconnection built in. Polling stays as the fallback tier, not the primary design.
The four transports in plain terms
- Regular (short) polling: the client sends
GET /updates?since=...every N seconds. Most responses are empty. Simple, works everywhere, wasteful. - Long polling: the client sends the same request, but the server does not answer until an event arrives or a timeout (say 25 seconds) passes. The client immediately re-requests after each response. Near-real-time over ordinary HTTP.
- SSE: the browser's
EventSourceAPI opens one HTTP request whose response never ends; the server writesdata: ...lines as events happen. One direction only (server to client). The client sends anything upstream with normal HTTP requests. - WebSocket: starts as an HTTP request with an
Upgrade: websocketheader; after the server answers101 Switching Protocols, the same TCP connection carries framed messages in both directions at any time.
Two terms the comparison needs. Full-duplex means both sides can send at the same moment on one connection (WebSocket). Half-duplex means one direction at a time: each HTTP request/response exchange is effectively half-duplex, so polling and long polling are half-duplex, and SSE is a one-way (simplex) stream.
Comparison across the five asked axes
| Axis | Regular polling | Long polling | SSE | WebSocket |
|---|---|---|---|---|
| Connection model | New request per interval (or reused keep-alive connection); half-duplex | One held request at a time; half-duplex | One long-lived HTTP response; server to client only | One upgraded TCP connection; full-duplex |
| Typical latency | Average delay is about half the interval plus one round trip (5 s interval: about 2.5 s average, up to 5 s) | About one round trip when an event arrives; an extra round trip if a second event lands while the client is re-requesting | One-way network latency | One-way network latency, lowest framing overhead per message |
| Browser / proxy support | Universal | Universal; some proxies time out held requests, so keep the hold below their idle timeout | All modern browsers; response-buffering proxies can delay events (disable buffering); on HTTP/1.1 browsers cap around 6 connections per origin, so use HTTP/2 | All modern browsers; some corporate proxies and old intermediaries block the Upgrade; load balancers need idle timeouts above your heartbeat |
| Server resources | No held connections, but request rate is high and mostly wasted | One held connection per client, plus a full request cycle per event | One held connection per client; cheap per message | One held connection per client; cheap per message; state (auth, subscriptions) lives with the connection |
| Common use cases | Low-urgency dashboards, status pages, fallback | Legacy real-time, environments where streaming is blocked | Live notifications, stock tickers, sports scores, log tails | Chat, multiplayer games, collaborative editing, anything with frequent client-to-server traffic |
A keep-alive connection (in the Connection model row) is a single TCP connection reused across several sequential requests instead of opening a new one each time; it still carries one request at a time, so reusing it does not make polling any less half-duplex.
Push versus pull, and who pays for it
Polling is pull: the client decides when to ask, so cost scales with the number of clients times the poll frequency, whether or not anything happened. SSE and WebSockets are push: cost scales with connection-time and with the number of real events. Long polling sits between (push semantics, pull mechanics).
This matters for the bill and for the network:
- Per-request pricing. Many managed gateways and load balancers charge per request or per processed unit. Polling generates requests even when there is no news; push generates billable messages only for real events plus heartbeats.
- Per-connection pricing. Managed WebSocket services typically charge for connection-minutes as well as messages. For clients that are connected all day but rarely receive anything, a long poll interval can be cheaper than a held connection. Run both numbers for your own traffic shape.
- Mobile radio and battery. Every poll wakes the cellular radio, and the radio stays in a high-power state for a while after each transfer. Frequent polling from a phone drains the battery far more than the tiny payloads suggest.
Worked cost example (1 million connected clients)
Assume an empty poll costs about 800 bytes of request plus response headers (an estimate; measure your own), and a WebSocket heartbeat (a small ping message either side sends on a timer, answered with a pong, so both sides know the connection is still alive) ping/pong pair costs about 100 bytes on the wire including TCP/IP overhead (also an estimate).
- Polling every 5 s: 1,000,000 / 5 = 200,000 requests per second, mostly empty. 200,000 × 800 B = 160 MB/s, which is 160 MB/s × 86,400 s ≈ 13.8 TB per day of overhead.
- Polling every 30 s: 1,000,000 / 30 ≈ 33,333 requests per second, with an average staleness of 15 s. That is 33,333 × 800 B ≈ 26.7 MB/s, which is 26.7 MB/s × 86,400 s ≈ 2.3 TB per day.
- WebSocket with a 30 s heartbeat: 1,000,000 / 30 × 100 B ≈ 3.3 MB/s, about 288 GB per day, and events arrive within one network latency.
The push option carries roughly 48 times less idle overhead than 5 s polling (13.8 TB vs 0.288 TB), but that comparison mixes a 5 s interval against a 30 s heartbeat, so do not read 48x as the inherent push-versus-pull gap. At the same 30 s interval, push still wins by roughly 8 times (2.3 TB vs 0.288 TB), because a heartbeat (about 100 bytes) is far smaller than a full HTTP poll's headers (about 800 bytes); push's real extra advantage on top of that is not needing a short interval at all, a WebSocket does not trade staleness for cost the way polling does. The price of push is a million held connections, which you must be able to run.
Worked hybrid: an intermittently connected mobile fleet
Consider a delivery-driver app for a globally distributed, intermittently connected fleet: drivers work worldwide, often on weak cellular signal and often with the app in the background. No single transport fits, so use three tiers that all feed one sync mechanism.
- OS push for battery-efficient wake. When the app is backgrounded, the OS suspends its sockets. Use the platform push service (Apple Push Notification service, APNs, or Firebase Cloud Messaging, FCM) to send a small "you have a new job offer" signal. The OS delivers it on a connection it already maintains, so the app pays no battery for its own connection.
- One pooled WebSocket when foregrounded and connected. When the app is in the foreground with a usable network, open a single WebSocket shared by every feature (job offers, chat with dispatch, route changes). "Pooled" here means one multiplexed (many independent logical streams of messages sharing one physical connection) connection per app, not one per screen.
- Adaptive-interval polling as the fallback. If the WebSocket cannot connect (a proxy strips the Upgrade, or the network flaps too fast to hold a connection), poll with an interval keyed to device and network state: for example 5 s on Wi-Fi in the foreground, 15 s on cellular, 60 s in battery-saver mode, always with random jitter (a small random offset added to each client's timer so they do not all fire at the same instant) so a fleet does not poll in lockstep.
The design move that makes this safe: every tier is only a trigger; the source of truth is one sync(since=cursor) endpoint. (A cursor is a marker, often a timestamp or sequence number, for the last update the client already has, so the server returns only what is newer.) A push notification, a WebSocket message and a poll response all cause the client to fetch everything after its last cursor. Missed or duplicated triggers then cannot lose or double-apply data, and switching tiers mid-shift is invisible to the driver.
The two recommendations
(a) Live chat: WebSockets. Users send and receive constantly (messages, typing indicators, read receipts). A full-duplex connection avoids a separate HTTP request per sent message and keeps per-message overhead to a few bytes of framing. Keep long polling or SSE-plus-POST as the fallback for networks that block the Upgrade.
(b) Server-to-many broadcasting: SSE. Traffic is one-way, so WebSocket's upstream channel buys nothing. SSE is plain HTTP, so it passes through proxies, HTTP-aware load balancers and authentication middleware unchanged; EventSource reconnects automatically and sends a Last-Event-ID header so the server can resume. Serve it over HTTP/2 so many streams share one connection. If the same product already runs a WebSocket fleet, reusing it is a reasonable choice; the fan-out architecture behind the transport (how one update is copied to many connections) matters more than the transport.
Trade-offs and pitfalls
- Choosing by latency alone. SSE and WebSockets have the same latency; the real differentiators are direction of traffic, intermediary support and operational cost.
- Forgetting that held connections are state. Push transports move cost from request handling to connection holding: file descriptors (the OS handle each open connection consumes; a machine can run out of these before it runs out of memory), memory per connection, and reconnect storms (when a server or node dies, every client it was holding tries to reconnect within the same few seconds, spiking load on whatever takes over). Plan for those.
- Polling in lockstep. A fleet that polls on the minute creates self-inflicted traffic spikes; always add jitter.
- Treating the transport as the delivery guarantee. Every transport drops messages across disconnects. Resumable cursors or sequence numbers, not the socket, are what make delivery reliable.
Describe a robust client-side feature detection and fallback strategy for environments where WebSocket is blocked or unreliable. Include how to detect failures/blocks (e.g., blocked Upgrade header, connection resets), how to choose between SSE and long-polling fallback, how to preserve session semantics and message ordering across fallback transitions, and how to keep server-side behavior consistent.
Sample Answer
Direct answer
Detect blockage by proving the channel end to end, not by trusting the open event: a WebSocket counts as working only once the server's first application message (a welcome) arrives within a short deadline. If it fails, try SSE (Server-Sent Events: a single HTTP response the server keeps open and writes events into), and prove that too by requiring a first event within a deadline; if SSE is buffered or blocked, drop to long polling (repeated HTTP requests, each held open by the server until there is news or a timeout), which works through almost anything. Keep behaviour identical across all three by putting a transport-agnostic session layer on the server: every session has an ID, every server-to-client message carries a per-session sequence number, and every client-to-server message carries a client message ID, so a transport switch becomes "reconnect and resume from sequence N" instead of "start over".
Key terms
- SSE (Server-Sent Events): a normal HTTP response that never ends; the server writes events into it as text. One direction only; the browser API (
EventSource) reconnects automatically and sends aLast-Event-IDheader saying the last event it saw. - Long polling: the client makes an HTTP request, the server holds it until there is data or a timeout (say 25 s), responds, and the client immediately asks again.
- Buffering proxy: an intermediary (corporate proxy, antivirus scanner, some load balancers) that waits for a response to finish before passing it on. It breaks SSE silently: the connection "succeeds" but no event ever arrives.
1. Detecting failure and blockage
Two terms the table below uses: close code 1006 is the WebSocket protocol's code for an abnormal closure, the connection ended with no proper close handshake, as if it was simply cut. A middlebox is any network device between client and server (a proxy, firewall, antivirus scanner or load balancer) that inspects or modifies traffic in transit; deep-packet inspection means such a device examines the actual content of packets, not just their headers, to decide whether to allow them, and a TLS-intercepting proxy goes further and decrypts, inspects and re-encrypts TLS traffic, effectively sitting as a man-in-the-middle for security scanning.
| Symptom the client sees | Likely cause | Treat as |
|---|---|---|
error then close with code 1006 before open fires | Upgrade header stripped, handshake rejected, port blocked | WebSocket blocked on this network |
open fires but no welcome within 5 s | A middlebox accepted the upgrade but is not relaying frames | WebSocket blocked |
| Works, then closes with 1006 after a fixed idle period (for example every 60 s) | Proxy or load balancer idle timeout | Fix with heartbeats, not fallback |
| Repeated resets within seconds of connecting, on this network only | Deep-packet inspection or TLS-intercepting proxy | Blocked; fall back |
| SSE request returns 200 but no event within 5 s | Buffering proxy | SSE blocked; use long polling |
Rules that make detection robust:
- Handshake plus proof deadline: open, then wait for the server's
welcome(it carries the session ID). No welcome in 5 s means failure, whatever the socket state says. - Don't fall back on the first failure: one failure could be a server deploy. Fall back after, say, 2 consecutive proven failures on the same network.
- Remember the verdict per network, keyed by something like the Wi-Fi network or a hash of the public IP range, for a few hours, so the next page load starts on the transport that worked instead of paying the detection delay again.
- Probe upward: while in fallback, retry a WebSocket in the background every 10 minutes or so; networks change (the user leaves the office VPN).
2. Choosing SSE vs long polling
Try SSE first when WebSocket is blocked, because it is a single streaming request with lower latency and less overhead than a request per message. Require an initial padding event from the server (send a comment line plus a ready event immediately) so a buffering proxy is detected in seconds, not after the first real message. Also ask your own proxy tier not to buffer: for example nginx (a widely used web server and reverse proxy) honours an X-Accel-Buffering: no response header.
Go straight to long polling when SSE's proof deadline fails, when the client is an old environment without EventSource, or when the user has many tabs on HTTP/1.1 (browsers allow only about 6 connections per host, and each SSE stream holds one permanently).
Upstream messages (client to server) go over ordinary POST requests in both SSE and long-polling modes.
3. Preserving session semantics and ordering across transitions
sequenceDiagram
participant C as Client
participant S as Session layer
C->>S: WebSocket hello, resume session 42 from seq 17
S-->>C: welcome, seq 18 to 20
Note over C: WebSocket dies, client switches to SSE
C->>S: SSE request, session 42, Last-Event-ID 20
S-->>C: replay seq 21 and 22, then live events
C->>S: POST message with client id m-9
S-->>C: ack m-9 as seq 23
- Session ID outlives the transport. The server keeps session state (subscriptions, auth context) for a grace period (say 2 minutes) after a transport drops.
- Server-assigned sequence numbers, per session. The server keeps the last few hundred messages per session in a replay buffer. Any transport resumes with "last seq I processed"; SSE gets this for free through
Last-Event-ID, WebSocket and long polling send it explicitly. If the requested seq has fallen out of the buffer, the server says so and the client does a full state refresh. - Client applies strictly in order: if it receives seq 25 while expecting 23, it holds 25 and waits (or asks for a replay); if it receives a seq it already applied, it drops it. That dedupe matters because a transition often delivers the same message twice.
- Client-to-server messages carry a client-generated ID (for example
m-9). If the client is unsure whether a send went through (the socket died mid-send), it resends with the same ID over the new transport and the server discards duplicates. This makes retries safe.
4. Keeping server behaviour consistent
- One session and delivery service; WebSocket, SSE and long-poll handlers are thin adapters that call the same
subscribe,send,ackandresumeoperations. Authorization, rate limits and message formats live in the shared layer, never in one adapter. - Same message envelope on every transport (
seq,type,payload). - Session state in a shared store (or sticky routing, a load balancer consistently sending the same client's requests to the same backend server, by session ID) because long polling and upstream
POSTs from one client can land on different servers. - Measure per transport: share of sessions on each, time-to-first-message, fallback rate by network, so you notice when a fallback becomes the majority path.
Worked example
A user in an office opens the app. WebSocket upgrade returns an error with close code 1006 twice. The client records "office network: WebSocket blocked", opens SSE with session 42, gets the padding event within 300 ms, and runs on SSE. The user sends a message as POST with ID m-9; the POST times out, the client retries with the same ID, the server recognises m-9 and returns the original ack (seq 23) rather than posting twice. The user walks to a cafe; the background probe succeeds, and the client moves back to WebSocket, resuming from seq 23 without a visible gap.
Pitfalls
- Treating
openas success (middleboxes that accept then silently drop frames). - Falling back permanently on a single error, which quietly moves most users to the most expensive transport after one bad deploy.
- Separate code paths per transport that drift apart (different auth check, different message format).
- No dedupe on the client, so every fallback shows a duplicated message.
Implement a simple per-connection fixed-size ring buffer in Python to hold the last N messages for replay after reconnection. Provide methods: append(message), get_since(sequence_id) which returns messages after a given sequence number, and trim to enforce capacity. Consider memory efficiency and how to represent sequence numbers.
Sample Answer
Direct answer
Give every message a sequence number (a counter that goes up by one per message on this connection), store messages in a fixed-size list used as a circle, and compute each message's slot as seq % capacity. Appending then overwrites the oldest entry for free, and get_since(n) is just "read slots n+1 up to the newest". The one case that must not be hidden: if the client asks for a sequence number older than anything still kept, the buffer raises a gap error so the server can tell the client to do a full resync instead of silently skipping messages.
Approach
A ring buffer (also called a circular buffer) is an array of fixed length where the write position wraps back to index 0 after the last slot. It suits reconnect replay because:
- the memory cost is fixed the moment the connection opens (no growth during a traffic burst);
- the oldest message is evicted automatically when a new one arrives;
- with monotonically increasing sequence numbers you never need to search: the slot for sequence
siss % capacity.
State kept per connection:
| Field | Meaning |
|---|---|
_slots | preallocated list of length capacity, each slot (seq, message) or None |
_next_seq | the sequence number the next append will assign (starts at 1) |
_count | how many slots hold live messages (grows to capacity, then stays there) |
Two derived values do all the work: last_seq = _next_seq - 1 and oldest_seq = _next_seq - _count.
Reconnect protocol it supports. Every message sent to the client carries its seq. The client remembers the highest seq it processed. On reconnect it sends resume_from=<that number>; the server calls get_since(resume_from) and replays the result, or answers "resync needed" on a gap.
Code
class ReplayGap(Exception):
"""The client asked for messages older than the buffer still holds."""
class ReplayBuffer:
__slots__ = ("_cap", "_slots", "_next_seq", "_count")
def __init__(self, capacity):
if capacity < 1:
raise ValueError("capacity must be >= 1")
self._cap = capacity
self._slots = [None] * capacity # preallocated: no growth, no per-append resize
self._next_seq = 1 # sequence number the NEXT append will get
self._count = 0 # how many slots currently hold a live message
def append(self, message):
seq = self._next_seq
self._slots[seq % self._cap] = (seq, message) # overwrites the oldest when full
self._next_seq += 1
if self._count < self._cap:
self._count += 1
return seq
@property
def oldest_seq(self):
return self._next_seq - self._count # == next_seq when empty
@property
def last_seq(self):
return self._next_seq - 1 # 0 means nothing sent yet
def get_since(self, seq):
"""Messages with sequence number > seq, oldest first."""
if seq > self.last_seq:
raise ValueError(f"client claims seq {seq}, server only issued {self.last_seq}")
if seq < self.oldest_seq - 1:
raise ReplayGap(f"need {seq + 1}, oldest kept is {self.oldest_seq}")
return [self._slots[s % self._cap] for s in range(seq + 1, self._next_seq)]
def trim(self, new_capacity=None, keep_last=None):
"""Shrink capacity and/or drop all but the newest keep_last messages."""
keep = self._count if keep_last is None else min(keep_last, self._count)
cap = self._cap if new_capacity is None else new_capacity
if cap < 1:
raise ValueError("capacity must be >= 1")
keep = min(keep, cap)
live = [self._slots[s % self._cap] for s in range(self._next_seq - keep, self._next_seq)]
self._cap, self._slots, self._count = cap, [None] * cap, keep
for seq, msg in live:
self._slots[seq % cap] = (seq, msg)
buf = ReplayBuffer(capacity=4)
for i in range(1, 7):
buf.append(f"m{i}")
print("oldest", buf.oldest_seq, "last", buf.last_seq)
print("since 3:", buf.get_since(3))
print("since 6:", buf.get_since(6))
try:
buf.get_since(1)
except ReplayGap as e:
print("gap:", e)
buf.trim(new_capacity=2)
print("after trim cap=2 oldest", buf.oldest_seq, "since 4:", buf.get_since(4))
buf.append("m7")
print("since 5:", buf.get_since(5))
Output of python3 ring.py:
oldest 3 last 6
since 3: [(4, 'm4'), (5, 'm5'), (6, 'm6')]
since 6: []
gap: need 2, oldest kept is 3
after trim cap=2 oldest 5 since 4: [(5, 'm5'), (6, 'm6')]
since 5: [(6, 'm6'), (7, 'm7')]
Reading the output: capacity 4 after six appends keeps m3 to m6; asking for everything after 3 returns 4, 5, 6; asking after 6 (the client is fully caught up) returns an empty list; asking after 1 needs m2, which was overwritten, so the call raises instead of returning a misleadingly partial list. After trim(new_capacity=2) only 5 and 6 survive, and the next append (7) lands in the smaller ring correctly.
Walking through trim()
trim() is the least obvious method because it rebuilds the whole buffer rather than mutating it in place, so here is what each part does:
keep = self._count if keep_last is None else min(keep_last, self._count): decide how many messages survive. Default to everything currently held; if the caller asked to keep fewer, take the smaller of the two.cap = self._cap if new_capacity is None else new_capacity: decide the new size, defaulting to the current one if onlykeep_lastwas given.keep = min(keep, cap): a buffer can never hold more live messages than its own capacity, so capkeepa second time against the (possibly smaller) new size.live = [self._slots[s % self._cap] for s in range(self._next_seq - keep, self._next_seq)]: read thekeepnewest messages out, still using the OLD capacity (self._cap) to compute slot positions, because this line runs before anything is resized.self._cap, self._slots, self._count = cap, [None] * cap, keep: only now does the buffer actually shrink or grow: a fresh, all-empty list of the new size replaces the old one.for seq, msg in live: self._slots[seq % cap] = (seq, msg): replay the saved messages back in, this time using the NEW capacity (cap) to compute each slot, so they land in the right place in the resized array.
The order matters: reading with the old capacity and writing with the new one is what makes shrinking safe without losing the newest messages.
Key points
- Sequence numbers are per connection (or per stream) and start at 1, so
0naturally means "I have seen nothing". Python integers never overflow; in a language with fixed-width integers use a 64-bit unsigned counter (at one million messages per second it takes over 500,000 years to wrap). - The slot index is derived, not stored. Because sequence numbers are consecutive,
seq % capacityis always the right slot and the storedseqin each tuple is only a safety check and a convenience for the caller. get_sincevalidates both ends. Aseqabovelast_seqmeans the client claims messages the server never sent (a bug, or a client that reconnected to the wrong stream). Aseqbelowoldest_seq - 1is a gap.trimhandles both meanings of trim: shrink capacity (for example when the server is under memory pressure) and keep only the newestkeep_lastitems. It rebuilds into a new list, which is O(kept) and happens rarely.__slots__removes the per-instance__dict__, which matters when there is one buffer object per open connection.
Complexity
append: O(1) time, no allocation beyond the stored tuple.get_since(seq): O(k) where k is the number of messages returned; no scanning of older entries.trim: O(kept) time, done on configuration change, not per message.- Memory: O(capacity) per connection, fixed up front.
Worked example: what the memory really costs
Measured with sys.getsizeof on the CPython 3.14 build used here (sizes differ slightly between Python versions): a list of 256 slots is 2,104 bytes; each stored entry costs an 8-byte list pointer, a 64-byte 2-tuple, a 28-byte integer and the payload. With a 200-character string payload (241 bytes) that is 8 + 64 + 28 + 241 = 341 bytes per message.
About 71 percent of each entry (241 of 341 bytes) is payload; the other 100 bytes is Python object overhead, and the real problem is the multiplier: 4 GiB of RAM on one node just for replay. Two ways to shrink it, in the order I would try them:
- Share payloads across connections. In a chat room or ticker, the same message goes to thousands of clients. Keep one immutable
bytesobject per message and store references in each ring, so 50,000 rings holding the same message cost 50,000 pointers (400 KB), not 50,000 copies. - Cap by bytes, not only by count. Track
bytes_usednext to_countand evict until it fits, so one client subscribed to large messages cannot hold 256 x 64 KB.
Edge cases
- Empty buffer:
oldest_seq == 1,last_seq == 0;get_since(0)returns[]. - Client exactly at the oldest boundary:
get_since(oldest_seq - 1)is legal and returns the full buffer. - Server restart: an in-memory ring is lost, so sequence numbers restart at 1. Pair the counter with a stream epoch (a random ID generated at startup); if the client's epoch differs from the server's, force a resync instead of trusting the numbers.
- Capacity 1: works (every append evicts the previous message); capacity 0 is rejected.
- Concurrency: this class is not thread-safe. In an asyncio server (Python's cooperative concurrency model, where a single thread's event loop runs many connections' code by switching between them only at
awaitpoints, never truly in parallel) that is fine, one event loop per connection's code path: nothing preempts a running piece of code partway through, so two calls into the same buffer cannot physically overlap unless there is anawaitbetween them, and neitherappendnorget_sincecontains one. With threads, which really can run at the same instant on different CPU cores, guardappendandget_sincewith one lock per buffer.
Trade-offs and pitfalls
- Count-based capacity is a proxy for time. 256 messages is ten minutes on a quiet chat and two seconds on a busy ticker. Many systems size by "last N seconds" instead and derive N from the stream's rate.
- Returning a partial list on a gap is the classic bug. The client applies messages 5 to 8, never learns 2 to 4 were lost, and shows a wrong state forever. The explicit gap error is the most important line in the design.
- Per-connection rings duplicate data. For broadcast streams, a single per-topic log plus a per-connection cursor (just the last acknowledged
seq) is cheaper; the per-connection ring is right when each connection's stream is unique (private notifications, per-user filtered feeds).
Unlock Full Question Bank
Get access to all 12 Real-Time and Streaming System Design interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.