Real-Time and Streaming System Design Questions
Designing client-facing, always-on live delivery systems: real-time communication transports (WebSockets, server-sent events, long-polling, WebRTC, MQTT), connection lifecycle and scaling for millions of persistent connections, presence, pub/sub fan-out to online users, chat and notifications, live feeds and tickers, real-time collaboration (CRDT vs OT, offline sync and reconciliation), and live and on-demand video delivery (ingest, transcoding, adaptive bitrate, CDN delivery, low-latency protocols, playback entitlement and content protection). Covers latency budgets, per-client ordering, delivery guarantees across disconnect and reconnect, per-client backpressure, capacity estimation for connection fleets, authentication, authorization, and revocation for real-time channels and premium content, and operational readiness (SLOs and error budgets, observability and incident response, rate limiting, safe rollout, and load and chaos testing) for live-delivery platforms. Data-pipeline stream processing (Kafka or Flink jobs, windowed aggregation, exactly-once pipelines, real-time analytics ingestion) is out of scope.
Design an offline-first synchronization architecture for a mobile client (for example a CRM app) that allows creating and editing records offline, supports conflict detection and resolution, and syncs efficiently across varying bandwidth, including partial syncs when bandwidth is limited. Describe data models, client change queues, batching thresholds, retry/backoff policies, the server-side reconciliation algorithm, and how you would present merges to users.
Sample Answer
Direct answer
The client treats its local database as the source of truth for the UI and records every user change in an outbox: a durable, ordered queue of operations waiting to be sent. A sync engine drains the outbox in batches when there is connectivity and pulls server changes since a cursor (a saved marker, such as a sequence number, recording exactly how far the last pull got, so the next pull resumes from there instead of re-fetching everything). The server applies each operation with a field-level three-way merge ("three-way" because it compares three values per field: the client's base version before it edited, the server's current version, and the client's new value), so edits to different fields never conflict and only same-field edits need a rule. Low-stakes fields resolve automatically; high-stakes fields (deal amount, stage) become a visible conflict the user resolves. On poor bandwidth the engine syncs in priority tiers: small, important changes first, large attachments last.
Offline-first means the app is designed to work fully without a network and treats the network as an optimisation, rather than bolting offline caching onto an online app.
Requirements I am designing to
- Sales reps create and edit accounts, contacts and opportunities offline for up to several days.
- Two people can edit the same record while one of them is offline.
- Networks range from office Wi-Fi to a weak 2G signal; sync must make progress even in short windows of connectivity.
- No edit is ever silently lost.
Data model
On the client (SQLite on the device):
| Table | Key columns |
|---|---|
records | id (a UUID generated on the device, so new records need no server round trip), type, fields (JSON), base_version (server version this copy came from), dirty flag |
outbox | op_id (UUID), client_seq (monotonic counter per device), record_id, op (create / update / delete), changed_fields with old and new values, base_version, attempts, priority |
sync_state | pull_cursor per record type, last successful sync time |
conflicts | record, field, local value, server value, who changed it |
On the server: each record has a version (increments on every committed change) and per-field field_version plus updated_by; a change log indexed by a global sequence gives pull cursors; an idempotency table keyed by op_id remembers operations already applied, so a retried batch is not applied twice.
Sync flow
sequenceDiagram
participant UI
participant DB as Local DB and outbox
participant Sync as Sync engine
participant API as Sync API
UI->>DB: edit field, append op to outbox
Sync->>DB: read next batch by priority
Sync->>API: push batch (op_ids, base versions)
API-->>Sync: per-op result: applied, merged, conflict
Sync->>DB: remove applied ops, store conflicts
Sync->>API: pull changes since cursor
API-->>Sync: changes and new cursor
Sync->>DB: apply changes, rebase pending ops
Push before pull, and rebase. "Rebase" here means: after pulling, any operations still in the outbox are re-applied on top of the fresh server data locally (the way you would redo your own edits on top of a document after fetching someone else's newer version), so the UI shows "server state plus my pending edits" without flicker.
Batching thresholds
Flush the outbox when any of these is true:
- 50 operations queued, or 64 KB of payload (keeps each request small enough to finish on a weak link);
- 5 seconds since the oldest unsent operation (bounded staleness when online);
- connectivity changes from offline to online, or the app is about to go to the background.
Each batch is atomic per operation, not per batch: the server returns a result for every op_id, so one bad operation does not block the other 49.
Retry and backoff
- Exponential backoff with full jitter: wait a random time between 0 and min(cap, base x 2^attempt), with base 1 s and cap 5 minutes. The jitter stops thousands of devices that lost signal together (say, a subway line) from retrying in lockstep when it returns.
- Retry only retryable failures: timeouts, 5xx (a server-side error status code, meaning the failure was the server's fault and retrying may succeed), and 429 ("too many requests", with a Retry-After header telling the client exactly how many seconds to wait before trying again). A 4xx (a client-side error, meaning the request itself was invalid) validation error goes to a "needs attention" state on that record instead of retrying forever.
- Retries are safe because each operation carries its
op_idand the server's idempotency table answers "already applied" for duplicates. - Reset backoff on any successful request, and trigger an immediate attempt on an OS connectivity-change event.
Partial sync when bandwidth is limited
Classify work into tiers and let the engine measure throughput (bytes per second of the last few requests) to decide how far down the list to go:
- Tier 1, always: outbox pushes of record fields, and pulls of records assigned to this rep changed since the cursor. Kilobytes.
- Tier 2, when throughput allows: other records in the rep's territory, activity history, related contacts.
- Tier 3, Wi-Fi or strong signal only: attachments, images, full history. Download as metadata first (name, size, thumbnail) and fetch content on demand.
Pulls use field projection (only the columns the mobile screens show) and compressed responses. Pagination by cursor means an interrupted pull resumes where it stopped rather than restarting.
Server-side reconciliation algorithm
For each incoming operation, in client_seq order per device:
-
If
op_idis in the idempotency table, return the stored result. -
If
base_version == current version, apply directly (the common case: nobody else touched the record). -
Otherwise, for each changed field: if the server's
field_versionfor that field is not newer than the client's base, apply the change (someone else edited a different field). If it is newer, it is a same-field conflict:- auto-resolve fields tagged low-stakes (notes append, phone number) with last-writer-wins by server receive time, keeping the loser in the audit history;
- flag high-stakes fields (amount, stage, close date, owner) as a conflict: keep the server value, store the client's value, and return
conflictfor that field.
Concrete trace: record
acmeis at serverversion12, withfield_version{phone: 12, stage: 10}. A rep's client last synced atbase_version12 (so its local base for every field is 12) and, while offline, editsstage. Meanwhile a colleague's earlier online edit already bumped the server'sstagefield_versionto 13. When the rep's queued op reaches the server withbase_version: 12: forstage, the server'sfield_version(13) is newer than the client's base (12), so this is a same-field conflict and the rep'sstagechange is flagged. If the rep had instead editedphone(serverfield_versionstill 12, unchanged since the rep's base), the server'sfield_version(12) is not newer than the client's base (12), so that change applies directly with no conflict. -
Deletes vs edits: an edit to a record deleted on the server returns
conflict: deletedand the client offers "restore with my changes" or "discard". -
Record the result under
op_id, bump version and field versions, append to the change log.
Presenting merges to users
- Non-conflicting merges are silent; the rep sees the colleague's changes appear with a small "updated by Priya, 10:42" note.
- Conflicts show a badge on the record and a side-by-side card: "You set Stage to Negotiate (offline, 9:10). Priya set it to Closed Lost (10:02)." Buttons: keep mine, keep theirs, edit.
- A sync status indicator ("3 changes waiting to sync") so reps trust that offline work is saved.
Worked example: a rep back from a site visit
The rep made 500 field edits offline over two days, each operation about 300 bytes as JSON.
payloadat 50 kbit/s (weak 2G)at 1 Mbit/s=500×300=150,000 bytes=150 KB=50,000150,000×8=24 s=1,000,000150,000×8=1.2 sAt 50 operations per batch that is 10 requests of 15 KB (about 2.4 s each on the weak link). If the signal drops after batch 6, 300 operations are safely committed and the next window resumes at batch 7. Compressing JSON typically shrinks it several-fold, so the real transfer is smaller; the batch size is what keeps each request short enough to finish.
Of those 500 edits, suppose 12 touched records a colleague also changed. Field-level merge resolves most of them automatically because the fields differ; only same-field edits on high-stakes fields reach the conflict card.
Trade-offs and pitfalls
- Whole-record last-write-wins is the classic data-loss bug: the rep's phone-number fix is overwritten by a colleague's stage change. Field-level merge fixes it cheaply.
- Server-assigned IDs break offline creation (the child contact cannot reference an account that has no ID yet). Device-generated UUIDs avoid the remapping problem.
- CRDTs would auto-merge everything but are the wrong tool for business fields like "deal amount", where a merged value is meaningless and a human decision is required. They fit free-text notes if collaborative editing is needed.
- Unbounded offline windows make the pull cursor and idempotency table grow; define a maximum (for example 30 days), after which the device does a full resync and its unsent operations are replayed against fresh data.
Explain eventual consistency in the context of real-time client updates. Describe the trade-offs between latency and strong consistency, examples of acceptable eventual consistency (social feeds) versus unacceptable cases (financial transfers), and how to design reconciliation processes and user-visible indicators for divergence.
Sample Answer
Direct answer
Eventual consistency means that if updates stop, every copy of the data (servers, caches, each user's screen) converges to the same value, but for a while different people can see different values. In real-time apps we accept it because it lets us show changes instantly and keep working during network trouble, instead of waiting for every replica to agree. It is fine where a briefly stale or out-of-order view harms nobody (a like count, a social feed) and unacceptable where a stale read lets someone break a rule that must always hold (spending money that is already gone). Where we accept it, we owe the user two things: a reconciliation process that brings copies back together, and visible indicators when what they see may not be final.
Strong consistency is the opposite promise: once a write is confirmed, every later read anywhere sees it. It costs latency (waiting for agreement between copies) and availability (refusing work when copies cannot reach each other).
The latency versus consistency trade-off
- To show a like instantly, the phone updates its own screen first and tells the server afterwards. Fast, but for a moment the phone and the server disagree.
- To be strongly consistent, the phone must wait until the server (and its replicas) confirm. With a 150 ms round trip to a distant region, every tap feels 150 ms slower, and when the network drops, the tap fails.
- The practical rule: choose per operation, not per app. One app can show likes optimistically and still require a confirmed server round trip for a money transfer.
Acceptable versus unacceptable
| Case | Eventual OK? | Why |
|---|---|---|
| Social feed ordering, like counts | Yes | Seeing 41 likes instead of 42 for two seconds harms nobody; it converges |
| Presence ("online" dot) | Yes | A few seconds of staleness is expected |
| Chat typing indicator | Yes | Loss is harmless |
| Financial transfer / account balance | No | Two devices each seeing a $100 balance could both approve a $100 transfer: an invariant (balance never below zero) is broken, not just a display |
| Seat or inventory booking | No, for the decision | Two users could both "win" the last seat; the display can be eventual, but the booking must be decided by one authority |
The test to apply: does a stale read lead to an irreversible decision that breaks an invariant? If yes, that decision needs strong consistency (a single authoritative check, a transaction, or a lock), even if everything around it stays eventual.
Worked example: a like button
- Priya taps like on a post showing 41. Her app immediately shows 42 and a filled heart (an optimistic update) and sends
like(post_id, request_id). - Meanwhile Sam, on another server replica, also likes it; his screen shows 42 too.
- Both writes reach the server, which stores likes as a set of user IDs, so the true count is 43. It pushes the new count over the real-time channel.
- Both screens converge to 43 within a second or two. Nobody was harmed by the brief 42.
If the server rejects Priya's like (she was blocked by the author), her app rolls back to 41 and shows a short message. That rollback path is part of the design, not an afterthought.
Designing reconciliation
- Server is authoritative. Clients hold optimistic, pending changes; the server's answer replaces them.
- Use mergeable data shapes where possible. A like stored as "set of user IDs" merges trivially (union), while "count += 1" applied twice after a retry double-counts. Structures designed to merge automatically are called CRDTs (conflict-free replicated data types); a set that only grows is the simplest one.
- Idempotent writes: every client write carries a request ID so a retry does not apply twice.
- Versions on objects: each object carries a version number; the client sends "I edited version 7". If the server is at 8, it either merges or returns a conflict for the client to resolve.
- Catch-up after reconnect: the client sends the last version or sequence it saw; the server returns changes since then, or a fresh snapshot if too much has changed.
- Periodic anti-entropy: a background comparison that repairs replicas that missed an update, for the rare case the real-time channel dropped one.
User-visible indicators for divergence
- Pending state: a sent-but-unconfirmed item shows a clock icon or lighter text ("Sending...").
- Stale data banner: "Offline. Showing data from 10:42" when the real-time connection is down.
- Conflict notice: "This document was changed elsewhere. Review changes" rather than silently overwriting.
- Rollback message: "Couldn't like this post" when an optimistic update is rejected.
- For money, no optimism: show "Processing" until the server confirms; never show a new balance before it is final.
Pitfalls
- Treating the whole app as either eventual or strong instead of deciding per operation.
- Optimistic updates with no rollback path, which leave screens permanently wrong after a rejection.
- Counters implemented as increments without request IDs, which drift upward on retries.
- Hiding divergence entirely, so users act on stale data without knowing it.
You need to implement a live, high-fan-out feed (for example a live sports score or a real-time dashboard) that is mostly one-way, server-to-client, and must reach up to 100k concurrent clients with low latency. Between SSE and WebSockets, which would you choose and why? Consider scalability, proxies/CDNs, mobile clients, TLS, automatic reconnection behavior, and how you would implement efficient fan-out to many clients.
Sample Answer
Direct answer
For a mostly one-way, high-fan-out feed I would choose Server-Sent Events (SSE), a one-way stream of text events sent over a single, long-lived, plain HTTP response, no protocol upgrade needed, for browser clients: it is plain HTTP, reconnects on its own with a resume cursor, and needs no upstream channel the use case does not use. The harder problem is not the transport but the fan-out tier: serialize each update once, push it to a small number of edge nodes, and let each node write the same bytes to its local connections with per-client backpressure. I would expose a WebSocket endpoint (a persistent, full-duplex connection upgraded from an HTTP request, so either side can send at any moment instead of only replying to requests) on the same edge nodes for native mobile apps, where WebSocket support is first-party and SSE is not.
Fan-out means delivering one published update to many subscribers. Backpressure means what the server does when a client reads more slowly than updates arrive.
Why SSE wins here, axis by axis
Two terms used below: a CDN (content delivery network) is a fleet of third-party edge servers that caches and relays HTTP traffic close to users, and TLS (Transport Layer Security) is the encryption layer behind HTTPS and wss://.
| Consideration asked | SSE | WebSocket | Verdict for this feed |
|---|---|---|---|
| Scalability | One held HTTP response per client; tiny per-event framing (data: ...\n\n) | One held TCP connection per client; tiny framing | Tie on raw cost; both are bounded by connections per node |
| Proxies / CDNs | Plain HTTP, so HTTP-aware load balancers, auth middleware and logging work unchanged; response-buffering proxies must be told not to buffer (for nginx, the X-Accel-Buffering: no response header) | Needs every hop to support the Upgrade (the HTTP mechanism that turns the same TCP connection from request/response into a raw two-way stream); some corporate proxies strip it | SSE is friendlier, as long as buffering is disabled |
| Mobile clients | Browsers have EventSource; native iOS and Android need a library | Native platform APIs exist on both | WebSocket for native apps, SSE for web |
| TLS | Runs as HTTPS | Runs as wss:// | Same; both pay a TLS handshake per connect, which matters during reconnect storms (many clients reconnecting at once, for example after a deploy; use TLS session resumption, where the client and server reuse cryptographic state from an earlier handshake with the same peer instead of paying the full handshake cost again) |
| Automatic reconnection | Built in: the browser reconnects, honors a server-sent retry: delay, and sends Last-Event-ID so the server can resume | You write it yourself (backoff, jitter, resume) | Clear SSE win |
| Upstream traffic | None (use normal HTTP requests) | Full-duplex | Not needed here |
One condition to handle: on HTTP/1.1, browsers allow about 6 concurrent connections per origin, and each SSE stream holds one. Serve SSE over HTTP/2, where many streams share one connection, or users with several tabs open will see requests hang.
What would flip the choice to WebSocket everywhere: clients that send frequent messages (chat, votes, cursor positions), binary payloads, or a user base concentrated behind proxies that buffer streamed responses and cannot be fixed.
The fan-out architecture
flowchart LR
P[Score or metrics publisher] --> B[Pub/sub bus]
B --> E1[Edge node 1]
B --> E2[Edge node 2]
B --> E3[Edge node N]
E1 --> C1[About 12.5k clients]
E2 --> C2[About 12.5k clients]
E3 --> C3[About 12.5k clients]
LB[HTTP load balancer] --> E1
LB --> E2
LB --> E3
- Publish once. The producer (score service, metrics aggregator) publishes each update to a pub/sub (publish/subscribe) bus, where senders publish to named topics and every current subscriber of a topic gets a copy. Redis Pub/Sub or NATS (both lightweight message brokers) work well; the topic here is something like
match:1234. - Subscribe per node, not per client. Each edge node subscribes to a topic once, the first time any of its clients asks for it, and keeps a local in-memory list of the connections interested in that topic. The bus therefore fans out to 8 nodes, not 100,000 clients.
- Serialize once per node. The node formats the SSE event bytes once and writes the same buffer to every local connection. Formatting per client is the most common reason a fan-out node runs out of CPU.
- Coalesce and bound per client. Each connection gets a small bounded queue. For a score or dashboard, only the latest value matters, so when a slow client's queue is full, replace the queued update with the newest one instead of growing the queue. A client that stays behind for long (say 30 s) is disconnected and reconnects to a fresh snapshot.
- Resume on reconnect. Every event carries an
id:(a per-topic sequence number). On reconnect the browser sendsLast-Event-ID; the node replays from a short in-memory ring buffer (a fixed-size buffer of the most recent events; once full, each new event overwrites the oldest) if it has those events, or sends a full snapshot if it does not.
Worked capacity example (100,000 concurrent clients)
Assumptions (state them in the room): one update per second per topic, 500 bytes per event after serialization, and each client on one topic.
- Egress: 100,000 × 500 B × 1/s = 50 MB/s = 400 Mbps total. Modest for a small fleet.
- Nodes: assume one node comfortably holds 20,000 connections (an ESTIMATE; confirm with a load test on your hardware). 100,000 / 20,000 = 5 nodes at full load. Run 8 so that losing one node leaves 7 nodes carrying 100,000 / 7 ≈ 14,300 connections each, well under the ceiling.
- Memory: if a connection costs about 30 KB (kernel socket buffers, TLS state, application buffer; an ESTIMATE to measure), 20,000 connections × 30 KB = 0.6 GB per node.
- File descriptors: the operating system's handle for each open socket; a process has a limit on how many it can hold at once. Each connection is one open file descriptor on the node. Default per-process limits are often 1,024, so raise
ulimit -n(the shell and OS setting for that per-process limit) and the system-wide limit well above 20,000. - Proxy-to-node ports: if a reverse proxy (a server sitting in front of your edge nodes that clients connect to, which opens its own separate connection onward to the real node) opens one upstream connection per client, it is limited by its ephemeral port range per destination address and port (every outbound TCP connection needs a locally unique source port for a given destination IP and port, and the OS only hands those out from a bounded range). Linux's default range (32768 to 60999) is 28,232 ports, so a proxy talking to a single node address cannot exceed that; spread across more node addresses or let clients hit nodes more directly.
Trade-offs and pitfalls
- Arguing about transports and ignoring fan-out. Both SSE and WebSocket fail the same way if each client triggers its own database query or its own serialization.
- Reconnect storms. A deploy or node crash disconnects 12,500 clients at once. Send
retry:values with random jitter (for example 1 to 10 seconds) and drain nodes (stop routing new connections to a node while its existing ones keep running, instead of killing it outright) gradually during deploys. - Unbounded per-client queues. One client on a train tunnel can grow a queue until the node runs out of memory. Always bound and coalesce.
- Silent proxy buffering. Events appear in bursts every few seconds; the cause is an intermediary buffering the stream, not your code.
Describe a robust client-side feature detection and fallback strategy for environments where WebSocket is blocked or unreliable. Include how to detect failures/blocks (e.g., blocked Upgrade header, connection resets), how to choose between SSE and long-polling fallback, how to preserve session semantics and message ordering across fallback transitions, and how to keep server-side behavior consistent.
Sample Answer
Direct answer
Detect blockage by proving the channel end to end, not by trusting the open event: a WebSocket counts as working only once the server's first application message (a welcome) arrives within a short deadline. If it fails, try SSE (Server-Sent Events: a single HTTP response the server keeps open and writes events into), and prove that too by requiring a first event within a deadline; if SSE is buffered or blocked, drop to long polling (repeated HTTP requests, each held open by the server until there is news or a timeout), which works through almost anything. Keep behaviour identical across all three by putting a transport-agnostic session layer on the server: every session has an ID, every server-to-client message carries a per-session sequence number, and every client-to-server message carries a client message ID, so a transport switch becomes "reconnect and resume from sequence N" instead of "start over".
Key terms
- SSE (Server-Sent Events): a normal HTTP response that never ends; the server writes events into it as text. One direction only; the browser API (
EventSource) reconnects automatically and sends aLast-Event-IDheader saying the last event it saw. - Long polling: the client makes an HTTP request, the server holds it until there is data or a timeout (say 25 s), responds, and the client immediately asks again.
- Buffering proxy: an intermediary (corporate proxy, antivirus scanner, some load balancers) that waits for a response to finish before passing it on. It breaks SSE silently: the connection "succeeds" but no event ever arrives.
1. Detecting failure and blockage
Two terms the table below uses: close code 1006 is the WebSocket protocol's code for an abnormal closure, the connection ended with no proper close handshake, as if it was simply cut. A middlebox is any network device between client and server (a proxy, firewall, antivirus scanner or load balancer) that inspects or modifies traffic in transit; deep-packet inspection means such a device examines the actual content of packets, not just their headers, to decide whether to allow them, and a TLS-intercepting proxy goes further and decrypts, inspects and re-encrypts TLS traffic, effectively sitting as a man-in-the-middle for security scanning.
| Symptom the client sees | Likely cause | Treat as |
|---|---|---|
error then close with code 1006 before open fires | Upgrade header stripped, handshake rejected, port blocked | WebSocket blocked on this network |
open fires but no welcome within 5 s | A middlebox accepted the upgrade but is not relaying frames | WebSocket blocked |
| Works, then closes with 1006 after a fixed idle period (for example every 60 s) | Proxy or load balancer idle timeout | Fix with heartbeats, not fallback |
| Repeated resets within seconds of connecting, on this network only | Deep-packet inspection or TLS-intercepting proxy | Blocked; fall back |
| SSE request returns 200 but no event within 5 s | Buffering proxy | SSE blocked; use long polling |
Rules that make detection robust:
- Handshake plus proof deadline: open, then wait for the server's
welcome(it carries the session ID). No welcome in 5 s means failure, whatever the socket state says. - Don't fall back on the first failure: one failure could be a server deploy. Fall back after, say, 2 consecutive proven failures on the same network.
- Remember the verdict per network, keyed by something like the Wi-Fi network or a hash of the public IP range, for a few hours, so the next page load starts on the transport that worked instead of paying the detection delay again.
- Probe upward: while in fallback, retry a WebSocket in the background every 10 minutes or so; networks change (the user leaves the office VPN).
2. Choosing SSE vs long polling
Try SSE first when WebSocket is blocked, because it is a single streaming request with lower latency and less overhead than a request per message. Require an initial padding event from the server (send a comment line plus a ready event immediately) so a buffering proxy is detected in seconds, not after the first real message. Also ask your own proxy tier not to buffer: for example nginx (a widely used web server and reverse proxy) honours an X-Accel-Buffering: no response header.
Go straight to long polling when SSE's proof deadline fails, when the client is an old environment without EventSource, or when the user has many tabs on HTTP/1.1 (browsers allow only about 6 connections per host, and each SSE stream holds one permanently).
Upstream messages (client to server) go over ordinary POST requests in both SSE and long-polling modes.
3. Preserving session semantics and ordering across transitions
sequenceDiagram
participant C as Client
participant S as Session layer
C->>S: WebSocket hello, resume session 42 from seq 17
S-->>C: welcome, seq 18 to 20
Note over C: WebSocket dies, client switches to SSE
C->>S: SSE request, session 42, Last-Event-ID 20
S-->>C: replay seq 21 and 22, then live events
C->>S: POST message with client id m-9
S-->>C: ack m-9 as seq 23
- Session ID outlives the transport. The server keeps session state (subscriptions, auth context) for a grace period (say 2 minutes) after a transport drops.
- Server-assigned sequence numbers, per session. The server keeps the last few hundred messages per session in a replay buffer. Any transport resumes with "last seq I processed"; SSE gets this for free through
Last-Event-ID, WebSocket and long polling send it explicitly. If the requested seq has fallen out of the buffer, the server says so and the client does a full state refresh. - Client applies strictly in order: if it receives seq 25 while expecting 23, it holds 25 and waits (or asks for a replay); if it receives a seq it already applied, it drops it. That dedupe matters because a transition often delivers the same message twice.
- Client-to-server messages carry a client-generated ID (for example
m-9). If the client is unsure whether a send went through (the socket died mid-send), it resends with the same ID over the new transport and the server discards duplicates. This makes retries safe.
4. Keeping server behaviour consistent
- One session and delivery service; WebSocket, SSE and long-poll handlers are thin adapters that call the same
subscribe,send,ackandresumeoperations. Authorization, rate limits and message formats live in the shared layer, never in one adapter. - Same message envelope on every transport (
seq,type,payload). - Session state in a shared store (or sticky routing, a load balancer consistently sending the same client's requests to the same backend server, by session ID) because long polling and upstream
POSTs from one client can land on different servers. - Measure per transport: share of sessions on each, time-to-first-message, fallback rate by network, so you notice when a fallback becomes the majority path.
Worked example
A user in an office opens the app. WebSocket upgrade returns an error with close code 1006 twice. The client records "office network: WebSocket blocked", opens SSE with session 42, gets the padding event within 300 ms, and runs on SSE. The user sends a message as POST with ID m-9; the POST times out, the client retries with the same ID, the server recognises m-9 and returns the original ack (seq 23) rather than posting twice. The user walks to a cafe; the background probe succeeds, and the client moves back to WebSocket, resuming from seq 23 without a visible gap.
Pitfalls
- Treating
openas success (middleboxes that accept then silently drop frames). - Falling back permanently on a single error, which quietly moves most users to the most expensive transport after one bad deploy.
- Separate code paths per transport that drift apart (different auth check, different message format).
- No dedupe on the client, so every fallback shows a duplicated message.
Draft a reconnection strategy for mobile clients connecting to a WebSocket real-time service that enforces server rate limits and must minimize battery usage. Include exponential backoff with jitter, maximum retry windows, progressive decay, fallback strategies to polling, and metrics to determine when to escalate or notify users.
Sample Answer
Direct answer
Reconnect with capped exponential backoff with full jitter (wait a random time between zero and a limit that doubles each failure), because it spreads a crowd of clients out so they do not all hit the server at once. Always obey the server's rate-limit signal (Retry-After) as a floor. Retry aggressively only for the first couple of minutes, then decay progressively to long intervals, and stop active retrying while the app is backgrounded or the OS says there is no network, relying on platform push to wake it. Reset the backoff immediately on real events (network change, app foregrounded, push received). If the network works but WebSockets keep failing, fall back to polling and probe back periodically. Measure time-to-reconnect, attempts per successful connect, and fallback share so you know when to tell the user and when to page someone.
Key terms
- Exponential backoff: after each failure, wait longer (1 s, 2 s, 4 s, ...), capped at a maximum.
- Jitter: randomness added to the wait. Full jitter picks uniformly between zero and the backoff limit.
- Thundering herd: many clients retrying in lockstep, for example after a server deploy disconnects them all at once.
- Rate limit: the server accepts at most N new connections per second and rejects the rest (HTTP 429 during the handshake, often with a
Retry-Afterheader saying how long to wait).
1. Backoff with jitter
For the n-th consecutive failure (starting at 0), with base 1 s and cap 60 s:
delayn=uniform(0, min(60, 1⋅2n)) seconds waitn=max(delayn, RetryAfter)- The server's
Retry-Afteralways wins if larger: the server knows its own load. - A deploy that deliberately closes connections should send a close reason telling clients to spread their reconnect over a window (for example "reconnect within 30 s"), so even the first attempt is jittered.
2. Maximum retry windows and progressive decay
| Phase | Condition | Behaviour |
|---|---|---|
| Fast | First 2 minutes of disconnection, app in foreground | Backoff as above, cap 60 s |
| Decay | 2 to 30 minutes | Cap raised to 5 minutes |
| Dormant | Over 30 minutes, or app backgrounded | No timed retries; reconnect only on a trigger |
| Offline | OS reports no network | No attempts at all until connectivity returns |
Why these particular cutoffs: most transient drops (a Wi-Fi to cellular handoff, a brief tunnel or elevator, a cell-tower handover) resolve in well under two minutes, so it is worth retrying hard during that window. A disconnection that outlasts 30 minutes is more likely a real outage, a backgrounded app, or the user out of range for a while, so retrying every 60 seconds and spending battery on mostly-failed guesses stops paying for itself; stretching the interval to 5 minutes still reconnects within a bounded time if no trigger ever fires, while costing far less battery over a long dormant stretch. These are starting points to tune from the metrics in section 4, not fixed constants.
Triggers that reset backoff to attempt 0: the OS reports a network change, the app comes to the foreground, a push notification arrives (APNs, the Apple Push Notification service, or FCM, Firebase Cloud Messaging), or the user taps retry. These are moments when success is actually likely, so they are worth a battery wake-up; a timer firing in a tunnel is not.
Battery reasoning: each attempt wakes the cellular radio, which then stays in a high-power state for a few seconds. The design minimises attempts that are likely to fail (offline, backgrounded, server shedding load) and concentrates attempts where success is likely.
3. Fallback to polling
- If attempts fail while ordinary HTTPS requests to the same host succeed (for example the handshake is rejected or the socket never delivers the first server message), the network is probably blocking WebSockets rather than being down.
- After 3 such failures, switch to long polling (repeatedly issuing a plain HTTP request that the server holds open until it has something to send, then the client immediately opens the next one) or SSE (Server-Sent Events, a one-way stream of events over a single long-lived HTTP response) for the session, and send client messages as
POSTs. Carry over the same sequence-number resume logic used on WebSocket reconnect: every message the server would have pushed also carries a per-connection sequence number, so on reconnect, or on a switch to a fallback transport, the client reports the last number it saw and the server resumes from there instead of replaying everything or leaving a gap. - Probe WebSocket again every 10 minutes in the background while in fallback, and remember the verdict per network so the next launch on that network does not pay the detection cost again.
- Do not fall back when the failure is a 429 or 503: that means the server is overloaded, and switching to polling adds more requests.
4. Metrics, and when to escalate or notify
Client-side metrics (sampled, sent when connected):
- Time from disconnect to reconnect (p50, p99: 50th and 99th percentile).
- Attempts per successful connection.
- Share of sessions in fallback mode, by network type and carrier.
- Rate-limit responses received.
- Radio wake-ups per hour attributable to reconnect logic.
Notify the user (in the foreground only):
- Show a quiet "Reconnecting..." indicator after 3 s disconnected, not immediately (most drops heal in a second or two).
- After 30 s, show "Offline. Messages will send when you're back online" and queue outgoing messages locally.
- Never show a modal error for connectivity.
Escalate to the on-call engineer (fleet-wide server alerts):
- Reconnect success rate across all clients drops below a threshold for 5 minutes.
- p99 time-to-reconnect exceeds its SLO (service-level objective).
- Fallback share jumps (a new network or proxy is blocking WebSockets).
- 429 volume stays elevated after a deploy has finished, meaning clients are not spreading out.
Worked example: why jitter matters
This runnable simulation models 100,000 clients disconnected at the same instant by a server deploy, with the server accepting at most 5,000 handshakes per second (so 20 s is the theoretical best).
import random, heapq
from collections import Counter
def simulate(strategy, clients=100_000, capacity_per_s=5_000, base=1.0, cap=60.0, seed=7):
"""All clients lose their connection at t=0 (e.g. a server deploy).
The server accepts at most capacity_per_s handshakes in each 1-second window;
extra attempts are rejected (HTTP 429) and the client schedules a retry."""
rng = random.Random(seed)
def delay(n): # n = number of failed attempts so far
if strategy == "fixed_1s":
return 1.0
exp = min(cap, base * 2 ** n)
if strategy == "exp_no_jitter":
return exp
if strategy == "exp_full_jitter":
return rng.uniform(0, exp)
events = [(delay(0), i, 0) for i in range(clients)] # first attempt
heapq.heapify(events)
per_second = Counter()
accepted = Counter()
attempts = 0
connected = 0
t_99 = None
while events:
t, cid, fails = heapq.heappop(events)
sec = int(t)
per_second[sec] += 1
attempts += 1
if accepted[sec] < capacity_per_s:
accepted[sec] += 1
connected += 1
if t_99 is None and connected >= 0.99 * clients:
t_99 = t
else:
heapq.heappush(events, (t + delay(fails + 1), cid, fails + 1))
return max(per_second.values()), attempts, t_99
for s in ["fixed_1s", "exp_no_jitter", "exp_full_jitter"]:
peak, attempts, t99 = simulate(s)
print(f"{s:16s} rejected handshakes={attempts - 100_000:7d} "
f"attempts per client={attempts / 100_000:.2f} 99% connected at t={t99:.1f}s "
f"peak attempts in one second={peak}")
Output:
fixed_1s rejected handshakes= 950000 attempts per client=10.50 99% connected at t=20.0s peak attempts in one second=100000
exp_no_jitter rejected handshakes= 950000 attempts per client=10.50 99% connected at t=903.0s peak attempts in one second=100000
exp_full_jitter rejected handshakes= 307225 attempts per client=4.07 99% connected at t=38.2s peak attempts in one second=124571
How to read it:
- Fixed 1 s retries recover in the theoretical minimum time but make each client try 10.5 times on average: 950,000 wasted handshakes the server must reject, and 10 radio wake-ups per phone.
- Exponential backoff without jitter is the worst of both: clients stay in lockstep, so each retry round is again 95,000 clients hitting a server that admits 5,000, and the rounds drift further apart as the delay grows. It takes 903 s to reconnect 99%.
- Full jitter reconnects 99% of clients in about 38 s with about 4 attempts per client and a third of the rejected handshakes of the other two.
The printed peak column gives a sharper, slightly counter-intuitive finding: full jitter's busiest single second (124,571 attempts) is actually higher than either lockstep strategy's (100,000 each). Every strategy's entire first wave of 100,000 clients tries within about a second of the disconnect either way, spread across [0 s, 1 s) with full jitter, or bunched at exactly t = 1.0 s with no spread at all when there is no jitter, since that delay is then a fixed number, not a random one. Jitter does not shrink that unavoidable first wave. What it changes is whether the wave re-forms: the two lockstep strategies give every rejected client the exact same next delay, so the roughly 95,000 clients turned away in round one all land on the same later second and get rejected together again, rebuilding a wave close to 100,000 on every round; full jitter spreads that same 95,000 thinly enough across a widening window that no later round ever rebuilds a wave anywhere near that size, which is the real reason it reaches 99% connected in 38 s instead of 903 s. The practical conclusion still holds: the server should spread the disconnects itself during a deploy rather than rely on client backoff alone to smooth that first, unavoidable wave.
Pitfalls
- Backoff that never resets, so a user returning to good coverage waits minutes.
- Retrying while offline or backgrounded, burning battery for nothing.
- Falling back to polling when the server is overloaded, which increases load.
- Treating every close as a failure: a server-initiated close with a reason should drive behaviour (for example "go to another region" or "reconnect in 20 s").
Unlock Full Question Bank
Get access to all 14 Real-Time and Streaming System Design interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.