Real-Time and Streaming System Design Questions

Direct answer

"Older clients ignore it" is an assumption to prove, not to trust, because some client decoders reject unknown fields by default. The safe plan: check every live client decoder against the new message first, make the server emit the field only to clients that say they understand it (capability negotiation), put emission behind a server-side feature flag (a switch stored on the server, outside your normal deploy, that turns a code path on or off for chosen users without shipping new code), ramp from 1% of connections upward while watching client parse errors and disconnects broken down by client version, and roll back by turning the flag off, which takes seconds and needs no deploy.

Why an "optional" field can still break live clients

A backward-compatible change is one old readers can process unchanged. Adding a field is backward compatible only if every reader is a tolerant reader (it ignores fields it does not know). Concrete counter-examples:

• Kotlin's kotlinx.serialization JSON decoder throws a SerializationException on an unknown key unless the app configured ignoreUnknownKeys = true; the default is false.
• A client that validates messages against a JSON Schema with additionalProperties: false rejects the message.
• Hand-written parsers that switch on the number of fields, or a client that re-serialises and forwards the message to a strict service.
• Size limits: a field that pushes messages over a client's frame or buffer limit.

Protobuf (Protocol Buffers, a binary message format where each field is numbered and a reader simply skips numbers it does not recognise) readers do keep unknown fields, and most JSON libraries ignore them by default, so the risk is usually a minority of clients. On mobile that minority can live for years, because old app versions stay installed.

The rollout plan

1. Inventory and test the readers (before any code ships)

• List every consumer of the real-time channel: each mobile app version still above a small usage threshold, web clients, partner SDKs, internal services that subscribe.
• Run a compatibility test: feed a recorded production message plus the new field into each client version's decoder (automated contract tests, meaning tests that check a real message against each client's real parser rather than a mock, in the CI (continuous integration) pipeline, plus a device lab run, physical or emulated devices running the oldest supported app builds, for anything CI cannot cover). The output is a list of versions that are safe, and versions that are not.

2. Capability negotiation

• Clients send a hello on connect with their app version and a list of capabilities, for example caps: ["msg.v2.priority"].
• The server stores the capabilities on the connection and includes the new field only for connections that declared it. Clients that did not declare it get the old shape forever, so an unfixable old build is never exposed.
• New client builds ship first, declaring the capability and reading the field: this is the readers before writers principle, deploy every consumer that can safely handle the new shape before anything produces it, so nothing ever receives data it cannot yet parse. The field starts flowing only after the server side is switched on.

3. Backward-compatible schema change

• The field is optional with a documented default meaning when absent; never repurpose or rename an existing field in the same change.
• Bump the schema's minor version in the registry (a central service that stores every version of every message schema, so a decoder can look up exactly what shape a given version is meant to have) or schema file; keep the old version valid.
• The server must also accept the old shape from clients (if the field ever travels upstream).

4. Feature flag and gradual ramp

The flag controls whether the server emits the field to capable connections. Ramp by connection hash (compute a stable number from each connection's ID, for example 0 to 99, and include it in the rollout only once the ramp percentage passes that number) so a given client stays in or out consistently across the whole step instead of flipping on every reconnect: 1% for an hour, then 5%, 25%, 50%, 100%, with each step gated on the metrics below. Prefer ramping by region or tenant only if the field's content itself varies by them.

5. Monitoring

Watch these, each split by client version and platform, comparing flag-on and flag-off cohorts:

• Client-reported parse or validation errors (clients should report the message type and version, not the payload).
• Server-side unexpected close codes: 1007 (invalid payload data) and 1003 (unsupported data) are the standard codes a strict client may send; also abnormal 1006 closes and reconnect rate.
• Delivery acknowledgements per message and time-to-ack.
• Crash-free session rate for mobile apps (a strict decoder can crash rather than error).

6. Rollback

• Flag off stops emission immediately to all connections; no deploy, no client update.
• Because readers were shipped first and the field is optional, rolling the flag back never leaves a client needing the field.
• Keep the flag for at least one full app-release cycle before deleting it.

Worked example

1,000,000 clients are connected. Testing finds that one Android build, 8% of the fleet, uses a strict decoder.

• If you skipped capability negotiation and turned the field on for everyone, 80,000 clients would fail to parse every message.
• With only the flag and no negotiation, the 1% step hits 1,000,000 x 1% = 10,000 connections, of which 10,000 x 8% = 800 fail. That spike is visible within minutes on the per-version error metric (the 8% failure rate on that one Android build stands out immediately, even though it is a rounding error, 0.08%, in the fleet-wide number), and flipping the flag off then takes seconds. But the 1% step is defined to run for a full hour before promoting to 5%, so unless someone is watching the dashboard live and reacts early, those 800 users spend up to that hour on a build that cannot parse the new field: rolling the flag back stops new failures, it does not undo the ones that already happened.
• With capability negotiation, that build never declared the capability and never receives the field: 0 failures, and the ramp exists only to catch what testing missed.

Pitfalls

• Trusting "optional means safe" without testing the actual decoders.
• Monitoring aggregate error rate only: 800 failures in a million connections is 0.08%, invisible in a global graph but 100% for that build.
• Tying the change to a server deploy so rollback means another deploy.
• Removing the old shape too early for clients that never upgrade.

Direct answer

Use the edge for the parts of a real-time feature that are per-connection and stateless: terminating the WebSocket (a long-lived, two-way connection a client keeps open with a server, so the server can push data without being asked) close to the user, verifying the login token, absorbing heartbeats, and fanning a message out to the sockets that happen to sit on that edge node. Keep anything that needs one agreed answer across the whole world (the authoritative presence record a friend in another continent reads, the durable notification inbox, per-user ordering, anything that moves money or permissions) in a central region or in a single designated owner object. My default is "edge terminates and aggregates, the centre decides", and I only move a decision to the edge when being briefly wrong is harmless and I can show the latency win is in the part of the path the edge actually removes.

The vocabulary, in one pass

• Edge / PoP (point of presence): a data centre run by a CDN (content delivery network) or cloud provider close to users, often dozens of milliseconds nearer than your main region. Edge functions are small programs the provider runs there on each request (Cloudflare Workers, CloudFront Functions, Lambda@Edge). Regional edge nodes are fewer, bigger sites between the PoPs and your origin.
• Presence: the "online / away / offline" dot next to a user, usually derived from whether they have a live connection and have sent a recent heartbeat (a small "still here" message every N seconds).
• WebSocket: a long-lived, two-way connection upgraded from an HTTP request, so the server can push without the client asking.
• Cold start: the delay the first time a function runs in a place where no warm copy exists (loading code, creating a runtime). Warm-up is everything else that is slow on first use: TLS sessions to origin (the encrypted-connection setup with a server, expensive to redo from scratch each time), connection pools (a set of connections to origin kept open and reused instead of opened fresh per request), caches.
• Fan-out: delivering one event to many recipients.

When the edge actually helps

A latency budget has three parts: client to nearest server (first mile), server to server (backbone), and processing. The edge only shortens the first mile and only removes backbone hops for work that can finish locally.

How I would build it

flowchart LR
  C[Phone or browser] -->|WebSocket, TLS| E[Edge PoP: verify token, hold socket, absorb heartbeats]
  E -->|state changes only| P[Presence owner per user: central region or one owner object]
  P --> B[Pub/sub bus in region]
  N[Notification service + durable inbox] --> B
  B -->|one copy per PoP| E
  E -->|local fan-out| C

1. Edge layer. The client connects to the nearest PoP. The edge verifies the token, keeps the socket open, and keeps a small in-memory table: user_id -> last_heartbeat. It forwards only transitions (online to offline, or a status change) to the presence owner, not every heartbeat.
2. Presence owner. For each user there is exactly one logical owner: a single process, wherever it happens to run, that is the one agreed-upon holder of that user's authoritative state, so two different answers for the same user can never coexist. It holds that state with a TTL (time-to-live: the record expires if not refreshed). The owner can be a sharded service (one logical service split across many machines, each owning a slice of the users, so no single machine has to hold all of them) in the home region, or a per-entity coordination object. Cloudflare Durable Objects are one concrete, illustrative example of the latter, not the only way to build one: each object has a globally unique name, is provisioned close to where it is first requested, and with the WebSocket Hibernation API (a feature that lets the object's process be unloaded from memory to save cost while its open sockets stay connected at the edge, then reloaded on the next message) it can be evicted from memory while its clients stay connected (its in-memory state is reset, so anything important must be in its storage).
3. Notifications are written centrally first (so they survive restarts and can be re-read after reconnect), then published to the pub/sub bus (a publish/subscribe channel: publishers send once and every interested subscriber gets a copy, with no need to know who is listening), which delivers one copy per edge node that has an interested subscriber, and the edge does the last-hop fan-out.
4. Reconnect goes to whichever PoP is nearest now; the client sends its last seen notification sequence number and the centre replays from the inbox. The edge never has to be the source of truth for "what did I miss".

Worked example (assumed inputs, arithmetic shown)

Inputs I would state out loud: 2,000,000 users online at peak, heartbeat every 30 s, average session 20 minutes.

• Heartbeats reaching the system: 2,000,000 / 30 = 66,667 per second.
• Presence transitions: each session produces 2 (come online, go offline). Sessions ending per second = 2,000,000 / 1,200 s = 1,667, so transitions are about 2 x 1,667 = 3,333 per second.
• If the edge absorbs heartbeats and forwards transitions only, the central presence tier sees 66,667 / 3,333 = 20x fewer writes. That is the difference between a large sharded presence cluster and a modest one, and it also cuts backbone traffic.
• Latency (illustrative round-trip times, not measurements): a user 20 ms from a PoP and 150 ms from the origin region needs about 3 round trips to open a socket (TCP handshake, TLS 1.3 handshake, HTTP upgrade to WebSocket). Edge termination: 3 x 20 = about 60 ms; origin-only: 3 x 150 = about 450 ms. A delivered notification still pays the backbone once (centre to PoP), so the edge does not make a cross-continent push faster; it makes connect and reconnect faster and cheaper.

That last point is the most common mistake: teams expect "edge" to make every push feel instant, but a message that originates centrally still travels the same distance.

Trade-offs on each axis

Consistency

Edge nodes see only their own connections. If a user has a phone on one PoP and a laptop on another, each PoP alone would get presence wrong, which is why the per-user owner merges them. Accept eventual consistency (all readers agree after a short delay) for presence, with an explicit bound: "a friend sees you go offline within 2 heartbeat intervals". Never let the edge be the last word on anything that has to be linearizable (every reader sees the latest write immediately), such as "has this notification been read" or a permission change.

Routing complexity

Clients land on the nearest PoP, but messages for a user must reach wherever that user is connected now. You need a connection registry (user_id -> PoP / node) or a subscription model where each PoP subscribes to the topics its sockets care about. Topic subscription is simpler and survives PoP moves; a registry allows direct sends but must be updated on every reconnect and goes stale during failover. I choose topic subscriptions keyed by user or room.

Cold starts and warm-up

• Isolate-based edge runtimes (many lightweight, isolated functions sharing one already-running process, the model Cloudflare Workers uses) start in a millisecond or two, much faster than container-based functions (each of which needs its own full, heavier sandboxed environment started from scratch), so per-request cold start is rarely the problem for the handshake path.
• Some edge runtimes cannot do this job at all: CloudFront Functions, for example, are sub-millisecond with no network access, so they suit token checks and header rewrites, not holding a socket or calling a presence service.
• The real warm-up cost in real time is after a deploy or a PoP failover: hundreds of thousands of clients reconnect at once, every new edge instance opens fresh connections to the origin, and hibernated objects rehydrate state from storage. Mitigate with jittered client backoff (each client waits a randomised delay before reconnecting, so a mass of clients does not retry in one synchronised wave), staged rollouts per PoP, and pre-opened origin connection pools.

Operational cost

• Pricing models built for short HTTP requests can punish long-lived sockets (duration billing, per-message billing). Model cost per connected user per month before committing.
• Debugging across hundreds of PoPs needs centralised logs and a per-connection trace ID; "works in my region" bugs are common.
• Two runtimes (edge and central) means two deployment pipelines and a compatibility contract between them. Budget for it.
• Data residency: some regulations restrict where personal data is processed. Presence and message content may flow through PoPs in other jurisdictions unless you pin them.

Safe at the edge vs must be central

The test I apply: if two edge nodes made this decision independently and disagreed, would anyone be harmed or would data be lost? If yes, it is central.

Pitfalls that mark a weaker answer

• Putting presence truth in each PoP and then discovering that multi-device users flicker between online and offline.
• Treating edge as a latency cure for pushes that originate in one region.
• Forgetting the reconnect storm: the edge makes connections cheap to open, which makes a synchronised reconnect larger, not smaller.
• Running authorisation of recipients at the edge from a stale cache, so a user removed from a room keeps receiving messages until the cache expires. Keep that TTL short or push invalidations.

What would flip the recommendation: a single-region user base (the first-mile saving disappears, so terminate sockets in the region and skip the edge tier), or a feature whose state is naturally local to one place, such as a live room where everyone is in one city, where a per-room owner object at the edge can be the authority.

Direct answer

An ABR (adaptive bitrate) algorithm runs in the player before every segment download and picks one rendition from the ladder (the same video encoded at several bitrates). It decides from two signals: measured throughput (how fast recent segments downloaded) and buffer level (how many seconds of video are downloaded but not yet played). For mobile networks with fluctuating throughput I would ship a hybrid: start on a low rung for fast startup, estimate throughput conservatively, scale how much of that estimate you trust by the buffer level, climb one rung at a time, and drop immediately when a download threatens to empty the buffer. Validate it with real-user metrics for startup time, rebuffering ratio, delivered bitrate and switch rate, in an A/B test.

How ABR algorithms decide

Players in production typically mix both signals. dash.js (a widely used open-source browser video-player library for DASH streaming), for instance, ships a throughput rule and a BOLA rule and switches between them by buffer level.

The three goals conflict. Startup time wants a small first segment. Quality wants high rungs. Rebuffering (the video freezing because the buffer ran dry) wants low rungs. On mobile, rebuffering is the one viewers punish most, so it gets priority.

Proposed strategy and player heuristics

1. Startup: request the lowest or second-lowest rung for the first segment, or a rung chosen from the historical median throughput for this network type and carrier if you have it. A 4 s segment at 0.4 Mbps is 1.6 Mbit, about 1 s on a 1.5 Mbps link. The same segment at 3 Mbps is 12 Mbit, 8 s on that link.
2. Conservative estimate: use the smaller of the last segment's throughput and the harmonic mean of the last three. The harmonic mean is dominated by slow samples, so one lucky fast download does not cause an overshoot.
3. Buffer-scaled safety factor: trust 70% of the estimate when the buffer is under 10 s, 90% between 10 and 20 s, 110% above 20 s (a full buffer can absorb a slightly optimistic choice).
4. Drain check: only choose a rung if its predicted download time would not pull the buffer below a reserve of about 4 s, with a small extra allowance of half a segment's playback time so the check does not reject a rung over a one- or two-second margin right at the boundary. In code that reads fetch_time <= max(buf - 4, 0) + SEG * 0.5: the buffer is allowed to be predicted down to (4 s minus half a segment) before a rung is rejected, not down to exactly 4 s. Without that allowance, a buffer sitting exactly at the 4 s reserve would reject every rung except an instant download and get stuck on the lowest rung until the buffer grew past 4 s on its own.
5. Climb one rung per segment; drop as far as needed at once. Asymmetric switching avoids oscillation upward and stalls downward.
6. Abandon slow downloads: if a segment in flight will clearly not finish before the buffer empties, cancel it and fetch the lowest rung.
7. Cap by device: never request resolution above the screen; respect a cellular data-saver setting.
8. Ladder design matters as much as the algorithm: on mobile, include a rung low enough for bad cells (for example 0.2 Mbps). No algorithm can avoid a stall when the network is slower than the lowest rung.

Worked example: a simulation you can run

The program below plays 5 minutes of video (75 four-second segments) over 200 synthetic mobile throughput traces, each pinned by its seed. The trace switches every 20 s between five network conditions, including a deliberately harsh 0.3 Mbps state. It compares a naive rate-based player (90% of the last segment's throughput) with the hybrid above, on two ladders.

import random

LADDER = [0.4, 0.8, 1.6, 3.0, 4.5]   # Mbps renditions (see the walk-through below for one real decision)
SEG = 4.0                             # seconds of video per segment
MAX_BUF = 30.0                        # player stops fetching above this
N_SEGS = 75                           # 5 minutes of video

def make_trace(seed=7, secs=900):
    """Mobile-like throughput in Mbps, one value per second: good/bad phases."""
    rng = random.Random(seed)
    trace, level = [], 3.0
    for t in range(secs):
        if t % 20 == 0:                       # regime change every 20 s
            level = rng.choice([0.3, 0.8, 2.5, 5.0, 8.0])
        trace.append(max(0.2, level * rng.uniform(0.5, 1.3)))
    return trace

def download(trace, t, bits_mbit):
    """Return seconds to fetch bits_mbit starting at wall time t."""
    start = t
    while bits_mbit > 1e-9:
        sec = int(t)
        rate = trace[sec % len(trace)]
        room = (sec + 1) - t              # time left in this 1-second slot
        if bits_mbit <= rate * room:
            t += bits_mbit / rate
            bits_mbit = 0
        else:
            bits_mbit -= rate * room
            t = sec + 1
    return t - start

def rate_based(est, buf, last):
    ok = [r for r in LADDER if r <= 0.9 * est]
    return ok[-1] if ok else LADDER[0]

def hybrid(est, buf, last):
    if last is None:
        return LADDER[0]                      # fast start on the lowest rung
    safety = 0.7 if buf < 10 else (0.9 if buf < 20 else 1.1)
    pick = LADDER[0]
    for r in LADDER:
        fetch_time = r * SEG / est            # predicted download time
        # take r only if throughput supports it AND fetching it would not
        # drain the buffer below a 4 s reserve
        if r <= safety * est and fetch_time <= max(buf - 4, 0) + SEG * 0.5:
            pick = r
    i_last = LADDER.index(last)
    if LADDER.index(pick) > i_last + 1:       # climb at most one rung per segment
        pick = LADDER[i_last + 1]
    return pick

def simulate(policy, trace, first_guess=3.0):
    t, buf, stall, startup = 0.0, 0.0, 0.0, None
    last, bits, switches, history = None, [], 0, []
    for i in range(N_SEGS):
        if buf > MAX_BUF - SEG:               # buffer full: idle, playback drains
            wait = buf - (MAX_BUF - SEG)
            t += wait; buf -= wait
        if not history:
            est = first_guess
        elif policy is rate_based:
            est = history[-1]                 # naive: last segment only
        else:                                 # conservative: min(last, harmonic mean of last 3)
            hm = len(history[-3:]) / sum(1 / x for x in history[-3:])
            est = min(history[-1], hm)
        r = policy(est, buf, last)
        dt = download(trace, t, r * SEG)
        history.append(r * SEG / dt)          # measured throughput of this segment
        t += dt
        if startup is None:
            startup = dt                      # playback starts after segment 1
        else:
            if dt > buf:
                stall += dt - buf; buf = 0.0
            else:
                buf -= dt
        buf += SEG
        if last is not None and r != last:
            switches += 1
        last = r; bits.append(r)
    return startup, stall, sum(bits) / len(bits), switches

N_SESSIONS = 200
for ladder in ([0.4, 0.8, 1.6, 3.0, 4.5], [0.2, 0.4, 0.8, 1.6, 3.0, 4.5]):
    LADDER = ladder                           # policies read the global ladder
    print("ladder", ladder)
    for name, p in [("rate-based", rate_based), ("hybrid", hybrid)]:
        rows = [simulate(p, make_trace(seed)) for seed in range(N_SESSIONS)]
        mean = lambda k: sum(r[k] for r in rows) / N_SESSIONS
        stalled = sum(1 for r in rows if r[1] > 0) / N_SESSIONS
        print(f"  {name:10s} startup={mean(0):5.2f}s rebuffer={mean(1):5.2f}s "
              f"stalled_sessions={stalled:.0%} bitrate={mean(2):4.2f}Mbps "
              f"switches={mean(3):4.1f}")

Walking through one real decision

Running just the hybrid policy on trace seed 7 (the first ladder above) and printing what happens inside the loop of step 4, at the second segment the buffer sits at exactly buf = 4.00 s, the just-established reserve floor, with est = 3.146 Mbps and safety = 0.70 (buffer under 10 s). The candidate loop evaluates:

• 0.4 Mbps: fetch_time = 0.51 s, within the 0.70x bandwidth budget, and 0.51 <= max(4.00-4,0) + 2.0 = 2.0, so it passes.
• 0.8 Mbps: fetch_time = 1.02 s, also passes both checks.
• 1.6 Mbps: fetch_time = 2.04 s, passes the bandwidth check but fails the drain check (2.04 > 2.0), so it is rejected.

The loop keeps the last rung that passed both checks, so it picks 0.8 Mbps. This is exactly where the half-segment slack in step 4 earns its keep: the strict, no-slack version of the rule (fetch_time <= buf - 4 = 0) would have rejected every rung except an instant download, forcing the player back onto the lowest rung (0.4 Mbps) even though 0.8 Mbps finishes in almost a second and the buffer starts refilling immediately afterward. The slack lets the player climb off the floor without letting it pick 1.6 Mbps, whose 2.04 s fetch really would eat too far into the reserve.

Output:

ladder [0.4, 0.8, 1.6, 3.0, 4.5]
  rate-based startup= 7.25s rebuffer= 7.60s stalled_sessions=60% bitrate=2.37Mbps switches=23.3
  hybrid     startup= 1.93s rebuffer= 5.90s stalled_sessions=53% bitrate=2.16Mbps switches=25.2
ladder [0.2, 0.4, 0.8, 1.6, 3.0, 4.5]
  rate-based startup= 7.25s rebuffer= 5.11s stalled_sessions=46% bitrate=2.27Mbps switches=24.4
  hybrid     startup= 0.97s rebuffer= 3.19s stalled_sessions=32% bitrate=2.00Mbps switches=27.9

What it shows, and what it does not:

• The hybrid cuts mean startup from 7.25 s to 1.93 s (0.97 s with the lower rung), because it begins on the lowest rung instead of trusting a 3 Mbps guess.
• It reduces rebuffering (5.90 s vs 7.60 s mean; 53% vs 60% of sessions stalled) and gives up some bitrate (2.16 vs 2.37 Mbps). That is the trade you want on mobile.
• Adding a 0.2 Mbps rung helps both players more than the algorithm change alone: the hybrid's stalled sessions drop from 53% to 32%. The harsh state regularly delivers less than 0.4 Mbps, which no choice of rung on the first ladder survives.
• It increases switching (25.2 to 27.9 vs 23.3 to 24.4). The one-rung climb makes it step up often. A production version would add hysteresis (require the estimate to stay above the next rung for two segments before climbing).
• The trace is synthetic and harsher than typical networks, so the absolute numbers are not a forecast. Only the direction of each comparison is the point; real validation needs field data.

Client metrics to validate the strategy

• Startup time: play request to first frame, as p50 and p90.
• Rebuffering ratio: stall time ÷ (stall time + play time), plus stalls per hour and the share of sessions with any stall.
• Delivered quality: time-weighted average bitrate and, better, average perceptual quality (for example the VMAF score of each rung, a 0 to 100 predicted viewer rating).
• Switch rate and switch magnitude (downswitches per minute).
• Exits before video start (the viewer gave up during startup) and playback error rate.
• Context dimensions: network type (cellular or Wi-Fi), carrier, device, region, player version and the experiment arm, so a regression can be found in one slice.
• Algorithm inputs, sampled: throughput estimates, buffer level at each decision and the chosen rung, so you can replay decisions offline.

Validate with an A/B test split by viewer, compare distributions rather than means, and set guardrails: ship only if rebuffering does not rise in any major network or device slice.

Pitfalls

• Tuning on Wi-Fi in the office; the mobile tail is where decisions matter.
• Optimising average bitrate as the headline metric; viewers weigh stalls far more heavily.
• Measuring throughput on tiny segments or cached responses, which overstate bandwidth.

Direct answer

Eventual consistency means that if updates stop, every copy of the data (servers, caches, each user's screen) converges to the same value, but for a while different people can see different values. In real-time apps we accept it because it lets us show changes instantly and keep working during network trouble, instead of waiting for every replica to agree. It is fine where a briefly stale or out-of-order view harms nobody (a like count, a social feed) and unacceptable where a stale read lets someone break a rule that must always hold (spending money that is already gone). Where we accept it, we owe the user two things: a reconciliation process that brings copies back together, and visible indicators when what they see may not be final.

Strong consistency is the opposite promise: once a write is confirmed, every later read anywhere sees it. It costs latency (waiting for agreement between copies) and availability (refusing work when copies cannot reach each other).

The latency versus consistency trade-off

• To show a like instantly, the phone updates its own screen first and tells the server afterwards. Fast, but for a moment the phone and the server disagree.
• To be strongly consistent, the phone must wait until the server (and its replicas) confirm. With a 150 ms round trip to a distant region, every tap feels 150 ms slower, and when the network drops, the tap fails.
• The practical rule: choose per operation, not per app. One app can show likes optimistically and still require a confirmed server round trip for a money transfer.

Acceptable versus unacceptable

The test to apply: does a stale read lead to an irreversible decision that breaks an invariant? If yes, that decision needs strong consistency (a single authoritative check, a transaction, or a lock), even if everything around it stays eventual.

Worked example: a like button

1. Priya taps like on a post showing 41. Her app immediately shows 42 and a filled heart (an optimistic update) and sends like(post_id, request_id).
2. Meanwhile Sam, on another server replica, also likes it; his screen shows 42 too.
3. Both writes reach the server, which stores likes as a set of user IDs, so the true count is 43. It pushes the new count over the real-time channel.
4. Both screens converge to 43 within a second or two. Nobody was harmed by the brief 42.

If the server rejects Priya's like (she was blocked by the author), her app rolls back to 41 and shows a short message. That rollback path is part of the design, not an afterthought.

Designing reconciliation

• Server is authoritative. Clients hold optimistic, pending changes; the server's answer replaces them.
• Use mergeable data shapes where possible. A like stored as "set of user IDs" merges trivially (union), while "count += 1" applied twice after a retry double-counts. Structures designed to merge automatically are called CRDTs (conflict-free replicated data types); a set that only grows is the simplest one.
• Idempotent writes: every client write carries a request ID so a retry does not apply twice.
• Versions on objects: each object carries a version number; the client sends "I edited version 7". If the server is at 8, it either merges or returns a conflict for the client to resolve.
• Catch-up after reconnect: the client sends the last version or sequence it saw; the server returns changes since then, or a fresh snapshot if too much has changed.
• Periodic anti-entropy: a background comparison that repairs replicas that missed an update, for the rare case the real-time channel dropped one.

User-visible indicators for divergence

• Pending state: a sent-but-unconfirmed item shows a clock icon or lighter text ("Sending...").
• Stale data banner: "Offline. Showing data from 10:42" when the real-time connection is down.
• Conflict notice: "This document was changed elsewhere. Review changes" rather than silently overwriting.
• Rollback message: "Couldn't like this post" when an optimistic update is rejected.
• For money, no optimism: show "Processing" until the server confirms; never show a new balance before it is final.

Pitfalls

• Treating the whole app as either eventual or strong instead of deciding per operation.
• Optimistic updates with no rollback path, which leave screens permanently wrong after a rejection.
• Counters implemented as increments without request IDs, which drift upward on retries.
• Hiding divergence entirely, so users act on stale data without knowing it.

Direct answer

A WebSocket is pinned to one server for its whole life, so "stateless WebSocket servers" really means keep only the socket on the gateway, and keep everything else (who is connected where, subscriptions, message history) outside it. For large-scale operations I recommend a combination: gateways that hold connections but no durable state, a routing registry (user to gateway, in an external store such as Redis, an in-memory key-value database fast enough for a lookup on every message) and a pub/sub layer that addresses gateways, not users, with resumable sessions so that a gateway failure costs a reconnect and nothing else. Live connection handoff is useful only on the same host during deploys; across machines, a client reconnect with a resume token is the practical form of handoff.

The four approaches

• (1) Sticky load balancer with state local to the server. The load balancer keeps sending a user to the same server (by cookie or IP hash, routing a client to a server chosen by hashing its IP address, so the same client lands on the same server every time). Everything about the session (subscriptions, unacknowledged messages) lives in that server's memory. To reach a user connected elsewhere, servers must talk to each other directly or broadcast.
• (2) Centralized session store. A store such as Redis holds user -> {gateway_id, connection_id, subscriptions}. Any service that wants to reach a user looks up the gateway, then calls or messages that gateway.
• (3) Stateless servers with tokens plus external pub/sub. Identity comes from a signed token presented at connect, so the gateway needs no session lookup to authenticate. Messages are routed through a pub/sub (publish/subscribe) system, where senders publish to named channels and every subscriber of a channel gets a copy (Redis Pub/Sub, NATS, or similar brokers); gateways subscribe on behalf of their connections.
• (4) Connection handoff across frontends. Moving a live connection from one frontend to another without the client noticing. On the same host this is real and routine: a new process receives the open socket's file descriptor (the small integer handle the operating system uses internally to refer to that open connection) from the old process over a Unix domain socket (a communication channel for processes on the same machine, faster than a network socket because it never has to leave the kernel), using SCM_RIGHTS file-descriptor passing (the specific Linux message type used to hand an open file descriptor from one process to another), which enables zero-downtime restarts. Across hosts, live TCP migration requires kernel-level connection repair (rewriting the TCP connection's state inside a different machine's operating system so packets keep arriving there, which most kernels and networks do not support) and is rarely worth it; the practical cross-host version is a server-initiated reconnect with a resume token.

Comparison

Recommended design for large scale

flowchart LR
  C[Clients] --> LB[TCP load balancer]
  LB --> G1[Gateway A]
  LB --> G2[Gateway B]
  G1 --> R[(Routing registry)]
  G2 --> R
  SVC[Chat or notify service] --> R
  SVC --> PS[Pub/sub: one inbox per gateway]
  PS --> G1
  PS --> G2
  SVC --> DB[(Message store with sequence numbers)]

1. Gateways hold sockets only. On connect: verify the signed token, write user -> gateway_id to the registry with a TTL, and subscribe the gateway (once) to its own inbox channel gw:<id>.
2. Route by gateway, not by user. A sender's service looks up the recipient's gateway and publishes to gw:<id>. The broker then carries a few hundred gateway channels, not millions of user channels, which keeps subscription churn off the broker when users reconnect.
3. Durability lives in the message store, not the pipe. Every message gets a per-conversation sequence number in a durable store before delivery. The pub/sub hop can lose a message; the client notices the sequence gap (or reconnects) and fetches from the store. That makes delivery reliable even though pub/sub is at-most-once (each message delivered zero or one time).
4. Gateway liveness. Each gateway renews a lease (a key that expires automatically unless its owner renews it) every few seconds. If it dies, its lease expires and a sweeper (a background job that scans for and removes expired entries) removes its registry entries, so senders stop publishing into a dead inbox.
5. Deploys use same-host handoff where possible, and drain-with-resume otherwise.

The load balancer is still "sticky" in the only sense that matters (a socket stays on its gateway), but nothing is lost when that gateway goes away. This is approach 3 for identity and message transport, approach 2 for routing, and approach 4 in its same-host form.

Worked capacity example

Assume 10 million concurrent connections and 50,000 connections per gateway (an ESTIMATE to confirm by load test).

• Gateways: 10,000,000 / 50,000 = 200 gateways at full load, plus headroom for failures and deploys.
• Registry memory: about 100 bytes per entry (user ID, gateway ID, timestamps) × 10,000,000 = about 1 GB, small enough for a modest sharded Redis cluster with replicas (extra copies of each shard kept in sync, so one node failing does not lose that data).
• Failure blast radius: one gateway crash disconnects 50,000 clients. If they reconnect with jitter spread over 30 s, the rest of the fleet must accept 50,000 / 30 ≈ 1,667 new connections per second, each needing a TLS handshake, a token check and a registry write. Size for this, and it is why jitter is non-negotiable.
• Broker load: with per-gateway inboxes, the broker holds about 200 subscriptions instead of 10 million.

Trade-offs and pitfalls

• Option 1 looks cheap until a hot server (a celebrity's followers all hashed to one box) cannot be rebalanced without disconnecting users.
• Per-user pub/sub topics create millions of subscribe/unsubscribe operations per reconnect surge and are a common broker-overload cause.
• Treating the registry as truth without TTLs produces "ghost" entries pointing at dead gateways and silently dropped messages.
• Believing pub/sub is durable. Redis Pub/Sub drops messages for disconnected subscribers; reliability must come from sequence numbers plus a durable store.
• Over-investing in cross-host live handoff. Clients must handle reconnects anyway (networks drop), so make reconnect cheap rather than trying to make it never happen.