Real-Time and Streaming System Design Questions

Direct answer

I would build one pipeline with two entry paths. VOD (video on demand: uploaded files) goes through a chunked, parallel batch encode: split the source into pieces, encode every piece of every quality level in parallel, and stitch. Live goes through always-on real-time encoders, one set per stream, running redundantly. Both paths produce the same output: CMAF fragments (Common Media Application Format, a fragmented-MP4 segment format that both HLS, HTTP Live Streaming, and DASH, Dynamic Adaptive Streaming over HTTP, players can use), with keyframes aligned across all qualities, packaged into HLS and DASH manifests. Codec choice follows audience and economics: H.264 for every title (it plays everywhere), H.265/HEVC for high resolutions on devices that decode it, and AV1 only where the bandwidth savings repay its much higher encode cost, which in practice means popular VOD first.

Some terms: transcoding is decoding a source video and re-encoding it at different settings; an ABR ladder (adaptive bitrate ladder) is the set of qualities (for example 1080p at 5 Mbps down to 240p at 300 kbps) a player switches between as bandwidth changes, with each individual quality level called a rung of that ladder; packaging wraps encoded video into segments plus manifests (HLS .m3u8 playlists, DASH .mpd files) that tell the player what exists.

Pipeline

flowchart LR
    U[VOD upload] --> SPL[Split at scene or GOP, group of pictures, boundaries]
    SPL --> Q[Chunk encode queue]
    Q --> W[Encoder workers per codec and rung]
    W --> ST[Stitch and verify]
    L[Live ingest RTMP or SRT] --> LE[Real-time encoders, redundant pair]
    ST --> MEZ[MEZ: mezzanine, the intermediate CMAF segments in object storage]
    LE --> MEZ
    MEZ --> PKG[Packager: HLS and DASH manifests, DRM]
    PKG --> CDN[CDN]

Live ingest typically arrives over RTMP (an older but ubiquitous streaming push protocol) or SRT (Secure Reliable Transport, a UDP-based protocol that tolerates lossy networks better).

Segment duration

• VOD: 6-second segments with a keyframe every 2 seconds. Apple's HLS authoring specification says target durations should be 6 seconds and key frames (IDR frames, which a decoder can start from without earlier data) should be present every two seconds. Longer segments mean fewer requests and better compression; shorter ones let the player switch quality sooner.
• Live, standard latency: 2 to 4 second segments. Players typically buffer about three segments before starting, so latency behind real time is roughly three segment durations plus encode and CDN time.
• Live, low latency: keep ordinary-length segments (a few seconds) but publish each one in sub-second partial chunks (Low-Latency HLS partial segments, or chunked CMAF for DASH), so a player can fetch the start of a segment while it is still being encoded. This keeps request counts and compression close to normal while cutting latency to a few seconds.
• The keyframe interval (GOP, group of pictures) must divide the segment duration and be identical across every rung and codec. A player can only start decoding a segment at a keyframe (a full frame that stands alone; every other frame is stored as a difference from earlier frames). If segments are 6 s and the keyframe interval is 2 s, every rung has a keyframe exactly on every segment boundary, so the player can switch from the 1080p rung to the 480p rung right there. If one rung instead used a 2.5 s keyframe interval, its keyframes would drift away from the 6-second segment lines, so the frame sitting at that boundary in the new rung would not be a keyframe: the player would either show a broken frame for an instant or stall until that rung's next real keyframe, which is the glitch.

Codecs and tracks

Audio: encode once per language (AAC stereo for compatibility, optionally a 5.1 surround track), independent of video. Audio renditions are listed as alternates (the specific manifest syntax is implementation detail, not something to memorize here: HLS calls them EXT-X-MEDIA TYPE=AUDIO groups, DASH calls them audio adaptation sets), so adding a language never re-encodes video.

Subtitles and captions: WebVTT for HLS and DASH (a plain-text subtitle format; IMSC1 text profile where richer styling is needed), delivered as separate segmented tracks; live closed captions (CEA-608/708, the broadcast-TV caption standards, one older and one newer) are carried in the video stream from the encoder rather than as a separate track. None of these format names change the design; they only matter when you implement packaging. Subtitles are tiny and can be added after publication.

Parallelization

• VOD, chunked encoding: split the source into chunks (say 30 to 60 seconds) at keyframe or scene boundaries, and encode each (chunk x rung x codec) as an independent job. A 2-hour film is 120 to 240 chunks per rendition, so wall-clock encode time is bounded by the slowest chunk, not the film's length. The stitcher verifies timestamps and keyframe alignment at chunk boundaries.
• Live: you cannot chunk the future. Parallelize across rungs (each rung or codec on its own encoder process or hardware encoder) and across streams. Run two encoders per important stream in different availability zones (physically separate data centers within one cloud region, each with independent power and networking, so a failure in one does not take out the other), producing identical segment numbers and timestamps, so the packager can fail over without the player noticing.
• Just-in-time packaging: store CMAF segments once and generate HLS or DASH manifests (and apply DRM, digital rights management, encryption) at request time, cached by the CDN. That avoids storing two copies of every segment.

Pre-warming encoders

• Scheduled live events: start encoders (and fill the GPU or hardware-encoder pool) 15 to 30 minutes early, run a test signal through the full chain, and have the ingest endpoint ready before the broadcaster connects.
• Unscheduled live: keep a warm pool of idle encoder instances with images already pulled and codecs initialised, sized to the peak rate of new streams per minute; a cold virtual machine plus container start is far slower than a user will wait at "Go live".
• VOD: pre-warming matters less; queue depth and a priority lane for fresh uploads keep time-to-publish predictable.

Worked example: when does AV1 pay for itself?

All inputs below are assumptions for illustration; substitute your measured bitrates and prices.

Assume a 1080p stream averages 4 Mbit/s in H.264, AV1 needs 30% fewer bits for the same quality, CDN delivery costs 0.02 dollars per GB, and AV1 encoding of a one-hour title costs 20 dollars more than H.264 alone.

Only AV1-capable viewers produce savings, so if half the audience can decode AV1 the break-even doubles to about 3,700 viewing hours. The rule this produces: encode H.264 for everything at upload, then re-encode to AV1 when a title's views cross a threshold. The same logic explains why AV1 for live usually waits for hardware encoders: a live stream cannot amortize a slow encode over future views.

Trade-offs: latency, quality, cost

Three encoder knobs behind the table below: an encoder preset trades encode speed for compression efficiency (a faster preset finishes sooner but needs slightly more bits for the same quality); lookahead frames are future frames the encoder examines before compressing the current one, so it can spend bits more wisely at the cost of extra buffering delay; multi-pass encodes the content twice, once to analyze it and once to compress using what it learned, which is why it is VOD-only (a live stream has no "later" frames to look ahead into).

Pitfalls: misaligned keyframes across rungs (quality switches stutter); re-encoding the audio per video rung (wasted compute and duplicate storage); one fixed ladder for all content (a cartoon needs far fewer bits than a sports match, so per-title ladders save bandwidth); and live encoders without redundancy, where one crashed virtual machine takes the stream down.

Direct answer

Put a single-writer sequencer per instrument at the centre: every update for an instrument passes through one process that stamps it with a gap-free, per-instrument sequence number, which is what "strict ordering" means in practice. From the sequencer, updates go out on UDP multicast (one packet copied by the network switches to every subscriber) inside the data centre, with a second, identical A/B feed on separate network paths so one lost packet is usually recovered from the other feed with no round trip. Missed updates are recovered from retransmission servers holding a 60-second in-memory ring buffer, and from snapshots when a subscriber is too far behind. Slow subscribers never slow the sequencer: they drop and recover. The "at-most-once" requirement (downstream applies each update no more than once, even though the transport itself may redeliver it) is met on the subscriber side, by applying each sequence number at most once.

Terms

• Instrument: a tradable security, such as a stock, option, or futures contract. Every message in this system is scoped to one instrument, which is what makes the system partitionable.
• Order book: the live list of buy and sell orders waiting to trade for one instrument, ordered by price. It is what a matching engine maintains and what this system broadcasts changes to.
• Matching engine: the system that receives buy and sell orders for an instrument and pairs (matches) them into trades, updating the order book.
• Book change: one update to the order book: an order was added, modified (price or quantity changed), deleted (cancelled), or resulted in a trade (matched against another order).
• Ring buffer: a fixed-size buffer that overwrites its oldest entries once full, used here to hold the last 60 seconds of messages for replay without unbounded memory growth.
• Multicast group: a network address that, once a device subscribes to it, causes network switches to deliver a copy of every packet sent to that address, without the sender addressing each recipient individually.
• Hot standby: a second instance that runs the same computation as the primary in real time but does not publish, so it can take over instantly if the primary fails.
• A/B feed / arbitration: sending the same data twice over two independent network paths (A and B); arbitration is picking whichever copy of a given message arrives first and discarding the duplicate.

Requirements and the one that needs interpreting

• Strict ordering per instrument (not globally: ordering across instruments is not required, which is what makes the system partitionable).
• Sub-10 ms tail latency (p99, the 99th percentile) within a region.
• Replay of missed updates for up to 1 minute.
• At-most-once delivery for downstream processing, and 100,000 updates per second.

"At-most-once" normally means "may be lost, never duplicated", while "replay missed updates" means "not lost". Taken together, the sensible reading is: the transport may deliver a message more than once (A/B feeds and replays guarantee duplicates), but downstream processing must apply each update at most once, and gaps must be repairable within 60 seconds. Each subscriber keeps last_applied_seq per instrument and discards anything at or below it. I would confirm this reading with the interviewer; if they truly meant "fire and forget, never replay", the retransmission tier disappears and the design gets simpler, but the order book would silently diverge after any loss.

Components

flowchart LR
  ME[Matching engines] --> SEQ[Sequencers by instrument partition]
  SEQ --> MCA[Multicast feed A]
  SEQ --> MCB[Multicast feed B]
  SEQ --> RING[(Retransmission ring buffer 60 s)]
  SEQ --> SNAP[(Snapshot service)]
  MCA --> SUB[Subscribers]
  MCB --> SUB
  SUB -. gap request .-> RING
  SUB -. resync .-> SNAP
  MCA --> EDGE[TCP fan-out tier for remote subscribers]

1. Ingest: matching engines (which already process orders per instrument) emit book changes (add, modify, delete, trade). Each engine owns a set of instruments.
2. Sequencer: instruments are partitioned (for example by instrument id) across sequencer processes. Within a partition, one thread stamps (instrument_id, seq) and writes the message to the feeds and to the ring buffer. Single-writer means no locks and no coordination on the hot path.
3. Transport: UDP multicast, one multicast group per partition, so a subscriber that only cares about some instruments joins only those groups.
4. Retransmission servers: listen to the feed, store the last 60 seconds, answer "resend instrument X, sequences 1001 to 1010".
5. Snapshot service: maintains the full book per instrument with the sequence number it reflects, so a subscriber that joins late or falls far behind can load the book and continue from the stream.
6. Fan-out tier (TCP or WebSocket) for subscribers outside the data centre that cannot receive multicast.

Why multicast: the bandwidth arithmetic

With 64-byte messages (a compact binary format: message type, instrument id, sequence, price, quantity, side, timestamp):

• One subscriber taking the full feed: 100,000 x 64 bytes = 6.4 MB/s = 51.2 Mbit/s.
• 2,000 subscribers over unicast (a separate copy per subscriber): 51.2 Mbit/s x 2,000 = 102.4 Gbit/s leaving the publisher. That is a hardware problem and a latency problem, because the last copy is sent long after the first.
• Multicast: the publisher sends 51.2 Mbit/s once; switches replicate it. Latency is the same for every subscriber.

Averages understate market data: bursts at the open or on news can be several times the average. I would provision the feed and every subscriber path for at least 5x (500,000 updates per second, 256 Mbit/s), and treat that multiplier as an assumption to confirm with historical peak data.

Sequencing and gap detection

• Each message carries (instrument_id, seq); sequences are per instrument, gap-free, starting at 1 per trading session.
• Packets also carry a partition-level sequence (a per-instrument sequence only reveals a gap when the next message for that same instrument arrives, and if that instrument stays quiet there may be no next message for a long time) so a subscriber detects a lost packet even for an instrument that then goes quiet. A periodic heartbeat per partition carries the latest sequence for the same reason.
• Subscriber logic: receive from A and B, take whichever copy arrives first, discard the duplicate by sequence. If a gap persists beyond a short window (say 1 ms, to absorb A/B skew), request the range from a retransmission server. If the gap is older than 60 s, or the request fails, load a snapshot and replay from its sequence.

Storage for replay

60 seconds x 100,000 updates per second = 6,000,000 messages x 64 bytes = 384 MB. At the 5x burst assumption it is 1.92 GB. This is an in-memory ring buffer, indexed by (partition, seq) to an offset, on at least two retransmission servers. No disk is on the replay path. Separately, the full session is written to durable storage asynchronously for audit and regulatory record-keeping, which is off the latency path.

Latency budget for the p99 under 10 ms

These are allocations, not measurements: each hop gets a histogram and an alert on its p99, so a regression is attributable to a hop. Keeping headroom matters because a gap repair costs one round trip to a retransmission server, and it must fit within the 10 ms target for the p99 to hold. Timestamps at each hop need clock synchronization much finer than the budget; PTP (Precision Time Protocol, hardware-assisted clock sync) is the standard choice.

Backpressure

The sequencer and the multicast feed never wait for any subscriber. Backpressure from one slow reader must not become latency for thousands of others.

• A slow multicast subscriber overflows its own socket buffer (the OS's fixed-size holding area for incoming network data before the application reads it) and loses packets (the OS drops new packets once that buffer is full, since there is nowhere to put them); its gap logic recovers via retransmission, and if it is persistently slow, it is disconnected from retransmission (rate-limited per subscriber) and told to use snapshots.
• In the TCP fan-out tier, each subscriber has a bounded outbound queue. On overflow, the tier either conflates (sends the latest book state for each instrument instead of every intermediate change, acceptable for display clients) or disconnects the subscriber with a "resync from snapshot" instruction (required for trading clients that need every change).
• Retransmission servers are rate-limited per subscriber, so one broken client cannot use the replay tier to hurt everyone else's recovery.

Operational considerations

• Sequencer failover: a hot standby consumes the same inputs and computes the same sequence numbers deterministically, but does not publish. On failure, it takes over publishing from the next sequence. A paused primary is dangerous precisely because it can resume: a garbage-collection pause, a frozen VM, or a network partition can make the standby correctly promote itself while the old primary is still running, and once unpaused the old primary still believes it is the leader and keeps publishing. Fencing (a monotonically increasing leadership epoch in every packet, so subscribers ignore the old primary if it wakes up) prevents two sequencers publishing the same numbers.
• Start of day: sequences reset per session; subscribers load the opening snapshot.
• Monitoring: per-hop p99, gap rate per subscriber, retransmission requests per second, A/B arbitration stats (how often B saved you), snapshot requests.
• Capacity tests with recorded peak days replayed at 5x.

Trade-offs and pitfalls

• Kafka or a similar log broker gives replay and ordering per partition for free, but adds a disk-backed, replicated write before delivery. That is fine for downstream analytics consumers (they can subscribe to a copy) and wrong for the sub-10 ms path.
• Global ordering across instruments would force one sequencer for everything and cap throughput; per-instrument ordering is what the requirement asks for.
• Treating duplicates as the enemy at the transport: A/B feeds deliberately create duplicates; dedup belongs at the subscriber, by sequence.
• What would change the design: if subscribers were mostly across the internet (retail apps), multicast is unavailable and the design becomes a TCP/WebSocket fan-out tree with conflation, and "sub-10 ms" would have to be redefined as within our network.

Direct answer

Eventual consistency means that if updates stop, every copy of the data (servers, caches, each user's screen) converges to the same value, but for a while different people can see different values. In real-time apps we accept it because it lets us show changes instantly and keep working during network trouble, instead of waiting for every replica to agree. It is fine where a briefly stale or out-of-order view harms nobody (a like count, a social feed) and unacceptable where a stale read lets someone break a rule that must always hold (spending money that is already gone). Where we accept it, we owe the user two things: a reconciliation process that brings copies back together, and visible indicators when what they see may not be final.

Strong consistency is the opposite promise: once a write is confirmed, every later read anywhere sees it. It costs latency (waiting for agreement between copies) and availability (refusing work when copies cannot reach each other).

The latency versus consistency trade-off

• To show a like instantly, the phone updates its own screen first and tells the server afterwards. Fast, but for a moment the phone and the server disagree.
• To be strongly consistent, the phone must wait until the server (and its replicas) confirm. With a 150 ms round trip to a distant region, every tap feels 150 ms slower, and when the network drops, the tap fails.
• The practical rule: choose per operation, not per app. One app can show likes optimistically and still require a confirmed server round trip for a money transfer.

Acceptable versus unacceptable

The test to apply: does a stale read lead to an irreversible decision that breaks an invariant? If yes, that decision needs strong consistency (a single authoritative check, a transaction, or a lock), even if everything around it stays eventual.

Worked example: a like button

1. Priya taps like on a post showing 41. Her app immediately shows 42 and a filled heart (an optimistic update) and sends like(post_id, request_id).
2. Meanwhile Sam, on another server replica, also likes it; his screen shows 42 too.
3. Both writes reach the server, which stores likes as a set of user IDs, so the true count is 43. It pushes the new count over the real-time channel.
4. Both screens converge to 43 within a second or two. Nobody was harmed by the brief 42.

If the server rejects Priya's like (she was blocked by the author), her app rolls back to 41 and shows a short message. That rollback path is part of the design, not an afterthought.

Designing reconciliation

• Server is authoritative. Clients hold optimistic, pending changes; the server's answer replaces them.
• Use mergeable data shapes where possible. A like stored as "set of user IDs" merges trivially (union), while "count += 1" applied twice after a retry double-counts. Structures designed to merge automatically are called CRDTs (conflict-free replicated data types); a set that only grows is the simplest one.
• Idempotent writes: every client write carries a request ID so a retry does not apply twice.
• Versions on objects: each object carries a version number; the client sends "I edited version 7". If the server is at 8, it either merges or returns a conflict for the client to resolve.
• Catch-up after reconnect: the client sends the last version or sequence it saw; the server returns changes since then, or a fresh snapshot if too much has changed.
• Periodic anti-entropy: a background comparison that repairs replicas that missed an update, for the rare case the real-time channel dropped one.

User-visible indicators for divergence

• Pending state: a sent-but-unconfirmed item shows a clock icon or lighter text ("Sending...").
• Stale data banner: "Offline. Showing data from 10:42" when the real-time connection is down.
• Conflict notice: "This document was changed elsewhere. Review changes" rather than silently overwriting.
• Rollback message: "Couldn't like this post" when an optimistic update is rejected.
• For money, no optimism: show "Processing" until the server confirms; never show a new balance before it is final.

Pitfalls

• Treating the whole app as either eventual or strong instead of deciding per operation.
• Optimistic updates with no rollback path, which leave screens permanently wrong after a rejection.
• Counters implemented as increments without request IDs, which drift upward on retries.
• Hiding divergence entirely, so users act on stale data without knowing it.

Direct answer

Let the server, not the network, define the order: a single sequencer per conversation assigns gap-free sequence numbers (1, 2, 3, ...) when it durably stores each message, and the broker is partitioned by conversation ID so one conversation's messages flow through one ordered lane. Because the broker is at-least-once (it retries until acknowledged, so it can deliver a message twice), the client treats sequence numbers as the truth: it drops anything it has already shown, holds messages that arrive early in a small reorder buffer, and after a short timer fetches any missing range from the server. On reconnect the client sends the last sequence it saw contiguously and the server replays everything after it. The trade-off is a small hold-back delay whenever a gap appears; I would tolerate a few hundred milliseconds before fetching, never hold the screen hostage, and show the sender's own message instantly as "pending".

Why the broker's order is not enough

• At-least-once means redelivery after a lost acknowledgement. The redelivered copy can arrive after newer messages.
• Several server instances publishing for one conversation can interleave.
• A client that reconnects to a different server gets live messages and replayed messages over two paths at once.

So ordering has to be a property of the data (a sequence number), checked at the edge, not a hope about the pipe.

Server-side strategies

1. Assign seq at write time. One writer per conversation increments last_seq and stores the message in the same atomic step (a database transaction, or, on a message broker, a partition leader: the one server responsible for a given partition, the only one allowed to accept writes for it). Timestamps are not an order: two servers' clocks drift apart by milliseconds or more.
2. Per-conversation partitions. Key broker messages by conversation_id, so every message of a conversation lands on the same partition (one ordered, independently-processed slice of the broker's overall stream, the "one ordered lane" from the direct answer above) and the same consumer (the one process reading that partition and handling its messages in order), which processes them in order. Different conversations spread across partitions and scale out.
3. Idempotent sends. The sender's client attaches client_msg_id (a UUID, a universally unique identifier, a 128-bit random value different clients can each generate independently with negligible chance of collision, generated once and reused on every retry). The server keeps a unique constraint on (conversation_id, client_msg_id), so a retried send returns the original seq instead of creating message 42 twice.
4. Every delivery carries seq, and the server answers "latest seq" in heartbeats and on reconnect.

Client-side strategy: dedupe, buffer, fill gaps

A runnable model of one conversation's client view. The simulated network loses 5% of messages (always including the final one, the hardest case), duplicates 20%, and locally reorders deliveries:

import random

class ConversationView:
    """Client-side reorder buffer for ONE conversation.
    Server assigns seq 1, 2, 3, ... per conversation; the broker may
    duplicate, delay or reorder deliveries (at-least-once)."""

    def __init__(self, fetch_range):
        self.last_shown = 0          # highest seq rendered contiguously
        self.pending = {}            # seq -> message, arrived early
        self.fetch_range = fetch_range
        self.shown = []
        self.dupes_dropped = 0
        self.gap_fetches = 0

    def on_message(self, seq, body):
        if seq <= self.last_shown or seq in self.pending:
            self.dupes_dropped += 1   # at-least-once duplicate: ignore
            return
        self.pending[seq] = body
        self._drain()

    def on_gap_timeout(self, server_latest=None):
        # a gap has persisted too long: ask the server for the missing range.
        # server_latest comes from a heartbeat or reconnect handshake; it is
        # the only way to notice a lost TAIL (nothing later arrives to expose it)
        hi = min(self.pending) - 1 if self.pending else server_latest
        if hi is not None and hi > self.last_shown:
            lo = self.last_shown + 1
            self.gap_fetches += 1
            for seq, body in self.fetch_range(lo, hi):
                self.pending.setdefault(seq, body)
            self._drain()

    def _drain(self):
        while self.last_shown + 1 in self.pending:
            self.last_shown += 1
            self.shown.append(self.pending.pop(self.last_shown))

random.seed(7)
server_log = {s: f"msg-{s}" for s in range(1, 201)}
fetch = lambda lo, hi: [(s, server_log[s]) for s in range(lo, hi + 1)]

# network: every message duplicated with p=0.2, dropped with p=0.05,
# then the whole stream is locally shuffled (reordering within a window of 5)
wire = []
for s, body in server_log.items():
    if random.random() < 0.05 or s == 200:
        continue                                  # lost (always lose the last one)
    wire.append((s, body))
    if random.random() < 0.2:
        wire.append((s, body))                    # duplicate
wire = sorted(wire, key=lambda m: m[0] + random.uniform(0, 5))

view = ConversationView(fetch)
for i, (seq, body) in enumerate(wire):
    view.on_message(seq, body)
    if i % 10 == 9:
        view.on_gap_timeout()                     # timer fires periodically
view.on_gap_timeout(server_latest=200)            # heartbeat: "latest seq is 200"

print("delivered on wire:", len(wire))
print("shown:", len(view.shown), "in order:", view.shown == [server_log[s] for s in range(1, 201)])
print("duplicates dropped:", view.dupes_dropped, "gap fetches:", view.gap_fetches)

Output:

delivered on wire: 240
shown: 200 in order: True
duplicates dropped: 58 gap fetches: 17

What the model demonstrates:

• Duplicates are free to handle: seq <= last_shown is already on screen. The 58 dropped include both network duplicates and late copies of messages already fetched to fill a gap.
• Early arrivals wait in pending until the missing seq appears.
• Gaps are filled from the server after a timer; that is what gap fetches counts.
• A lost final message is invisible to gap detection, because nothing later arrives to reveal it. Only an explicit "latest seq is 200" signal (heartbeat or reconnect handshake) exposes it, which is why the server must send one. The model deliberately loses seq 200 and still shows all 200 in order only because of that heartbeat call.

Dealing with gaps in practice

• Short hold (about 200 to 500 ms): most gaps are just reordering and close on their own.
• Then fetch the range (last_shown, min(pending)) over a normal request. Cap the fetch size; if the gap is enormous, fall back to reloading the conversation's latest page.
• Never block forever: if the fetch fails, render what you have with a visible "loading earlier messages" marker, and keep retrying.
• Permanent gaps: if a seq was assigned but its write failed, the server stores an explicit tombstone for that seq (a stored marker recording that this sequence number will never have a message, unlike an ordinary row) so clients do not wait for a message that will never exist.

Reconciling after reconnect

1. Client reconnects and sends {conversation_id: last_contiguous_seq} for its open conversations (or, for the inbox list, a single per-user cursor: the highest seq across all of that user's conversations this device has fully processed, so the inbox can catch up in one read instead of one per conversation).
2. Server replays everything after each cursor, paged, then switches the connection to live delivery. Live messages that arrive during the replay simply go through the same dedupe and buffer, so the overlap is harmless.
3. Outbound messages written while offline are resent with their original client_msg_id. Any that the server had already stored before the disconnect come back with their existing seq rather than being duplicated.
4. Local echo: a message the user typed is shown immediately at the bottom with a "sending" clock icon. When its seq arrives it is moved to its true position. If someone else's message was sequenced first, the user's message visibly moves up by one: correct, and rare.

Latency versus ordering

For chat the right trade is a bounded hold: humans do not notice 300 ms, but they do notice messages rearranging while they read. For a stock ticker or live game state, where only the newest value matters, I would do the opposite: skip stale sequence numbers and never wait.

Pitfalls

• Tracking only "highest seq seen" instead of "highest contiguous seq": the client then silently skips a gap forever.
• Unbounded pending buffers: cap them, and on overflow reload the page of recent messages.
• Relying on the broker's partition order but letting multiple consumer threads process one partition in parallel, which reintroduces reordering after the broker.
• Forgetting that a lost last message has no gap to reveal it.

Direct answer

I would build the editor on a CRDT (conflict-free replicated data type: a data structure whose copies merge automatically to the same result regardless of the order edits arrive) using a mature library such as Yjs (an open-source CRDT library built for real-time collaborative text editing), rather than hand-building OT (operational transformation: a server-ordered scheme that rewrites concurrent edits against each other). Each open document is served by one document-session server chosen by hashing the document ID; it relays edits over WebSockets, batches them into an append-only update log (a log that only ever has entries added to its end, never edited or removed in place, so replaying it rebuilds history exactly), and periodically compacts the log into snapshots in object storage (a storage service for whole files addressed by key, such as Amazon S3, rather than a filesystem or database). Presence and cursors travel on the same socket but are never persisted. I would roll it out in stages behind flags (feature flags: config switches that turn a new code path on for a chosen subset of traffic without a deploy), starting in shadow mode, so every piece is proven before it carries real documents.

CRDT or OT: the decision

Recommendation: for most teams, a mature CRDT library beats hand-built OT on complexity, latency and developer effort, especially with offline editing in the requirements. What would flip it: an online-only product with an existing, battle-tested OT stack.

Architecture

flowchart LR
  C[Editor client with local CRDT and offline cache] --> LB[Load balancer]
  LB --> DS[Document-session server]
  DS --> RR[(Routing registry doc to server)]
  DS --> LOG[(Update log)]
  DS --> OBJ[(Snapshots in object storage)]
  DS --> AW[Awareness: cursors and presence]
  CMP[Compaction workers] --> LOG
  CMP --> OBJ
  VH[Version history service] --> OBJ

Document sharding

• Documents are sharded by document ID using consistent hashing (a scheme where adding a server moves only a small fraction of documents). A routing registry records which session server currently owns each open document, with a lease so a crashed owner's documents are reassigned within seconds.
• All editors of one document connect to its owner. CRDTs do not strictly require a single server, but one owner per document makes broadcasting, batching writes and access checks simple and keeps a document's update log single-writer.

Broadcasting edits

• A client edit is applied locally, encoded as a small binary update, and sent over the WebSocket. The session server checks the user's permission (editor, commenter, viewer), appends the update to an in-memory buffer, and broadcasts it to the other connected editors.
• Clients batch keystrokes (for example every 50 to 100 ms while typing) so one busy editor sends a few updates per second, not one per character.

Persistence and compaction

• The session server flushes each document's buffered updates to the durable update log every 1 to 2 seconds as one append.
• Clients keep every local update until the server acknowledges it as persisted; on reconnect they resend unacknowledged updates. CRDT merges ignore duplicates, so resending is safe and the 2-second buffer cannot lose acknowledged work.
• Compaction: after about 1,000 updates (or 5 minutes), a worker merges the snapshot plus the log tail into a new snapshot in object storage and truncates the log. Loading a document is then "latest snapshot plus a short tail".

Offline edits

• The client stores the document and pending updates in local storage; in browsers this is IndexedDB, a persistent database built into the browser that survives a page reload. Offline, the user keeps editing; on reconnect, the client and server exchange state vectors (a compact summary of which updates each side already has) and send only what the other side is missing.

Version history

• Named and automatic versions (for example hourly while edited) are pointers to snapshots, with author and time metadata. Restoring a version applies a new edit that makes the document match the old content, so history is never rewritten and other clients merge the restore like any edit.

Presence and cursor sync at scale

• Cursor positions, selections and "who is here" travel as an ephemeral awareness channel on the same socket. They are throttled (a few updates per second per user), never written to the log, and dropped on disconnect.
• For a document with hundreds of viewers, show a capped avatar list and only broadcast cursors of users near the viewer's scroll position.

Merge UX: surfacing conflicts without blocking

CRDTs guarantee every replica converges, but not that the result is what both authors intended (two people rewriting the same sentence produce a merged sentence). So the UI must make concurrent edits visible:

• Presence and cursor cues: colored cursors and name labels show who is editing where, so people avoid collisions in the first place.
• Inline change indicators: text changed by others while you were offline or away is highlighted with the author's color for a short period.
• An accept/revert review panel, not a blocking modal: after an offline session, a side panel lists incoming changes that touched paragraphs you edited; each can be kept or reverted (a revert is a new edit). The user keeps typing while reviewing.

Worked example: why offline edits merge for free

Say a shared meeting-notes document starts as "Notes: TBD". Two editors go offline on different devices. Editor A inserts " - discuss budget" right after TBD, intending "Notes: TBD - discuss budget". Editor B, working from the same starting text, deletes TBD and types "3pm Thursday" in its place, intending "Notes: 3pm Thursday". Neither device has seen the other's edit.

With a plain text diff (no CRDT), reconciling these two edits on reconnect is exactly the git merge-conflict problem: both edits touch the same few characters, so a naive three-way merge either has to guess or ask a human. A CRDT avoids this because every character has a permanent ID, and an edit refers to an ID, not a position. A's insert is stored as "insert ' - discuss budget' after the ID of the last character of TBD". B's edit is stored as two operations against those same character IDs: "delete the characters with TBD's IDs" (each becomes a tombstone, a marker that the character existed and is now gone, rather than being erased outright) and "insert '3pm Thursday' after the ID of the character before TBD". Merging is then just replaying every operation from both devices in ID order: B's delete tombstones the TBD characters without removing their IDs, B's insert places 3pm Thursday where TBD used to be, and A's insert still finds its anchor, the last (now-tombstoned) character of TBD, and lands right after it. Both replicas converge on "Notes: 3pm Thursday - discuss budget", deterministically, regardless of which device's operations are replayed first. Nobody had to choose a winner; both edits survived because they were never really competing for the same position, only for the same anchor, and the CRDT's IDs kept that anchor valid even after B's delete.

Worked capacity example

Assumptions: 5 million documents, 1% open at peak, 3 active editors per open document, 2 batched updates per second per typing editor.

• Open documents: 5,000,000 × 1% = 50,000. Active editors: 50,000 × 3 = 150,000.
• Updates received: 150,000 × 2 = 300,000 per second; each is broadcast to the 2 other editors, so 600,000 outbound messages per second.
• Durable writes: with a flush every 2 s per document, 50,000 / 2 = 25,000 log appends per second, instead of 300,000 if every update were written separately.
• Session servers: at 5,000 open documents per server (an ESTIMATE to load-test), 50,000 / 5,000 = 10 servers, run with headroom.
• Compaction cadence: a document receiving 3 × 2 = 6 updates per second reaches 1,000 updates in about 167 seconds.

Incremental rollout with a testing plan

1. CRDT core, offline test harness. Property-based tests (tests that generate thousands of random inputs and check a rule that must always hold) generate random concurrent edit sequences across simulated replicas with random delivery order, drops and duplicates, and assert all replicas converge to identical content. Run the same harness against the persistence layer (snapshot plus log replays to the same state).
2. Shadow mode, meaning the new sync path runs for real but nothing user-facing depends on it yet: keep the existing save path authoritative. Run the new sync path in parallel for internal documents and compare a hash of the document from both paths on every save; any mismatch is logged with the update sequence for replay.
3. Internal dogfood (the team uses the new path on its own real documents, before any customer does) with the new path authoritative, including deliberate failure drills: kill a session server mid-edit, partition a client for an hour, restore from snapshot. Pass criteria: zero lost acknowledged edits and zero divergence reports.
4. Canary by document (release to a small slice of real traffic first, so a bug affects few documents and is cheap to reverse), starting at 1% of newly created documents, behind a flag with a kill switch (a manual override that immediately disables the new path and reverts to the old one) that turns affected documents read-only and falls back. Watch p99 (99th percentile) edit propagation latency, divergence reports (clients periodically send a hash of their state), snapshot size growth and reconnect rates.
5. Migrate existing documents lazily: convert a document to a CRDT snapshot the first time it is opened after its cohort (the group of documents or users a given rollout stage currently applies to) is enabled, keeping the original as a restorable version.
6. Enable offline editing last, because it produces the longest divergent branches and exercises the merge UX hardest.

Trade-offs and pitfalls

• Tombstone growth: deleted content leaves markers so late-arriving edits can still be placed; very old, heavily edited documents grow. Compaction plus library garbage collection (the CRDT library's own process for discarding tombstones and other bookkeeping metadata it no longer needs) keeps this bounded.
• Server-side validation is awkward with CRDTs: once a client applied an edit, rejecting it at the server means sending a compensating edit (a new edit that reverses or corrects the earlier one, rather than deleting it from history), not refusing it. Enforce permissions at connect and on each update, before broadcast.
• Persisting cursor traffic multiplies write load for no value.
• Big-bang migration of millions of documents risks unrecoverable corruption; lazy per-document migration with the original kept is the safe path.