API and Interface Design for Distributed Services Questions

The real tension is not "flexible versus strict," it is where you spend the strictness. Keep the contract strict and machine-verified at the point that governs breaking-change detection (the schema itself, in a source-controlled registry with automated compatibility checks), while giving consumers ergonomic flexibility inside that boundary, such as optional fields, generated clients, and a query language for shape selection. Which protocol you pick changes where that boundary naturally falls: gRPC (a binary remote-procedure-call framework built on protocol buffers) pushes strictness to compile time, GraphQL pushes flexibility to query time with server-side cost controls, and REST with JSON Schema or OpenAPI sits in between and depends entirely on how disciplined the team is about additive-only changes.

Comparing the three

Policy that applies regardless of protocol

• Additive-only by default: new fields are optional with sane defaults; removing or renaming a field requires a deprecation window, not an immediate change.
• Automated compatibility checks in CI: a schema-diff tool fails the build on a breaking change (removed field, changed type, renamed required field), so "did this break someone" is answered by a machine before merge, not discovered by a downstream team in production.
• Consumer-driven contract tests: each real consumer publishes the exact shape it depends on, and the provider's CI runs against every published contract, so a change that would break one specific downstream team fails in the team that made the change, not weeks later.
• Isolate the loosely-typed part: if a team wants to pass an arbitrary blob for now, scope that flexibility to genuinely exploratory, non-critical fields (a metadata bag), never to fields that drive billing, authentication, or routing. That is the real lever here, it is a per-field decision, not one dial for the whole API.
• For GraphQL specifically, query depth and complexity limits are not optional if ergonomics should not come with an operational blast radius, since the same flexibility that makes GraphQL appealing (arbitrary nested queries) is what lets one client accidentally request an exponential fan-out.

Worked example

A User type currently has a required email field. A team wants to add multi-email support without breaking a mobile client that only reads email.

• Wrong move: rename email to primaryEmail and remove email. Old clients that expect email fail immediately with a missing-field error.
• Additive move: keep email unchanged (still returns the primary address) and add a new optional emails list alongside it. Old clients ignore the new field entirely. A schema-diff check for this change reports "1 field added, 0 removed, 0 type changes" and passes automatically; the same check on the wrong move reports "1 field removed" and fails the build before merge.

Trade-offs and pitfalls

Over-indexing on strictness everywhere (gRPC for every surface, no exploratory field at all) genuinely slows down iteration on non-critical, evolving data; the fix is a scoped metadata field, not abandoning the strict contract for the whole API. Over-indexing on flexibility (accepting arbitrary blobs everywhere "so developers can move fast") pushes the cost of catching breaking changes from build time to production-incident time, which is a far more expensive place to catch them. A common wrong turn is treating this purely as a protocol choice, as in "switch to GraphQL and the problem goes away": GraphQL solves over-fetching and under-fetching ergonomics, it does not automatically solve breaking-change detection. Whichever protocol is chosen still needs the same discipline, a schema registry, a CI compatibility gate, and a deprecation window, or it inherits the same risk under different syntax.

Direct answer

Intermittent duplicates and missing rows in a paginated list under concurrent writes are almost always a symptom of a pagination contract that counts position (offset) rather than anchoring to a specific row's identity, reading a list that keeps moving underneath it. The fix path is: collect telemetry that lets you correlate a client's exact requests against what changed on the backend, reproduce it deterministically with a synthetic test rather than chasing it live, ship a contained mitigation, then replace the contract itself.

Telemetry to collect

• Per-request: the exact pagination parameters used (cursor or offset, limit), a request ID, and the item IDs actually returned, logged on both client and server so the two logs can be joined.
• Per-write: a monotonic write sequence number or commit timestamp for every insert, update, or delete against the transaction table, so you can reconstruct exactly which writes landed between two page fetches.
• Aggregate dashboards: rate of duplicate-ID and gap incidents (a page that skips an ID present in a request one page earlier) bucketed by client, endpoint, and page size, so you can tell whether the rate correlates with write volume or a specific client behavior (e.g. slow scrolling, background tab refetches).

Synthetic reproduction

The fastest way to confirm the hypothesis is a small, fully deterministic script rather than a live capture: insert a row between two page fetches and check whether the second fetch repeats or skips an item. This is directly runnable and needs no production access:

class Store:
    def __init__(self, items):
        self.items = items  # newest first

    def insert_at_head(self, item):
        self.items.insert(0, item)

    def page_by_offset(self, offset, limit):
        return self.items[offset: offset + limit]

    def page_by_cursor(self, after_id, limit):
        start = 0 if after_id is None else self.items.index(after_id) + 1
        return self.items[start: start + limit]


store = Store(["txn-5", "txn-4", "txn-3", "txn-2", "txn-1"])
page1 = store.page_by_offset(0, 2)        # fetch page 1 before the insert
store.insert_at_head("txn-6")             # concurrent write lands
page2 = store.page_by_offset(2, 2)        # fetch page 2 after the insert
print("OFFSET page1:", page1)
print("OFFSET page2:", page2)
print("txn-4 in both pages:", "txn-4" in page1 and "txn-4" in page2)
print("txn-2 skipped entirely:", "txn-2" not in page1 and "txn-2" not in page2)

Running this prints OFFSET page1: ['txn-5', 'txn-4'] and OFFSET page2: ['txn-4', 'txn-3'], with both follow-up checks printing True: txn-4 appears in both pages (duplicate) and txn-2 never appears in either page (skipped entirely), reproduced with a handful of lines and no timing dependency.

Re-running the same scenario against a cursor-anchored implementation instead:

store2 = Store(["txn-5", "txn-4", "txn-3", "txn-2", "txn-1"])
page1c = store2.page_by_cursor(None, 2)
store2.insert_at_head("txn-6")
page2c = store2.page_by_cursor(page1c[-1], 2)
print("CURSOR page1:", page1c)
print("CURSOR page2:", page2c)
print("no overlap between pages:", set(page1c).isdisjoint(page2c))

This prints CURSOR page1: ['txn-5', 'txn-4'], CURSOR page2: ['txn-3', 'txn-2'], and no overlap between pages: True. This is the reproducible evidence that anchors the fix, not a guess.

Short-term mitigation vs long-term fix

Trade-offs and pitfalls

It is tempting to chase this as a backend consistency problem (replica lag, read quorum) rather than a pagination contract problem. If the storage layer is genuinely eventually consistent, that is worth knowing, but the API-level fix here does not require tuning consensus or replication internals: it requires the endpoint to commit to an explicit, anchored ordering contract and, if staleness is possible, to say so in the response (for example a snapshot or version marker the client can compare) so the client can detect it rather than silently rendering wrong data. Conflating "our replication is laggy" with "our pagination contract is unanchored" leads teams to spend weeks on the wrong layer.

A related pitfall: fixing this only on the read path while leaving deletes unhandled. A cursor anchored on a row that gets deleted between pages needs an explicit signal (do not silently skip past it or error opaquely) so the client knows to discard that cursor rather than retry it forever.

Direct answer

For a collaborative dashboard with thousands of concurrent viewers, use WebSockets as the primary transport if edits genuinely flow both directions (a user's own edit needs to reach the server as fast as other users' edits reach them); use Server-Sent Events (SSE), a one-directional push protocol built on plain HTTP, if the client mostly receives updates and sends edits through ordinary POST requests. Either way, the API can only honestly promise at-least-once delivery, never exactly-once, across an unreliable network; the contract that makes that safe is a per-channel sequence number in every event, so the client can detect duplicates and gaps itself. Clients that cannot hold a persistent connection (old browsers, restrictive corporate proxies, flaky mobile carriers) fall back to long-polling against the same event schema, so the client's rendering code never has to branch on transport.

Structured elaboration

Transport comparison, as an interface decision (not as an internal scaling mechanism):

Delivery guarantee as an API contract (this is the part the caller actually needs documented, not how the backend achieves it internally):

• Every event carries a monotonically increasing seq per channel/dashboard, assigned once by the server.
• The contract the API documents is: "events are delivered at least once; the same seq may arrive more than once, the client must deduplicate by seq, and a client resuming after a disconnect will replay from its last acknowledged seq." That is exactly-once application behavior built on top of an honestly at-least-once transport guarantee; the API should never claim exactly-once delivery, because nothing about the network between server and client can make that promise.
• Resumption is exposed to the caller through a documented mechanism: SSE's native Last-Event-ID request header, an equivalent since_seq query parameter for WebSocket/long-polling clients, so any transport can ask "replay everything after N."

Fallback contract for older or unreliable clients:

• One canonical event envelope ({ seq, type, data }) is used across all three transports, so switching transport at the client is a connection-layer decision, not a data-shape decision.
• The client negotiates transport through a lightweight capability probe (for example, attempt the WebSocket upgrade; on failure or repeated drop, fall back to SSE; on SSE failure, fall back to long-polling against GET /dashboards/{id}/updates?since_seq=N), all reading the same event stream by sequence number.
• This is the concrete home for the fallback angle: an older browser or a mobile network that kills idle sockets degrades gracefully to polling and still gets a correct, ordered, deduplicated view, just at higher latency.

Worked example

SSE event as the server would emit it (mirrors the WebSocket message body and the long-polling response item):

event: dashboard-update
id: 482991
data: {"seq":482991,"type":"cell-update","dashboardId":"dash_77","cell":"B12","value":481200,"editedBy":"u_582"}

Resumption after a dropped connection: the browser's EventSource automatically resends the last id it saw as a request header:

GET /dashboards/dash_77/stream
Last-Event-ID: 482991

The server looks up its durable event log for that channel starting at seq 482992 and replays from there; nothing is skipped and nothing already-seen needs to be discarded by the client beyond ordinary dedup.

WebSocket frame carrying the identical event (same schema, different transport):

{"seq":482991,"type":"cell-update","dashboardId":"dash_77","cell":"B12","value":481200,"editedBy":"u_582"}

Long-polling fallback for a client that cannot hold either connection open:

GET /dashboards/dash_77/updates?since_seq=482991

{
  "events": [
    {"seq":482992,"type":"cell-update","dashboardId":"dash_77","cell":"B12","value":481500,"editedBy":"u_311"}
  ],
  "poll_after_ms_hint": 2000
}

Same envelope, same seq field, so a client that fell back from WebSocket to long-polling mid-session can keep deduplicating exactly the same way.

Trade-offs and pitfalls

• Claiming exactly-once delivery is the most common mistake here; the honest, documentable contract is at-least-once transport plus client-side dedup by sequence number, which behaves like exactly-once from the caller's point of view without making a promise the network can't keep.
• SSE is simpler to operate and gets reconnection for free in the browser, but it is one-directional; a dashboard where users also edit needs a separate POST path for outbound edits, and that path needs its own idempotency story (a duplicate POST from a retried edit should not double-apply).
• WebSocket's biggest real-world failure mode is not code complexity, it's the network path: some mobile carriers and corporate proxies silently drop idle WebSocket connections without a clean close frame, which looks to the client like a hang rather than a disconnect. The contract needs an application-level heartbeat/ping so the client can detect a dead connection instead of waiting on a TCP timeout.
• A pitfall on the server-facing side: shipping a different event schema per transport (say, camelCase for WebSocket and snake_case for the long-poll fallback) forces client code to branch on transport, which defeats the point of having a documented, transport-agnostic contract in the first place.

Direct answer

For a feed with a high insert rate at the head, pick cursor-based (keyset) pagination. It anchors each page to the last item's sort key instead of a row count, so it does not skip or repeat rows when new items appear ahead of the client's read position. Offset-based pagination (OFFSET n LIMIT m) is simpler but silently shifts under concurrent inserts, which is exactly what a frequently-updated feed does continuously.

Framework

What each approach actually does

• Offset pagination: the client asks for "skip N, take M." The server re-runs the ordering on every request and counts N rows in before returning results. A page's identity is a position in a list, and that position can move.
• Cursor (keyset) pagination: the client sends the sort-key value(s) of the last item it saw. The server issues a seek query (WHERE (created_at, id) < (:last_created_at, :last_id) ORDER BY created_at DESC, id DESC LIMIT M). A page's identity is anchored to a specific row's key, not a position.

Sorting by created_at alone is not enough if two rows can share a timestamp. The seek predicate needs a secondary, unique tie-breaker column (commonly id) so (created_at, id) is a total order with no ties.

Worked example

A concrete trace of why offset duplicates a row when an item is inserted at the head, and why cursor pagination does not.

Five existing rows, ordered newest-first by created_at:

Offset pagination, page size 2:

• Page 1: OFFSET 0 LIMIT 2 returns positions 1-2: [105, 104].
• A new row 106 (created_at 2026-01-01T10:05:00Z) is inserted at the head. Every existing row shifts down one position: 105 is now position 2, 104 is position 3, and so on.
• Page 2: OFFSET 2 LIMIT 2 returns whatever now sits at positions 3-4, which is [104, 103]. The client already saw 104 on page 1, so it is duplicated, and the client has no way to notice.

Cursor pagination, same scenario:

• Page 1: no cursor supplied, returns [105, 104]; next_cursor encodes {"id": "104", "created_at": "2026-01-01T10:03:00Z"}.
• Row 106 is inserted at the head.
• Page 2: WHERE (created_at, id) < ('2026-01-01T10:03:00Z', 104) ORDER BY created_at DESC, id DESC LIMIT 2. Row 106's created_at is greater than 104's, so it fails the < predicate and is correctly excluded. The query still returns [103, 102], exactly the rows that follow 104, regardless of what was inserted ahead of it.

Sample cursor-based response shape (page 1 above):

{
  "items": [
    { "id": "105", "created_at": "2026-01-01T10:04:00Z" },
    { "id": "104", "created_at": "2026-01-01T10:03:00Z" }
  ],
  "next_cursor": "eyJpZCI6IjEwNCIsImNyZWF0ZWRfYXQiOiIyMDI2LTAxLTAxVDEwOjAzOjAwWiJ9.f0e3e3b16f56ab33",
  "has_more": true
}

How that cursor was actually built (reproducible, step by step):

1. Payload: {"id": "104", "created_at": "2026-01-01T10:03:00Z"}.
2. Compact JSON serialization: {"id":"104","created_at":"2026-01-01T10:03:00Z"}.
3. Base64url-encode the bytes, no padding: eyJpZCI6IjEwNCIsImNyZWF0ZWRfYXQiOiIyMDI2LTAxLTAxVDEwOjAzOjAwWiJ9.
4. Sign the raw JSON bytes with a server-side hash-based message authentication code (HMAC-SHA256) using a secret only the server holds, and take the first 16 hex characters of the digest as a tamper check: f0e3e3b16f56ab33.
5. Join encoded payload and signature with a period: <base64url>.<signature>.

A production cursor payload would also carry an issued-at timestamp and a time-to-live (TTL): the server checks issued-at plus TTL on every request and, if the cursor has expired, rejects it (for example with a 400 response) and asks the client to restart from the first page, rather than trying to resume mid-sequence from stale coordinates.

Trade-offs and pitfalls

• Cursor pagination cannot jump to an arbitrary page. If the product genuinely needs "go to page 43," you need offset, or a hybrid where a search step lands the client near a position and cursor paging takes over from there.
• If the tie-breaker column is not unique, or is not indexed together with the primary sort key, the seek query degrades toward a scan instead of an index lookup.
• Signing the cursor matters for more than tamper-proofing: an unsigned or unencrypted cursor can leak internal ids and ordering, and a forged cursor could be used to skip access checks that were only applied on page 1.
• A common pitfall is anchoring the cursor to a mutable field, such as an "updated at" timestamp instead of creation time. An edit to any row changes its position under that ordering, so items can be revisited or skipped after an edit, not just after an insert. Prefer an immutable creation-time key plus a stable id tie-breaker.

Direct answer

Give the contract a semantic version, major.minor.patch, and enforce schema compatibility automatically in continuous integration (CI) using a schema registry with a declared compatibility mode. Treat "does an older client still work against the newer server" as something measured continuously through per-client-version error-rate alerts, not something assumed just because the versioning rules were followed. When retiring a version, announce the exact stop date with the Sunset response header (a standardized header, RFC 8594), and once that date passes, switch the status code to 410 Gone, since 410 tells the caller the resource is intentionally and permanently gone, unlike the ambiguous 404 Not Found.

Framework

Versioning and schema evolution

• Major means breaking, minor means additive only, patch means no contract change.
• Prefer additive schema changes: new optional fields, new endpoints. For a field that must be removed or retyped, run a transitional period where the server accepts and emits both the old and new shape (a tolerant reader and writer), gated behind the caller's declared version.
• If the wire format uses a typed interface definition language, such as protobuf (Protocol Buffers, a binary serialization format defined by a .proto schema) or Avro (a similar binary schema format common in data-pipeline and messaging systems), declare an explicit compatibility mode, backward (a new schema can read data written with the old schema), forward (an old schema can read data written with the new schema), or full (both directions hold at once), in a schema registry, and enforce it in CI so an incompatible change fails the build before it ships, not after a client breaks in production.

Compatibility testing

• Maintain a small matrix of currently supported client versions. In CI, run each supported version's expectations against the new server build (consumer-driven contract tests), so a change that would break v1 clients is caught in the pull request, not in a bug report.
• Before a risky change reaches all production traffic, replay a sample of real production requests from each supported client version against the new build in a non-production environment and compare responses field by field.

Deprecation and Sunset headers, 410 vs 404

• Sunset is a standardized response header (RFC 8594) carrying an HTTP date: the point at which the resource, here the old API version, is expected to stop responding. Send it on every response from a deprecated version once a sunset date is set, so any client or tool that reads response headers programmatically, not just the docs, can see it.
• There is also a Deprecation response header, standardized in RFC 9745 (March 2025) on the IETF Standards Track (Proposed Standard status), that simply flags a resource as deprecated, with or without a date. That status is actually stronger than Sunset's own RFC 8594, which is Informational rather than Standards Track. Used together, they say two separable things: "this is deprecated now" and "here is exactly when it stops."
• After the sunset date passes, return 410 Gone, not 404 Not Found. 404 is ambiguous: it could mean a typo in the URL, something that never existed, or something temporarily missing, and it gives the caller no signal about whether retrying is worthwhile. 410 says, unambiguously, this used to exist here, it was removed on purpose, and it is not coming back, which is exactly the signal an intentionally sunset version needs to send, and it is what well-behaved client tooling treats as permanent rather than as a transient error worth retrying.

Automated detection and alerting

• Tag every request with the client version that made it, from a header or an API key's declared version, and emit per-version metrics: error rate, latency, traffic volume.
• Alert when a specific client version's error rate crosses a threshold clearly above that version's own historical baseline, not a single global threshold, since an old client version naturally has a different steady-state error rate than the newest one.
• Alert separately on the inverse signal: a deprecated version's traffic staying flat or rising as the sunset date approaches, which means the deprecation notice is not reaching the people still depending on it.

flowchart LR
    C1[Client v1] --> GW[Version-aware gateway]
    C2[Client v2] --> GW
    C3[Client v3] --> GW
    GW --> SVC[Backend service]
    SVC --> REG[(Schema registry, compatibility mode enforced)]
    SVC --> MET[Per-client-version metrics]
    MET --> AL{Error rate above that version's baseline?}
    AL -- yes --> INC[Alert and open incident]
    AL -- no --> CONT[Continue serving]
    REG --> CI[CI contract tests per supported client version]

Worked example

Suppose v1's historical error rate is a steady 0.5%, older clients send slightly malformed requests more often than newer ones, and that is already baselined as normal. After a server change, v1's error rate jumps to 4% while v2 and v3 stay flat. Because the alert threshold is set relative to v1's own 0.5% baseline, for example alerting at 3 times baseline sustained for 5 minutes, this is caught immediately as a v1-specific regression. A single global error-rate threshold tuned for the whole API's blended traffic might not trip at all if v1 is a small fraction of total volume.

Trade-offs and pitfalls

• Per-client-version baselines are more accurate but need more setup than one global threshold, since you need enough historical data per version to know what normal looks like. For a brand-new version with no history yet, fall back to a conservative global threshold until it accumulates its own baseline.
• A tolerant reader and writer transitional period keeps old and new clients both working but adds real branching complexity to the server. The discipline that makes this safe is a hard deadline to delete the transitional code once the old version is fully retired, not letting it become permanent.
• A common pitfall is applying 410 Gone too early, before the sunset date, out of a sense that the migration is basically done. Clients coded to retry on 404 or 5xx but to treat 410 as fatal will stop retrying earlier than intended, which is a much harsher outcome than the team meant to cause.