API and Interface Design for Distributed Services Questions

Use the expand, migrate, contract pattern applied to the API contract itself, not a flag-day cutover. Expand the contract by adding the new shape alongside the old one, so both are simultaneously valid; let consumers migrate to the new shape at their own pace while both are served from the same underlying logic; then contract by removing the old shape only after telemetry proves nobody is still using it, backed by a documented deprecation window and a hard sunset date announced in advance.

Phase 1: Expand

Add the new field or shape without touching the old one. If the change is a genuine reshape rather than a plain addition (for example flattening a nested object), serve both shapes from the same endpoint at once, either through content negotiation (the client requests a version via an Accept header or a version path segment) or by adding the new shape as an additional field alongside the old one, letting consumers switch field by field.

Phase 2: Coexist and migrate

This is the long middle phase, and where most of the risk lives.

• Tolerant-reader consumers: strongly encourage, and where possible enforce through a shared client SDK, that consumers read only the fields they need and ignore unknown ones, rather than deserializing strictly against an exact schema. A consumer that fails hard on an unexpected new field will break the day any new field is added, even a purely additive one, so this is worth fixing before the migration even starts.
• Version negotiation: old consumers keep requesting the old version explicitly, or get it by default if no version header is sent, and receive the old shape; migrated consumers request the new version and receive the new shape. Both are served by the same backing logic wherever possible, translated to two shapes at the response layer, so two independent implementations are not maintained and cannot drift apart.
• Consumer-driven contract testing: every downstream team registers the exact shape it depends on, and the provider's continuous integration runs against every registered contract before any deploy, so whether a change breaks team X is answered in the API owner's own pipeline, not discovered by team X in production days later.
• Usage telemetry: instrument the old shape's usage per consumer identity (a request header, API key, or service identifier), not just "is the old shape used at all," since 100 consumers migrated with 3 stragglers looks identical to a full migration in an aggregate "any usage" metric, but requires a very different retirement decision.

Phase 3: Contract, retiring the old shape

Only after usage telemetry shows the old shape's request volume has been at or below an agreed noise threshold for a sustained window, long enough to catch low-frequency batch consumers that call only monthly, publish a firm sunset date via a Sunset response header or a Deprecation header on the old shape's responses, give a fixed notice period, and only then remove it.

Verifying it is actually safe to retire

• A dashboard or query that answers "which consumer, by identity, called the old shape in the last N days," not just an aggregate count.
• A dry-run period where the old shape is served from the same new-shape-backed logic, so its data is guaranteed consistent with the new shape, purely to catch a consumer still silently depending on an old-shape quirk before it is physically removed.
• A short dark-launch window where the old shape returns an explicit, documented error instead of data, so any straggler consumer fails loudly and immediately, with a clear pointer to the migration guide, rather than silently, surfacing stragglers telemetry missed.

Worked example

An inventory API changes warehouseLocation from a single free-text string like "SFO-3" into a structured object with a code and a region.

• Expand: add warehouseLocationV2: {code, region} alongside the existing warehouseLocation: string. Both fields are populated from the same underlying data on every response.
• Coexist: consumers migrate to warehouseLocationV2 at their own pace; contract tests registered by 40 known consumer teams all continue passing, since none of them broke, they simply have an extra field they can ignore or adopt.
• Telemetry over 6 weeks shows the old warehouseLocation field requested by 3 remaining consumer identities, down from all 40 at expand time.
• Those 3 teams are contacted directly, since telemetry names exactly who, not just a non-zero number; once they confirm migration, the old field is removed with a Sunset header giving 30 days' notice, then dropped.

Trade-offs and pitfalls

Serving both shapes simultaneously from the same underlying logic costs real engineering time, a translation layer and dual test coverage, for the entire coexistence window, which can run months for an API with hundreds of consumers on different release cadences; that cost is the price of avoiding a flag-day cutover, not a shortcut around it. A common wrong turn is treating "usage dropped to near zero" as sufficient to retire without per-consumer identity: aggregate near-zero usage can still be one high-value consumer's monthly batch job, and retiring on the aggregate alone risks silently breaking exactly the consumer least likely to notice quickly. Skipping the tolerant-reader requirement on consumers is the single most common root cause of "we made an additive, backward-compatible change and it still broke someone": a consumer that deserializes strictly, treating an unknown field as an error, turns every additive change into a breaking one for that specific consumer, regardless of how careful the API owner is.

Direct answer

Model the collection with a resource-oriented URL (GET/POST /v1/products), cursor pagination for listing, structured filter and sort query parameters, and a URI-path major version (/v1/). To roll out a breaking change without a hard cutover, let the old and new response shapes run side by side under different version paths, keep the new field additive first, and retire the old version only once usage telemetry shows it is safe, not on a calendar date alone.

Framework

Endpoints and shapes

GET /v1/products?limit=2&sort=price:asc
200 OK
{
  "data": [
    { "id": "p1", "name": "Widget", "price": 9.99, "status": "active" },
    { "id": "p2", "name": "Gadget", "price": 14.50, "status": "active" }
  ],
  "page": { "limit": 2, "next_cursor": "<opaque-cursor-for-p2>", "has_more": true }
}

POST /v1/products
{ "name": "New Widget", "price": 19.99, "category": "tools" }

201 Created
Location: /v1/products/p3
{ "id": "p3", "name": "New Widget", "price": 19.99, "category": "tools", "status": "active" }

Pagination strategy: cursor-based, for the same reason as any list that can grow and be written to concurrently: stable under inserts, roughly constant query cost at depth, at the cost of not supporting a direct "jump to page 43." Offer limit and an opaque after cursor rather than page/offset.

Filtering and sorting: plain query parameters for common fields (status=active, category=tools), and a single sort=field:direction parameter, comma-separated for multiple fields, for example sort=price:asc,name:desc. The filterable field vocabulary is documented per field in the OpenAPI spec (a machine-readable API description) so clients know exactly what is filterable rather than guessing.

Versioning: the major version lives in the URL path (/v1/), because it is simple to route at a gateway or load balancer without inspecting headers, and it is trivially cacheable per version. Additive changes, new optional fields, new optional query parameters, ship inside v1 without a version bump; anything that removes a field, changes a field's type, or changes default sort or filter behavior goes into v2.

Rolling out a breaking change without a hard cutover: suppose v2 needs to rename price to unit_price and change it from a decimal amount to an integer number of cents.

1. Ship v2 alongside v1 on the same deployment. v2's handler reads and writes the same underlying data as v1, so there is exactly one source of truth behind two response shapes.
2. Mark v1's price field deprecated via a response header and a documented sunset date, while it keeps working exactly as before. Nothing breaks yet.
3. Instrument both versions: tag every request with which version served it, and track the fraction of traffic still hitting v1.
4. Migrate internal clients first, since you can coordinate with them directly, then notify external and partner clients with a fixed migration window, for example 90 days, pointing at the v2 docs and a short code sample.
5. Remove v1 only once telemetry shows its remaining traffic has dropped to a level you have explicitly decided is safe to force-migrate, for example only a handful of clients you can contact individually, not on the calendar date alone. If a significant client is still on v1 at the deadline, extend the window rather than break them, and treat that as a signal the migration tooling or communication needs work.

Worked example

Concretely, at the moment v2 ships, v1 carries 100% of traffic. After internal clients migrate in the first month, v1's share drops to 70%. Announcing the 90-day window to external partners brings it down further; by day 60, v1 is at 8%, all from three named partner integrations already contacted directly about their remaining migration steps. That 8%-and-named state, not the passage of 60 days on its own, is what tells you it is close to safe to set a hard removal date, once those three integrations confirm.

Trade-offs and pitfalls

• Renaming price to unit_price and changing its unit, dollars to cents, in the same release conflates two changes into one migration. A client that only cared about the rename still has to handle the unit change, which raises the odds the migration is done wrong. Prefer landing one breaking change at a time when volume allows it.
• A common pitfall is announcing deprecation only in documentation and not in the response itself. Clients that never read a changelog will not notice until the sunset date arrives, so a machine-readable response header, not prose, is what actually drives safe removal.
• Cursor pagination combined with sort needs care: the opaque cursor usually encodes the sort key's value, so changing sort mid-pagination, fetching page 1 by price then asking for page 2 by name, should be rejected or restarted from page 1, since a cursor from one sort order is meaningless under a different one.

Direct answer

A batch-write endpoint needs two independent idempotency scopes, not one, because a caller retries for two different reasons: the whole call might have timed out and the caller genuinely doesn't know if anything landed, or the caller deliberately wants to resubmit only the items that failed after fixing them. A single batch-level key collapses those two cases together and breaks "retry only what failed," so the contract needs a batch-level key for exact whole-request retries and a separate, client-supplied identifier on each item for per-item deduplication that survives across separate calls.

Structured elaboration

Two idempotency scopes:

Response shape when only part of the batch succeeds:

• Some APIs reuse the WebDAV 207 Multi-Status code for a batch response; a more conservative and widely compatible choice is to always return 200 for the batch call itself (the server successfully processed the batch as a request) and carry per-item outcomes in the body. Either is defensible; the important part is that the body always has one explicit result per submitted item, never a single pass/fail for the whole batch.
• Each item's result carries the caller's own client_ref (so the caller can match results back to what it sent), a status, and, on failure, a structured error object: a machine-readable code, a human-readable message, and the batch's request_id for support correlation.

Safe retry of just the failed portion:

• The client builds a new batch containing only the items that came back failed, using each item's original idempotency_key unchanged. The server treats each one on its own merits: if that key had previously succeeded, replay the stored success; if it had previously failed with a retryable error, actually re-attempt it; if it had failed with a non-retryable validation error, fail it again with the same code so the caller doesn't loop forever.
• This is why only successful (or permanently rejected) outcomes get cached against an item's idempotency key. Caching a transient failure would turn a temporary problem into a permanent one, since every retry would just replay the cached failure instead of trying again.

Worked example

Batch request:

// POST /v1/batch-items   Idempotency-Key: batch-e920
{
  "items": [
    { "client_ref": "a1", "idempotency_key": "k-a1", "op": "create", "payload": { "sku": "TSHIRT-M", "quantity": 5 } },
    { "client_ref": "a2", "idempotency_key": "k-a2", "op": "create", "payload": { "sku": "TSHIRT-M", "quantity": -1 } }
  ]
}

Response, partial success:

// 200 OK
{
  "batch_id": "batch_e920",
  "results": [
    { "client_ref": "a1", "status": "succeeded", "resource_id": "s_501" },
    {
      "client_ref": "a2",
      "status": "failed",
      "error": { "code": "VALIDATION_ERROR", "message": "quantity must be positive", "request_id": "req_9c31" }
    }
  ]
}

Client fixes item a2 and resubmits only that item, reusing its original idempotency_key:

// POST /v1/batch-items   Idempotency-Key: batch-e921
{
  "items": [
    { "client_ref": "a2", "idempotency_key": "k-a2", "op": "create", "payload": { "sku": "TSHIRT-M", "quantity": 1 } }
  ]
}

The server sees k-a2 has no cached successful outcome (its only prior outcome was a validation failure, which is not cached as a terminal replay target), so it processes the corrected payload fresh:

// 200 OK
{ "batch_id": "batch_e921", "results": [ { "client_ref": "a2", "status": "succeeded", "resource_id": "s_502" } ] }

Item a1 was never resent and is untouched, exactly the "retry only the failed portion" behavior the contract is meant to provide.

Trade-offs and pitfalls

• Relying on the batch-level key alone forces all-or-nothing retries: a client unsure whether anything succeeded would have to resend everything, and without a separate item-level key, previously-succeeded items risk being re-applied. Both scopes are needed together, not as alternatives.
• Item-level keys must be supplied by the client, not generated by the server, because the client needs a stable identity for the same logical item across two separate HTTP calls; a server-generated id would differ on every request and defeat cross-call dedup entirely.
• Ordering dependencies inside a batch are a real trap: if item 2 logically depends on item 1's created resource and item 1 fails, item 2 should come back with an explicit dependency-failure code rather than being silently attempted (and possibly succeeding against a stale or missing reference).
• Using 207 Multi-Status is a legitimate choice but is a WebDAV-originated status code being repurposed for a plain REST batch API; naming that honestly (rather than presenting it as a universal REST convention) and offering the "always 200, per-item status in the body" alternative shows awareness that this part of HTTP semantics is genuinely unsettled across APIs.

Direct answer

Intermittent duplicates and missing rows in a paginated list under concurrent writes are almost always a symptom of a pagination contract that counts position (offset) rather than anchoring to a specific row's identity, reading a list that keeps moving underneath it. The fix path is: collect telemetry that lets you correlate a client's exact requests against what changed on the backend, reproduce it deterministically with a synthetic test rather than chasing it live, ship a contained mitigation, then replace the contract itself.

Telemetry to collect

• Per-request: the exact pagination parameters used (cursor or offset, limit), a request ID, and the item IDs actually returned, logged on both client and server so the two logs can be joined.
• Per-write: a monotonic write sequence number or commit timestamp for every insert, update, or delete against the transaction table, so you can reconstruct exactly which writes landed between two page fetches.
• Aggregate dashboards: rate of duplicate-ID and gap incidents (a page that skips an ID present in a request one page earlier) bucketed by client, endpoint, and page size, so you can tell whether the rate correlates with write volume or a specific client behavior (e.g. slow scrolling, background tab refetches).

Synthetic reproduction

The fastest way to confirm the hypothesis is a small, fully deterministic script rather than a live capture: insert a row between two page fetches and check whether the second fetch repeats or skips an item. This is directly runnable and needs no production access:

class Store:
    def __init__(self, items):
        self.items = items  # newest first

    def insert_at_head(self, item):
        self.items.insert(0, item)

    def page_by_offset(self, offset, limit):
        return self.items[offset: offset + limit]

    def page_by_cursor(self, after_id, limit):
        start = 0 if after_id is None else self.items.index(after_id) + 1
        return self.items[start: start + limit]


store = Store(["txn-5", "txn-4", "txn-3", "txn-2", "txn-1"])
page1 = store.page_by_offset(0, 2)        # fetch page 1 before the insert
store.insert_at_head("txn-6")             # concurrent write lands
page2 = store.page_by_offset(2, 2)        # fetch page 2 after the insert
print("OFFSET page1:", page1)
print("OFFSET page2:", page2)
print("txn-4 in both pages:", "txn-4" in page1 and "txn-4" in page2)
print("txn-2 skipped entirely:", "txn-2" not in page1 and "txn-2" not in page2)

Running this prints OFFSET page1: ['txn-5', 'txn-4'] and OFFSET page2: ['txn-4', 'txn-3'], with both follow-up checks printing True: txn-4 appears in both pages (duplicate) and txn-2 never appears in either page (skipped entirely), reproduced with a handful of lines and no timing dependency.

Re-running the same scenario against a cursor-anchored implementation instead:

store2 = Store(["txn-5", "txn-4", "txn-3", "txn-2", "txn-1"])
page1c = store2.page_by_cursor(None, 2)
store2.insert_at_head("txn-6")
page2c = store2.page_by_cursor(page1c[-1], 2)
print("CURSOR page1:", page1c)
print("CURSOR page2:", page2c)
print("no overlap between pages:", set(page1c).isdisjoint(page2c))

This prints CURSOR page1: ['txn-5', 'txn-4'], CURSOR page2: ['txn-3', 'txn-2'], and no overlap between pages: True. This is the reproducible evidence that anchors the fix, not a guess.

Short-term mitigation vs long-term fix

Trade-offs and pitfalls

It is tempting to chase this as a backend consistency problem (replica lag, read quorum) rather than a pagination contract problem. If the storage layer is genuinely eventually consistent, that is worth knowing, but the API-level fix here does not require tuning consensus or replication internals: it requires the endpoint to commit to an explicit, anchored ordering contract and, if staleness is possible, to say so in the response (for example a snapshot or version marker the client can compare) so the client can detect it rather than silently rendering wrong data. Conflating "our replication is laggy" with "our pagination contract is unanchored" leads teams to spend weeks on the wrong layer.

A related pitfall: fixing this only on the read path while leaving deletes unhandled. A cursor anchored on a row that gets deleted between pages needs an explicit signal (do not silently skip past it or error opaquely) so the client knows to discard that cursor rather than retry it forever.

Direct answer

Model the upload as a stateful resource, an upload session with its own id, and make idempotency work at two different levels: session creation is protected by a client-supplied Idempotency-Key so a retried "start upload" call cannot spin up two competing sessions for the same file, and each chunk is naturally idempotent because it is addressed by its byte range plus a checksum, so re-sending the same range is a safe no-op rather than a duplicate write. Everything the server needs to persist is small: the session's metadata and the set of byte ranges already received; the file bytes themselves live in object storage, not in the session record.

Structured elaboration

Endpoints:

1. POST /uploads (initiate). Body: filename, total_size, content_type. Headers: Idempotency-Key (recommended). Response 201: { upload_id, expires_at, chunk_size_hint }.
2. PATCH /uploads/{upload_id} (upload a chunk). Headers: Content-Range: bytes {start}-{end}/{total}, a chunk checksum header. Response 200: { received_ranges: [...], next_offset }. Accepts chunks out of order.
3. GET /uploads/{upload_id} (resume). Returns received_ranges and next_offset so a reconnecting mobile client knows exactly what is missing.
4. POST /uploads/{upload_id}/complete (finalize). Body: client-computed whole-file checksum. Server assembles/validates and returns the final file id.
5. DELETE /uploads/{upload_id} (abandon explicitly).

Server state persisted:

• Session row: upload_id, user_id, filename, total_size, content_type, state (initiated / in_progress / completed / aborted), created_at, last_activity_at, expires_at.
• Received-ranges record: a compact list of {start, end, checksum} entries per chunk actually stored, not the chunk bytes themselves (those go straight to object storage).
• Nothing here requires holding an open connection between chunks; a mobile client can drop off the network for minutes and resume against the same upload_id.

Idempotency at each step:

• Initiate: Idempotency-Key plus a fingerprint of (filename, total_size, content_type) maps to one upload_id for a bounded window (documented, e.g., 24 hours); replaying the same initiate call returns the same session instead of creating a second one.
• Chunk upload: idempotent by construction through Content-Range. If a range that was already received arrives again with a matching checksum, the server responds 200 and does nothing further (safe retry after a lost response). If the same range arrives with a different checksum, that is a real conflict (409), not a retry, because the client is now sending different bytes for a range it already committed.
• Finalize: idempotent on upload_id; calling complete twice after success returns the same final file id both times rather than re-assembling or erroring, so a client that couldn't tell whether its first complete call landed can safely call it again.

Handling intermittent mobile connectivity:

• Small chunk sizes (server hints a size, e.g., a few hundred kilobytes to a few megabytes) so a dropped connection loses at most one chunk's worth of progress, not the whole upload.
• GET /uploads/{upload_id} is the resume contract: the client asks what ranges are already received and only re-sends what's missing, rather than restarting from byte zero.

Abandoned uploads and abuse:

• expires_at on the session (returned at initiate time so the client can see its own deadline); a background sweeper deletes sessions and their stored chunks only after expires_at has passed, never touching a session with recent last_activity_at.
• Per-user quotas (maximum concurrent sessions, maximum total bytes in flight) enforced at initiate time, returning 429 when exceeded, the same documented-quota pattern used for any other rate-limited endpoint.
• total_size is validated against the declared 200MB ceiling at initiate time, and each chunk's checksum is verified on arrival so a corrupted or tampered chunk is rejected before it is ever assembled into a final file.

Worked example

Initiate:

// POST /uploads   Idempotency-Key: up-init-9f21
{ "filename": "trip-video.mp4", "total_size": 52428800, "content_type": "video/mp4" }

// 201 Created
{ "upload_id": "up_7c14", "expires_at": "2026-07-20T09:00:00Z", "chunk_size_hint": 262144 }

Upload the first chunk (256KB, matching the hinted chunk size, so the byte range is 0 through 262143 of the 52428800-byte total):

PATCH /uploads/up_7c14
Content-Range: bytes 0-262143/52428800

// 200 OK
{ "received_ranges": [[0, 262143]], "next_offset": 262144 }

Client drops off the network, reconnects later, and asks what it still needs:

GET /uploads/up_7c14

{ "upload_id": "up_7c14", "received_ranges": [[0, 262143]], "next_offset": 262144, "total_size": 52428800 }

It resumes exactly at byte 262144, no re-upload of the first chunk. Finalize once all ranges are in:

// POST /uploads/up_7c14/complete
{ "checksum_algorithm": "sha256", "checksum": "<client-computed full-file digest>" }

// 200 OK
{ "file_id": "file_31a0", "state": "completed" }

Trade-offs and pitfalls

• Server-proxied chunk uploads (as shown) are simpler to validate per-chunk but cost server bandwidth; direct-to-object-store multipart uploads (client uploads parts straight to storage using short-lived credentials) save that bandwidth but push part-tracking and checksum bookkeeping onto a more complex client-storage handshake. For 10k concurrent clients, the bandwidth savings usually win, at the cost of that added complexity.
• Chunk size is a real trade-off: smaller chunks tolerate flaky mobile networks better (less to re-send after a drop) but add per-chunk request overhead at scale; larger chunks are more efficient per byte but riskier on an unreliable connection.
• A pitfall specific to idempotency here: treating "same range, different checksum" as a silent overwrite instead of a 409 hides a real client bug (a chunk that got corrupted or a retry that picked up stale data) and can quietly assemble a corrupted final file.
• Garbage-collecting by expires_at alone, without checking last_activity_at, risks deleting a session that is still actively (if slowly) being uploaded to from a poor connection right at the boundary of its TTL; the sweeper should treat both signals together, the same discipline as garbage-collecting any other long-lived job resource.