API and Interface Design for Distributed Services Questions

Direct answer

The client supplies the idempotency key as a header, commonly Idempotency-Key, on the mutating request. The server scopes it to (authenticated caller, key value) so the same string cannot be reinterpreted across unrelated callers. On a duplicate, the server looks up that composite key: if the original request already completed, it replays the stored response instead of re-executing the operation; if the original is still in flight, it tells the caller to wait rather than starting a second execution. A key becomes safe to garbage-collect once its retention window has covered every retry the client could plausibly still make, a figure you size from the caller's own retry and backoff policy, not from a round number that feels safe.

Framework

1. How the client supplies the key
The server documents Idempotency-Key: <client-generated-token> as required on every mutating request (POST, PUT, PATCH). The header string alone is not the whole lookup key: the server combines it with the caller's authenticated identity, and, if the same caller could reuse a key across two different endpoints, the operation name too, so one caller's create-order-88 cannot accidentally collide with another caller's identical string. The server also stores a hash of the request body next to the key, to tell "a genuine retry of the same request" apart from "the key was reused for a different request," which is a client bug and should get a 409 Conflict rather than a guess about which body was real.

2. How the server recognizes and responds to a duplicate

stateDiagram-v2
    [*] --> InProgress: first request with key K, atomic create
    InProgress --> Completed: operation succeeds, response cached
    InProgress --> Failed: operation fails, error cached
    Failed --> InProgress: client retries same key, operation re-attempted
    Completed --> [*]: retention window expires, record garbage collected
    Failed --> [*]: retention window expires, record garbage collected

• Atomic create-if-absent on the composite key when the first copy of a request arrives, storing status IN_PROGRESS.
• Duplicate while IN_PROGRESS: tell the caller the original is still processing rather than starting a second execution in parallel.
• Duplicate after COMPLETED: replay the stored response verbatim, same status code, same body, so retrying looks identical to having received that response the first time.
• Duplicate after FAILED: the operation never actually happened, so a fresh attempt on the same key is allowed to re-run it.

3. Deciding when a key is safe to garbage-collect
The retention window is a function of how long a legitimate retry could still show up, not an arbitrary round number.

• Start from the client's documented retry policy. For example, "retry up to 5 times with exponential backoff starting at 1 second, capped at 30 seconds between attempts" resolves in a couple of minutes end to end, not hours.
• Add margin for what the client-side retry budget alone does not cover: the caller's own process crashing and restarting, a queued request redelivered after an upstream outage, clock skew between services. These push the realistic "a legitimate duplicate could still show up" window from minutes to a few hours for most systems.
• Pick a retention window an order of magnitude above that worst case, so garbage collection is not happening on a knife's edge: if the worst-case legitimate retry window is a few hours, 24 to 48 hours leaves comfortable margin without holding records indefinitely.
• The one exception this reasoning does not cover on its own: some domains need a much longer window for reasons unrelated to retries, such as audit or dispute rules. That is a separate retention decision layered on top of the dedup window, usually enforced by a durable ledger rather than the hot dedup store, not a substitute for sizing the dedup window itself.

Worked example

A concrete race, two copies of the same client request, a genuine retry, not two separate clicks, both carrying Idempotency-Key: create-order-88, arriving 40 milliseconds apart at two different instances of the same service because the client's original request timed out on the network but actually landed.

• Request A: the create-if-absent on (caller_42, create-order-88) succeeds, status IN_PROGRESS, begins creating the order.
• Request B, 40 milliseconds later: the create-if-absent fails because the key now exists. Request B reads the existing record, sees IN_PROGRESS, and returns "in progress, retry shortly" instead of creating a second order.
• 300 milliseconds later, request A finishes: order ord_9911 is created, the record is updated to COMPLETED with the response cached.
• A third retry after that still fails the create-if-absent, but now the record is COMPLETED, so the server replays the cached response containing ord_9911 directly, without touching the order-creation code path at all.

The atomic create-if-absent plus the IN_PROGRESS/COMPLETED state is the mechanism that prevents a double order; the retention window only controls how long that protection lasts after the fact.

Trade-offs and pitfalls

• Making the header optional is tempting for a softer rollout, but it removes the server's ability to distinguish "no key supplied, protect me anyway" from "no key supplied, this is fire-and-forget." Most teams that go this route end up requiring the header for all mutating endpoints anyway.
• A retention window picked from "what feels safe" instead of the caller's actual retry policy tends to be wrong in one of two costly directions: too short, and a legitimate late retry double-executes the operation; too long, and the dedup store grows without bound and starts to look like an accidental second database.
• Replaying a COMPLETED response verbatim is only correct if the operation's result does not depend on something that changes between the original call and the retry, like a current inventory count embedded in the response. If the response includes time-sensitive derived data, decide up front whether a retry should get the original snapshot or fresh data recomputed around a fixed transaction id.
• Do not conflate this dedup window with how long the underlying business record itself is kept. Garbage-collecting the idempotency key should never delete the order, payment, or resource it protected, only the cheap "have I seen this exact request before" pointer.

The real tension is not "flexible versus strict," it is where you spend the strictness. Keep the contract strict and machine-verified at the point that governs breaking-change detection (the schema itself, in a source-controlled registry with automated compatibility checks), while giving consumers ergonomic flexibility inside that boundary, such as optional fields, generated clients, and a query language for shape selection. Which protocol you pick changes where that boundary naturally falls: gRPC (a binary remote-procedure-call framework built on protocol buffers) pushes strictness to compile time, GraphQL pushes flexibility to query time with server-side cost controls, and REST with JSON Schema or OpenAPI sits in between and depends entirely on how disciplined the team is about additive-only changes.

Comparing the three

Policy that applies regardless of protocol

• Additive-only by default: new fields are optional with sane defaults; removing or renaming a field requires a deprecation window, not an immediate change.
• Automated compatibility checks in CI: a schema-diff tool fails the build on a breaking change (removed field, changed type, renamed required field), so "did this break someone" is answered by a machine before merge, not discovered by a downstream team in production.
• Consumer-driven contract tests: each real consumer publishes the exact shape it depends on, and the provider's CI runs against every published contract, so a change that would break one specific downstream team fails in the team that made the change, not weeks later.
• Isolate the loosely-typed part: if a team wants to pass an arbitrary blob for now, scope that flexibility to genuinely exploratory, non-critical fields (a metadata bag), never to fields that drive billing, authentication, or routing. That is the real lever here, it is a per-field decision, not one dial for the whole API.
• For GraphQL specifically, query depth and complexity limits are not optional if ergonomics should not come with an operational blast radius, since the same flexibility that makes GraphQL appealing (arbitrary nested queries) is what lets one client accidentally request an exponential fan-out.

Worked example

A User type currently has a required email field. A team wants to add multi-email support without breaking a mobile client that only reads email.

• Wrong move: rename email to primaryEmail and remove email. Old clients that expect email fail immediately with a missing-field error.
• Additive move: keep email unchanged (still returns the primary address) and add a new optional emails list alongside it. Old clients ignore the new field entirely. A schema-diff check for this change reports "1 field added, 0 removed, 0 type changes" and passes automatically; the same check on the wrong move reports "1 field removed" and fails the build before merge.

Trade-offs and pitfalls

Over-indexing on strictness everywhere (gRPC for every surface, no exploratory field at all) genuinely slows down iteration on non-critical, evolving data; the fix is a scoped metadata field, not abandoning the strict contract for the whole API. Over-indexing on flexibility (accepting arbitrary blobs everywhere "so developers can move fast") pushes the cost of catching breaking changes from build time to production-incident time, which is a far more expensive place to catch them. A common wrong turn is treating this purely as a protocol choice, as in "switch to GraphQL and the problem goes away": GraphQL solves over-fetching and under-fetching ergonomics, it does not automatically solve breaking-change detection. Whichever protocol is chosen still needs the same discipline, a schema registry, a CI compatibility gate, and a deprecation window, or it inherits the same risk under different syntax.

Direct answer

Use consumer-driven contract testing: each consuming team publishes the exact requests and expected responses it actually depends on, and the provider team's own continuous integration (CI) pipeline verifies the real API against every published consumer contract before a change can merge or deploy. This inverts the usual failure mode, where a consumer discovers a break after the provider has already shipped; here the provider's own pipeline fails first, and the consumer team is notified automatically through the same system that stores the contracts, not by someone noticing something broke in production.

Structured elaboration

The contract and where it lives:

• Each consumer team writes a small set of interactions ("when I call this endpoint with this request, I expect a response shaped like this") and publishes them to a shared, versioned contract registry (a broker), tagged with which environment or release they apply to.
• This is deliberately narrower than a full schema: the consumer only asserts the exact fields and shapes it actually reads, so the contract reflects real usage rather than the provider's entire response shape.

Automating verification in the provider's CI:

• On every pull request to the provider's API code, a CI step fetches the latest contracts tagged for the relevant environment, spins up (or points at a running instance of) the provider, replays each consumer's recorded requests against it, and asserts the actual responses still satisfy what each consumer expects. Any mismatch fails the build and blocks the merge.
• Before an actual deploy (not just before merge), a second gate checks whether the version about to be deployed is compatible with whichever consumer versions are currently running in production, not just the newest consumer contract, since a provider change can be compatible with a consumer's latest contract while still breaking a consumer instance that hasn't picked up its own latest release yet.

How the affected team finds out:

• The contract broker's webhook posts directly to the consumer team's own channel (chat integration or issue tracker) whenever a provider verification against their specific contract fails, naming the exact interaction that broke and linking to a diff between what was expected and what the provider now returns. The team learns about the break from an automated, targeted message tied to their own contract, not from a shared build dashboard they'd have to be watching.

Worked example

A consumer's published contract (one interaction, in a contract-broker style format):

{
  "consumer": { "name": "checkout-web" },
  "provider": { "name": "orders-api" },
  "interactions": [
    {
      "description": "fetching an order by id",
      "request": { "method": "GET", "path": "/orders/500" },
      "response": {
        "status": 200,
        "body": { "id": "500", "status": "PAID", "total_cents": 4200 }
      }
    }
  ]
}

Provider CI step (illustrative, tool-agnostic pseudocode) that fails the build on a mismatch:

verify-contracts:
  script:
    - fetch-contracts --provider orders-api --tag production
    - contract-verify --provider-base-url http://localhost:8080 --contracts ./contracts/*.json
  # non-zero exit code on any interaction mismatch blocks the merge

Suppose a developer on the provider team renames total_cents to totalCents in a refactor. The verification step replays the recorded request, gets back a body missing the field the consumer's contract asserted, and the build fails with a message like: "interaction 'fetching an order by id' failed: expected field total_cents, field not present in actual response." The broker then posts that same message, with a link to the failing interaction, straight to the checkout-web team's channel, so they know about the incompatibility before it ever reaches a shared environment.

Trade-offs and pitfalls

• Contract testing only catches what a consumer actually asserted. If a consumer's published contract never checked the total_cents field in the first place, removing it silently passes; the technique is only as good as how faithfully each contract reflects real usage, which puts real responsibility on consumer teams to keep their contracts current as their own code changes.
• Verifying against the latest contract at merge time is necessary but not sufficient; gating the actual deploy against what is compatible with consumer versions currently running in production (not just their newest published contract) is what catches the case where two independently-safe changes combine badly because of deploy ordering.
• This approach is faster and cheaper than full end-to-end integration testing, but it is not a replacement for it: contract tests verify shape and basic behavior, not real-world concerns like load, timing, or business-logic correctness across a full user flow.
• A pitfall on the process side: if publishing and updating contracts is treated as optional or bolted on late, teams tend to let contracts drift stale, at which point the safety net silently degrades and nobody notices until a real break gets through anyway.

Direct answer

Intermittent duplicates and missing rows in a paginated list under concurrent writes are almost always a symptom of a pagination contract that counts position (offset) rather than anchoring to a specific row's identity, reading a list that keeps moving underneath it. The fix path is: collect telemetry that lets you correlate a client's exact requests against what changed on the backend, reproduce it deterministically with a synthetic test rather than chasing it live, ship a contained mitigation, then replace the contract itself.

Telemetry to collect

• Per-request: the exact pagination parameters used (cursor or offset, limit), a request ID, and the item IDs actually returned, logged on both client and server so the two logs can be joined.
• Per-write: a monotonic write sequence number or commit timestamp for every insert, update, or delete against the transaction table, so you can reconstruct exactly which writes landed between two page fetches.
• Aggregate dashboards: rate of duplicate-ID and gap incidents (a page that skips an ID present in a request one page earlier) bucketed by client, endpoint, and page size, so you can tell whether the rate correlates with write volume or a specific client behavior (e.g. slow scrolling, background tab refetches).

Synthetic reproduction

The fastest way to confirm the hypothesis is a small, fully deterministic script rather than a live capture: insert a row between two page fetches and check whether the second fetch repeats or skips an item. This is directly runnable and needs no production access:

class Store:
    def __init__(self, items):
        self.items = items  # newest first

    def insert_at_head(self, item):
        self.items.insert(0, item)

    def page_by_offset(self, offset, limit):
        return self.items[offset: offset + limit]

    def page_by_cursor(self, after_id, limit):
        start = 0 if after_id is None else self.items.index(after_id) + 1
        return self.items[start: start + limit]


store = Store(["txn-5", "txn-4", "txn-3", "txn-2", "txn-1"])
page1 = store.page_by_offset(0, 2)        # fetch page 1 before the insert
store.insert_at_head("txn-6")             # concurrent write lands
page2 = store.page_by_offset(2, 2)        # fetch page 2 after the insert
print("OFFSET page1:", page1)
print("OFFSET page2:", page2)
print("txn-4 in both pages:", "txn-4" in page1 and "txn-4" in page2)
print("txn-2 skipped entirely:", "txn-2" not in page1 and "txn-2" not in page2)

Running this prints OFFSET page1: ['txn-5', 'txn-4'] and OFFSET page2: ['txn-4', 'txn-3'], with both follow-up checks printing True: txn-4 appears in both pages (duplicate) and txn-2 never appears in either page (skipped entirely), reproduced with a handful of lines and no timing dependency.

Re-running the same scenario against a cursor-anchored implementation instead:

store2 = Store(["txn-5", "txn-4", "txn-3", "txn-2", "txn-1"])
page1c = store2.page_by_cursor(None, 2)
store2.insert_at_head("txn-6")
page2c = store2.page_by_cursor(page1c[-1], 2)
print("CURSOR page1:", page1c)
print("CURSOR page2:", page2c)
print("no overlap between pages:", set(page1c).isdisjoint(page2c))

This prints CURSOR page1: ['txn-5', 'txn-4'], CURSOR page2: ['txn-3', 'txn-2'], and no overlap between pages: True. This is the reproducible evidence that anchors the fix, not a guess.

Short-term mitigation vs long-term fix

Trade-offs and pitfalls

It is tempting to chase this as a backend consistency problem (replica lag, read quorum) rather than a pagination contract problem. If the storage layer is genuinely eventually consistent, that is worth knowing, but the API-level fix here does not require tuning consensus or replication internals: it requires the endpoint to commit to an explicit, anchored ordering contract and, if staleness is possible, to say so in the response (for example a snapshot or version marker the client can compare) so the client can detect it rather than silently rendering wrong data. Conflating "our replication is laggy" with "our pagination contract is unanchored" leads teams to spend weeks on the wrong layer.

A related pitfall: fixing this only on the read path while leaving deletes unhandled. A cursor anchored on a row that gets deleted between pages needs an explicit signal (do not silently skip past it or error opaquely) so the client knows to discard that cursor rather than retry it forever.

Direct answer

Ship the server change so it accepts requests both with and without the new field, defaulting it server-side when absent, so nothing breaks the moment the change deploys. Roll enforcement in behind a flag keyed by caller or client version, expanding gradually while watching a "requests missing this field" metric drop toward zero, and only flip the field to actually required once that metric shows it is safe for the population of clients you cannot update directly, not on a fixed calendar date.

Framework

Phase 0: ship tolerant, observe only
Deploy server code that accepts the field as optional, applies a sensible default when it is missing, and logs, tagged by caller and client version, every request that arrives without it. Nothing is rejected in this phase. This is what makes the eventual rollout zero-downtime: the deploy that adds validation never also has to add the field, because those are two separate deploys.

Phases 1-3: progressive enforcement
Behind a server-side feature flag keyed by caller id or client version, start rejecting requests missing the field for a small, known-safe slice of traffic first, for example your own internal callers, who you can fix immediately if they break. Widen the slice in stages, a percentage of external traffic, then all traffic from clients above a known-good version, watching the missing-field rate and the resulting error rate at each stage before widening further.

Phase 4: flip to required
Once the missing-field rate across all remaining traffic is at, or has been driven to, zero, or the only remaining non-compliant callers are ones identified by name who have either been fixed directly or have explicitly accepted the breakage, remove the tolerant path and the flag. The field is now genuinely required, not just documented as required while the server quietly still accepts its absence.

Worked example

Concretely: at Phase 0, telemetry shows 12% of requests lack the new field. Enabling strict validation for 1% of traffic, Phase 1, internal callers only, shows 0% errors, since internal callers were already updated first. Widening to 25% of external traffic, Phase 2, surfaces a 2% error rate concentrated in exactly 3 identifiable partner API keys still on an old integration version; those 3 keys are contacted directly and updated. Phase 3 widens to 100% traffic and the error rate returns to 0%, confirming it is now safe to make Phase 4's flip permanent and delete the tolerant path.

Rollback plan, defined before the rollout starts, not improvised during an incident: the fastest rollback is flipping the enforcement flag back to "accept missing field" for the affected slice, which takes effect without a new deployment. If the underlying server change itself is the problem, not just the validation strictness, redeploy the previous server version behind the same gradual-traffic mechanism used to roll it forward, rather than an all-at-once revert.

Trade-offs and pitfalls

• Splitting "accept the field" and "require the field" into two separate deploys costs an extra release cycle compared to shipping both at once, but it is what makes the rollout genuinely zero-downtime instead of a bet that every client updates before a single deploy lands.
• A common pitfall is widening the enforcement percentage on a fixed schedule, day 1 at 10%, day 7 at 50%, day 14 at 100%, instead of on the observed missing-field rate at each stage. A schedule that ignores what the data is saying can either drag on longer than necessary, or worse, widen past a real problem simply because a date arrived.
• Another pitfall is treating "0% errors so far" as proof of safety when the current traffic slice does not yet include the population most likely to be non-compliant, batch jobs that run monthly, or mobile clients on very old versions most users have already updated away from. Widen enforcement along dimensions that specifically include the slowest-moving clients before declaring the rollout complete.