API and Interface Design for Distributed Services Questions

Use the expand, migrate, contract pattern applied to the API contract itself, not a flag-day cutover. Expand the contract by adding the new shape alongside the old one, so both are simultaneously valid; let consumers migrate to the new shape at their own pace while both are served from the same underlying logic; then contract by removing the old shape only after telemetry proves nobody is still using it, backed by a documented deprecation window and a hard sunset date announced in advance.

Phase 1: Expand

Add the new field or shape without touching the old one. If the change is a genuine reshape rather than a plain addition (for example flattening a nested object), serve both shapes from the same endpoint at once, either through content negotiation (the client requests a version via an Accept header or a version path segment) or by adding the new shape as an additional field alongside the old one, letting consumers switch field by field.

Phase 2: Coexist and migrate

This is the long middle phase, and where most of the risk lives.

• Tolerant-reader consumers: strongly encourage, and where possible enforce through a shared client SDK, that consumers read only the fields they need and ignore unknown ones, rather than deserializing strictly against an exact schema. A consumer that fails hard on an unexpected new field will break the day any new field is added, even a purely additive one, so this is worth fixing before the migration even starts.
• Version negotiation: old consumers keep requesting the old version explicitly, or get it by default if no version header is sent, and receive the old shape; migrated consumers request the new version and receive the new shape. Both are served by the same backing logic wherever possible, translated to two shapes at the response layer, so two independent implementations are not maintained and cannot drift apart.
• Consumer-driven contract testing: every downstream team registers the exact shape it depends on, and the provider's continuous integration runs against every registered contract before any deploy, so whether a change breaks team X is answered in the API owner's own pipeline, not discovered by team X in production days later.
• Usage telemetry: instrument the old shape's usage per consumer identity (a request header, API key, or service identifier), not just "is the old shape used at all," since 100 consumers migrated with 3 stragglers looks identical to a full migration in an aggregate "any usage" metric, but requires a very different retirement decision.

Phase 3: Contract, retiring the old shape

Only after usage telemetry shows the old shape's request volume has been at or below an agreed noise threshold for a sustained window, long enough to catch low-frequency batch consumers that call only monthly, publish a firm sunset date via a Sunset response header or a Deprecation header on the old shape's responses, give a fixed notice period, and only then remove it.

Verifying it is actually safe to retire

• A dashboard or query that answers "which consumer, by identity, called the old shape in the last N days," not just an aggregate count.
• A dry-run period where the old shape is served from the same new-shape-backed logic, so its data is guaranteed consistent with the new shape, purely to catch a consumer still silently depending on an old-shape quirk before it is physically removed.
• A short dark-launch window where the old shape returns an explicit, documented error instead of data, so any straggler consumer fails loudly and immediately, with a clear pointer to the migration guide, rather than silently, surfacing stragglers telemetry missed.

Worked example

An inventory API changes warehouseLocation from a single free-text string like "SFO-3" into a structured object with a code and a region.

• Expand: add warehouseLocationV2: {code, region} alongside the existing warehouseLocation: string. Both fields are populated from the same underlying data on every response.
• Coexist: consumers migrate to warehouseLocationV2 at their own pace; contract tests registered by 40 known consumer teams all continue passing, since none of them broke, they simply have an extra field they can ignore or adopt.
• Telemetry over 6 weeks shows the old warehouseLocation field requested by 3 remaining consumer identities, down from all 40 at expand time.
• Those 3 teams are contacted directly, since telemetry names exactly who, not just a non-zero number; once they confirm migration, the old field is removed with a Sunset header giving 30 days' notice, then dropped.

Trade-offs and pitfalls

Serving both shapes simultaneously from the same underlying logic costs real engineering time, a translation layer and dual test coverage, for the entire coexistence window, which can run months for an API with hundreds of consumers on different release cadences; that cost is the price of avoiding a flag-day cutover, not a shortcut around it. A common wrong turn is treating "usage dropped to near zero" as sufficient to retire without per-consumer identity: aggregate near-zero usage can still be one high-value consumer's monthly batch job, and retiring on the aggregate alone risks silently breaking exactly the consumer least likely to notice quickly. Skipping the tolerant-reader requirement on consumers is the single most common root cause of "we made an additive, backward-compatible change and it still broke someone": a consumer that deserializes strictly, treating an unknown field as an error, turns every additive change into a breaking one for that specific consumer, regardless of how careful the API owner is.

Direct answer

Scope the idempotency key to the pair (authenticated caller, key value), not the key alone, since two different callers could otherwise pick the same string. Have the client generate the key once per distinct payment intent, for example a universally unique identifier (UUID) minted when the user initiates the payment and reused on every retry of that same intent, and send it in an Idempotency-Key header. Keep two separate stores with two separate retention policies: a short-lived dedup record that exists only to catch retries, and a permanent, append-only transaction ledger that is the actual system of record, retained for as long as audit and regulatory rules require, independent of the dedup window.

Framework

Key scoping and generation

• Composite lookup key: (tenant_id or caller_id, Idempotency-Key header value). The caller's identity comes from authentication, not the header, so a buggy or malicious client cannot pick another tenant's key and collide with, or read, their transaction.
• The client mints the key once per business intent (for example, once when the user clicks "Pay," reused automatically by the client's own retry logic on a network failure). A UUID, or an HMAC (hash-based message authentication code) of stable request fields, both work; what matters is that a fresh user action always gets a fresh key.
• The server stores a hash of the request body alongside the key. A retry with the same key and the same body hash is a genuine retry. A retry with the same key and a different body hash means the client reused a key for a different operation, a client bug, and the server should reject it with 409 Conflict rather than guess which body was the real one.

Request flow

flowchart TD
    A[POST with Idempotency-Key] --> B{Key + caller seen before?}
    B -- No --> C[Atomic insert: status=IN_PROGRESS, body hash]
    C --> D[Run charge/authorize]
    D --> E[Update record: status=COMPLETED, cache response]
    E --> F[Return response]
    B -- Same body hash, COMPLETED --> G[Replay cached response, no re-charge]
    B -- Same body hash, IN_PROGRESS --> H[Tell client to retry shortly]
    B -- Different body hash --> I[409 Conflict: key reused for a different request]

Dedup retention vs. ledger retention
These are two different stores doing two different jobs, and conflating them is the most common mistake in this design:

1. Hot dedup store: a fast key-value store with a conditional, create-if-absent write, holding just enough history to catch realistic retries. A time-to-live (TTL) of 24 to 72 hours comfortably covers a client retrying after a network blip, a redelivered request after an outage, or a user reopening the app to resubmit.
2. Cold transaction ledger: an append-only record of every completed transaction, keyed by its own transaction id, retained for the audit window (commonly several years for financial records, a figure set by finance or compliance, not by engineering). This ledger, not the dedup record, is what an auditor or a dispute investigation actually reads.

Because these are separate, letting the hot dedup record expire is safe: once it is gone, a very late duplicate looks like a new request to the dedup store, but a uniqueness constraint on (caller_id, idempotency_key) in the ledger itself can still catch it, more slowly, as a database constraint violation instead of a fast cache hit, and refuse to double-write.

Worked example

Storage growth is the concrete cost of these two retention windows. Assume, for this calculation: 2,000,000 payment requests per day, a 300-byte hot dedup record (key, body hash, status, timestamps), a 3-day hot TTL, and a 150-byte compact ledger record kept for 5 years. These are illustrative inputs chosen to make the arithmetic concrete, not measured production numbers.

Hot store size at steady state:

Ledger size over the 5-year audit window:

The hot store stays small and bounded, 1.8 GB, regardless of how long the business has been running, because the TTL caps it. The ledger grows without bound over the audit window and reaches roughly half a terabyte by year five at this volume, which is why it belongs in cheaper, append-friendly, cold storage rather than the same fast store used for dedup lookups.

Trade-offs and pitfalls

• A short hot-store TTL keeps storage and lookup latency small but pushes more responsibility onto the ledger's own uniqueness constraint for very late duplicates. That check is slower and surfaces as a database error rather than a clean cache hit, so decide up front how the API surfaces that case (still 409, just discovered later and by a different mechanism).
• Choosing a strongly consistent store for the hot dedup lookup, so the create-if-absent race always has exactly one winner, is a deliberate trade of some write latency for correctness. For payment creation this is almost always the right trade, since a double-execution bug here is a chargeback and a compliance incident, not a cosmetic glitch.
• A common pitfall is treating the dedup TTL as the retention policy for the underlying transaction. Garbage-collecting the dedup record must never delete or affect the ledger entry it protected, only the cheap "have I seen this exact request" pointer.
• Another pitfall is generating the idempotency key server-side, for example by hashing only the request body, instead of letting the client mint it. Two genuinely different user actions that happen to produce an identical body, the same amount to the same recipient, submitted twice on purpose, would then collide, and the second, legitimate payment would be silently dropped.

Direct answer

This is the classic N+1 fan-out: fetching a list of related items triggers one call for the list plus one additional call per item, instead of a single call that fetches all the related data at once, so latency and downstream load scale with the size of the result set rather than staying flat. The fix is to collapse the per-item calls into a single batched call, or a request-scoped loader that performs that collapsing automatically, and only reach for a precomputed denormalized store when the read path is hot enough to justify the added consistency cost.

Diagnosing it

Confirm it is really N+1 and not just "a slow downstream service" by counting calls per request, not just measuring total time: instrument the aggregation layer to log how many downstream calls a single incoming request triggers, then check whether that count grows linearly with the size of the requested list. A request for 5 related items making 5 downstream calls and a request for 200 items making 200 calls is the signature; a flat call count regardless of list size means the problem is elsewhere.

Fix patterns

A "DataLoader" here means a request-scoped utility, popularized by GraphQL server implementations, that defers execution until every resolver in the current request has registered the key it needs, then fires one batched downstream call instead of many individual ones.

Worked example

flowchart LR
    subgraph Naive["Naive: one call per item"]
        G1[API Gateway] --> I1[Item 1 call]
        G1 --> I2[Item 2 call]
        G1 --> I3[Item N call]
    end
    subgraph Batched["Fixed: single batched call"]
        G2[API Gateway] --> B1[Batch call: all item IDs]
    end

Reasoning about the scaling, not wall-clock timing (real timings depend on hardware and network and are not reproducible from an answer alone): if a request needs data for $N$ related items and each downstream round trip costs a fixed unit $c$, a fully serial naive fan-out costs $N \times c$ total round-trip time. A single batched call collapses this to a constant cost of $c$ regardless of $N$. A concurrency-capped fan-out with cap $k$ costs $\lceil N/k \rceil \times c$: for example, with $N=200$ items and $k=20$ concurrent calls, that is $\lceil 200/20 \rceil = 10$ rounds, versus 1 round batched or 200 rounds fully serial. The batched call is the only option whose cost does not grow with the result set at all.

Trade-offs and pitfalls

Batching changes the failure model: a single bad ID in a batch call must not fail the entire batch. The batched response needs a shape that supports partial success (per-item status alongside a shared envelope), or one missing related record takes down an otherwise-successful response for every other item in the batch.

A request-scoped loader only helps inside one request's lifetime: if the same key is requested by two separate incoming requests seconds apart, the loader does not share that saved work across them, that is what a cache is for. Reaching for a denormalized store as the first fix, before trying batching or a loader, is a common overreach: it is the most expensive option operationally (a whole pipeline to build and keep in sync) and should be reserved for paths where the simpler fixes have already been tried and the traffic genuinely justifies it.

Direct answer

Version the payload schema so that most evolution never requires a new API version at all: additive, optional-field changes ship into the existing version, and only a genuinely breaking change (removing or renaming a field, changing a type, changing meaning) earns a new major version. Communicate and execute deprecation through machine-readable signals, not just documentation and email, so that automated partner integrations, not just the humans who read the changelog, find out a version is going away and when.

Structured elaboration

Classifying changes so most of them don't need a version bump:

Rollout stages for a genuinely breaking change:

1. Preview. The new major version is available opt-in (a new URI path or an explicit request header) so partners can test against it before it is the default.
2. Dual-run. Both versions are served in production for a defined overlap window; usage of the old version is measured per partner so the team knows who still needs to migrate, not just that traffic to it is declining in aggregate.
3. Deprecation announced. The still-serving old version starts returning a Deprecation header on every response (standardized in RFC 9745, March 2025, on the IETF Standards Track) carrying the date deprecation took effect, plus a Sunset header (RFC 8594, Informational status) carrying the exact date the version will stop serving entirely. The Deprecation header's Standards Track status is actually the stronger of the two. Both are machine-readable, so a partner's own monitoring can alert on them without a human reading a changelog.
4. Sunset. After the Sunset date passes, the old version stops serving.

410 versus 404 after sunset, the definitional point: once a version is actually sunset, the correct response is 410 Gone, not 404 Not Found. 404 means "nothing is here," which is ambiguous to a caller debugging a broken integration, it could be a typo in the URL just as easily as a real removal. 410 says explicitly "this used to exist, on purpose, and it is not coming back," which is exactly the information a caller needs to know it must migrate rather than fix a typo. The 410 response body should still explain what happened and where to go next.

Worked example

While the old version is deprecated but still serving:

HTTP/1.1 200 OK
Deprecation: Wed, 01 Jul 2026 00:00:00 GMT
Sunset: Thu, 31 Dec 2026 23:59:59 GMT
Link: <https://api.example.com/v2/ingest>; rel="successor-version"

After the sunset date has passed:

HTTP/1.1 410 Gone
Content-Type: application/json

{
  "error": {
    "code": "endpoint_sunset",
    "message": "API v1 was sunset on 2026-12-31. Migrate to /v2/ingest; see the migration guide linked in this response.",
    "request_id": "req_9f21"
  }
}

The dates in the headers and the body agree, so a partner's monitoring, its on-call engineer reading a log line, and its API client's own error message all tell the same story.

Trade-offs and pitfalls

• Relying only on a changelog and email is the most common failure mode; automated partner integrations do not read email, and by the time a human notices the deprecation notice it may already be close to the sunset date. Machine-readable headers close that gap.
• Returning 404 for a sunset endpoint (instead of 410) is a small-looking choice with a real cost: it makes "this API version is gone on purpose" indistinguishable from "you mistyped the URL," which slows down every partner's own debugging.
• Dual-running two versions is real, ongoing engineering cost (two schemas, two sets of compatibility tests, two things that can each break), not a one-time expense; the overlap window needs a firm end date from the start, or "temporary" dual-running quietly becomes permanent.
• A subtler pitfall: bumping the major version for a change that could have been additive (for example, renaming a field instead of adding a new one alongside the old, deprecated one) creates breaking-change churn that erodes partner trust in the version number actually meaning something.

Make the consistency guarantee a first-class, documented, per-endpoint contract field, not prose buried in a general "about consistency" page: state it in the endpoint reference as a plain-language sentence plus a machine-readable annotation, return it as response metadata on every call rather than only in the docs, and give explicit client guidance for each documented failure mode. The docs, the response shape, and the client SDK all say the same thing, so a developer never has to infer a guarantee from behavior they happened to observe once.

Per-endpoint documentation structure

Every endpoint that carries a consistency guarantee gets the same four-part structure, so a developer scanning multiple endpoints is not re-taught the format each time: a one-sentence plain-language summary, a machine-readable contract block that tooling (including client SDKs) can act on, worked examples of a normal response and each documented failure mode, and explicit guidance for what a client should do on each response.

Two concrete endpoint contracts

Error and status-code semantics, using standard HTTP rather than invented codes

• 409 Conflict: a linearizable write lost a race with a concurrent write to the same resource. Client guidance: re-read the current state, then decide whether to retry with the fresh state or surface the conflict to the end user; never blindly retry the original write unchanged.
• 503 Service Unavailable with a Retry-After header (a standard header telling the client how many seconds to wait before trying again): the system cannot currently meet the documented guarantee at all, for example during a network partition. Client guidance: respect Retry-After, do not tight-loop.
• A stale-but-within-bound read is not an error at all. It returns a normal 200 OK with stalenessMs populated, since serving a bounded-stale read is the correct, expected behavior for that endpoint's documented contract, not a failure. If a read happens to fall outside its documented bound, for example during a replica lag spike, it is still a normal 200 OK, since refusing to return data would be worse than returning a flagged stale value, but the response's stalenessMs exceeds the documented bound, so the client can detect the violation itself rather than trusting the service-level agreement (SLA: a documented commitment about a specific response characteristic) blindly. Reusing an unrelated HTTP status code to signal this would be a real mistake: status 425 is a real, registered code, defined in RFC 8470 to mean "Too Early," used specifically to reject requests that risk a TLS replay attack during 0-RTT connection setup. Repurposing 425 to mean "stale read" would collide with its actual, unrelated meaning and could mislead HTTP-aware infrastructure sitting in the request path (proxies, gateways) that already interprets that code correctly.

Client-side guidance, concretely

A typed client exposes the guarantee in its method signature, for example a read call that accepts a maximum acceptable staleness and raises a client-side error if the returned stalenessMs exceeds what the caller asked for, even when the server's documented bound was technically met. Retry guidance is written per status code, not generically: a 409 requires a fresh read before any retry, never a blind retry of stale write data; a 503 retry must honor Retry-After; a 200 with a high stalenessMs is not something to retry at all, since retrying a read does not fix replication lag, it is something to surface to the caller or fall back to a stronger-guarantee endpoint if one exists.

Worked example, full round trip

Request: GET /v1/analytics/summary?userId=u_501. Normal response:

{
  "userId": "u_501",
  "totalOrders": 42,
  "asOf": "2026-07-18T09:00:00Z",
  "stalenessMs": 1800
}

Here stalenessMs: 1800 (1.8 seconds) is well within the documented 5000 millisecond bound, so a client comparing this number against its own tolerance can proceed with confidence, without knowing anything about how the 5-second bound is achieved internally.
Degraded response, bound exceeded, still 200 since the data itself is valid, just late:

{
  "userId": "u_501",
  "totalOrders": 42,
  "asOf": "2026-07-18T08:59:52Z",
  "stalenessMs": 8400
}

A client-side check of stalenessMs > 5000 tells this specific caller, on this specific response, that the documented guarantee was not met this time, without waiting for an incident report or a status page update.

Trade-offs and pitfalls

Returning staleness metadata on every response has a real cost: the read path must always know and report its own commit or replication timestamp, which is not free to compute or propagate on every code path, and some teams skip this and only document a static bound, leaving the client no way to detect a violation in the moment, only after the fact from an incident report. The common wrong turn worth calling out explicitly is inventing a custom HTTP status code, or repurposing a real but differently-scoped one, to signal an application-level condition like a stale read: standard status codes have registered meanings that other HTTP-aware infrastructure already interprets, and a normal 200 with documented body or header metadata is both more correct and more compatible with that infrastructure than borrowing an unrelated code. The main long-term risk is documentation and the runtime contract drifting apart: if the docs say a 5 second bound but nothing enforces or alerts on stalenessMs exceeding it in production, the documented guarantee quietly becomes fiction. The fix is treating the machine-readable contract block and the response metadata as the same source of truth, ideally generated from one schema, rather than hand-maintained prose alongside a separately hand-maintained response shape.