API and Interface Design for Distributed Services Questions

Direct answer

Use semantic versioning at the contract level, major.minor.patch, but expose only the major version to callers, in the URL path (for example /v2/). Treat minor and patch changes as invisible, additive changes. Size deprecation windows by who is actually affected rather than a single fixed number, and confirm a version is safe to remove using measured usage telemetry crossing an explicit, pre-agreed threshold, not the passage of a deprecation date by itself.

Framework

Versioning scheme
Major means a breaking change, minor means additive and backward-compatible, patch means a bug fix with no contract change. Only the major version is ever exposed to callers.

Pick URI path for the major version. With thousands of clients across many regions and independent release cadences, the version needs to be something every client, in every language, with zero special tooling, routes on correctly by default. Reserve a custom header for an optional, opt-in early-access preview of the next major version.

Deprecation windows, tied to who is affected, not one number

• Internal services you can coordinate with directly: shorter windows, weeks, since you can track and push migration yourself.
• External partners integrated through your own SDKs: medium windows, a few months, since you control the SDK's default behavior.
• Long-tail third-party integrators with no direct relationship: the longest windows, 6 to 12 months, since you are relying entirely on them reading a changelog.

Progressive rollout via feature flags
This is the mechanism that turns "ship v2" from one risky release into a series of small, reversible steps. Put the new behavior behind a server-side flag keyed by caller id or traffic percentage, so it can be enabled for a handful of internal callers first, then a widening slice, with an instant, code-free way to turn it back off, before it is ever exposed as the new major version to the general client population.

Knowing when removal is actually safe
Instrument every request with the version that served it and watch usage of the old version fall toward zero. Safe to remove means the remaining traffic on the old version is either genuinely zero, or belongs only to callers identified by name who have either been force-migrated or have explicitly accepted the breakage. A deprecation date passing is not, by itself, evidence of safety; it is a trigger to go check the actual telemetry.

Worked example

Illustrative telemetry over a deprecation window: at announcement, v1 carries 100% of traffic. In month 1, enabling v2 behind the flag for 5% of callers drops v1's share to 95%. By month 4, with the flag widened to 60% of callers, v1's share is down to 40%. By month 6, v1's share is 3%, and that 3% is fully attributable to two named legacy integrators who have already been contacted directly. It is that 3%-and-named state, not the passage of 6 months on its own, that makes removal safe: the remaining traffic is fully accounted for.

Trade-offs and pitfalls

• A single fixed deprecation window for every client type is easy to communicate but wrong in both directions: too short for a long-tail integrator who never reads a changelog, too long for an internal service that could migrate in a week.
• Feature-flag rollout adds real operational complexity, the service now behaves differently depending on who is calling, and that complexity has to be retired too. Once a flag reaches 100% and stays there, it should be deleted, not left as permanent branching logic.
• A common pitfall is declaring victory once "most" traffic has moved and removing the old version anyway. The callers left behind are disproportionately the ones least able to adapt quickly, unmaintained integrations, infrequently updated mobile clients, so a surprise removal hurts exactly the group least equipped to absorb it.

Direct answer

Scope the idempotency key to the pair (authenticated caller, key value), not the key alone, since two different callers could otherwise pick the same string. Have the client generate the key once per distinct payment intent, for example a universally unique identifier (UUID) minted when the user initiates the payment and reused on every retry of that same intent, and send it in an Idempotency-Key header. Keep two separate stores with two separate retention policies: a short-lived dedup record that exists only to catch retries, and a permanent, append-only transaction ledger that is the actual system of record, retained for as long as audit and regulatory rules require, independent of the dedup window.

Framework

Key scoping and generation

• Composite lookup key: (tenant_id or caller_id, Idempotency-Key header value). The caller's identity comes from authentication, not the header, so a buggy or malicious client cannot pick another tenant's key and collide with, or read, their transaction.
• The client mints the key once per business intent (for example, once when the user clicks "Pay," reused automatically by the client's own retry logic on a network failure). A UUID, or an HMAC (hash-based message authentication code) of stable request fields, both work; what matters is that a fresh user action always gets a fresh key.
• The server stores a hash of the request body alongside the key. A retry with the same key and the same body hash is a genuine retry. A retry with the same key and a different body hash means the client reused a key for a different operation, a client bug, and the server should reject it with 409 Conflict rather than guess which body was the real one.

Request flow

flowchart TD
    A[POST with Idempotency-Key] --> B{Key + caller seen before?}
    B -- No --> C[Atomic insert: status=IN_PROGRESS, body hash]
    C --> D[Run charge/authorize]
    D --> E[Update record: status=COMPLETED, cache response]
    E --> F[Return response]
    B -- Same body hash, COMPLETED --> G[Replay cached response, no re-charge]
    B -- Same body hash, IN_PROGRESS --> H[Tell client to retry shortly]
    B -- Different body hash --> I[409 Conflict: key reused for a different request]

Dedup retention vs. ledger retention
These are two different stores doing two different jobs, and conflating them is the most common mistake in this design:

1. Hot dedup store: a fast key-value store with a conditional, create-if-absent write, holding just enough history to catch realistic retries. A time-to-live (TTL) of 24 to 72 hours comfortably covers a client retrying after a network blip, a redelivered request after an outage, or a user reopening the app to resubmit.
2. Cold transaction ledger: an append-only record of every completed transaction, keyed by its own transaction id, retained for the audit window (commonly several years for financial records, a figure set by finance or compliance, not by engineering). This ledger, not the dedup record, is what an auditor or a dispute investigation actually reads.

Because these are separate, letting the hot dedup record expire is safe: once it is gone, a very late duplicate looks like a new request to the dedup store, but a uniqueness constraint on (caller_id, idempotency_key) in the ledger itself can still catch it, more slowly, as a database constraint violation instead of a fast cache hit, and refuse to double-write.

Worked example

Storage growth is the concrete cost of these two retention windows. Assume, for this calculation: 2,000,000 payment requests per day, a 300-byte hot dedup record (key, body hash, status, timestamps), a 3-day hot TTL, and a 150-byte compact ledger record kept for 5 years. These are illustrative inputs chosen to make the arithmetic concrete, not measured production numbers.

Hot store size at steady state:

Ledger size over the 5-year audit window:

The hot store stays small and bounded, 1.8 GB, regardless of how long the business has been running, because the TTL caps it. The ledger grows without bound over the audit window and reaches roughly half a terabyte by year five at this volume, which is why it belongs in cheaper, append-friendly, cold storage rather than the same fast store used for dedup lookups.

Trade-offs and pitfalls

• A short hot-store TTL keeps storage and lookup latency small but pushes more responsibility onto the ledger's own uniqueness constraint for very late duplicates. That check is slower and surfaces as a database error rather than a clean cache hit, so decide up front how the API surfaces that case (still 409, just discovered later and by a different mechanism).
• Choosing a strongly consistent store for the hot dedup lookup, so the create-if-absent race always has exactly one winner, is a deliberate trade of some write latency for correctness. For payment creation this is almost always the right trade, since a double-execution bug here is a chargeback and a compliance incident, not a cosmetic glitch.
• A common pitfall is treating the dedup TTL as the retention policy for the underlying transaction. Garbage-collecting the dedup record must never delete or affect the ledger entry it protected, only the cheap "have I seen this exact request" pointer.
• Another pitfall is generating the idempotency key server-side, for example by hashing only the request body, instead of letting the client mint it. Two genuinely different user actions that happen to produce an identical body, the same amount to the same recipient, submitted twice on purpose, would then collide, and the second, legitimate payment would be silently dropped.

Use the expand, migrate, contract pattern applied to the API contract itself, not a flag-day cutover. Expand the contract by adding the new shape alongside the old one, so both are simultaneously valid; let consumers migrate to the new shape at their own pace while both are served from the same underlying logic; then contract by removing the old shape only after telemetry proves nobody is still using it, backed by a documented deprecation window and a hard sunset date announced in advance.

Phase 1: Expand

Add the new field or shape without touching the old one. If the change is a genuine reshape rather than a plain addition (for example flattening a nested object), serve both shapes from the same endpoint at once, either through content negotiation (the client requests a version via an Accept header or a version path segment) or by adding the new shape as an additional field alongside the old one, letting consumers switch field by field.

Phase 2: Coexist and migrate

This is the long middle phase, and where most of the risk lives.

• Tolerant-reader consumers: strongly encourage, and where possible enforce through a shared client SDK, that consumers read only the fields they need and ignore unknown ones, rather than deserializing strictly against an exact schema. A consumer that fails hard on an unexpected new field will break the day any new field is added, even a purely additive one, so this is worth fixing before the migration even starts.
• Version negotiation: old consumers keep requesting the old version explicitly, or get it by default if no version header is sent, and receive the old shape; migrated consumers request the new version and receive the new shape. Both are served by the same backing logic wherever possible, translated to two shapes at the response layer, so two independent implementations are not maintained and cannot drift apart.
• Consumer-driven contract testing: every downstream team registers the exact shape it depends on, and the provider's continuous integration runs against every registered contract before any deploy, so whether a change breaks team X is answered in the API owner's own pipeline, not discovered by team X in production days later.
• Usage telemetry: instrument the old shape's usage per consumer identity (a request header, API key, or service identifier), not just "is the old shape used at all," since 100 consumers migrated with 3 stragglers looks identical to a full migration in an aggregate "any usage" metric, but requires a very different retirement decision.

Phase 3: Contract, retiring the old shape

Only after usage telemetry shows the old shape's request volume has been at or below an agreed noise threshold for a sustained window, long enough to catch low-frequency batch consumers that call only monthly, publish a firm sunset date via a Sunset response header or a Deprecation header on the old shape's responses, give a fixed notice period, and only then remove it.

Verifying it is actually safe to retire

• A dashboard or query that answers "which consumer, by identity, called the old shape in the last N days," not just an aggregate count.
• A dry-run period where the old shape is served from the same new-shape-backed logic, so its data is guaranteed consistent with the new shape, purely to catch a consumer still silently depending on an old-shape quirk before it is physically removed.
• A short dark-launch window where the old shape returns an explicit, documented error instead of data, so any straggler consumer fails loudly and immediately, with a clear pointer to the migration guide, rather than silently, surfacing stragglers telemetry missed.

Worked example

An inventory API changes warehouseLocation from a single free-text string like "SFO-3" into a structured object with a code and a region.

• Expand: add warehouseLocationV2: {code, region} alongside the existing warehouseLocation: string. Both fields are populated from the same underlying data on every response.
• Coexist: consumers migrate to warehouseLocationV2 at their own pace; contract tests registered by 40 known consumer teams all continue passing, since none of them broke, they simply have an extra field they can ignore or adopt.
• Telemetry over 6 weeks shows the old warehouseLocation field requested by 3 remaining consumer identities, down from all 40 at expand time.
• Those 3 teams are contacted directly, since telemetry names exactly who, not just a non-zero number; once they confirm migration, the old field is removed with a Sunset header giving 30 days' notice, then dropped.

Trade-offs and pitfalls

Serving both shapes simultaneously from the same underlying logic costs real engineering time, a translation layer and dual test coverage, for the entire coexistence window, which can run months for an API with hundreds of consumers on different release cadences; that cost is the price of avoiding a flag-day cutover, not a shortcut around it. A common wrong turn is treating "usage dropped to near zero" as sufficient to retire without per-consumer identity: aggregate near-zero usage can still be one high-value consumer's monthly batch job, and retiring on the aggregate alone risks silently breaking exactly the consumer least likely to notice quickly. Skipping the tolerant-reader requirement on consumers is the single most common root cause of "we made an additive, backward-compatible change and it still broke someone": a consumer that deserializes strictly, treating an unknown field as an error, turns every additive change into a breaking one for that specific consumer, regardless of how careful the API owner is.

Direct answer

Split the surface into two contracts that share nothing but the partner's identity: OAuth2 (an authorization framework that lets a partner application act with specific, scoped permissions without ever seeing a user's password) for inbound calls the partner makes to you, and signed webhooks for outbound calls you make to the partner. The piece that actually lets the partner trust a webhook is a signature computed with a keyed hash over the exact bytes delivered, using a secret only the two of you share, so the partner can prove the request originated from you and was not tampered with in transit or replayed from an old capture.

Structured elaboration

Inbound: OAuth2 grant types for partner applications

• client_credentials, for server-to-server calls where there is no end user in the loop (a partner's backend pulling data on its own behalf). The partner authenticates with a client id and secret directly against a token endpoint.
• authorization_code with Proof Key for Code Exchange (PKCE), for calls made on behalf of one of your users through a partner's app; PKCE binds the eventual token exchange to the same client that started the authorization flow, closing off interception of the authorization code by anything else running on the user's device.
• Access tokens carry explicit scopes (for example orders:read, orders:write) so a partner's stolen or leaked token is limited to exactly what it was issued for, not full account access.
• Token endpoint: POST /oauth/token with grant_type, credentials, and requested scope; response includes access_token, expires_in, and, for the delegated flow, a refresh_token.

Outbound: webhook registration and delivery

• POST /webhooks lets the partner register a callback url and the event types it wants (for example order.created); response returns a webhook_id and a delivery secret used only for signing, never transmitted with each delivery.
• Each delivery is an ordinary POST to the partner's registered URL carrying the event body plus three headers: a delivery id (for the partner's own dedup), a timestamp, and a signature.
• Retries on failure: a 2xx response marks the delivery done; a 4xx (other than 429) is treated as a permanent rejection and is not retried; a 429 or 5xx is retried with exponential backoff and jitter, up to a bounded number of attempts, after which the delivery moves to a dead-letter state the partner can inspect through an API rather than losing the event silently.

How the partner verifies a webhook actually came from you (the signature contract):

• The signed material is the timestamp concatenated with the exact request body, hashed with a keyed-hash message authentication code, HMAC-SHA256, using the shared secret. Including the timestamp in what gets signed, and requiring the partner to reject anything outside a small tolerance window (for example five minutes), is what stops a captured, valid request from being replayed later.
• The partner recomputes the same hash locally from the raw bytes it received and the shared secret, and compares it to the signature header using a constant-time comparison, never a plain string equality check, since a naive comparison can leak timing information about how many leading bytes matched.

Worked example

A fully pinned, reproducible signature. Fixed inputs: secret whsec_9f8a1c2d3e4f5061728394a5b6c7d8e9, timestamp 1737550496, and this exact JSON body:

{"type":"order.created","data":{"order_id":"ord_7841","amount_cents":4899}}

The signed payload is {timestamp}.{body} (a literal period joining the two), and the signature is the hex-encoded HMAC-SHA256 of that string:

const crypto = require("crypto");
const secret = "whsec_9f8a1c2d3e4f5061728394a5b6c7d8e9";
const timestamp = "1737550496";
const body = JSON.stringify({"type":"order.created","data":{"order_id":"ord_7841","amount_cents":4899}});
const signedPayload = timestamp + "." + body;
const signature = crypto.createHmac("sha256", secret).update(signedPayload).digest("hex");
console.log(signature);
// bdc32fe8ed1f55d6912031fca37ed221ad285aa2177d6e2b11c4d92a8981af5c

Running this exact snippet against these exact inputs reproduces bdc32fe8ed1f55d6912031fca37ed221ad285aa2177d6e2b11c4d92a8981af5c every time; that is what the partner would compute from the raw request it receives, and it must match the header the request was sent with:

POST https://partner.example/webhook
Content-Type: application/json
X-Webhook-Id: wh_2210
X-Delivery-Id: del_7734
X-Timestamp: 1737550496
X-Signature: sha256=bdc32fe8ed1f55d6912031fca37ed221ad285aa2177d6e2b11c4d92a8981af5c

{"type":"order.created","data":{"order_id":"ord_7841","amount_cents":4899}}

If the partner receives the same X-Delivery-Id twice (a legitimate retry after a dropped response), it dedupes on that id rather than re-processing the order a second time.

Trade-offs and pitfalls

• Signing with a shared secret (HMAC) is simple to implement on both sides but requires securely distributing and rotating that secret; an alternative is asymmetric signing where you publish a public key (or a JSON Web Key Set) and sign with a private key, which removes the shared-secret distribution problem at the cost of more moving parts, worth naming as the trade-off even if HMAC is the simpler default.
• A pitfall specific to the signature: signing only the body and not the timestamp lets an attacker replay a captured, validly-signed request indefinitely; the timestamp-in-the-signed-material plus a tolerance window is what closes that gap, and it is easy to omit by accident.
• Retrying 4xx responses (other than 429) as if they were transient is a common mistake: a 400 usually means the partner's endpoint rejected the payload shape and will keep rejecting it, so retrying just burns delivery attempts and delays discovery of a real integration bug; only 429 and 5xx are genuinely worth retrying.
• Rotating the webhook secret is disruptive if not planned for: the contract should support two active secrets during a rotation window (deliveries signed with either are accepted) so the partner can update its verification code without a coordinated cutover instant.