API and Interface Design for Distributed Services Questions

Direct answer

Split the surface into two contracts that share nothing but the partner's identity: OAuth2 (an authorization framework that lets a partner application act with specific, scoped permissions without ever seeing a user's password) for inbound calls the partner makes to you, and signed webhooks for outbound calls you make to the partner. The piece that actually lets the partner trust a webhook is a signature computed with a keyed hash over the exact bytes delivered, using a secret only the two of you share, so the partner can prove the request originated from you and was not tampered with in transit or replayed from an old capture.

Structured elaboration

Inbound: OAuth2 grant types for partner applications

• client_credentials, for server-to-server calls where there is no end user in the loop (a partner's backend pulling data on its own behalf). The partner authenticates with a client id and secret directly against a token endpoint.
• authorization_code with Proof Key for Code Exchange (PKCE), for calls made on behalf of one of your users through a partner's app; PKCE binds the eventual token exchange to the same client that started the authorization flow, closing off interception of the authorization code by anything else running on the user's device.
• Access tokens carry explicit scopes (for example orders:read, orders:write) so a partner's stolen or leaked token is limited to exactly what it was issued for, not full account access.
• Token endpoint: POST /oauth/token with grant_type, credentials, and requested scope; response includes access_token, expires_in, and, for the delegated flow, a refresh_token.

Outbound: webhook registration and delivery

• POST /webhooks lets the partner register a callback url and the event types it wants (for example order.created); response returns a webhook_id and a delivery secret used only for signing, never transmitted with each delivery.
• Each delivery is an ordinary POST to the partner's registered URL carrying the event body plus three headers: a delivery id (for the partner's own dedup), a timestamp, and a signature.
• Retries on failure: a 2xx response marks the delivery done; a 4xx (other than 429) is treated as a permanent rejection and is not retried; a 429 or 5xx is retried with exponential backoff and jitter, up to a bounded number of attempts, after which the delivery moves to a dead-letter state the partner can inspect through an API rather than losing the event silently.

How the partner verifies a webhook actually came from you (the signature contract):

• The signed material is the timestamp concatenated with the exact request body, hashed with a keyed-hash message authentication code, HMAC-SHA256, using the shared secret. Including the timestamp in what gets signed, and requiring the partner to reject anything outside a small tolerance window (for example five minutes), is what stops a captured, valid request from being replayed later.
• The partner recomputes the same hash locally from the raw bytes it received and the shared secret, and compares it to the signature header using a constant-time comparison, never a plain string equality check, since a naive comparison can leak timing information about how many leading bytes matched.

Worked example

A fully pinned, reproducible signature. Fixed inputs: secret whsec_9f8a1c2d3e4f5061728394a5b6c7d8e9, timestamp 1737550496, and this exact JSON body:

{"type":"order.created","data":{"order_id":"ord_7841","amount_cents":4899}}

The signed payload is {timestamp}.{body} (a literal period joining the two), and the signature is the hex-encoded HMAC-SHA256 of that string:

const crypto = require("crypto");
const secret = "whsec_9f8a1c2d3e4f5061728394a5b6c7d8e9";
const timestamp = "1737550496";
const body = JSON.stringify({"type":"order.created","data":{"order_id":"ord_7841","amount_cents":4899}});
const signedPayload = timestamp + "." + body;
const signature = crypto.createHmac("sha256", secret).update(signedPayload).digest("hex");
console.log(signature);
// bdc32fe8ed1f55d6912031fca37ed221ad285aa2177d6e2b11c4d92a8981af5c

Running this exact snippet against these exact inputs reproduces bdc32fe8ed1f55d6912031fca37ed221ad285aa2177d6e2b11c4d92a8981af5c every time; that is what the partner would compute from the raw request it receives, and it must match the header the request was sent with:

POST https://partner.example/webhook
Content-Type: application/json
X-Webhook-Id: wh_2210
X-Delivery-Id: del_7734
X-Timestamp: 1737550496
X-Signature: sha256=bdc32fe8ed1f55d6912031fca37ed221ad285aa2177d6e2b11c4d92a8981af5c

{"type":"order.created","data":{"order_id":"ord_7841","amount_cents":4899}}

If the partner receives the same X-Delivery-Id twice (a legitimate retry after a dropped response), it dedupes on that id rather than re-processing the order a second time.

Trade-offs and pitfalls

• Signing with a shared secret (HMAC) is simple to implement on both sides but requires securely distributing and rotating that secret; an alternative is asymmetric signing where you publish a public key (or a JSON Web Key Set) and sign with a private key, which removes the shared-secret distribution problem at the cost of more moving parts, worth naming as the trade-off even if HMAC is the simpler default.
• A pitfall specific to the signature: signing only the body and not the timestamp lets an attacker replay a captured, validly-signed request indefinitely; the timestamp-in-the-signed-material plus a tolerance window is what closes that gap, and it is easy to omit by accident.
• Retrying 4xx responses (other than 429) as if they were transient is a common mistake: a 400 usually means the partner's endpoint rejected the payload shape and will keep rejecting it, so retrying just burns delivery attempts and delays discovery of a real integration bug; only 429 and 5xx are genuinely worth retrying.
• Rotating the webhook secret is disruptive if not planned for: the contract should support two active secrets during a rotation window (deliveries signed with either are accepted) so the partner can update its verification code without a coordinated cutover instant.

Direct answer

Frame the decision around the actual constraints rather than the protocols' reputations: which languages across the polyglot stack need first-class client support, whether the traffic is latency-sensitive service-to-service calling or something a human needs to inspect, and whether streaming is a hard requirement. Build a small prototype for the leading candidate against a real workload, compare it structurally rather than by a single benchmark number, and validate the decision with the teams who will own the resulting code before committing company-wide.

Evaluation criteria

• Wire efficiency: gRPC serializes with protocol buffers, a binary format that omits field names and uses variable-length integers, so a given message is typically meaningfully smaller on the wire than the equivalent JSON; REST/JSON is larger but human-readable without any tooling.
• Connection behavior: gRPC runs over HTTP/2, which multiplexes many concurrent requests over a single connection; this avoids head-of-line blocking, the delay where one slow request stuck at the front of a connection holds up every other request queued behind it, a problem plain HTTP/1.1 REST can suffer under high concurrency.
• Streaming support: gRPC has first-class client-streaming, server-streaming, and bidirectional streaming built into the protocol; REST needs bespoke chunking or a separate protocol (such as server-sent events or websockets) bolted on.
• Language and tooling coverage across the actual stack: check every language you run in production, not just the popular ones. gRPC's code generation and runtime maturity vary meaningfully by language.
• Debuggability: a REST/JSON payload is directly inspectable with a browser or curl; gRPC needs a reflection service and a tool like grpcurl to get the same visibility.
• Operational and observability tooling: both integrate with modern tracing systems, but gRPC's HTTP/2 transport sometimes needs extra configuration for proxies and load balancers that were written assuming HTTP/1.1.

What I would benchmark, and how

Build matching prototype services in the two or three languages that make up most of the real traffic, exercise both protocols with the same synthetic load generator and connection count, and compare relative, structural behavior (does one need meaningfully less CPU per request at the same throughput, does one handle the streaming case directly versus requiring a workaround) rather than quoting a single absolute number as if it generalizes. Any specific latency or throughput figure from a single environment is a snapshot of that environment, not a portable fact, so treat it as directional evidence for this specific decision rather than a reusable benchmark.

Worked example

Situation: a polyglot environment (Python data workers, Go services, Java-based batch jobs) needed a new internal path for high-throughput data ingestion. The requirement list included sustained streaming from Go into Python, and low overhead at high request volume. Action: scored REST and gRPC against the criteria above, built matching prototype services in Go and Python, ran both through the same load generator, and reviewed the comparative results with the two teams that would own the resulting code day to day. gRPC's built-in server-streaming matched the ingestion use case directly, while REST would have needed a bespoke chunked-transfer workaround; that single structural fit mattered more in the final call than any specific number either prototype produced. Result: adopted gRPC for the internal high-throughput and streaming paths, kept REST/JSON for a small admin API where human debuggability mattered more than raw efficiency, and put a translating gateway in front so a team that could not migrate immediately was not blocked.

Trade-offs and pitfalls

gRPC has a steeper learning curve than REST, and its debugging story is worse out of the box: budget time for a reflection service and team familiarity with grpcurl rather than assuming curl will still work. Verify gRPC client maturity for every language actually in your stack before committing, not just the two or three most common ones; an internal tool with weak gRPC support becomes a maintenance tax nobody accounted for at decision time.

REST remains the better default for public or third-party-facing APIs, where you cannot force external consumers to adopt a specific client library the way you can for an internal service. The most common mistake in this kind of evaluation is running a benchmark in one environment on one payload shape and generalizing that result to the whole company without a real pilot: always validate with a scoped rollout to one or two teams before treating a prototype's numbers as proof for every future use case.

Direct answer

For a feed with a high insert rate at the head, pick cursor-based (keyset) pagination. It anchors each page to the last item's sort key instead of a row count, so it does not skip or repeat rows when new items appear ahead of the client's read position. Offset-based pagination (OFFSET n LIMIT m) is simpler but silently shifts under concurrent inserts, which is exactly what a frequently-updated feed does continuously.

Framework

What each approach actually does

• Offset pagination: the client asks for "skip N, take M." The server re-runs the ordering on every request and counts N rows in before returning results. A page's identity is a position in a list, and that position can move.
• Cursor (keyset) pagination: the client sends the sort-key value(s) of the last item it saw. The server issues a seek query (WHERE (created_at, id) < (:last_created_at, :last_id) ORDER BY created_at DESC, id DESC LIMIT M). A page's identity is anchored to a specific row's key, not a position.

Sorting by created_at alone is not enough if two rows can share a timestamp. The seek predicate needs a secondary, unique tie-breaker column (commonly id) so (created_at, id) is a total order with no ties.

Worked example

A concrete trace of why offset duplicates a row when an item is inserted at the head, and why cursor pagination does not.

Five existing rows, ordered newest-first by created_at:

Offset pagination, page size 2:

• Page 1: OFFSET 0 LIMIT 2 returns positions 1-2: [105, 104].
• A new row 106 (created_at 2026-01-01T10:05:00Z) is inserted at the head. Every existing row shifts down one position: 105 is now position 2, 104 is position 3, and so on.
• Page 2: OFFSET 2 LIMIT 2 returns whatever now sits at positions 3-4, which is [104, 103]. The client already saw 104 on page 1, so it is duplicated, and the client has no way to notice.

Cursor pagination, same scenario:

• Page 1: no cursor supplied, returns [105, 104]; next_cursor encodes {"id": "104", "created_at": "2026-01-01T10:03:00Z"}.
• Row 106 is inserted at the head.
• Page 2: WHERE (created_at, id) < ('2026-01-01T10:03:00Z', 104) ORDER BY created_at DESC, id DESC LIMIT 2. Row 106's created_at is greater than 104's, so it fails the < predicate and is correctly excluded. The query still returns [103, 102], exactly the rows that follow 104, regardless of what was inserted ahead of it.

Sample cursor-based response shape (page 1 above):

{
  "items": [
    { "id": "105", "created_at": "2026-01-01T10:04:00Z" },
    { "id": "104", "created_at": "2026-01-01T10:03:00Z" }
  ],
  "next_cursor": "eyJpZCI6IjEwNCIsImNyZWF0ZWRfYXQiOiIyMDI2LTAxLTAxVDEwOjAzOjAwWiJ9.f0e3e3b16f56ab33",
  "has_more": true
}

How that cursor was actually built (reproducible, step by step):

1. Payload: {"id": "104", "created_at": "2026-01-01T10:03:00Z"}.
2. Compact JSON serialization: {"id":"104","created_at":"2026-01-01T10:03:00Z"}.
3. Base64url-encode the bytes, no padding: eyJpZCI6IjEwNCIsImNyZWF0ZWRfYXQiOiIyMDI2LTAxLTAxVDEwOjAzOjAwWiJ9.
4. Sign the raw JSON bytes with a server-side hash-based message authentication code (HMAC-SHA256) using a secret only the server holds, and take the first 16 hex characters of the digest as a tamper check: f0e3e3b16f56ab33.
5. Join encoded payload and signature with a period: <base64url>.<signature>.

A production cursor payload would also carry an issued-at timestamp and a time-to-live (TTL): the server checks issued-at plus TTL on every request and, if the cursor has expired, rejects it (for example with a 400 response) and asks the client to restart from the first page, rather than trying to resume mid-sequence from stale coordinates.

Trade-offs and pitfalls

• Cursor pagination cannot jump to an arbitrary page. If the product genuinely needs "go to page 43," you need offset, or a hybrid where a search step lands the client near a position and cursor paging takes over from there.
• If the tie-breaker column is not unique, or is not indexed together with the primary sort key, the seek query degrades toward a scan instead of an index lookup.
• Signing the cursor matters for more than tamper-proofing: an unsigned or unencrypted cursor can leak internal ids and ordering, and a forged cursor could be used to skip access checks that were only applied on page 1.
• A common pitfall is anchoring the cursor to a mutable field, such as an "updated at" timestamp instead of creation time. An edit to any row changes its position under that ordering, so items can be revisited or skipped after an edit, not just after an insert. Prefer an immutable creation-time key plus a stable id tie-breaker.

Make the consistency guarantee a first-class, documented, per-endpoint contract field, not prose buried in a general "about consistency" page: state it in the endpoint reference as a plain-language sentence plus a machine-readable annotation, return it as response metadata on every call rather than only in the docs, and give explicit client guidance for each documented failure mode. The docs, the response shape, and the client SDK all say the same thing, so a developer never has to infer a guarantee from behavior they happened to observe once.

Per-endpoint documentation structure

Every endpoint that carries a consistency guarantee gets the same four-part structure, so a developer scanning multiple endpoints is not re-taught the format each time: a one-sentence plain-language summary, a machine-readable contract block that tooling (including client SDKs) can act on, worked examples of a normal response and each documented failure mode, and explicit guidance for what a client should do on each response.

Two concrete endpoint contracts

Error and status-code semantics, using standard HTTP rather than invented codes

• 409 Conflict: a linearizable write lost a race with a concurrent write to the same resource. Client guidance: re-read the current state, then decide whether to retry with the fresh state or surface the conflict to the end user; never blindly retry the original write unchanged.
• 503 Service Unavailable with a Retry-After header (a standard header telling the client how many seconds to wait before trying again): the system cannot currently meet the documented guarantee at all, for example during a network partition. Client guidance: respect Retry-After, do not tight-loop.
• A stale-but-within-bound read is not an error at all. It returns a normal 200 OK with stalenessMs populated, since serving a bounded-stale read is the correct, expected behavior for that endpoint's documented contract, not a failure. If a read happens to fall outside its documented bound, for example during a replica lag spike, it is still a normal 200 OK, since refusing to return data would be worse than returning a flagged stale value, but the response's stalenessMs exceeds the documented bound, so the client can detect the violation itself rather than trusting the service-level agreement (SLA: a documented commitment about a specific response characteristic) blindly. Reusing an unrelated HTTP status code to signal this would be a real mistake: status 425 is a real, registered code, defined in RFC 8470 to mean "Too Early," used specifically to reject requests that risk a TLS replay attack during 0-RTT connection setup. Repurposing 425 to mean "stale read" would collide with its actual, unrelated meaning and could mislead HTTP-aware infrastructure sitting in the request path (proxies, gateways) that already interprets that code correctly.

Client-side guidance, concretely

A typed client exposes the guarantee in its method signature, for example a read call that accepts a maximum acceptable staleness and raises a client-side error if the returned stalenessMs exceeds what the caller asked for, even when the server's documented bound was technically met. Retry guidance is written per status code, not generically: a 409 requires a fresh read before any retry, never a blind retry of stale write data; a 503 retry must honor Retry-After; a 200 with a high stalenessMs is not something to retry at all, since retrying a read does not fix replication lag, it is something to surface to the caller or fall back to a stronger-guarantee endpoint if one exists.

Worked example, full round trip

Request: GET /v1/analytics/summary?userId=u_501. Normal response:

{
  "userId": "u_501",
  "totalOrders": 42,
  "asOf": "2026-07-18T09:00:00Z",
  "stalenessMs": 1800
}

Here stalenessMs: 1800 (1.8 seconds) is well within the documented 5000 millisecond bound, so a client comparing this number against its own tolerance can proceed with confidence, without knowing anything about how the 5-second bound is achieved internally.
Degraded response, bound exceeded, still 200 since the data itself is valid, just late:

{
  "userId": "u_501",
  "totalOrders": 42,
  "asOf": "2026-07-18T08:59:52Z",
  "stalenessMs": 8400
}

A client-side check of stalenessMs > 5000 tells this specific caller, on this specific response, that the documented guarantee was not met this time, without waiting for an incident report or a status page update.

Trade-offs and pitfalls

Returning staleness metadata on every response has a real cost: the read path must always know and report its own commit or replication timestamp, which is not free to compute or propagate on every code path, and some teams skip this and only document a static bound, leaving the client no way to detect a violation in the moment, only after the fact from an incident report. The common wrong turn worth calling out explicitly is inventing a custom HTTP status code, or repurposing a real but differently-scoped one, to signal an application-level condition like a stale read: standard status codes have registered meanings that other HTTP-aware infrastructure already interprets, and a normal 200 with documented body or header metadata is both more correct and more compatible with that infrastructure than borrowing an unrelated code. The main long-term risk is documentation and the runtime contract drifting apart: if the docs say a 5 second bound but nothing enforces or alerts on stalenessMs exceeding it in production, the documented guarantee quietly becomes fiction. The fix is treating the machine-readable contract block and the response metadata as the same source of truth, ideally generated from one schema, rather than hand-maintained prose alongside a separately hand-maintained response shape.

Give each stage (intent detection, slot-filling, response generation) its own independently versioned schema and its own deployment, and compose them through a stage contract every stage implements: accept a documented input shape for its declared version, return a documented output shape tagged with the version that produced it, and be able to return a degraded result instead of erroring outright when it cannot produce a full answer. The caller-facing response aggregates each stage's status and version rather than presenting one all-or-nothing result, so upgrading slot-filling's schema never requires redeploying intent-detection or response-generation in lockstep.

Per-stage versioning

Each stage publishes its own schema version (for example intent-detector.v2), independent of the others. The orchestrator's contract with each stage is version-negotiated: it declares the version it expects, and the stage either serves that version or responds with a documented version_unsupported status the orchestrator can react to, by falling back to a default or failing just that stage.

flowchart LR
  A[Caller request] --> O[Orchestrator]
  O -->|intent-detector.v2| B[Intent Detection]
  O -->|slot-filler.v3| C[Slot Filling]
  O -->|response-gen.v1| D[Response Generation]
  B --> O
  C --> O
  D --> O
  O --> R[Pipeline response envelope]

Schema evolution rule per stage

Within one version, changes are additive-only (new optional fields do not bump the version); a genuinely breaking change to one stage's input or output bumps only that stage's version. Because the orchestrator negotiates version per stage independently, slot-filling can move from v2 to v3 on its own release schedule while intent-detection stays on v2 and response-generation stays on v1, neither of the other two needs a coordinated redeploy.

The response envelope: what the caller sees on a degraded stage

This is the part of the design that most directly answers what the caller sees:

• status: success (every stage returned a complete result), partial (at least one stage returned a lower-confidence or fallback result but the pipeline still produced an answer), or failed (a required stage could not produce any usable result).
• Each stage's result is tagged with its own version and a degraded boolean, so the caller can tell not just that something degraded, but which stage, and under which schema version, which matters for tracing a regression back to a specific stage's rollout.
• A degraded stage returns a documented fallback shape instead of erroring: for response-generation, a clarifying question instead of a personalized reply; for slot-filling, the slots it did manage to resolve, with unresolved slots explicitly null rather than omitted, so the caller can tell "we don't have your departure city" apart from "we didn't attempt to resolve departure city."

Worked example

Pipeline response when slot-filling degrades to only partially resolving slots:

{
  "status": "partial",
  "traceId": "trace_88f2",
  "intent": { "version": "intent-detector.v2", "degraded": false, "name": "book_flight", "confidence": 0.92 },
  "slots": { "version": "slot-filler.v3", "degraded": true, "resolved": { "departure": "SFO", "arrival": null } },
  "response": { "version": "response-gen.v1", "degraded": true, "text": "I have your departure as SFO, what's the arrival city?" }
}

A caller reading this envelope can see specifically that slot-filling is the degraded stage, not response-generation, even though response-generation's output also changed as a consequence, and can decide whether to retry that stage, show the clarifying question as-is, or escalate to a human agent, without knowing anything about why slot-filling degraded internally.

Trade-offs and pitfalls

Per-stage independent versioning adds real coordination overhead: the orchestrator has to know which version combinations are actually validated together, since v2 intent-detector paired with v3 slot-filler might never have been tested against v1 response-generation, so a compatibility matrix or a contract-test suite across stage-version combinations is not optional, it is the real cost of buying independent deploys. A common wrong turn is designing the envelope so a degraded stage simply omits its field instead of returning an explicit degraded marker: an omitted field is indistinguishable from "this stage does not exist in this pipeline version" to a client, whereas an explicit flag is unambiguous. Scope note: this design covers the contract and versioning boundary between stages; deciding when to mark a stage degraded in the first place, timeout policy, retry budget, circuit-breaker state, is a reliability-engineering decision that sits outside the interface contract itself and is not something this answer prescribes.