API and Interface Design for Distributed Services Questions

Direct answer

Design this as a small, versioned, resource-oriented contract: one endpoint for synchronous scoring, one for asynchronous batch scoring, and one for model metadata, each with an explicit JSON schema, a URL-based major version, bearer-token authentication scoped by tenant, and a structured error body so every client in every region gets the same shape whether the request succeeds or fails. The goal is to promise the client just enough that the model behind the endpoint can change without the promise changing.

Framework

Endpoints

Sample request/response

POST /v1/models/reco-v2/score
{
  "request_id": "b6b8f8f0-6e2b-4a35-9b2e-6b6a2b6e2b6a",
  "features": {
    "user_id": "u_123",
    "country": "US",
    "candidate_items": ["A", "B", "C"]
  },
  "options": { "top_k": 2 }
}

200 OK
{
  "request_id": "b6b8f8f0-6e2b-4a35-9b2e-6b6a2b6e2b6a",
  "model_version": "2026-05-15",
  "predictions": [
    { "item_id": "B", "score": 0.92 },
    { "item_id": "A", "score": 0.77 }
  ]
}

Authentication and authorization

• Bearer tokens: JSON Web Tokens (JWT), issued by a central identity service, carrying a tenant_id claim and scopes such as predict:read and predict:batch.
• Authorization: role-based access control (RBAC). The gateway checks the token's scopes against the endpoint being called, and the tenant_id claim against the model's owning tenant, before forwarding the request.
• Transport: mutual Transport Layer Security (mTLS) between regional callers and the gateway, so both sides authenticate each other at the connection level, in addition to the per-request bearer token.
• Tokens are short-lived and refreshed through an OAuth 2.0 client-credentials flow (a machine-to-machine token exchange where the calling service, not a human user, proves its own identity to obtain a fresh token), so a leaked token has a bounded blast radius.

Versioning strategy

• The major version lives in the URL (/v1/) because it is the version of the contract shape, what fields exist and what they mean, not the version of the model. The model's own version is a separate response field (model_version), since a model can change on its own schedule while the contract stays stable for months.
• Minor, backward-compatible changes (new optional fields) ship without a version bump.
• A breaking change to the contract shape gets a new major version, and the old version is marked deprecated once the new one is stable.

Error codes

{
  "error_code": "MISSING_FEATURE",
  "message": "features.user_id is required",
  "request_id": "b6b8f8f0-6e2b-4a35-9b2e-6b6a2b6e2b6a",
  "retryable": false
}

Quota is communicated, not enforced, at this layer: each response carries X-RateLimit-Limit and X-RateLimit-Remaining headers (a widely used convention, though not from a single ratified standard), and a caller that exceeds its quota gets 429 with a Retry-After value telling it how long to wait. Which limiting algorithm sits behind that response is an implementation detail of the service, not part of the contract a client codes against.

Backward compatibility

• Additive only within a major version: new optional request fields, new optional response fields. Clients are expected to ignore fields they do not recognize (a tolerant reader).
• A field's meaning or type is never repurposed within a version.
• An OpenAPI document (a machine-readable API description) is published per major version, so client libraries can be generated and diffed automatically against it.

Worked example

A concrete, backward-compatible evolution: version 1's response is predictions: [{item_id, score}]. A later minor release adds an optional explanation field without bumping the version:

{
  "request_id": "b6b8f8f0-6e2b-4a35-9b2e-6b6a2b6e2b6a",
  "model_version": "2026-06-01",
  "predictions": [ { "item_id": "B", "score": 0.92 } ],
  "explanation": { "method": "shap", "top_features": ["country", "recency"] }
}

Old clients that only read predictions are unaffected. New clients that know to read explanation get more detail. This is what "additive, optional field" means in practice, and it is why either side treating the field's absence as an error would be a bug, not a feature.

Trade-offs and pitfalls

• Keeping model version only in the response makes it hard for a client to pin an exact model for reproducible results. If that matters, add an optional model_version request field and return 409 Conflict if the pinned version no longer exists.
• Tying authorization purely to a coarse scope like predict:batch is simple but blunt. If a tenant later needs per-model access control, retrofitting that into the token shape is a breaking change, so it is worth reserving a model-scoped claim from the start even if it is unused initially.
• A common pitfall is treating an async batch job as "the same request, just later." Batch has different failure semantics, partial success across many rows, that a single success/failure flag cannot represent, so the job-status endpoint needs its own richer schema with per-row outcome counts, not just a status string.

Give each stage (intent detection, slot-filling, response generation) its own independently versioned schema and its own deployment, and compose them through a stage contract every stage implements: accept a documented input shape for its declared version, return a documented output shape tagged with the version that produced it, and be able to return a degraded result instead of erroring outright when it cannot produce a full answer. The caller-facing response aggregates each stage's status and version rather than presenting one all-or-nothing result, so upgrading slot-filling's schema never requires redeploying intent-detection or response-generation in lockstep.

Per-stage versioning

Each stage publishes its own schema version (for example intent-detector.v2), independent of the others. The orchestrator's contract with each stage is version-negotiated: it declares the version it expects, and the stage either serves that version or responds with a documented version_unsupported status the orchestrator can react to, by falling back to a default or failing just that stage.

flowchart LR
  A[Caller request] --> O[Orchestrator]
  O -->|intent-detector.v2| B[Intent Detection]
  O -->|slot-filler.v3| C[Slot Filling]
  O -->|response-gen.v1| D[Response Generation]
  B --> O
  C --> O
  D --> O
  O --> R[Pipeline response envelope]

Schema evolution rule per stage

Within one version, changes are additive-only (new optional fields do not bump the version); a genuinely breaking change to one stage's input or output bumps only that stage's version. Because the orchestrator negotiates version per stage independently, slot-filling can move from v2 to v3 on its own release schedule while intent-detection stays on v2 and response-generation stays on v1, neither of the other two needs a coordinated redeploy.

The response envelope: what the caller sees on a degraded stage

This is the part of the design that most directly answers what the caller sees:

• status: success (every stage returned a complete result), partial (at least one stage returned a lower-confidence or fallback result but the pipeline still produced an answer), or failed (a required stage could not produce any usable result).
• Each stage's result is tagged with its own version and a degraded boolean, so the caller can tell not just that something degraded, but which stage, and under which schema version, which matters for tracing a regression back to a specific stage's rollout.
• A degraded stage returns a documented fallback shape instead of erroring: for response-generation, a clarifying question instead of a personalized reply; for slot-filling, the slots it did manage to resolve, with unresolved slots explicitly null rather than omitted, so the caller can tell "we don't have your departure city" apart from "we didn't attempt to resolve departure city."

Worked example

Pipeline response when slot-filling degrades to only partially resolving slots:

{
  "status": "partial",
  "traceId": "trace_88f2",
  "intent": { "version": "intent-detector.v2", "degraded": false, "name": "book_flight", "confidence": 0.92 },
  "slots": { "version": "slot-filler.v3", "degraded": true, "resolved": { "departure": "SFO", "arrival": null } },
  "response": { "version": "response-gen.v1", "degraded": true, "text": "I have your departure as SFO, what's the arrival city?" }
}

A caller reading this envelope can see specifically that slot-filling is the degraded stage, not response-generation, even though response-generation's output also changed as a consequence, and can decide whether to retry that stage, show the clarifying question as-is, or escalate to a human agent, without knowing anything about why slot-filling degraded internally.

Trade-offs and pitfalls

Per-stage independent versioning adds real coordination overhead: the orchestrator has to know which version combinations are actually validated together, since v2 intent-detector paired with v3 slot-filler might never have been tested against v1 response-generation, so a compatibility matrix or a contract-test suite across stage-version combinations is not optional, it is the real cost of buying independent deploys. A common wrong turn is designing the envelope so a degraded stage simply omits its field instead of returning an explicit degraded marker: an omitted field is indistinguishable from "this stage does not exist in this pipeline version" to a client, whereas an explicit flag is unambiguous. Scope note: this design covers the contract and versioning boundary between stages; deciding when to mark a stage degraded in the first place, timeout policy, retry budget, circuit-breaker state, is a reliability-engineering decision that sits outside the interface contract itself and is not something this answer prescribes.