API Documentation and Developer Experience Questions

A good API doc for a model inference endpoint should be concise, machine- and developer-friendly, and include the following sections:

• Summary / Purpose: one-line description of what the endpoint does and intended use.
• Endpoint URL and path params: e.g., POST https://api.example.com/v1/models/{model_id}/infer
• HTTP method: POST (or GET/PUT as appropriate)
• Headers: required headers (Authorization: Bearer <token>, Content-Type: application/json, Accept: application/json), optional headers (X-Request-ID, X-Trace-Id)
• Authentication: scheme (Bearer token, API key, OAuth2), token scope, example
• Rate limits & quotas: requests/minute, burst policy, behavior on limit (429), headers returned (X-RateLimit-Remaining)
• Request body schema: JSON schema or example; list required fields, types, max sizes, validation rules, preprocessing expectations
• Query params & path params: explain purpose and defaults
• Response schema: success structure with field descriptions (predictions, scores, model_version, timestamps), content-type
• Status codes & error responses: 200, 400, 401, 403, 404, 429, 500 with example error payloads
• Examples: sample request and response for common use-cases
• Latency / SLAs: typical p95 latency, timeouts
• Retry semantics & idempotency: when to retry, backoff guidance
• Versioning & changelog: model_version, API versioning policy
• Security & privacy notes: PII handling, data retention, logging
• SDKs & cURL examples, FAQs

Example JSON request (backend integrator style):

{
  "input": {
    "text": "Classify sentiment for this customer review: 'The product arrived late but works well.'"
  },
  "parameters": {
    "top_k": 3,
    "temperature": 0.0
  },
  "request_id": "abc123",
  "model_version": "sentiment-v2.1"
}

Example JSON response:

{
  "request_id": "abc123",
  "model_version": "sentiment-v2.1",
  "predictions": [
    { "label": "neutral", "score": 0.72 },
    { "label": "positive", "score": 0.20 },
    { "label": "negative", "score": 0.08 }
  ],
  "processing_time_ms": 84
}

Include concrete examples for errors (400 with validation errors, 429 with retry-after), and a short curl/snippet showing Authorization header.

Overview
/v1/generate — POST — synchronous text-generation endpoint. Accepts JSON body; returns generated text and metadata. Auth: Bearer token via Authorization header. Content-Type: application/json.

Request body schema (application/json)

• prompt (string) — required
  • description: input text or instruction for the model
  • max length: 50,000 chars
• max_tokens (integer) — optional, default: 256
  • valid range: 1–4096
• temperature (number) — optional, default: 1.0
  • valid range: 0.0–2.0 (float)
  • semantics: higher = more random
• top_p (number) — optional, default: 1.0
  • valid range: 0.0–1.0 (float)
  • semantics: nucleus sampling
• stop (array[string] or string) — optional, default: null
  • semantics: token(s) where generation stops; max 5 items, each ≤ 100 chars

Headers

• Authorization: Bearer <API_KEY> — required
• Content-Type: application/json — required
• Accept: application/json — recommended

Successful response (200)
Body (application/json):
{
"id": "gen_12345",
"model": "gpt-xyz",
"prompt": "<echoed prompt or id>",
"generated_text": "<string>",
"tokens_used": {
"prompt_tokens": 12,
"completion_tokens": 45,
"total_tokens": 57
},
"finish_reason": "stop" | "length" | "eos",
"created_at": "2025-11-22T12:34:56Z"
}

Error responses
400 Bad Request — invalid input

• Cause examples: missing prompt, invalid types, values outside ranges, too-long prompt, malformed JSON.
• Body:
  {
  "error": {
  "code": "invalid_request",
  "message": "prompt is required and must be a string",
  "details": {"field":"prompt"}
  }
  }

429 Too Many Requests — rate limit / quota exceeded

• Headers: Retry-After: seconds
• Body:
  {
  "error": {
  "code": "rate_limit_exceeded",
  "message": "Rate limit exceeded. Retry after 30 seconds."
  }
  }

500 Internal Server Error — server-side

• Body:
  {
  "error": {
  "code": "server_error",
  "message": "An unexpected error occurred. Try again later.",
  "trace_id": "abc123" // for support/debugging
  }
  }

Validation and behavior notes

• If both temperature and top_p provided, they are combined per sampling rules; use either or both.
• If stop matched mid-token, model returns finish_reason "stop".
• max_tokens caps generated tokens; may return fewer tokens.
• Rate-limiting and per-account quotas apply; prefer exponential backoff on 429.

Curl example

curl -X POST "https://api.example.com/v1/generate" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "Write a concise summary of transformers in NLP.",
    "max_tokens": 120,
    "temperature": 0.7,
    "top_p": 0.9,
    "stop": ["\n\n"]
  }'

Expected example response (200)

{
  "id": "gen_01F...",
  "model": "gpt-xyz",
  "prompt": "Write a concise summary of transformers in NLP.",
  "generated_text": "Transformers use self-attention to model long-range dependencies, enabling parallel training and state-of-the-art performance across NLP tasks...",
  "tokens_used": {
    "prompt_tokens": 8,
    "completion_tokens": 42,
    "total_tokens": 50
  },
  "finish_reason": "stop",
  "created_at": "2025-11-22T12:34:56Z"
}