Requirements:
- Real-time streaming text completions via WebSocket
- Support reconnect + resume partial responses
- Backpressure/throttling, fair usage
- Versioning strategy that avoids client interruption
High-level contract:
- Connect: wss://api.example.com/v1/stream?api_key=… or use Sec-WebSocket-Protocol: "stream-v1"
- Auth via bearer token header or initial JSON auth message
- Messages are JSON framed with "type" field.
Core message types:
- client -> server:
- "start": {request_id, model, prompt, params:{max_tokens, temperature}, resume_token?:null}
- "ack": {request_id, seq} — client consumed up to seq (flow control)
- "cancel": {request_id}
- server -> client:
- "started": {request_id, resume_token, initial_metadata}
- "delta": {request_id, seq, text_chunk}
- "error": {request_id, code, message}
- "complete": {request_id, seq, finish_reason}
- "heartbeat": {ts}
Resume semantics:
- Server assigns monotonic seq numbers per request. Each delta includes seq.
- resume_token encodes request_id + last_seq + signature + ttl. On reconnect client sends "start" with resume_token; server validates and resumes sending from last_seq+1. Server retains partial state for TTL (e.g., 5 minutes) or persists for longer tiers.
Throttling / backpressure:
- Per-connection token bucket + per-request concurrency limits.
- Default: 60 requests/min, 4 concurrent streaming requests per user; burst capacity 10.
- Flow-control via client "ack(seq)" — server stops sending when unacked window exceeds N (e.g., 64 KB or 32 messages). If window full, server pauses generation until ACK or drops/errored with 429.
- On overload return "error" with code "rate_limited" and Retry-After.
Reconnect behavior:
- Client reconnects, re-auths, resends start with resume_token. If token expired, client may re-send full request; server can restart generation with new request_id.
- Recommend exponential backoff with jitter.
Idempotency and consistency:
- request_id client-generated UUID to dedupe repeated starts.
- resume_token signed to prevent tampering.
Versioning strategy:
- Use major-version in WebSocket subprotocol and URL (v1, v2). Maintain backward compatibility within patch/minor. New major version introduced for breaking changes.
- Support negotiation: client may send Sec-WebSocket-Protocol list; server picks highest compatible. Provide a "compatibility" header in initial "started" message for feature detection.
- Deprecation policy: announce at least 90 days, dual-run endpoints (v1 still available), migration guide and automated tooling to translate between versions where feasible.
Observability & safety:
- Include request-level IDs in logs, expose usage headers periodically (e.g., "quota": {remaining, reset}).
- Monitor slow consumers and apply backpressure or soft disconnects.
Trade-offs:
- Keeping resume state server-side increases memory — mitigate via TTL/persistence or storing checkpoints.
- Embedding major version in URL simplifies routing; subprotocols allow negotiated compatibility.
