Requirements & constraints:
- Low added latency (sub-10ms goal)
- Support model routing by version and experiment (A/B, canary, rollout)
- Auth, per-customer throttling, observability, minimal churn for ML teams
High-level architecture:
Client → Edge LB → API Gateway (stateless, hot path) → Model Router → Authz Service (cached) → Experiment Service (decision & config cache) → Throttler → Model Serving (gRPC/HTTP) → Logging/Obs pipeline (async)
Component responsibilities:
- API Gateway (fast, in-process): validate schema, authenticate token (JWT/mTLS; verify signature with locally cached keys), perform request normalization, add trace id, consult local in-memory routing cache, enforce throttling tokens, forward enriched request.
- Model Router: evaluates routing rules (version header, user attribute, experiment assignment). Runs locally with periodic sync of rules from Experiment Config Store.
- Experiment Service (control plane): defines experiments, traffic split rules, sticky assignment logic, and exposes k/v configs. Persists evaluation metrics for experiment analysis.
- Throttler: token-bucket per-customer and global rate limits. Use Redis/Lua for distributed rate limits when needed.
- Authz Service (control plane): issues/rotates keys, maps tokens to tenant/quota.
- Observability pipeline: async exporter (Kafka) to metrics/traces/logs store.
Routing by version or experiment:
- Priority order: explicit client-specified version > sticky experiment assignment > default model.
- Assignment algorithm: deterministic hashing on stable key (user_id, device_id). For weighted splits, hash mod 100 vs thresholds. Sticky cookie/header ensures consistent user experience.
- Canary/rollout: support rules like “X% of requests for tenant T to version v2 where latency improvement > threshold” driven by Experiment Service flags. Gateway decision is local (no network call) using cached flag + short TTL (e.g., 1s) to minimize latency.
- Header-based overrides for debugging (admin header whitelist).
Minimizing latency:
- Keep decision logic in-process, use read-through caches for auth keys and experiment configs with background refresh.
- Use binary protocols (gRPC) to model servers, keep connections warm (connection pools).
- Async, batched logging to reduce blocking; sample traces under high load.
Telemetry to emit at gateway (per-request labels: tenant, model_version, experiment_id, route_outcome):
- Metrics (prometheus):
- request_count{tenant,model_version,experiment,endpoint,status}
- latency_ms_histogram{...} (client->gateway, gateway->model, end-to-end)
- error_count{...} (4xx/5xx, auth failures, throttle rejections, model errors)
- throttled_count{tenant}
- routed_count_by_rule{rule_id}
- Traces (distributed): trace id across gateway → model server; spans for gateway decision, auth, throttling, model inference.
- Logs: structured request logs (id, tenant, model_version, experiment, input_size, resp_size, latency_breakdown). Keep PII out; redact sensitive fields.
- Feature-distribution sketches/sample payloads (privacy aware): periodically emit sample of inputs per model for drift detection (store in separate telemetry pipeline with access controls).
- Experiment metrics: per-experiment conversion/quality signals and model-specific latencies/error rates, aggregated and exported to Experiment Service.
Example flow:
- Gateway receives request, validates JWT (local cache).
- Lookup routing config locally; hash user_id → assignment (e.g., experiment E: user -> v2).
- Check throttler token; if exceeded return 429 with X-RateLimit headers.
- Forward to model v2 via gRPC, add trace headers.
- Asynchronously push metrics and structured log to observability pipeline.
Trade-offs:
- Local caches add eventual consistency on config changes; mitigate with short TTL and push invalidation for urgent rollouts.
- Heavy telemetry sampling reduces visibility; tune sampling and aggregate histograms for scale.
This design gives ML teams fast, flexible routing and rich telemetry while keeping the gateway latency minimal and the control plane expressive for experiments and rollouts.
