API Gateways and API Management Questions

Requirements:

• Expose REST/JSON endpoints externally; internally call gRPC services (unary & bi/streaming).
• Translate headers, payloads and errors.
• Per-route auth (JWT/OAuth/mTLS).
• Graceful fallback when gRPC unavailable.
• Support large streaming uploads from browsers.

High-level architecture:

• External edge (TLS, WAF, CDN) → API Gateway (HTTP/1.1 & HTTP/2 + gRPC-Web) → Service Mesh / gRPC backends.
• Gateway implemented with Envoy or custom gateway using gRPC client libs; runs in k8s, autoscaled.

Core components & responsibilities:

1. Protocol translation: gateway accepts REST/JSON and uses mapping rules (OpenAPI + protobuf annotations) to convert to gRPC calls. For browser streams use gRPC-Web or chunked transfer mapping to gRPC streaming RPCs.
2. Header & metadata translation: whitelist/transform headers to gRPC metadata; map gRPC trailers to HTTP responses. Normalize trace/traceparent headers.
3. Error mapping: map gRPC status codes to HTTP statuses (e.g., NOT_FOUND→404), include structured JSON body {code, message, details}. Preserve retryable flags.
4. Per-route auth: policy engine (e.g., Istio/Envoy ext_authz) enforces JWT verification, OAuth introspection, or mTLS; attach caller identity as gRPC metadata.
5. Fallback mechanism: circuit breaker + health-checks. If backend unhealthy, gateway can:
   • serve cached responses (for idempotent routes)
   • return a degraded-but-meaningful response (HTTP 503 + Retry-After)
   • route to a standby service or read-only replica
   • enqueue requests into durable queue for async processing (for uploads)
6. Streaming uploads from browsers:
   • Preferred: use gRPC-Web (Envoy gRPC-Web filter) so browsers stream to gateway and gateway forwards to gRPC streaming RPC.
   • For large files: support chunked/resumable uploads (tus protocol or signed URLs):
     • Option A: Client uploads directly to object storage (S3) via signed upload URL; gateway orchestrates and sends final reference to gRPC backend.
     • Option B: Chunked upload over gRPC streaming with resumable offsets; gateway buffers minimally, streams to backend; use flow-control and backpressure, limit memory by writing to temp storage (local disk or object storage) if needed.
   • Implement rate limits, per-upload size quotas, virus scanning pipeline.

Scalability & reliability:

• Use connection pools for gRPC, HTTP/2 multiplexing, and keepalives.
• Centralized config (xDS) for routing, retries, timeouts.
• Observability: structured logs, metrics (request latencies, stream durations), tracing.
• Trade-offs: gRPC-Web adds complexity but enables browser streaming; direct upload to storage reduces gateway load and simplifies retries.

This design provides protocol translation, secure per-route auth, robust error handling, streaming support, and graceful degraded behavior when backends fail.

Requirements:

• Functional: authentication (JWT/OAuth2), per-route authorization, request validation, global + per-client rate limiting, retries, circuit breakers.
• Non-functional: low latency (<50ms overhead), high throughput (millions RPS), multi-region HA, hot-reloadable policies, strong telemetry.

High-level architecture:
Client → (edge LB) → API Gateway Fleet (stateless containers) → Authz/Authn service & RateLimit service & Policy service → Backend services
Support components: distributed cache (Redis/cluster), config store (etcd/consul), metrics backend (Prometheus+Grafana), tracing (Jaeger), service mesh optional (Envoy).

Core components & responsibilities:

1. Gateway (stateless): performs TLS termination, request parsing, JWT validation (with local public key cache), request validation (JSON Schema), enforces per-route policies, forwards requests. Runs circuit-breaker and retry logic per upstream host.
2. Authn/Authz: token introspection, OAuth flows, userinfo; returns claims. Heavy checks done asynchronously; gateway caches results short-lived.
3. Rate limiter: a distributed, strongly consistent quota service using token-bucket implemented with Redis Lua scripts or a CRDT-based global limiter for multi-region. Supports global, per-client, per-route limits.
4. Policy/config store: authoritative configs in etcd; gateway nodes watch for changes and hot-reload via versioned configs with validation and canary rollout.
5. Telemetry & tracing: gateway emits structured logs, metrics (per-route latency, error rates), and spans to tracing backend. Alerting based on SLOs.
6. Resilience: circuit-breaker per upstream instance (sliding window failure rate), bulkhead isolation (per-upstream worker pools), retries with exponential backoff and idempotency checks.

Scaling & HA:

• Make gateways stateless; scale horizontally with autoscaling groups behind anycast or regional LBs.
• Multi-region deployment with active-active; use consistent hashing or geo-DNS to route.
• Redis/RateLimit: shard and use HA Redis (sentinel/cluster) or a scalable limiter like Envoy rate-limiting service with local tokens for burst handling.
• No single point of failure: critical state (configs, rate counters) stored in distributed stores with replication; fail-open or degrade gracefully policies (e.g., if authz service down, fallback to cached policy with tight TTL).

Per-route policy & hot-reload:

• Policies are versioned JSON/YAML objects stored in etcd with schema validation and signed commits.
• Gateways subscribe to watch endpoints; on change, they fetch new version, run local syntactic+semantic validation, run in-memory dry-run for a fraction of traffic (canary) and switch atomically by swapping pointer to policy tree.
• Rollback supported by keeping prior versions; changes applied via CI/CD pipeline that runs unit/integration tests and policy linting.
• Use feature flags & gradual rollout: percentage-based canary, metrics-driven promotion.

Safety considerations:

• Validate policies server-side before publishing; limit complexity/resources per policy to avoid DoS.
• Rate-limit config changes and put circuit-breaker on policy application to prevent cascading failures.
• Ensure idempotent retries and preserve client identity across retries via idempotency keys.

Trade-offs:

• Strong global rate limiting increases coordination cost/latency — mitigate with local bursting and asynchronous reconciliation.
• Caching auth/token info reduces latency but slightly increases risk of stale permissions — use short TTLs and event-driven invalidation.

This design achieves low-latency, high-availability API gateway with safe, observable, and dynamically configurable policy management.

An API gateway is a single entry point that sits between clients and a set of backend microservices, consolidating cross-cutting concerns so services can remain focused on business logic.

Core responsibilities:

• Routing: maps incoming requests to the correct service and endpoint, supports path/version-based routing, can implement request/response transformations.
• Authentication & authorization: centralizes token validation (OAuth/JWT), session handling, and role-based access control to avoid duplicating logic.
• Rate limiting & throttling: enforces per-client or per-key limits to protect backend services from abuse and spikes.
• Protocol translation: translates between client-facing protocols (HTTP/JSON, gRPC, WebSocket) and internal protocols; can do payload/format conversions.
• Observability & logging: aggregates metrics, traces, and centralized access logs for monitoring.
• Caching, load balancing, and request aggregation (fan-out / composition) where appropriate.

Benefits:

• Simplifies client code and decouples clients from service topology
• Centralizes security and policy enforcement
• Reduces duplicated code across services
• Enables consistent observability and operational control

Drawbacks:

• Single point of failure/latency if not highly available
• Added operational complexity and maintenance burden
• Can become a “god layer” that accumulates too much logic
• Potential performance bottleneck and increased deployment surface

Operational considerations:

• Run gateways redundantly across AZs, use health checks and circuit breakers
• Monitor latency, error rates, and capacity; autoscale under load
• Keep gateway lightweight—push heavy business logic into services
• Secure gateway management plane and rotate keys/certs

When premature:

• Small team/project with few services or monolith-to-microservices migration in early stages
• When client needs map 1:1 to services and cross-cutting needs are minimal
• If you lack operational capacity to run and secure the gateway

Introduce a gateway when you need centralized security, routing/versioning, or to reduce client complexity and you can commit to its operational requirements.

Requirements:

• Prevent one noisy backend from impacting other routes or gateway stability.
• Fast failure for clients (latency SLAs), observable degradation, automatic recovery.

High-level architecture:
API Gateway → Routing Layer with per-route policies → Circuit Breaker & Timeout Layer → Bulkhead (thread/pool/connection) per backend → Rate/Concurrency limiter → Fallback/Graceful response.

Core components & configs (examples and rationale):

1. Timeouts

• Client-facing (gateway → client): short overall timeout, e.g. 2s.
• Upstream per-call: conservative, e.g. 800ms for synchronous services.
  Rationale: fail fast, reduce head-of-line blocking.

2. Circuit Breakers

• Per-backend and per-route CB (sliding window):
  • failureThreshold=50% over last 10s, minRequests=20
  • openDuration=30s, half-open trial=5 requests
    Rationale: isolate failing service automatically, allow retry probes.

3. Bulkheads

• Isolated resources per backend:
  • Dedicated thread pool (or async semaphore) size per route, e.g. 50 threads
  • Connection pool caps to backend: max 100
    Rationale: prevents one backend consuming gateway threads/connections.

4. Resource limits per route

• Concurrency limits (e.g. 100 concurrent requests)
• Rate limits (per-second quotas, per-tenant)
• Queue depth caps; if full, return HTTP 429 or fast fallback.
  Rationale: backpressure and predictable resource usage.

5. Graceful degradation / fallbacks

• Tiered strategy:
  • Strong fallback: cached response or stale-but-acceptable data (HTTP 200 with header X-Cache: stale)
  • Soft fallback: lightweight error with action (HTTP 503 + Retry-After)
  • Client hints: add headers like X-Service-Status, X-Degraded: true, and a machine-readable JSON payload explaining partial data.
    Rationale: maintain UX while being transparent.

6. Observability & automated actions

• Metrics: p95/p99 latency, error rate, CB state, queue depths, thread pool utilization.
• Alerts and automated scaling: if healthy attempts spike, scale downstream or gateway pools.
• Logging traces with request-id for debugging.

Trade-offs:

• Tight timeouts reduce latency but may increase apparent errors; mitigate with retries on idempotent calls and cached fallbacks.
• More isolation (threads, pools) increases memory/complexity.

Example YAML snippet (gateway policy):

routes:
  - id: payments
    timeout_ms: 800
    circuit_breaker:
      failure_threshold: 0.5
      sliding_window_sec: 10
      min_requests: 20
      open_sec: 30
      half_open_requests: 5
    bulkhead:
      type: thread_pool
      max_threads: 50
      queue_limit: 100
    concurrency_limit: 100
    fallback:
      type: cache
      max_stale_sec: 300
      status_on_fallback: 200

This design isolates failures via per-route CBs and bulkheads, enforces predictable resource usage with limits, surfaces degraded behavior via headers/fallbacks, and keeps operators informed with metrics for recovery and scaling.

Definition & primary intent:

• API gateway: a north‑south edge component that exposes backend services to clients (mobile/web/third parties). Acts as a single ingress, handling routing, protocol translation, request aggregation, auth at boundary, rate limiting, and coarse observability.
• Service mesh: an east‑west infrastructure layer inside the cluster made of sidecar proxies that handle inter‑service networking: mTLS, fine‑grained routing/retries/timeout, per‑service telemetry, and policy enforcement.

Responsibilities:

• Routing: Gateway does external routing, host/path-based routing, and can do API composition. Mesh provides service-to-service routing (service discovery, canary, traffic shifting).
• Auth: Gateway handles authentication/authorization for external clients (OAuth, API keys, JWT validation). Mesh enforces mutual TLS, service identity, RBAC between services and can check end-to-end policies.
• Observability: Gateway provides request-level logs, metrics and traces for incoming traffic. Mesh produces dense telemetry (per-call latency, retries, success rates) and distributed tracing for internal calls.

Performance characteristics:

• Gateway: single or scaled gateway instances; adds one hop for external requests. Optimized for throughput but can become bottleneck if not scaled.
• Mesh: sidecar introduces per‑hop proxy overhead (CPU/memory + latency ~0.5–5ms depending on proxy). Overhead multiplies with call graph depth.

Operational overhead:

• Gateway: relatively low; manage fewer instances, certificates, external routing rules.
• Mesh: higher; deploy sidecars for every service, manage control plane (policy, config), more complex observability data volume and RBAC rules.

When to use which:

• API gateway preferred when exposing APIs to clients, performing auth at edge, TLS termination, request aggregation, and protecting backend from abusive clients.
• Service mesh preferred when you need fine‑grained service identity, progressive delivery (traffic shifting, mirroring), observability between services, and uniform resilience patterns.

Combined architecture example:

• Internet → CDN → API Gateway (Kong/Envoy as edge) → Ingress → Kubernetes cluster with sidecar service mesh (Istio/Linkerd)
  Division of responsibilities:
• API Gateway: authenticate external users (OAuth/JWT), rate limit, WAF, protocol translation, coarse metrics, request routing to appropriate internal service.
• Service Mesh: enforce mTLS, per‑service RBAC, circuit breakers, retries, canary traffic splitting, per‑call metrics/traces, fine‑grained policy.
  Justification: Edge handles client concerns (auth, aggregation, exposure) and reduces attack surface; mesh handles internal reliability/security and observability where you need per-service control. This separation keeps external policy centralized and internal policies decoupled, reducing blast radius and enabling independent scaling and evolution.