Service Communication and API Design Questions
Design and specify APIs and interservice communication patterns for systems that must scale. Topics include REST design and best practices, API versioning and deprecation strategies, backward compatibility and API evolution, contract design between services, choosing protocols such as gRPC versus HTTP for internal or external communication, security and authentication patterns at the API level, performance and scalability considerations, observability and monitoring of service interactions, and strategies for evolving interfaces without breaking consumers.
HardTechnical
48 practiced
Describe end-to-end idempotency and distributed transaction patterns for ensuring correctness across multiple microservices when processing payments (SAGA, compensating actions, and idempotency tokens). As a Solutions Architect, explain when to use SAGA vs two-phase commit and how to handle partial failures and compensation semantics.
Sample Answer
Situation/Requirements: For payment flows you must guarantee correctness (no double-charges, money conserved), visibility (audit/trace), and availability (services stay responsive). You also need to support distributed operations across Payment Gateway, Ledger, Inventory, and Notification microservices.When to use SAGA vs Two-Phase Commit (2PC)- Use 2PC only when strong, synchronous ACID across multiple databases is mandatory and latency/availability trade-offs are acceptable (rare in microservices; requires XA-compliant resources and central coordinator).- Prefer SAGA for microservices: it’s asynchronous, scales, tolerates partial failures, and aligns with bounded contexts. SAGA gives eventual consistency with explicit compensations.SAGA patterns- Choreography: Each service emits events (PaymentAuthorized → LedgerDebit → InventoryReserve). Simple, no central orchestrator, but complexity grows for many participants.- Orchestration: Central saga orchestrator issues commands and tracks state. Easier to reason, simpler compensation flows, better for complex payment flows.Compensating actions and semantics- Design compensations to be idempotent and reversals of the forward action (e.g., RefundCharge for ChargeCustomer). Record saga state and invariants (transaction id, step, status) and persist compensation payloads.- Compensations should be compensatory, not undoing side effects outside your control (use retries, alerts, manual reconciliation for external third-party settlement finality).Idempotency tokens and deduplication- Require clients or orchestrator to provide an idempotency key for the whole payment transaction. Persist keys with result status and timestamps in an idempotency store (fast key-value DB).- On retry, lookup key: return prior result or resume idempotent operation. For commands to downstream services, include per-step idempotency tokens to prevent duplicate side effects caused by retries.Handling partial failures- Implement exponential backoff + circuit breaker for transient errors.- Saga orchestrator detects timeouts and triggers compensations for completed steps. Partial success => run compensating steps in reverse order; mark saga terminal states (Succeeded, Compensated, FailedNeedsManualAction).- For non-compensatable external steps (bank settlements), isolate them and design for eventual reconciliation and human-in-loop resolution.Operational concerns- Tracing: use a correlation id across services; export saga events to distributed tracing and audit logs.- Monitoring: alert on long-running sagas, compensation rates, idempotency store size, and orphaned transactions.- Security & consistency: sign/verify messages, use exactly-once delivery patterns where possible (e.g., Kafka with consumer offsets persisted), and enforce schemas/versioning.Example flow (Orchestrator)1. Client POST /payments with idempotency-key K.2. Orchestrator records saga (K, state=Started).3. Orchestrator -> PaymentService (charge) with step-token S1.4. If success -> LedgerDebit with S2; if failure -> trigger Refund if charge occurred.5. On any transient failure, retry using S1/S2; on permanent failure trigger compensations; update saga state and store audit events.Trade-offs- 2PC: strong consistency, brittle, poor availability—only for tightly-coupled systems.- SAGA: scalable and resilient but requires careful compensation design, observability, and acceptance of eventual consistency.This architecture balances correctness (idempotency, persisted saga state), scalability (asynchronous SAGA), and operational practicality (tracing, alerts, manual reconciliation for irreversible external effects).
HardTechnical
53 practiced
Describe how you would implement end-to-end tracing and distributed context propagation for a mixed protocol environment (HTTP/JSON, gRPC, and async messages). Include header names, semantic attributes to propagate (user id, correlation id, tenant id), and how to handle propagation when messages cross protocol boundaries.
Sample Answer
Goals: preserve distributed trace and business context (trace id, span id, sampling, plus semantic attributes: user_id, correlation_id, tenant_id) across HTTP/JSON, gRPC, and async messaging so we can reconstruct end-to-end flows and perform multi-tenant debugging.Standards & headers- Use W3C Trace Context as primary trace header: - traceparent: "00-<trace-id>-<span-id>-<flags>" - tracestate: vendor-specific hints- Use W3C Baggage for lightweight propagated key/value pairs: - baggage: "user-id=...,correlation-id=...,tenant-id=..."- Optionally support B3 (X-B3-TraceId, X-B3-SpanId, X-B3-Sampled) for legacy systems and translate to/from traceparent.- For gRPC: transport via metadata keys: "traceparent", "tracestate", "baggage" (lowercase).- For HTTP/JSON: standard headers (traceparent, tracestate, baggage). For incoming JSON payloads if headers stripped, include a standard metadata envelope (e.g., _meta.traceparent).- For async messages (Kafka, RabbitMQ): use message headers/properties named traceparent, tracestate, baggage. If broker limits header types/size, compress/encode baggage (base64 or compact key set) and store larger attributes (e.g., user profile) in a side-store with correlation_id reference.Semantic attributes to propagate- correlation_id: short immutable request id for business tracing (also in baggage).- user_id: minimal identifier for request-scoped user; avoid PII, prefer user_hash.- tenant_id: multi-tenant routing/authorization key.- service.name, service.instance.id, span.kind (set by instrumentation, not in propagation payload).Propagation rules & crossing protocol boundaries- Preserve traceparent across all transports; if missing, start a new trace but log origin.- Map W3C headers <-> gRPC metadata <-> message headers 1:1. Convert B3 when detected.- When crossing to async (sync->async): set outgoing span as producer span, inject traceparent + baggage into message headers, set correlation_id = existing correlation_id or generate a child correlation id (store relation in logs).- When async->sync consumer: create consumer span as child of traceparent from message; continue baggage.- If headers are lost in JSON body-only clients: require a gateway that extracts/embeds trace context from headers into body _meta section (and removes on egress).- Honor size limits: keep baggage minimal (correlation_id, tenant_id, user_id). Avoid large tokens; reference large objects via correlation_id -> side-store (e.g., Redis) when needed.Sampling & security- Sampling decisions encoded in traceparent flags; sampling should be made at edge and respected downstream. Provide dynamic sampling via tracestate.- Treat baggage as untrusted; validate and sanitize (no raw PII).- Encrypt or avoid propagating sensitive fields. Use short opaque identifiers for user/tenant.Instrumentation & operational practices- Use OpenTelemetry SDKs to standardize injection/extraction and translate B3 <-> W3C as needed.- Implement middleware at gateways, API proxys, message producers/consumers to ensure context injection/extraction.- Add observability: log trace_id, correlation_id, tenant_id in structured logs and metrics to enable linking.- Test: end-to-end integration tests that assert trace continuity across protocol boundaries and header-loss scenarios.Example header values (HTTP):traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01baggage: correlation_id=abc123,user_id=u-42,tenant_id=t-7This approach maximizes interoperability, minimizes vendor lock-in, and keeps propagated context small and secure while enabling full end-to-end tracing across mixed protocols.
MediumTechnical
48 practiced
You must design REST endpoints for a product catalog used by client-facing apps: listing products with filters, retrieving product details, and bulk updates. As a Solutions Architect, propose resource URLs, query parameters for filtering/pagination/sorting, and explain design decisions for bulk operations and error reporting in bulk APIs.
Sample Answer
Requirements clarification:- Client-facing read APIs: list with filters, sort, pagination; read single product by id; bulk updates for inventory/price/attributes with partial failure reporting.- Non-functional: high read scale, eventual consistency acceptable for catalogs, idempotent updates preferred.Resource URLs and methods:- GET /v1/products — list products (supports filters, pagination, sorting)- GET /v1/products/{productId} — product details- POST /v1/products/bulk/updates — submit bulk update job (async)- GET /v1/products/bulk/updates/{jobId} — job status & result- PATCH /v1/products/{productId} — single partial updateQuery parameters for GET /v1/products:- filter by attributes: category=electronics, brand=sony- range/price: price_min=100&price_max=1000- availability: in_stock=true- text search: q=wireless+headphones- facets: facet=brand,category (for UI)- pagination: page[size]=50&page[number]=2 (or cursor: cursor=eyJvcmQ...)- sorting: sort=price,-rating (prefix - for desc)Design notes: support both page and cursor for clients needing stable pagination; default sort relevance then id.Bulk operations design:- Use async job API (POST creates job, returns 202 + jobId). Payload accepts NDJSON or JSON array with per-item upsert commands: { "items": [ { "productId": "P1", "op":"update", "fields": {"price":99.99, "stock":10} }, ... ] }- Enforce idempotency: client-supplied idempotency-key header for retry-safe submission.- Size limits: max 5k items or 10MB per request; recommend batching.Error reporting and results:- Job status includes counts: submitted, succeeded, failed, partial.- Downloadable result file URL (signed S3) with per-item outcome: { "productId":"P1", "status":"success" } or { "productId":"P2", "status":"failed", "code":"VALIDATION_ERROR", "message":"price must be >=0", "retryable":false }- Use structured error codes and HTTP 200 for job status responses; only initial POST returns 202. For fatal job-level errors return 4xx/5xx appropriately.- Provide retry guidance and backoff recommendations for retryable failures.Security, observability, and performance:- Auth via OAuth2/Bearer tokens; RBAC for update scopes.- Rate limits and throttling; bulk endpoints queued and processed by worker pool.- Emit audit events per item for traceability; expose requestId/jobId for support.Trade-offs:- Async bulk adds complexity but prevents timeouts and scales; NDJSON optimizes streaming large payloads. Synchronous bulk (atomic) not supported to avoid large transactions; could offer small-atomic endpoint for <=50 items if needed.
MediumTechnical
41 practiced
Compare JSON Schema with Protocol Buffers for API contract definitions. As a Solutions Architect, discuss differences in schema evolution, type expressiveness, tooling, size/performance, and browser/client support. Recommend scenarios where one is preferable to the other and how to manage cross-team interoperability.
Sample Answer
High-level: JSON Schema (JS) is a JSON-native DSL for validating JSON documents and documenting contracts; Protocol Buffers (Protobuf) is a compact, language-neutral IDL + binary serialization with generated code. Both define contracts but target different trade-offs.Schema evolution- Protobuf: Designed for forward/backward compatibility via field numbers, optional/required semantics, and reserved ranges. Safe additive changes and removing fields if numbers are reserved. Stronger guarantees when teams follow proto upgrade rules.- JSON Schema: No built-in versioning semantics — evolution is process-driven (adding optional properties is safe; changing types or removing properties requires coordination and versioning strategies). Use semantic versioning or message envelopes.Type expressiveness- Protobuf: Rich scalar types, enums, nested messages, oneof, repeated fields, maps, but limited to typed primitives and lacks arbitrary JSON structures without Any or well-known types.- JSON Schema: Very expressive for JSON shapes — supports arbitrary nesting, patternProperties, conditional schemas, formats, and constraints for strings/numbers; better for heterogeneous or loosely-typed payloads.Tooling- Protobuf: Strong codegen across many languages, RPC frameworks (gRPC), serializers/deserializers, linters, and CI hooks. Tooling automates client/server stubs.- JSON Schema: Wide ecosystem for validators in JS/Java/Python, documentation generators (Swagger/OpenAPI), but less uniform codegen quality; OpenAPI bridges HTTP APIs.Size / performance- Protobuf: Binary, compact, faster parsing and lower wire size — good for high-throughput, mobile, IoT, internal microservices.- JSON Schema/JSON: Textual, larger payloads, slower parsing, but human-readable, debuggable, and cacheable by HTTP.Browser / client support- JSON: Native in browsers, trivial to use in JS clients and RESTful APIs.- Protobuf: Requires generated code or protobuf-js; gRPC-Web exists but needs proxy (Envoy) or special client libraries. More friction in plain-browser usage.Recommendations / when to use- Use Protobuf when: high performance/low bandwidth required, strict type contracts, multi-language server-client stubs, gRPC ecosystems, internal microservices or mobile/IoT clients.- Use JSON Schema / JSON when: public HTTP REST APIs, human-readable contracts, rapid iteration, heterogeneous payloads, or when browser-first clients dominate.Cross-team interoperability- Establish a contract governance process: single source-of-truth repo, PR reviews, change policies (compatible vs breaking), and automated CI checks (linting, compatibility tests).- Provide dual artifacts where needed: generate OpenAPI from proto (or produce proto from canonical definitions) and vice versa; publish codegen artifacts/packages.- Versioning strategy: semantic versioning for API surfaces, explicit deprecation windows, and changelogs.- Bridge tooling: use schema translators (proto-to-jsonschema, openapi generators) and API gateways that can transform formats (gRPC <-> JSON/HTTP).- Document evolution rules, run automated compatibility tests, and provide SDKs to reduce client drift.Final note: pick based on client needs — favor JSON/JSON Schema for external, human-facing REST; favor Protobuf for high-performance, internal RPC and strongly-typed ecosystems.
HardSystem Design
38 practiced
You must migrate a monolithic public API to microservices without breaking external clients. Propose an incremental migration plan that includes an API façade, routing, data ownership transfer, and monitoring to ensure parity. As a Solutions Architect, identify risks and rollback strategies for each migration phase.
Sample Answer
Requirements & constraints:- Preserve external API contracts (URLs, auth, payloads)- Zero-downtime incremental cutover- Maintain data consistency and ownership transfer- Observability and quick rollback for each step- Low risk to SLAsHigh-level approach: Strangler-Fig + API façade (Gateway/BFF) that routes by endpoint/version; incremental extract of services and data ownership with anti-corruption layers, dual-read/write and sync jobs; strong monitoring, contract tests, and phased rollbacks.Phases1) Stabilize façade & traffic steering- Deploy API façade (API Gateway + BFF) that can route by path/header to monolith or new service. Add feature-flagging and traffic-split capability.- Monitoring: request/response schemas, latency, 5xx rate, per-route metrics, contract tests (consumer-driven).- Risks: gateway misrouting, auth regressions.- Rollback: flip flags to route all traffic back to monolith; use health checks to automate.2) Read-only extraction (read replicas / CQRS)- Implement a microservice that serves read APIs using replicated read-model (CDC -> event stream -> materialized views).- Keep writes to monolith.- Monitoring: data freshness lag, query parity, discrepancy alerts.- Risks: stale data, eventual consistency surprises.- Rollback: disable facade routing to new read service; revert clients to monolith reads.3) Write-path façade & anti-corruption- Introduce anti-corruption layer in façade to transform calls; implement dual-write (façade writes to monolith and to new service/queue) or write-to-monolith-with-async-replay.- Monitoring: write success ratio, queue depth, reconciliation jobs, idempotency metrics.- Risks: duplication, partial writes, ordering issues.- Rollback: switch façade to only write to monolith; drain/stop queues; replay logs to sync new service later.4) Data ownership transfer & canonicalization- Promote microservice to canonical owner for specific bounded context. Migrate authoritative data via bulk migration + CDC catch-up, validate with reconciliation.- Monitoring: data parity %age, reconciliation reports, business KPIs.- Risks: data loss, schema mismatch, business rule divergence.- Rollback: demote new service to read-only, replay authoritative writes to monolith, or restore from backups; feature-flag business routing back.5) Iterate, decommission, optimize- Remove monolith endpoints once coverage and business acceptance achieved. Harden SLAs, autoscaling, security, and observability.- Risks: hidden dependencies, performance regressions.- Rollback: retain façade routing capability to reintroduce monolith endpoints temporarily.Cross-cutting safeguards- Consumer-driven contract tests + CI gating for façade and services- Blue/green & canary deployments, progressive traffic shift (1%, 10%, 50%, 100%)- End-to-end integration and business tests in staging that mirror production traffic- Robust tracing (distributed traces with trace IDs), per-request logs, and dashboards for SLOs- Reconciliation jobs, idempotent APIs, sequence numbers for deterministic replay- Runbook and automated rollback scripts for each phase; database backups and snapshot retention policyWhy this works- Minimizes blast radius by separating routing from business logic- Allows measurable verification at each step (monitoring + reconciliation)- Provides clear rollback points and automated ways to revert routing, writes, or ownership while preserving client contracts.
Unlock Full Question Bank
Get access to hundreds of Service Communication and API Design interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.