Microservices Architecture and Service Design Questions
Covers the principles, patterns, and trade offs for designing, decomposing, operating, and evolving microservice and service oriented architectures. Candidates should be able to define service boundaries and decomposition strategies, explain domain driven design influences, and describe safe approaches to break a monolith into independently deployable services. Topic coverage includes application programming interface design and versioning, synchronous and asynchronous inter service communication patterns such as representational state transfer, remote procedure call frameworks, and messaging systems, as well as event driven architecture patterns. It includes data ownership and distribution, consistency models, distributed transaction patterns including the saga pattern and two phase commit trade offs, and resilience patterns such as circuit breakers, retries, and bulkheads. Operational concerns include service discovery, gateway and service mesh patterns, deployment and rollout strategies for independent services, observability and distributed tracing, monitoring, testing and debugging across services, failure handling and network latency considerations. The topic also covers organizational impacts including Conway's law, service choreography versus orchestration, team boundaries and operational complexity, and guidance on when to choose a monolith versus microservices.
HardTechnical
113 practiced
Design a schema evolution and governance strategy for event topics in Kafka used by multiple teams. Cover schema registry usage, compatibility modes (backward/forward/full), migration strategy for breaking changes, and how you'd coordinate cross-team changes without blocking delivery.
Sample Answer
**Situation & goals**Design a practical schema-evolution + governance strategy so multiple backend teams can change Kafka event schemas safely without blocking delivery, while preserving consumer compatibility and auditability.**Schema Registry & metadata**- Use Confluent Schema Registry (or Apicurio) as single source of truth per environment; register Avro/JSON Schema/Protobuf.- Enforce ACLs so only schema service accounts can mutate subjects; store owner/team, contact, semantic version, and change-requests in subject metadata.**Compatibility policy**- Default to backward compatibility for event topics (producers can add fields with defaults; old consumers still parse).- Use forward or full compatibility only for special topics with strict needs; document exceptions in governance board.- CI enforces compatibility checks via Registry API before merge.**Migration strategy for breaking changes**- Prefer non-breaking changes (add optional fields, default values).- For unavoidable breaking changes: 1. Create new subject/topic version: topic.v2 or topic-new with new schema. 2. Deploy dual-writing at producer side (write v1 and v2) or use a CDC/translation service to transform v1 -> v2. 3. Update consumers gradually to support v2; run both consumers in parallel; monitor consumers lag/errors. 4. Decommission old topic after verification and a TTL window.**Cross-team coordination without blocking**- Use lightweight change requests: PR to a central "schema-changes" repo containing .avsc files + migration plan + owner + rollout timeline.- Automated checks: schema compatibility, contract tests, and sample serialization in CI.- Governance Slack + weekly schema sync for risky changes; allow asynchronous approval with SLOs (e.g., auto-approve non-breaking after 24h).- Provide SDKs/helpers for dual-write, adapter layers, and transformation templates to reduce burden.**Observability & rollback**- Add schema-aware logging, metrics (schema version used), and end-to-end contract tests in CI.- Feature flags to toggle new behavior; rollback by switching producers back or re-pointing consumers.**Trade-offs**- Topic-per-version increases operational overhead but isolates risk. Dual-write/transformation adds code but enables non-blocking migration.This balances developer velocity with safe, auditable schema evolution suitable for backend teams.
MediumSystem Design
72 practiced
You are asked to design observability for 50 microservices. Describe a minimal set of telemetry to collect (traces, metrics, logs), naming conventions for spans and metrics, and how you would use correlation IDs and distributed tracing (e.g., OpenTelemetry) to debug a request that crosses multiple services.
Sample Answer
**High-level approach**Collect a minimal, consistent set: structured logs, key metrics, and distributed traces (OpenTelemetry SDKs). Use correlation IDs to tie logs/metrics to traces.**Minimal telemetry**- Traces: 1 span per incoming request + spans for DB calls, external HTTP/rpc, cache, and significant sync/async work.- Metrics: request_count{service,endpoint,status}, request_latency_ms_histogram{service,endpoint}, error_count{service,endpoint,type}, queue_depth, DB_connection_pool_usage.- Logs: structured JSON with timestamp, level, message, service, span_id, trace_id, trace_flags, correlation_id, user_id, request_path, error details.**Naming conventions**- Services: service.name = kebab-case (e.g., auth-service)- Spans: operation-style verbs: service.operation.resource — e.g., auth-service.http.post /login, orders-service.db.query get_order_by_id- Metrics: dot-separated: service.metric.type — e.g., orders-service.request.count, auth-service.request.latency_ms**Correlation & distributed tracing workflow**- Generate a single correlation_id at the edge (API gateway / load balancer). Inject into incoming request header (e.g., X-Correlation-ID) and into OpenTelemetry trace context (traceparent).- Every service: extract context, start spans as child of incoming span; include correlation_id as span attribute and in all logs.- Debugging: follow the trace in tracing UI (Jaeger/Tempo). Use trace_id to filter logs/metrics. Inspect slow spans (latency histogram + span durations) to find hotspots; check logs linked by span_id for errors and DB/third-party timings.**Practical tips**- Sample traces at adaptive rate (100% for errors, 1-5% for success).- Enforce SDK instrumentation and CI linting for naming conventions.- Export telemetry to backend (OTel collector -> backend like Grafana Tempo + Prometheus + Loki).
MediumTechnical
63 practiced
Design SLOs and alerting for a public-facing payments API. Specify at least three SLIs (e.g., availability, p95 latency, error rate), reasonable SLO targets for a payments service, and alert thresholds that balance noise and actionable incidents. Explain how you would tie alerts to runbooks.
Sample Answer
**Context & goals** I’d design SLOs to protect user money flow and trust: prioritize correctness (error rate), responsiveness (p95 latency), and availability.**SLIs & SLOs (examples)** - Availability (successful auth + payment processing): SLI = % of successful HTTP 200/202 for payment endpoints over 30d. SLO = 99.95% (≈ 22 min downtime/month). - p95 latency: SLI = p95 end-to-end API time for /payments and /authorize over 7d. SLO = ≤ 300 ms. - Error rate: SLI = % of client-visible failures (5xx or domain errors like insufficient funds/duplicate detection) over 30d. SLO = ≤ 0.1%.**Alerting strategy & thresholds** - Page (sev-1): Burn rate alert when error rate or availability violates SLO at 3x allowed burn for 1h OR absolute availability < 99.5% for 5m. Immediate paging. - Ops/On-call (sev-2): error rate spike > 0.5% sustained 15m OR p95 > 600 ms for 15m. Notify via chat/alert. - SRE/Dev (sev-3): SLO approaching burn (e.g., 80% of monthly error budget used in 24h) or transient latency blips (5m). Create ticket.Balance: short windows for paging critical incidents, longer windows for noisy metrics to reduce false positives.**Runbooks & playbooks** Each alert links to a runbook with: impact summary, dashboards, quick checks (service health, queue depth, DB errors, payment gateway status), mitigation steps (circuit breaker, rollback, rate-limiting), escalation steps, and postmortem template. I’d embed automated remediation hooks (e.g., toggle feature flag, scale workers) and require owners to update runbooks after incidents.
MediumTechnical
58 practiced
Propose a minimal chaos engineering experiment to validate resilience of an order-processing microservice. Define the hypothesis, failure injection (e.g., network latency, downstream DB failure), observability signals to collect, and safe rollback/abort criteria for the experiment.
Sample Answer
**Hypothesis**If the order-processing microservice’s downstream PostgreSQL becomes transiently unavailable or experiences increased latency, the service will (a) continue to accept requests via a bounded retry/queue mechanism, (b) surface meaningful 5xx/429 errors within SLA, and (c) recover without data loss when DB returns.**Failure injection**- Inject 30s of increased DB latency (add 200–500ms RTT) on the primary DB replica for a single availability zone OR- Simulate a short DB error (RETURNING 500 on connections) for 30s using a chaos tool or iptables DROP on DB port from app nodes.Start with 1 instance of the service (or a canary deployment) to limit blast radius.**Observability signals**- Application: request rate, 5xx/429 rate, request latency (p95/p99), retry counts, queue/backpressure length.- DB: connection errors, active connections, query latency.- Business: orders created vs. orders failed / orders queued (event store).- Infrastructure: CPU, memory, thread pool exhaustion, error logs with stack traces.- Traces: end-to-end distributed traces showing retries and where latency accumulates.**Abort / rollback criteria**Abort immediately and rollback chaos if any of:- 5xx rate > 2% above baseline sustained for >1 minute- p99 latency > SLA threshold (e.g., 5s) sustained >1 minute- Queue/backlog grows > 3x baseline or exceeds safe capacity- Any data-loss indicator: orders marked as processed but not persistedRollback steps: remove injected rules, revert canary to normal traffic, notify on-call, run automated health checks and replay queued items only after verification.**Post-mortem & learning**Capture traces/logs, confirm retry/backpressure behavior, tune retry/backoff/queue-size or add circuit breaker if needed. Repeat with scaled traffic after fixes.
EasyTechnical
67 practiced
Define idempotency in the context of HTTP APIs and message processing. Why is idempotency important in distributed systems? Provide two concrete techniques a backend developer can use to implement idempotent operations.
Sample Answer
**Definition (HTTP APIs & message processing)** Idempotency means performing the same operation multiple times has the same effect as doing it once. For HTTP: a repeated PUT or DELETE should not change state beyond the first call; POST is not inherently idempotent unless designed so. For message processing: delivering the same message multiple times should not produce duplicate side effects.**Why it matters in distributed systems** Networks, retries, and at-least-once delivery cause duplicate requests/messages. Idempotency prevents duplicate charges, duplicate records, and inconsistent state, simplifying error handling and improving reliability.**Two concrete techniques**1) Idempotency keys (request deduplication) - Client sends a unique idempotency key (UUID) in header for operations like “create payment”. - Server stores key → result mapping (status, resource id) in a durable store with TTL. If same key received, return stored response instead of re-executing. - Example: POST /payments with header Idempotency-Key: 123e...; server returns same payment id on retries.2) Declarative de-duplication using natural keys / upserts - Use a unique business key (order_number, invoice_id) and database constraints (unique index) or upsert operations. - Implement create-or-get semantics: attempt insert; on unique-constraint violation, fetch existing record and return it. This is simple, transactional, and leverages DB guarantees.**Notes & trade-offs** - Idempotency keys require storage and GC; upserts depend on having a stable natural key. Combine both for best coverage.
Unlock Full Question Bank
Get access to hundreds of Microservices Architecture and Service Design interview questions and detailed answers.
