Backend Architecture and Service Design Questions

Focuses on designing backend systems and services with clear responsibilities, interfaces, and manageable dependencies. Topics include service oriented architecture and microservices trade offs, service boundaries and domain driven design, data ownership and consistency models, idempotency and error handling, application programming interface design and versioning, separation of concerns and modular code organization, testing strategies, deployment and release patterns, scalability and fault tolerance patterns, observability and monitoring, operational concerns including capacity planning and dependency management, and maintainability practices to enable teams to extend systems without constant refactoring. Senior level discussion includes designing for evolution, team ownership models, and balancing complexity versus simplicity across the service landscape.

HardSystem Design

61 practiced

Design an architecture for a checkout/payment pipeline that guarantees exactly-once payment processing when interacting with an external payment gateway that may send duplicate callbacks. Focus on idempotency, deduplication, compensation actions, ordering guarantees, and SLO impacts.

Sample Answer

Requirements & constraints:- Exactly-once semantics for payments despite duplicate callbacks from external gateway and network retries.- Strong consistency for user-visible order/payment status; ability to compensate (refund/cancel) when needed.- Low latency checkout (SLO: 200ms happy-path API), durable processing, high availability.

High-level architecture:- Checkout API → Order Service → Payment Orchestrator → External Payment Gateway.- Persistent stores: Order DB (primary source of truth), Idempotency/Transaction Store, Outbox table, Event Log (append-only).- Worker(s) read Outbox and perform gateway calls; gateway responses (including duplicate callbacks) hit a Webhook Receiver which writes to Idempotency Store and Enqueues events for processing.

Key components & guarantees:1. Client-generated idempotency keys: Checkout API requires client to send idempotency-key (UUID). Order Service creates order in DB only if key unseen (unique constraint). This prevents duplicate orders from client retries.2. Transactional Outbox: Order creation and writing a payment command to Outbox in same DB transaction. Ensures durable handoff to async worker (no lost writes).3. At-least-once delivery worker: reads Outbox, attempts payment call. Worker stores payment attempt metadata (gateway request-id) in Idempotency Store indexed by (order_id, idempotency-key).4. Webhook Receiver deduplication: On callback, lookup by gateway request-id OR order_id+payment_id in Idempotency Store. If seen, acknowledge and ignore; if new, atomically update Order state via conditional write (compare-and-set / DB transaction) and append event to Event Log.5. Exactly-once state transition: Order status updates performed with idempotent state machine transitions (e.g., PENDING → CHARGED). Use DB constraints or compare-and-swap to ensure transitions applied once.6. Compensation & Sagas: For failures (partial success), orchestrator runs compensating actions (refund, cancel shipment). Compensation steps are also idempotent and recorded in Outbox to retry until acknowledged.7. Ordering guarantees: Use per-order sequencing via a queue keyed by order_id so concurrent callbacks or worker retries are processed sequentially; leverage DB row-level locking or optimistic concurrency control to serialize transitions.8. Observability & SLO impacts: - Monitor dedup rates, callback latency, outbox lag, retry counts. - SLOs: API latency 200ms; async finalization (payment settled) SLO 99.9% within 30s; error budget for external gateway failures. - Backpressure: if Outbox backlog grows beyond threshold, degrade non-critical features and alert on-call.Trade-offs:- Exactly-once requires persistence + coordination; adds latency for strong consistency but keeps user-facing API quick by returning provisional status and finalizing asynchronously.- Complexity vs. reliability: transactional outbox + idempotency store increases operational surface but yields safe retries and clear audit trail.

Failure scenarios:- Duplicate callbacks → deduped via gateway request-id + idempotency store.- Lost webhook → worker reconciling by polling gateway using stored request-id and reconciling via idempotent updates.- Partial failure after charge → compensating refund recorded and retried until confirmed.

This design provides end-to-end exactly-once processing by combining client idempotency, transactional handoff, idempotent state transitions, deduplicating webhook handling, serialized per-order processing, and explicit compensation with strong observability.

HardTechnical

76 practiced

Design a strategy for service evolution across many teams to minimize costly refactors. Describe governance (API catalogs, design docs), technical patterns (stable contracts, adapters), and operational practices (deprecation schedules, discovery tools) to enable safe, incremental evolution.

Sample Answer

Situation: At scale, many teams independently change services, causing brittle integrations and expensive refactors that impact availability and SLOs.

Strategy (goal): Enable safe, incremental service evolution that minimizes refactors while preserving reliability and clear ownership.

Governance- API catalog + registry: central searchable catalog (OpenAPI/AsyncAPI) with ownership, SLOs, compatibility guarantees, and SDK links. Enforce publishing via CI.- Lightweight design docs: template requiring contract, compatibility matrix, migration plan, and SLO impact. Docs reviewed by a cross-team API review board (monthly rotating members).- Contract policy: define allowed non-breaking vs breaking changes and approval paths for breaking changes (RFC + migration window).

Technical patterns- Stable contract-first APIs: require schema-first APIs (OpenAPI/Protobuf) and semantic versioning for contracts.- Adapters and anti-corruption layers: encourage client-side adapters or facade services for boundary changes to isolate internal refactors.- Backward/forward compatibility: use tolerant readers, defaulted fields, and feature flags for server-side rollouts.- Contract tests: provider & consumer contract tests run in CI; publish verification results to catalog.

Operational practices- Deprecation schedule: standard 3-stage lifecycle—announce (30 days), soft-deprecated (90 days; logs/metric alerts), hard-deprecated (180+ days) with escalation if high usage persists.- Discovery and observability: traffic-level discovery tools (service mesh, API gateway metrics) and dashboards showing active clients per endpoint to prioritize migrations.- Release controls: require canary + automated rollback and monitor SLO/error budget impact during migration windows.- Runbooks & playbooks: SRE-run templates for migration incidents, rollback, and postmortems.- Onboarding & incentives: training, SDK generators, and rewards for teams that maintain compatibility and low break rates.

Measure success- Track number of breaking changes per quarter, average migration time, refactor cost (engineer-hours), and SLO incidents attributable to API changes; aim to reduce breaking-change incidents by 80% in 12 months.

This approach balances governance with developer velocity, uses technical isolation to reduce cross-team coupling, and embeds operational controls so SREs can preserve reliability during evolution.

HardSystem Design

57 practiced

Design a chaos engineering plan focused on the critical payment path. Include types of failure injections to run, the steady-state hypothesis, blast radius controls, metrics to monitor during experiments, and the process for automating experiments in CI or staging.

Sample Answer

Requirements:- Protect the critical payment path end-to-end (checkout → payment gateway → ledger) while safely validating resilience under realistic load.- Non-production-first (staging) experiments; some low-risk production experiments with strict blast-radius controls.- Maintain SLOs: payment success rate ≥ 99.5%, p99 latency < 800ms for checkout API.

Steady-state hypothesis:- Under normal conditions, payment success rate ≥ 99.9%, checkout p95 < 300ms, queue/backlog lengths are stable, no error spikes, and no customer-visible retries > 0.5%.

High-level approach:- Phased plan: tabletop → staging automated experiments → canary production experiments (very limited) → scaled production experiments if safe.- Use chaos tooling (Litmus, Chaos Mesh, Gremlin) integrated with CI and orchestration (K8s) and service mesh (Istio) for traffic control.

Failure injections (ranked by risk):1. Network latency and packet loss between checkout service and payment gateway (staging first).2. Gateway HTTP 5xx and rate-limit responses (simulate degraded upstream).3. DB write latency and transient failures to ledger (inject errors on write path).4. Message queue/backpressure: delay or drop messages from payment processor to reconciliation worker.5. Instance or pod termination during in-flight transactions.6. Time skew on auth/signing services (simulate JWT expiry mismatches).7. Resource exhaustion: CPU/cpu- steal and disk I/O on payment service pods.

Blast-radius controls:- Run in staging by default; production experiments limited to a single canary region or 1% of traffic using feature flags/traffic-splitting.- Use circuit-breakers and feature toggles to instantly abort experiments.- Auto-rollback policies tied to SLO thresholds and alerting.- Schedule experiments during low-traffic windows; require on-call approval.

Metrics to monitor (real-time + dashboards + alerts):- Business-level: payment success rate, declined vs failed counts, conversion rate, revenue impact estimate.- SRE: checkout API latency (p50/p95/p99), payment-gateway latency, DB write latency, queue lengths, error rates (4xx/5xx), retry counts, circuit-breaker opens, CPU/memory, pod restarts.- User experience: time-to-confirmation, rollback rate, abandoned carts.- Correlate logs and traces (distributed tracing) to identify where failures propagate.

Experiment automation in CI/staging:- Build experiment definitions as code (e.g., Kubernetes Chaos CRDs or Gremlin scripts) stored in repo alongside tests.- Pipeline stages: 1. Unit + integration tests. 2. Deploy to ephemeral staging environment (identical infra-as-code). 3. Run smoke tests to assert steady-state. 4. Trigger chaos experiment (parameterized) via pipeline job. 5. Run functional tests and synthetic transactions during and after injection. 6. Run automated analysis: check SLOs, compare metrics to baseline using automated canary analysis (e.g., Kayenta). 7. Fail the pipeline on SLO breaches; capture artifacts (traces, logs, metrics), and create runbook-linked incident if needed.- For production canaries: gating via manual approval and automated guardrails; use Kubernetes namespace isolation + service mesh routing to limit traffic.- Maintain a CI job suite of progressive experiments (small, medium, large) with change-control and audit logging.

Post-experiment process:- Blameless postmortem for any SLO breaches with action items (runbook updates, code fixes, improved retries/backoff).- Maintain experiment catalog, risk score, owners, and re-run frequency.- Continually tune steady-state and experiment parameters; feed learnings into SLA/SLO adjustments.

Trade-offs:- Conservative blast radius reduces discovery of rare cascading failures vs larger-scale tests that risk customer impact.- Prefer automation in staging to move fast; reserve production for very targeted, well-instrumented experiments.

HardTechnical

62 practiced

You suspect a memory leak in a Java microservice causing intermittent OOM kills in 1% of pods. Describe a step-by-step investigation: what metrics and artifacts you’d collect, how to reproduce locally, mitigation to reduce impact on production, and rollout plan for a fix.

Sample Answer

Situation: A Java microservice is intermittently OOM-killed in ~1% of pods. I’d follow a reproducible, low-risk investigation + mitigation + staged rollout.

1) Collect metrics & artifacts (immediately)- Pod/container events and kubelet logs (kubectl describe pod, kubectl logs -p).- JVM metrics: heap usage, metaspace, GC pause time, GC throughput (Prometheus jmx-exporter or Micrometer). Query time-series for anomalous pods.- Node metrics: memory pressure, OOM killer logs (dmesg).- GC logs (enable if not already): -Xlog:gc*:file=/var/log/gc.log:time,level,tags- Heapdumps and thread dumps from affected pod(s): jcmd <pid> GC.heap_info, jmap -dump:live,format=b,file=/tmp/heapdump.hprof, jstack for threads.- Native memory and classloader growth: jcmd VM.native_memory, jcmd GC.class_histogram- App logs correlated with heap/GC spikes and recent deploys/config changes.

2) How to reproduce locally- Build same container image and JVM flags used in prod; run locally or in an isolated k8s namespace with same resource limits.- Start the app with diagnostics flags: -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -Xlog:gc*:gc.log:file=gc.log- Replay traffic (recorded or synthetic) matching prod patterns; use load generator (k6, vegeta) including long-lived connections or high-cardinality inputs.- Use stress scenarios: gradual memory leak by repeating the suspicious code path, high concurrency, or long retention scenarios.- Attach async-profiler or flight recorder (jfr) to capture allocation flame graphs to find allocation sites.

3) Analysis steps- Analyze heapdump in Eclipse MAT: dominator tree, leak suspects, retained set, suspicious growth (e.g., many instances of cache/key, classloader, ThreadLocal, javax.servlet.AsyncContext).- Correlate GC logs: increasing full GC frequency, long STW pauses, promotion failure patterns.- Inspect native memory (NIO direct buffers, JNI) with jcmd VM.native_memory.

4) Short-term mitigations (reduce production impact)- Add pod-level safety: lower JVM max heap (-Xmx) relative to container limit to avoid container OOM; enable -XX:+UseContainerSupport and set -XX:MaxRAMPercentage if needed.- Add or tighten liveness probe to restart stuck pods quickly (but ensure not to thrash).- Increase replica count and autoscaling to reduce per-pod load.- Configure graceful restart (preStop + terminationGracePeriod) to avoid request loss.- If leak is tied to a feature, toggle feature flag or route traffic away.- Enable HeapDumpOnOutOfMemoryError to capture future failures.

5) Fix and rollout plan- Implement code fix identified by heapdump/profile; add unit/integration tests and memory regression tests (stress tests).- Dev/QA validation: reproduce fix locally and in staging under same stress.- Canary rollout: deploy fix to small percentage (e.g., 5%) with observability checks: heap usage, GC frequency, error rates, latency. Use automated alerting to halt rollout on regressions.- Gradual ramp: 5% → 25% → 100% with 10–30 minute observation windows depending on traffic patterns.- Post-deploy: monitor for at least 24–72 hours; collect heapdumps and GC logs automatically for a sampling of pods to validate.- Rollback plan: immediate rollout abort plus automated rollback to previous image if OOMs or error SLO breaches detected.- Prevent recurrence: add JVM and app-level metrics to SLIs/SLOs (e.g., sustained heap growth > threshold), memory regression testing in CI, and run periodic heap/alloc profiling in pre-prod.

This approach prioritizes data collection, safe mitigation, reproducible testing, and a cautious canary release with clear rollback criteria.

EasySystem Design

71 practiced

Given an online store domain (catalog, inventory, checkout, payments, recommendations), propose service boundaries using domain-driven design principles. For each proposed service, state who owns the data, the API contract shape at a high level, and how you would minimize cross-service coupling.

Sample Answer

I'll propose bounded contexts and service boundaries that balance DDD principles with SRE concerns (availability, independent deployability, clear ownership).

1) Catalog Service- Data owner: product definitions, metadata, categories, SKUs.- API (high-level): GET /products/{id}, GET /products?query, POST/PUT for internal authoring.- Coupling minimization: push product read-models to a materialized view; publish product-change events (product.created/updated) on an event bus.

2) Inventory Service- Data owner: per-SKU stock levels, reservations.- API: GET /inventory/{sku}, POST /inventory/reserve {sku, qty, orderId}, POST /inventory/release.- Minimize coupling: authoritative single source of truth; expose idempotent reserve/release; subscribe to order.created and order.cancelled events rather than sync calls.

3) Order/Checkout Service- Data owner: order lifecycle (cart, payment status, fulfillment).- API: POST /orders {items, userId}, PATCH /orders/{id}/confirm, GET /orders/{id}.- Minimize coupling: orchestrate via async events; use sagas/compensating actions (e.g., if payment fails, release inventory).

4) Payments Service- Data owner: payment transactions, provider tokens (masked).- API: POST /payments {orderId, amount, method}, GET /payments/{id}/status.- Minimize coupling: synchronous call only from Order during checkout with clear timeouts/retries; emit payment.succeeded/failed events.

5) Recommendations Service- Data owner: recommendation models, user interaction signals (derived store).- API: GET /recommendations?userId=..., GET /recommendations?sku=...- Minimize coupling: pull-only read API; consume events (views, purchases) for offline model updates; provide cacheable responses.

Cross-cutting SRE considerations- Use event bus (Kafka) for domain events and to decouple services.- Define SLOs per service (e.g., Catalog 99.95 read, Inventory 99.9 reserve latency), circuit breakers, bulkheads.- API contracts: small, versioned REST/HTTP + protobuf for high-throughput RPC; include correlation IDs.- Observability: standardized metrics (request latency, error rates), structured logs, distributed tracing for sagas, alerting on SLO breaches.- Deployment: independently deployable containers, CI/CD, and feature flags for schema evolution.This keeps ownership clear, favors async communication for coupling reduction, and aligns with reliability and operational needs.

Unlock Full Question Bank

Get access to hundreds of Backend Architecture and Service Design interview questions and detailed answers.

Join thousands of developers preparing for their dream job.