Microservices Architecture and Service Design Questions
Covers the principles, patterns, and trade offs for designing, decomposing, operating, and evolving microservice and service oriented architectures. Candidates should be able to define service boundaries and decomposition strategies, explain domain driven design influences, and describe safe approaches to break a monolith into independently deployable services. Topic coverage includes application programming interface design and versioning, synchronous and asynchronous inter service communication patterns such as representational state transfer, remote procedure call frameworks, and messaging systems, as well as event driven architecture patterns. It includes data ownership and distribution, consistency models, distributed transaction patterns including the saga pattern and two phase commit trade offs, and resilience patterns such as circuit breakers, retries, and bulkheads. Operational concerns include service discovery, gateway and service mesh patterns, deployment and rollout strategies for independent services, observability and distributed tracing, monitoring, testing and debugging across services, failure handling and network latency considerations. The topic also covers organizational impacts including Conway's law, service choreography versus orchestration, team boundaries and operational complexity, and guidance on when to choose a monolith versus microservices.
HardSystem Design
62 practiced
Design a secure inter-service communication and secrets management plan for microservices deployed across multiple clusters and cloud regions. Address mutual TLS for service identity, certificate rotation, automated secret distribution, least-privilege access control for secrets, and recovery procedures if credentials are leaked.
Sample Answer
Requirements & constraints:- Strong service identity across clusters/regions, automatic mutual TLS (mTLS)- Automated certificate/secret issuance and rotation with minimal downtime- Least-privilege access to secrets (auditable)- Fast recovery and revocation if credentials leak- Multi-cloud & multi-cluster support, low operational overheadHigh-level architecture:- Centralized identity & secret authority: HashiCorp Vault (or Cloud KMS-backed Vault) deployed HA across regions with replication (performance-replication + DR-replication).- Workload identity: SPIFFE/SPIRE agents running on each node to provision short-lived X.509s for services.- Secret distribution: Vault with Kubernetes/CNI CSI driver + Vault Agent injector for non-K8s workloads; use Vault PKI for certs and dynamic secrets (DB creds).- Policy & auth: OIDC + Kubernetes/IAM auth to Vault; Vault policies implement least privilege.- Observability: Audit logs streamed to centralized SIEM (ELK/Datadog/Splunk); telemetry for SPIRE/Vault.Core components & responsibilities:1. SPIFFE/SPIRE - Issue workload SVIDs (short-lived certs) bound to node + pod identity. - Enforce attestation (node, k8s SA, cloud instance metadata) to prevent spoofing.2. Vault (PKI, Transit, Secrets Engine) - PKI: issue short-lived service certificates on demand. - Dynamic secrets: DB creds, cloud IAM tokens with TTLs and automatic revocation. - Policies: fine-grained RBAC per service/namespace/cluster.3. Secret Distribution - Kubernetes: CSI Secrets Store driver + Vault provider to mount secrets as files; or Vault Agent sidecar for caching and renewal. - VMs/legacy: Vault Agent + local caching with auto-renewal and re-auth.4. Certificate rotation - Use very short TTLs (minutes to hours) for service certs; automated renewal via SPIRE/Vault Agents. - CRL/OCSP: enable OCSP stapling/OCSP responder or integrate Vault revocation and SPIRE revocation lists.Data flow (example):- Pod start -> SPIRE attests workload -> SPIRE issues SVID -> Service uses SVID to mTLS with peers via Envoy sidecar (or native TLS).- When service needs DB creds -> authenticates to Vault via mTLS/OIDC -> Vault returns dynamic creds with TTL -> Vault records audit event.- Sidecars (Envoy) enforce mTLS for all inter-service traffic using SVID-derived certs.Least-privilege controls:- Vault policies scoped to path: e.g., secret/data/service-a/*, allowed methods = read/list only.- Use Kubernetes service accounts mapped to Vault roles (K8s auth method) so pods only get secrets for their SA.- Transit encryption keys limited to services that require crypto.- Network segmentation & service mesh RBAC (e.g., Istio AuthorizationPolicies) to limit which services can call which.Certificate rotation & automation:- All certs are short-lived; agents auto-renew before TTL expiry.- Rotation workflow: issue new cert -> rolling restart not required if agent hot-reloads; Envoy supports SIGHUP or SDS (Secret Discovery Service) to push rotated certs.- Automated testing: CI pipelines validate certs and rotation behavior in staging.Revocation & recovery if leaked:- Immediate actions: 1. Revoke certificates/credentials via Vault (revoke by lease ID or revoke path). 2. Rotate any upstream secrets (DB passwords, cloud keys) using Vault dynamic creds or automation to generate new credentials. 3. Update network/firewall rules if specific IPs were abused. 4. Force re-attestation: evict node/pod, restart workload agents to obtain new SVIDs.- Longer-term: - Rotate all affected Vault root/unwrap tokens if breached; use sealed/unseal with HSM/KMS. - For compromised Vault instance, failover to DR replica and rotate master keys. - Post-incident audit: query Vault audit logs + SPIRE logs to scope access and timeline.Monitoring, auditing & compliance:- Centralized audit trail for Vault, SPIRE, Envoy mTLS logs, K8s audit logs.- Alerts on suspicious patterns: high frequency secret reads, many revocation requests, failed attestations.- Periodic penetration tests and automated secret scanning (prevent hard-coded secrets).Scalability & trade-offs:- Short TTLs improve security but increase load on Vault/SPIRE—use horizontal scaling, caching via agents, and regional replicas.- Centralized Vault simplifies policy but is a single control-plane; mitigate via HA and cross-region replication.- SPIRE + service mesh adds complexity; trade-off is much stronger identity guarantees and easier mTLS enforcement.Key recovery playbooks (summarized):- Leak detected -> revoke leases & certs -> rotate secrets (dynamic when possible) -> force re-attest/redeploy workloads -> audit & remediate infra -> postmortem & policy changes.This plan provides machine-verifiable service identity (SPIFFE), automated short-lived secrets and certs (Vault + PKI), least-privilege access (Vault policies + K8s SA mapping), automated rotation and in-place renewal (agents + SDS), and clear revocation/recovery procedures with auditing for incident response.
MediumTechnical
59 practiced
Describe caching strategies appropriate for different layers in a microservices architecture: client-side, CDN/edge, API gateway aggregation, service-side in-memory caches, and database-side caches. Explain cache invalidation strategies (TTL, versioning, pub/sub invalidation) and trade-offs for read-heavy services with strict latency SLAs.
Sample Answer
Start by matching cache scope to who benefits and consistency requirements.Client-side (browser/mobile)- Purpose: reduce round-trips and perceived latency for identical requests (UI assets, user-specific reads).- Strategy: strong use of Cache-Control (max-age, stale-while-revalidate), ETag/If-None-Match. Conservative caching for user-sensitive data (short TTL or private caches).CDN / Edge- Purpose: serve static content and hotspot API responses close to users.- Strategy: long TTLs for immutable assets; cache-control + surrogate keys for targeted invalidation. Use origin shield and tiered caching to protect origin.- Trade-off: great latency reduction, eventual staleness for dynamic data.API gateway aggregation- Purpose: cache composed responses (combined data from many services) to avoid repeated fan-out.- Strategy: short TTLs, Vary headers, request fingerprinting (user scope), and cache keys that include query params and auth scope.- Trade-off: complexity in cache coherence; stale aggregated views if underlying services change.Service-side in-memory caches (e.g., Redis, memcached, local LRU)- Purpose: microservice-level hot data for sub-ms reads.- Strategy: combination of local in-process LRU for ultra-low latency and shared Redis for cross-instance coherence.- Trade-off: capacity vs eviction, cache warming needed after deploys.Database-side caches (materialized views, read replicas, DB proxy cache)- Purpose: reduce expensive queries, support complex read patterns.- Strategy: materialized views refreshed on schedule or via change capture; read replicas for scaling reads.- Trade-off: replication lag, complexity in keeping materialized views up-to-date.Invalidation strategies- TTL: simplest; bounds staleness but may serve stale data until expiry.- Versioning (immutable objects / cache-busting keys): safe for assets and responses; requires producers to change keys on updates.- Pub/Sub invalidation: services publish change events to invalidate/refresh caches (use Redis pub/sub, Kafka). Low latency invalidation but requires reliable delivery and handling missed events (fallback TTL).- Hybrid: use pub/sub for strong invalidation plus TTL as safety net.Trade-offs for read-heavy services with strict latency SLAs- Prefer multi-tier caching: edge + API gateway + local in-memory + shared cache to hit fastest layer first.- Ensure high cache hit ratio: cache hot keys proactively (pre-warm), use consistent hashing to reduce cold starts.- Favor eventual consistency with near-real-time invalidation (pub/sub) for most data; use versioning/strong invalidation for critical fields needing strict correctness.- Monitor cache metrics (hit/miss, tail latency) and implement circuit-breakers: if cache miss storm occurs, protect DB with request coalescing (singleflight) and rate limits.- Operational trade-offs: more aggressive caching reduces latency and DB load but increases staleness and invalidation complexity. Choose policy per data criticality: strong consistency => short TTL + sync invalidation; high-read/low-update => long TTL + versioning.
EasyTechnical
57 practiced
What is the circuit breaker pattern and why is it useful in microservice ecosystems? Describe the main states (CLOSED, OPEN, HALF-OPEN), what thresholds or metrics drive transitions, and provide a concrete scenario where opening the circuit prevents cascading failures.
Sample Answer
The circuit breaker pattern protects services from repeated failures by stopping calls to a failing dependency for a period, allowing the failing service to recover and preventing cascading outages.States:- CLOSED: normal operation; requests pass through and failures are recorded.- OPEN: requests fail-fast (or return fallback); no calls to the downstream service for a configured timeout.- HALF-OPEN: after the timeout, allow a limited number of trial requests to probe the dependency; success -> CLOSED, failure -> OPEN.Metrics and thresholds driving transitions:- Failure rate threshold (e.g., >50% errors over last N requests or time window) triggers CLOSED -> OPEN.- Consecutive failures threshold (e.g., 5 failures in a row) can also trigger OPEN.- Open timeout (fixed or exponential backoff) controls when to move to HALF-OPEN.- Success threshold during HALF-OPEN (e.g., 3 consecutive successes) moves to CLOSED; a single failure can revert to OPEN.- Optional: latency thresholds (high latency counts as failure) and concurrency limits.Concrete scenario:Service A calls payment-service B. B starts timing out due to DB issues. A’s calls pile up waiting for timeouts, consuming thread pool and connection slots, causing A to become slow and then fail. A circuit breaker on A detects error rate >60% over 30s and opens: A immediately returns a cached/fallback response or meaningful error. This frees A’s resources, prevents thread exhaustion, and gives B time to recover. Once B is healthy, HALF-OPEN probes succeed and normal traffic resumes—avoiding a cascading failure across the system.Why useful:- Limits blast radius of failures- Improves overall system responsiveness and stability- Enables graceful degradation and faster recovery decisions
HardTechnical
74 practiced
Describe strategies to perform backward-compatible schema and API changes across multiple microservices without downtime. Cover patterns such as expand/contract (add then remove), dual-writing, consumer-driven migration, feature flags, and migration orchestration, including rollout and rollback considerations.
Sample Answer
Start with the guiding principle: every change must preserve existing contracts (requests/responses and DB reads) while allowing new behavior to be introduced incrementally. Use a multi-step, observable, reversible migration strategy.1) Expand–Contract (add then remove)- Expand: add new fields/columns, new nullable columns, new API response fields, or new event attributes. Keep defaults so old consumers ignore extras. For DB, use online schema tools (gh-ost, pt-online-schema-change) or native ALTERs that don’t block.- Migrate consumers to understand/use the new field.- Contract: once all producers and consumers are migrated, remove old fields or deprecated code paths in a controlled release.- Example: add user.preferred_name column (nullable) → update services to write it → update read services to prefer it → remove old fallback.2) Dual-writing / Dual-read- Dual-write: new and old schemas/services written in parallel. Ensure idempotency (use write ids/timestamps) and implement reconciliation jobs to detect drift.- Dual-read (feature-gated): route a subset of reads to the new path (shadow reads) and compare outcomes without impacting users.- Pitfall: divergence—have automated diffing, sampling, and backfills.3) Consumer-driven migration- Let consumers opt-in to new versions. Use semantic versioning for APIs/events; provide clear deprecation windows.- Contract tests and consumer-driven contract tools (Pact) to validate compatibility before deployment.- Use version negotiation: accept both v1 and v2 payloads, or provide adapters.4) Feature flags & gradual rollout- Expose changes behind feature flags so you can enable per-service, per-region, or percentage-based rollout.- Combine with canary deployments: enable to small subset (5-10%) and monitor errors, latency, error budgets, and business metrics.- Flags allow fast rollback by flipping off the flag.5) Migration orchestration, rollout & rollback- Orchestrate using a migration plan: pre-checks (schema/backfill readiness), phased deployment order (producers first for schema adds, consumers first for schema removes), backfill jobs, post-checks.- Use CI/CD pipelines with automated integration tests and contract tests. Automate metrics collection (SLOs), tracing, and sampling diffs.- Rollback: prefer feature-flag-based toggles; for DB changes, rollback often means stop writes to new column and revert application code—schema rollbacks are hard, so prefer non-destructive adds and prepare a forward-compatible compensating migration instead.- Have runbooks: metrics thresholds that trigger rollback, automated reconcilers, and a communication plan.Operational notes:- Always treat schema migrations as deploy-time and runtime concerns. Use idempotent writes, multi-versioned deserializers, and clear deprecation timelines.- Test with shadow traffic and production-like data. Monitor reconciliation counters, consumer contract test pass rates, and business metrics.This combination — expand/contract, dual-writing/reading, consumer-driven contracts, feature flags, and careful orchestration — enables zero-downtime, reversible migrations across many microservices.
HardSystem Design
59 practiced
For a global microservices architecture with regional read replicas, design a cache coherency and invalidation strategy that minimizes stale reads while preserving low read latency. Discuss TTLs, change-data-capture-based invalidation, pub/sub invalidation, read-your-writes requirements, and fallback reconciliation when invalidations are missed.
Sample Answer
Requirements and constraints:- Global read-heavy microservices with regional read replicas; writes are global single source of truth (or routed to leader).- Goal: minimize stale reads, keep sub-10ms regional read latency, support read-your-writes within a session, tolerate occasional missed invalidations with reconciliation.High-level approach:- Use multi-layer caching: regional in-memory caches (L1) per service instance + regional distributed cache (L2, e.g., Redis cluster) backing reads. Central authoritative DB(s) with CDC emitting changes to an invalidation bus.Core components & responsibilities:1. CDC publisher: DB transaction log -> change events (key, version/timestamp, op). Ensure at-least-once delivery and include commit sequence number.2. Global pub/sub invalidation bus (e.g., Kafka + geo-replicated topics or regional Kafka with MirrorMaker/Confluent Replicator): routes CDC events to regional consumers.3. Regional invalidation consumer: applies events to regional L2 cache (evict or update), then notifies local in-process caches.4. Read path: check L1 → L2 → DB; attach version/timestamp when returning cached data.5. Read-your-writes: client/session attaches session token with last-write-seq; reads check cached item’s seq >= session seq or bypass cache to DB (or request a lightweight validation).6. Fallback reconciliation: periodic background validation scanning hot keys (compare cached version vs. latest seq from lightweight metadata store or a “version map” in Redis). If mismatch, refresh from DB.TTL and invalidation strategy:- Short TTLs for highly dynamic data (1–5s); longer TTLs for stable data (minutes). TTLs are safety net for missed invalidations.- CDC-driven invalidation is primary: on event, regionally evict/update key and propagate to process caches via local pub/sub (e.g., Redis keyspace notifications or in-memory event bus).- Apply "update-in-place" when possible (atomic write to cache with version check) to avoid read window where key absent.Handling eventual consistency & missed invalidations:- Use versioned cache entries (sequence numbers). Reads detect stale data by comparing cached seq to session seq or a “min-accepted-seq” published per region.- On a cache miss or detected staleness, fetch from DB and write back with current seq.- Implement exponential backoff retries for failed invalidation deliveries and monitor lag; metrics/alerts for CDC consumer lag.Trade-offs and operational notes:- CDC + pub/sub minimizes stale reads with near-real-time invalidations; costs: complexity, cross-region replication latency.- TTLs reduce impact of missed invalidations at cost of additional DB reads.- Read-your-writes enforces strong session consistency at small extra read cost; can be tuned per endpoint.- Reconciliation jobs catch rare edge cases; keep them incremental and scoped to hot keys to limit DB impact.This design balances low-latency regional reads with robust, versioned CDC invalidation and session guarantees; TTLs and background reconciliation provide safety nets when invalidations are missed.
Unlock Full Question Bank
Get access to hundreds of Microservices Architecture and Service Design interview questions and detailed answers.
