Microservices Architecture and Service Design Questions
Covers the principles, patterns, and trade offs for designing, decomposing, operating, and evolving microservice and service oriented architectures. Candidates should be able to define service boundaries and decomposition strategies, explain domain driven design influences, and describe safe approaches to break a monolith into independently deployable services. Topic coverage includes application programming interface design and versioning, synchronous and asynchronous inter service communication patterns such as representational state transfer, remote procedure call frameworks, and messaging systems, as well as event driven architecture patterns. It includes data ownership and distribution, consistency models, distributed transaction patterns including the saga pattern and two phase commit trade offs, and resilience patterns such as circuit breakers, retries, and bulkheads. Operational concerns include service discovery, gateway and service mesh patterns, deployment and rollout strategies for independent services, observability and distributed tracing, monitoring, testing and debugging across services, failure handling and network latency considerations. The topic also covers organizational impacts including Conway's law, service choreography versus orchestration, team boundaries and operational complexity, and guidance on when to choose a monolith versus microservices.
HardTechnical
71 practiced
A production model-serving microservice suddenly shows a 20% drop in a key business metric along with increased error rate. As the on-call ML engineer, outline your incident response plan: initial triage steps, data to collect, decision criteria for rollback vs mitigation, communication with stakeholders, and post-incident analysis actions.
Sample Answer
Initial triage (first 0–15 minutes)- Acknowledge alert and page relevant owners (infra, SRE, product lead). Set a 15-minute incident window.- Quickly confirm scope: which endpoints/models, regions, clients, and time window show the drop. Check deployment events (recent rollouts, config changes, infra changes) in the last 24h.- Halt automated retraining/rollouts if running. If traffic-splitting (canary/A-B), shift 100% to control if safe.Data to collect (first 15–60 minutes)- Business metric time series and model metrics: inference rates, error rate, latency, input feature distributions, model confidence scores, prediction drift, loss/score distributions.- Logs and traces: request/response logs, stack traces, dependency failures (DB, feature store, feature-serving latencies), container/VM metrics (CPU, memory, OOM, restarts).- Recent commits, config changes, docker image tags, feature-store ingestion delays, schema changes.- Sample inputs and corresponding outputs for failed and successful requests.- Affected customer list and error percentages by client.Triage analysis & hypotheses (15–90 minutes)- Check if issue is data-related (feature drift/missing features), model-related (bug/regression from new weights), or infra-related (latency/timeouts, resource exhaustion).- Validate model predictions offline on recent samples to see if predictions changed vs. expected baseline.- Reproduce error locally or on staging with captured inputs.Decision criteria: rollback vs mitigation- Rollback if: a recent model deploy or config change coincides with metric drop; offline evaluation shows regression; rollback can be executed safely and quickly (<15–30 min) and reduces risk.- Mitigate if: infra/dependency issue (DB outage, feature-store lag) or partial failure where circuit-breakers, rate-limiting, or fallbacks (use cached predictions or model-A) restore service while investigating.- Never rollback blindly if root cause unclear and rollback might worsen (e.g., feature schema changes). Prefer traffic switch to previous stable model/version.Immediate actions- If rollback chosen: execute blue/green or revert to stable image, validate health checks, monitor business metric and error rate for recovery.- If mitigation chosen: enable fallback model, increase autoscaling, throttle suspect clients, or serve cached responses; apply temporary feature validation filters.Communication- Send initial incident bridge invite and one-line summary to stakeholders (what, scope, owner, ETA for next update).- Provide updates every 15–30 minutes with status, actions taken, impact, and ETA for next update. Use dedicated channel (Slack + incident doc).- Once service stable, send severity, timeline, and next steps to PMs/customer support for external comms.Post-incident (within 24–72 hours)- Run blameless post-mortem: timeline, root cause, contributing factors, detection & response gaps.- Action items with owners and deadlines: deployment safety (require canary durations, automated rollback thresholds), improved monitoring (feature drift alerts, prediction-accuracy proxies), runbooks, automated tests (schema/contract checks), and chaos tests for dependencies.- Add regression tests to CI that caught the issue; implement gated deploys and better observability (per-feature histograms, data contracts).- Communicate final report and preventive actions to stakeholders.This plan prioritizes restoring business function safely, collecting evidence to identify root cause, and preventing recurrence through automation and improved monitoring.
MediumTechnical
99 practiced
Design a system to guarantee feature parity between offline training features and online serving features. Include options like single codepath for feature computation, feature versioning, storage formats, validation checks, and automated reconciliation processes to detect divergence before training or deployment.
Sample Answer
Situation/Goal: Guarantee feature parity between offline training and online serving to avoid training-serving skew and ensure reproducible models.Design overview:- Single codepath for feature computation: - Implement feature logic as portable functions/libraries (Python) that run both in batch pipelines and in streaming/online feature servers. Package as versioned artifacts (pip wheel or container). - Use same tests and CI to validate the library before release.- Feature versioning and contract: - Assign each feature a stable identifier and semantic version (feature_id:v#). Store schema (name, dtype, nullability, units, transformation logic pointer, owner). - Treat breaking changes as new feature versions; keep older versions available for model reproducibility.- Storage formats and serving: - Offline store: columnar, immutable Parquet/Delta Lake tables partitioned by date + feature_version metadata. - Online store: key-value low-latency store (Redis, Cassandra, HBase) with entries keyed by entity_id + feature_version. Use TTLs and materialized views updated by streaming jobs (Kafka/Beam/Flink). - Ensure serialization parity (Avro/Protobuf) and canonical typing.- Validation checks: - Static checks: schema conformance, unit tests, type checks, null-rate thresholds. - Runtime checks: shadow reads—when serving, read feature from online store and also compute/lookup the corresponding offline value for a sample subset to compare. - Data-quality monitors: distributions, cardinality, freshness, schema drift; raise alerts on deviation thresholds.- Automated reconciliation: - Nightly job compares latest offline feature snapshots to online store (sampled or full key-scan) and reports mismatches per feature_version. - Pre-deploy gate: CI runs end-to-end pipeline to compute training features using the library and compares a held-out sample against online serving values; block deployment if divergence > threshold. - Reconciliation actions: auto-roll back feature library version, create incident, or trigger re-computation.Trade-offs and operational notes:- Single codepath increases reliability but requires careful isolation of env differences (e.g., numpy vs streaming windows).- Versioning and immutable offline store enable reproducibility at cost of storage.- Use observability dashboards and automated alerts to catch regressions early.
EasyTechnical
56 practiced
Explain the trade-offs between REST (JSON/HTTP) and gRPC (protocol buffers) for ML inference services. Consider factors such as latency, payload size, streaming, browser/mobile compatibility, polyglot clients, proxies, compatibility with service meshes, and operational complexity.
Sample Answer
Short answer: REST/JSON over HTTP is simple, human-readable, widely compatible and easy to operate; gRPC with Protocol Buffers is binary, more compact and faster, especially for low-latency, high-throughput ML inference, but adds some operational and compatibility complexity. Below are trade-offs across the requested dimensions.- Latency: gRPC typically has lower latency due to HTTP/2, multiplexing, and binary serialization. REST/JSON has higher parsing and header overhead; acceptable for batch or infrequent calls.- Payload size: Protobufs are compact and faster to serialize; JSON is verbose (larger bandwidth) which matters for large model inputs/outputs (images, vectors).- Streaming: gRPC natively supports client/server and bidi streaming via HTTP/2—great for live inference streams or progressive results. REST requires workarounds (chunked responses, websockets).- Browser/mobile compatibility: REST/JSON works natively in browsers and is easiest on mobile. gRPC-Web exists but requires a proxy (envoy) and has some limitations; mobile native gRPC clients are available.- Polyglot clients: REST is language-agnostic (HTTP + JSON). gRPC has first-class multi-language support via generated stubs—excellent if you can adopt protobuf tooling.- Proxies and intermediaries: Many HTTP proxies and CDNs understand JSON/HTTP well. gRPC over HTTP/2 may be blocked or need proxy support; gRPC-Web or Envoy often required.- Service meshes: gRPC integrates well with modern meshes (mTLS, tracing) but watch for HTTP/2 settings; many meshes support both, but gRPC benefits from HTTP/2 connection reuse.- Operational complexity: REST is simpler to debug (curl, inspect JSON). gRPC requires proto management, tooling, and sometimes extra proxy layers; faster performance can justify added ops complexity.Recommendation: For low-latency, high-throughput internal ML inference between services, prefer gRPC/protobuf. For public APIs, browser clients, simpler ops, or when human-readable payloads help, prefer REST/JSON. Hybrid approach common: gRPC inside the cluster, REST or gRPC-Web at the edge.
HardTechnical
64 practiced
Design ML microservices and processes that handle PII-sensitive inputs while meeting GDPR requirements. Cover data minimization, encryption at rest and in transit, consent logging, right-to-be-forgotten, pseudonymization/tokenization in caches, audit trails, and multi-region considerations for data residency.
Sample Answer
Requirements & constraints:- Handle PII inputs for ML while complying with GDPR: purpose limitation, data minimization, lawful basis (consent/contract), data residency, right to rectify/erase, and auditability. Low latency for model inference; scalable, multi-region.High-level design:- Ingest API (edge) → Preprocessing & Privacy Layer → Model Serving (stateless) → Postprocess → Audit & Consent Store → Secure Data Lake (for training) / Key Management Service.Key controls and components:- Data minimization: accept only fields strictly needed for the model. Enforce schema validation and a “purpose” tag per feature; drop ancillary PII at ingest. Use feature flags to avoid collecting new PII without review.- Consent logging & lawful basis: central Consent Service recording user consent with timestamps, scope, TTL, and source (UI/API). Tokenize requests with consent-token; services check consent via cached signed JWTs to avoid latency while maintaining revocation list.- Encryption in transit & at rest: TLS 1.2+/mTLS for all internal and external comms. All storage encrypted with envelope encryption (KMS-managed DEKs and CMKs). Rotate keys regularly; audit KMS access.- Pseudonymization / tokenization: replace direct identifiers with stable pseudonyms for model features. Use deterministic keyed HMAC or reversible tokenization in a secured Token Service when re-identification is needed (accessed only by authorized jobs).- Caches: never cache raw PII. Cache only pseudonymized tokens or hashed feature vectors. Use short TTLs and encrypt cache stores (Redis with TLS and ACLs). For sensitive lookup caches, use tokenization service to return tokens, not PII.- Right-to-be-forgotten / erasure: maintain mapping from pseudonym -> raw PII in a secured, auditable store. On erasure request, remove raw PII, revoke associated pseudonyms (or mark as deleted), propagate revocation to feature stores, caches, and model retraining pipelines. Implement eventual consistency with strict SLAs; surface errors to Privacy Ops.- Audit trails: immutable append-only audit logs (WORM) for data access, consent changes, key operations, and model inference decisions that involve PII. Logs include who/what/why/timestamp and are hashed/chained. Retain per retention policy; encrypt and restrict access.- Training pipelines: use differential privacy where feasible (DP-SGD) or aggregate-only outputs. For transfer of datasets across regions, use anonymization & data minimization; if raw PII required, apply data residency rules and confine processing to permitted regions.- Multi-region & data residency: tag records with region/residency constraints. Route ingest and storage to region-local clusters; replicate only pseudonymized or aggregated exports across regions. Use geo-fenced KMS keys per region.- Access control & governance: RBAC + ABAC for services and human access; strong IAM policies, just-in-time elevated access, approval workflows, and break-glass with logged justification. Automated periodic attestation.- Monitoring & validation: anomaly detection for unusual access patterns; periodic privacy risk assessments and model explainability checks to detect that models aren't inferring unauthorized PII. Automated tests for erasure propagation, consent revocation, and key rotation.- Incident response & legal alignment: predefined procedures for data breaches, data subject requests SLA (e.g., 30 days), and liaison with Legal/Privacy teams. Provide data subject exports with interpretable feature mapping and model decision logs.Trade-offs & reasoning:- Deterministic pseudonymization enables joining across events but increases re-identification risk; mitigate with strong key management and limited access.- DP reduces utility; apply selectively to training sets where privacy/utility trade-off favors privacy.- Real-time consent checks increase latency; mitigate with signed short-lived tokens and revocation cache.This design balances model utility with GDPR requirements through minimization, cryptographic protections, controlled re-identification, auditable flows, and region-aware data handling—operationalized by services (Consent, Tokenization, KMS, Audit) and enforced by CI/CD, tests, and governance.
HardTechnical
63 practiced
You must design an organizational proposal to reduce operational complexity caused by many small ML services. Propose criteria to decide which services to consolidate, a migration plan, cost thresholds, service ownership models, and platform tooling investments (shared libraries, centralized observability, deployment templates) to reduce cognitive load while retaining team autonomy.
Sample Answer
Situation: Our org has dozens of small, independently owned ML services causing high ops overhead, duplicated work, and reliability drift.Proposal — goals: reduce cognitive load and cost by consolidating where it increases ROI, while preserving team autonomy for product innovation.Criteria to consolidate (decision matrix, score 0–5 each):- Low traffic / low QPS (<=10 rps) — candidate for consolidation- High functional overlap (shared model inputs/features)- Shared SLA requirements / latency tolerance- High maintenance burden (>=2 engineers/month)- Low release velocity dependency on other teamsConsolidate if weighted score > threshold (e.g., 12/25).Migration plan (phased):1. Audit & prioritize top 20% services by ops cost and overlap.2. Create shared-hosting blueprints (API contracts, feature stores).3. Pilot: move 2 services into a consolidated runtime (3 months) with rollback.4. Migrate in waves, pairing owning team with platform engineers.5. Validate: SLOs, cost, latency, and runbook readiness; deprecate old infra.Cost thresholds:- Consolidate if estimated 6–12 month payback (TCO reduction >=30%).- If consolidation adds >20% tail-latency or jeopardizes SLAs → keep separate.- Manual-maintenance >10 engineer-hours/week triggers review.Service ownership models:- Product-aligned ownership for model logic and validation.- Platform-owned runtime and infra (deploy, autoscale, common ML infra).- Clear RACI: teams own model code + tests; platform owns deployments, observability, and security.Platform tooling investments (priorities):- Shared libraries: standard model serialization, input validation, feature access, drift detection hooks.- Centralized observability: standardized telemetry (predictions, inputs, feature distributions), centralized dashboards and alerting templates.- Deployment templates: Terraform/K8s Helm charts, CI/CD pipelines with canary and automated rollback.- Automation: autoscaling policies, cost tagging, quota enforcement.- Developer DX: local emulators, templates, and migration guides.Metrics & governance:- Track mean time to onboard, incidents/month, infra cost per model, duplicate feature definitions.- Quarterly review board to re-evaluate consolidations and exceptions.This balances cost and cognitive load while keeping teams responsible for model quality and product outcomes.
Unlock Full Question Bank
Get access to hundreds of Microservices Architecture and Service Design interview questions and detailed answers.
