Microservices Architecture and Service Design Questions
Covers the principles, patterns, and trade offs for designing, decomposing, operating, and evolving microservice and service oriented architectures. Candidates should be able to define service boundaries and decomposition strategies, explain domain driven design influences, and describe safe approaches to break a monolith into independently deployable services. Topic coverage includes application programming interface design and versioning, synchronous and asynchronous inter service communication patterns such as representational state transfer, remote procedure call frameworks, and messaging systems, as well as event driven architecture patterns. It includes data ownership and distribution, consistency models, distributed transaction patterns including the saga pattern and two phase commit trade offs, and resilience patterns such as circuit breakers, retries, and bulkheads. Operational concerns include service discovery, gateway and service mesh patterns, deployment and rollout strategies for independent services, observability and distributed tracing, monitoring, testing and debugging across services, failure handling and network latency considerations. The topic also covers organizational impacts including Conway's law, service choreography versus orchestration, team boundaries and operational complexity, and guidance on when to choose a monolith versus microservices.
HardTechnical
68 practiced
Design a monitoring and alerting system to detect eventual consistency violations between a write service and multiple read targets (search index, materialized views). Define the metrics to track (lag, divergence count), acceptable staleness thresholds, automated reconciliation strategies, and alert logic that balances noise and timely detection.
Sample Answer
Requirements & constraints:- Detect violations where reads (search index, materialized views) diverge from the authoritative write service.- Low false positives, timely detection (< minutes for critical data).- Auto-reconcile where safe; alert humans for persistent or risky cases.- Scale to high write/read QPS, multi-region.Metrics to track (per entity/key and aggregated):- Write version / write timestamp (last-write-ts)- Read observed version / last-seen-ts (per read target)- Lag (time): last-seen-ts_read_target - last-write-ts- Version divergence count: count of keys where read_version != write_version- Missing document count: keys present in write service but absent in read target- Reconciliation attempts/success rate, and retry/backoff state- TTL of divergence (how long a key has been divergent)Acceptable staleness thresholds:- Critical keys: 0–5s lag; alert at >5s, incident at >60s- User-visible but non-critical: 0–60s; alert at >60s, incident at >15m- Batch-only data: alert at >1hDefine SLOs per data class (e.g., 99.9% keys within threshold).Automated reconciliation strategies:- Fast-path: idempotent upsert to read target (push latest version) triggered when missing or version mismatch detected.- Backfill queue: for bulk divergences use rate-limited worker pool with exponential backoff and prioritization (critical keys first).- Read-compare on access: on user read, compare read target vs write service and, if mismatch, serve authoritative value and enqueue reconciliation.- Safe-rollbacks: if reconciliation fails repeatedly, mark for human review and stop automated attempts to avoid thrash.Alert logic (balance noise vs detection):- Multi-tier alerts: - Warning: short spikes — divergence rate > X keys/min sustained > 2m but below SLO impact. - Pager (P1): divergence affecting >Y% of traffic or critical-key lag > incident threshold OR reconciliation success rate < 90% for 5m. - Pager (P2): persistent per-key TTL > incident threshold for top N keys by traffic.- Use anomaly detection + static thresholds: baseline divergence rate via rolling-window percentiles to reduce noisy alerts during known maintenance windows.- Silence/auto-mute during deployments, region failovers; require deployment tagging to disable alerts.- Include enriched context in alerts: top offending keys, read-target, recent reconciliation attempts, topology (region), and runbook link.Implementation notes:- Sampling + aggregation: detect per-key for critical keys; sample otherwise and compute extrapolated divergence rates to scale.- Store versions in a compact change-log (Kafka/CDC) to drive detection and reconciliation.- Observability: dashboards showing lag percentiles, divergence heatmap, reconciliation KPIs; audit logs for reconciliation actions.Trade-offs:- Per-key monitoring gives accuracy but is expensive; sampling + prioritization balances cost.- Aggressive auto-reconciliation reduces user-visible errors but risks write amplification — throttle and circuit-breaker recommended.
MediumSystem Design
57 practiced
Design a distributed rate-limiting system enforcing per-user and global quotas for APIs across stateless service instances and regions. Discuss algorithms (token bucket vs leaky bucket), state placement (centralized Redis vs local token caches), fairness across regions, degraded modes when stores are unavailable, and SRE operational concerns like monitoring denial-of-service and quota evasion.
Sample Answer
Requirements & constraints:- Enforce per-user (per API key) and global quotas across stateless service instances and multi-region deployment.- Low latency on request path (<ms level), high availability, eventual consistency across regions, and protection against bursts/DoS.High-level architecture:- API gateway / sidecar on each instance performs fast local checks and calls a central store for authoritative accounting when needed.- Centralized strongly-consistent counter store per region (Redis cluster with replication) + cross-region sync/leader for global quotas.- Optional global coordinator (sharded) for global quotas.Algorithms:- Token bucket for per-user quotas: supports bursts by maintaining token refill rate + capacity. Easier to reason about bursts allowed.- Leaky bucket if strict smoothing required (constant outflow) — less bursty but higher latency for initial requests.- Use token bucket locally for fast decisioning; central store maintains authoritative tokens.State placement & hybrid approach:- Primary state in regional Redis (authoritative per-region quotas and global aggregated counters).- Local in-memory token caches with conservative refill: on first request, fetch tokens from Redis; allow local serving of small bursts (configurable tokens per user/instance).- Periodic reconciliation: local cache drains synced back to Redis (atomic decrement via Lua scripts) to prevent double spend.- For global quotas, use a two-tier: regional quotas allocated from global pool via periodic lease (e.g., allocate X tokens/sec to region). For strict global limits, use a global Redis or consistent metadata service (e.g., Spanner) for allocation, but only on contention path.Fairness across regions:- Allocate global capacity to regions proportional to traffic SLA or dynamic load-based allocation. Use token leasing windows (e.g., 1s–1m) so regions get fair shares and can borrow with coordination.- Use deterministic hashing to route client keys to a preferred region for accounting to reduce cross-region coordination (sticky routing).Degraded modes:- If Redis unavailable regionally: fall back to local token cache with reduced capacity (fail-open policy with strict throttling), emit high-severity alerts. For global store failures, apply conservative per-user limits and reject requests exceeding local reserve.- If central store degraded but read-only: continue serving until local tokens exhausted then 429.- If store unreachable for extended time, shift to aggressive fail-closed for suspected abuse clients (blacklist) and allow whitelisted clients via emergency overrides.Operational & SRE concerns:- Monitoring: per-user/global throughput, token refill/consumption rates, Redis latency/ops, error rates (429), burst patterns, and rate of fallbacks to degraded mode.- Alerting: high percentage of fallbacks, sudden token consumption spikes, cross-region imbalance, increased 429s.- Detecting DoS & quota evasion: correlate spikes across API keys/IPs, detect many keys with identical patterns, fingerprint clients, rate-limit by IP/subnet and CAPTCHAs for suspicious patterns, apply anomaly detection (threshold + ML).- Security: protect Redis with auth, network isolation, and rate-limit the control plane APIs to prevent manipulation.- Testing & rolling deploys: feature flags for local cache sizes; chaos testing for Redis failures; synthetic traffic to validate fairness and reduced modes.- Observability: expose telemetry (tokens left per user sample, lease durations) and maintain replayable logs for postmortems.- Capacity planning: size Redis for peak token ops (Lua scripts atomically decrement + TTL), provision cross-region bandwidth for sync.Trade-offs:- Centralized Redis gives accuracy but higher latency and single-point risk; hybrid local caches reduce latency but increase complexity and temporary over-allocation risk.- Token bucket favored for flexibility; leaky bucket if strict smoothing required.This design balances low-latency enforcement, regional availability, and global fairness while providing clear degraded behaviors and SRE observability to operate safely.
EasyTechnical
72 practiced
Explain the delivery guarantees in event-driven systems: at-most-once, at-least-once, and exactly-once. As an SRE, what operational patterns and safeguards would you implement to ensure correct processing (deduplication, idempotency, ordering guarantees) and to recover from duplicates or missing events across services?
Sample Answer
At-most-once, at-least-once, exactly-once — what they mean and how an SRE operationalizes them:Definitions- At-most-once: messages are delivered zero or one time. Fast, but can lose messages (no retries).- At-least-once: messages are retried until acknowledgement → possible duplicates.- Exactly-once: each message is processed once from the application perspective (often implemented via idempotency + dedup + transactional sinks; true distributed exactly-once is expensive).Operational patterns & safeguards- Idempotency: require consumers/actions to be idempotent. Use idempotency keys (request IDs, message IDs) persisted in a dedup store (Redis with TTL, DynamoDB) so replays detect and skip duplicates.- Deduplication: keep a sliding window of processed message IDs or sequence numbers; evict by TTL to bound storage.- Ordering guarantees: partition by key so messages for one entity land in same partition (Kafka partitioning). Ensure single-threaded processing per partition or use sequence numbers + reordering buffer.- Reliable offsets/acks: commit consumer offsets only after processing and durable side-effects are persisted (use two-phase commit patterns or Kafka transactions where supported).- Retry/backoff and DLQs: implement exponential backoff for transient failures and route poison messages to a Dead Letter Queue with visibility into failures.- Transactional sinks: use broker-supported transactions (Kafka exactly-once semantics) or idempotent writes to downstream stores (upserts with idempotency key).- Observability & runbooks: monitor duplicate rate, lag, retry counts, DLQ volume, and create alerts/SLOs. Provide automated replay tools to reprocess ranges of offsets and safe rollbacks.- Schema & contract hygiene: use versioned schemas, validation, and compatibility checks to prevent silent drop or misprocessing.- Recovery procedures: documented steps to analyze DLQ entries, dedup store TTL tuning, controlled replays (replay into a staging consumer), and coordination (feature flags or maintenance windows) when repairing stateful consumers.Example: for an order service, assign order_id as partition key, persist processed message_id in a Redis set before applying payment state change; commit the consumer offset only after Redis+DB transaction completes; on failure, retry with backoff and if repeated failure send message to DLQ and alert on-call.These patterns combined let you choose practical guarantees: prefer at-least-once with idempotency and monitoring in most systems; adopt broker transactions when true exactly-once is required and justified by cost.
HardTechnical
83 practiced
You get alerts: API gateway 5xx error rate rising and downstream services showing latency spikes. Walk through a detailed SRE incident response for this cross-service outage: triage steps, queries and tools you would use (metrics, logs, traces), immediate mitigation options to reduce customer impact, and post-incident actions to prevent recurrence.
Sample Answer
Situation: Alerts indicate API Gateway 5xx rate rising and downstream services showing latency spikes — a cross-service outage with user impact.Triage (first 0–5 minutes)- Acknowledge alert, declare incident severity, notify on-call and stakeholders.- Confirm blast radius: check client error vs server error counts, affected regions, and traffic volume.- Quick questions: Is traffic abnormal? Recent deploys/rollouts? Any infra events (AZ/host failures)?Immediate data collection (5–15 minutes)- Metrics: Grafana/Prometheus queries: - sum(rate(api_gateway_http_requests_total{code=~"5.."}[1m])) by (service, region) - histograms for request latencies: api_gateway_request_duration_seconds_bucket - downstream service p50/p95/p99: service_response_time_seconds{service=...} - instance CPU/memory, connection errors, retries, queue length- Logs: Kibana/ELK query for 5xx traces: - request_id and full error stack: status:5.. AND service:api-gateway AND timestamp:[now-15m TO now] - downstream error codes/timeout messages- Traces: Jaeger/Zipkin lookup by high-latency traces and sampled 5xx traces; inspect causal path and time spent in each span.- Events: Check deployment pipelines, config changes, infra autoscaling, rate-limiter or circuit-breaker state, and DNS/LB health.Initial diagnosis patterns (15–30 minutes)- Gateway producing 5xx while downstream latency spikes → likely downstream overload, timeouts, or resource exhaustion.- If traces show long wait on a specific downstream call, target that service.- If many new clients or replayed traffic, consider traffic surge/DoS.Immediate mitigation options (while diagnosing)- Fail fast and reduce customer impact: - Enable circuit breakers or increase timeout/queue limits to avoid resource exhaustion. - Return graceful degraded responses for non-critical endpoints (cached responses, 503 with Retry-After). - Throttle or rate-limit noisy clients (per-IP or API-key), apply emergency rate limits at API gateway or WAF. - Roll back recent deploys if correlated with incident. - Scale up capacity: increase replicas/instance count, provision additional nodes, or scale up DB read replicas. - Enable routing to healthy regions or use feature flags to disable heavy features.- Communicate status to stakeholders and surface user-facing message.Resolution and verification (30–90 minutes)- Apply mitigations stepwise, monitor 5xx and latency metrics for improvement.- Use traces to confirm the critical span latency is reduced.- When stable, gradually remove mitigations (throttles, temporary fallbacks) and monitor.Post-incident (post-mortem, 24–72 hours)- Run blameless postmortem: timeline, root cause analysis (using metrics/traces/logs), decisions made, what worked/failed.- Identify fixes: - Code: fix root cause bugs, add timeouts/retries/backoff where missing. - Capacity: adjust autoscaling policies, increase resource headroom, add circuit breakers. - Observability: add SLO-based alerts (p95 latency and error budget burn), create dashboards showing top downstream latencies, add synthetic checks for critical paths. - Runbook: update incident playbook with exact runbook steps (queries, mitigations, thresholds) and owner. - Testing: add chaos/surge testing for that failure mode and load tests for the problematic endpoint.- Follow-up: schedule implementation tasks, track through tickets, and review with teams. Share learnings and update on-call training.Key principles used: reduce blast radius quickly, gather evidence (metrics/logs/traces) to find the causal span, apply reversible mitigations, and codify fixes into automation and runbooks to prevent recurrence.
HardTechnical
67 practiced
Create an incident postmortem template and a runbook for recurring cross-service outages caused by cascading failures. Include required sections: timeline, impact, root cause analysis (beyond symptoms), SLO impact, action items with owners and due dates, communications timeline, and automated mitigations that should be implemented. Provide a short runbook play for isolating failing services.
Sample Answer
Incident Postmortem Template — Cascading Cross-Service Outage1) Summary- Date/time window:- Services affected:- Severity / incident ID:- Short description: (one-line cause + impact)2) Timeline (precise, with timestamps)- Detection (who/alert/time)- Key events (failures, mitigations, config changes)- Service degradations and recoveries- Communications sent (times & channels)3) Impact- User-visible impact (requests/min lost, errors, regions)- Internal impact (batch jobs, downstream systems)- Business metrics affected (revenue, SLAs, campaigns)4) Root Cause Analysis (beyond symptoms)- Component-level failure(s)- Dependency graph showing cascading path- Why initial failure occurred (bug/config/resource/behavior)- Why failure propagated (retry storms, poor circuit breakers, shared resource, backpressure absent)- Why detection/mitigation failed (alert thresholds, runbook gaps)5) SLO Impact- Affected SLOs, windows, error budget consumed (%)- Historical context (is this recurring? trend analysis)- Customer segments impacted6) Action Items (owner / due date / priority / verification)- Example: Implement per-service circuit breakers — Owner: @alice — Due: 2025-01-15 — Verify: chaos test and staging rollout- Example: Rate-limit retries at client — Owner: @bob — Due: 2024-12-01 — Verify: Load test- Example: Add dependency topology & automated fault injection — Owner: SRE Platform — Due: 2025-02-01 — Verify: runbook drills(Include mitigation, detection, prevention, observability tasks; mark “done” with evidence)7) Communications Timeline- Initial detection alert (timestamp)- Internal incident channel opened (timestamp)- External customer notification (when, content owner)- Status updates cadence- Post-incident comms and RCA publication schedule8) Automated Mitigations to Implement- Auto-tripping circuit breakers per service when downstream latency > X- Global backoff orchestrator for retries (exponential with jitter + token bucket)- Prioritized request queuing and graceful degradation (feature flags)- Autoscale policies tied to queue depth and tail latency- Automated dependency isolation (see runbook play)- Health-check driven traffic-shed in ingress layer9) Lessons Learned- Recurrence reasons- Process gaps- Next review dateRunbook Play — Isolate Failing Service (short)Goal: Stop propagation quickly, restore global stability.Steps:1) Detect & confirm: Check alerts/trace spikes and dependency graph. Confirm service S as source.2) Throttle incoming traffic: Apply ingress rate-limit for S (API gateway / LB) to 10–20% of normal to reduce pressure.3) Enable circuit breakers: Push config to set breaker to OPEN for downstream calls from clients to S.4) Divert or shed non-critical traffic: Apply routing rule to send non-essential endpoints to degraded mode.5) Scale or rollback: If config/deploy caused issue, rollback; else scale S cautiously (horizontal) if CPU/queues high.6) Quarantine dependent services: For each downstream D experiencing retries, put D in degraded mode (reject or return cached responses).7) Monitor & stabilize: Validate error rate and latency drop for entire topology for 2× MTTD window.8) Post-action: Keep isolation until root cause fixed and tested; document actions and times.Notes & Ownership:- Incident commander orchestrates steps; platform SRE executes rate-limit/CB changes; service owner approves rollbacks.- Automate steps 2–4 as playbooks with a single run command and safety checks.
Unlock Full Question Bank
Get access to hundreds of Microservices Architecture and Service Design interview questions and detailed answers.
