Architectural Patterns and Anti-Patterns Questions

Core SLIs to track for a public HTTP service:

• Availability (successful responses / total requests, e.g., 2xx/3xx ratio)
• Latency distributions (p50, p95, p99 for request duration) split by critical endpoints
• Error rate by class (5xx rate, client vs server errors)
• Rate of traffic (RPS) and saturation indicators (CPU, memory, queue depth)
• Request success with full functionality (synthetic end-to-end checks)
• Time to first byte (TTFB) and tail latency for real-user monitoring

Reasonable SLOs (example, adjust to business needs):

• Availability: 99.95% over a 28-day rolling window (≈ 22 minutes downtime/month)
• Latency: p95 < 200 ms, p99 < 1 s for API read endpoints; p95 < 500 ms for heavier write endpoints
• Error rate: 5xx < 0.1% over 28 days

Using an error budget operationally:

• Define error budget = 1 - SLO (e.g., 0.05% budget for availability). Track consumption continuously.
• Release policy: full releases allowed while budget healthy (<50% consumed). If consumption passes thresholds (e.g., 50%, 80%), apply progressive restrictions: reduce release velocity, enable canary with longer evaluation, require postmortem for risky changes.
• Incident response: prioritize remediation when budget burn spikes. If burst consumption threatens SLO, enact mitigation playbooks, rollback or disable risky features, and divert capacity to stability work.
• Prioritization: convert remaining budget into sprint quota for reliability vs feature work—when budget low, prioritize reliability tickets.
• Alerts & dashboards: create actionable alerts for fast burn rates (e.g., error budget burn rate alarm) and provide runbooks mapping burn state to actions.
• Post-incident: perform blameless postmortem, update SLI instrumentation, and refine SLOs or capacity planning if patterns repeat.

This approach balances user experience, developer velocity, and measurable risk.

Requirements & constraints:

• Functional: synchronous checkout confirmation within <2s perceived by user; async fulfillment, shipping notifications.
• Scale: 10,000 orders/min peak (≈167 orders/sec sustained, bursty traffic).
• Non‑functional: high availability, idempotent processing, observable, bounded error budget.

High-level architecture:

• Frontend/API Gateway → Checkout Service (sync path) → Order Acceptance Queue (durable) → Order Processor workers (async fulfillment) → Downstream: Inventory, Payments, Shipping, Notification services. Use backing stores: Order DB (primary durable store), Event store/CDC for downstream integrations.

Sync checkout flow (user-visible):

1. Client calls Checkout API → Checkout Service validates, reserves inventory (optimistic/local cache or 2-phase async reservation) and charges payment via Payments service (sync or preauthorization).
2. Checkout Service writes an Order record with status=ACCEPTED to Order DB and publishes an "order.accepted" event to durable message broker (Kafka/RabbitMQ/SQS FIFO).
3. API returns confirmation (order id, status) within SLA.

Async fulfillment:

• Consumer group of Order Processors subscribes to "order.accepted", performs fulfillment steps: finalize charge/capture, decrement inventory (idempotent), create fulfillment tasks, call Shipping API, send notifications. Each step emits events and updates Order DB.

Idempotency & deduplication:

• Use client-supplied or server-generated idempotency key for checkout RPCs (persisted in Idempotency table with result and TTL).
• Every event contains order_id and sequence/version. Consumers use at-least-once delivery but implement dedupe by checking order status and processed message IDs in a idempotency log (per-order processed steps).
• For external retries (payment/shipping), use operation idempotency keys and implement exponential backoff with circuit breakers.

Failure handling & retries:

• Sync path: if persistent failure (DB unavailable) return transient error with client guidance; implement local cache for inventory reads to reduce DB load.
• Async: retries with backoff, dead-letter queues for items failing after N attempts; DLQ messages trigger alerts and human-in-the-loop remediation.
• Transactions: use saga pattern for distributed transactions with compensating actions (refunds, inventory re-reserve) on step failures.
• Bulk backpressure: API Gateway throttles and rate-limits; if queue backlog grows beyond thresholds, shed non-essential traffic and enable degraded mode (e.g., delayed notifications).

Scalability & capacity:

• Message broker partitioned to support throughput; scale Order Processor workers horizontally via autoscaling (k8s).
• Order DB: partitioning / sharding by order_id, read replicas for reporting.
• Cache (Redis) for hot-writes: idempotency keys, inventory reservations.

SLOs & observability placement:

• SLOs:
  • Checkout success rate: 99.9% of checkouts return ACCEPTED within 2s.
  • End-to-end fulfillment (order shipped): 99% shipped within X hours.
  • Queue processing latency: 99% of messages processed within Y seconds.
• KPIs/traces/metrics:
  • API latency, error rate, throttle rate at gateway.
  • Queue depth, consumer lag, processing rate, DLQ count.
  • Per-order state transitions, payment success/failure rates.
  • External dependency latencies (payments, shipping, inventory).
  • Idempotency hits/misses, dedupe conflicts.
• Alerts:
  • Queue depth above threshold, sustained consumer lag, rising DLQ, elevated payment failures, SLO burn rate alerts.
• Tracing & logs:
  • Distributed tracing (trace per order_id) across services for troubleshooting.
  • Structured logs with order_id, operation_id, idempotency_key.
  • Dashboards showing end-to-end latency SLOs, error budgets.

Observability-driven ops:

• Runbooks for DLQ remediation, payment provider outages, large inventory inconsistencies.
• Automated remediation: scale-up consumers, circuit-breaker resets, auto-retries for transient errors.
• Post-incident: blameless RCA, adjust SLOs/thresholds, capacity planning.

Trade-offs:

• Strict synchronous inventory lock increases latency and reduces throughput; choose optimistic reservation + background retries for most merchants.
• Kafka (high throughput) vs FIFO queue (ordering guarantees) — use partitioning keys (order_id) to maintain per-order ordering.

This design provides synchronous user confirmation while moving heavy downstream processing async, uses durable queues, idempotent operations, sagas for consistency, and places SLOs/observability where they map to user experience and system health.

Requirements & constraints:

• 200k RPS globally, 95th-percentile read latency <50ms
• Multi-region writes (eventual), reads must be fast and highly available
• SRE focus: reliability, monitoring, automated failover, capacity planning

High-level architecture:

• Global front layer: Anycast + geo-DNS to route clients to nearest edge POP
• Edge POPs: read-only caches (L1 Redis or in-memory) + regional read replicas
• Regional write ingestion services accept writes, persist to a local WAL and primary storage (distributed KV or document DB)
• Cross-region async replication via per-partition change streams (CDC) into a global replication bus (Kafka/Change Data Capture)
• Conflict resolution layer applies deterministic resolution before applying to read replicas

Core components:

1. Edge layer (L1): CDN/anycast + local cache for ultra-low latency reads. TTLs, conditional gets.
2. Regional read-replicas (L2): horizontally sharded storage (e.g., CockroachDB/ScyllaDB/Cassandra or cloud Spanner) tuned for reads.
3. Write service + WAL: durable, ack local write quickly.
4. Replication bus: partitioned Kafka with per-shard topics and high throughput across regions.
5. Apply workers: idempotent consumers that apply changes to replicas using vector clocks/CRDT merges.
6. Control plane: leader-election for partition masters (optional) and orchestration.

Cross-region replication:

• Leaderless, append-only change events with causal metadata (vector clocks or logical timestamps).
• Use per-shard monotonic sequence + origin-region tag; replicate via Kafka MirrorMaker2 or geo-replicated streaming.
• Apply events at destination idempotently; ensure at-least-once delivery and dedupe.

Conflict resolution:

• Prefer CRDTs for commutative ops (counters, sets). For general objects use hybrid approach:
  • Last-writer-wins with globally synchronized hybrid logical clocks (HLC) for most fields.
  • Field-level merges with application-specific merge hooks for complex types.
• Maintain version vectors for causal ordering where necessary; surface unresolved conflicts to higher-level reconciliation if automated rules inadequate.

Read routing & latency optimization:

• Serve reads from nearest edge cache; fallback to regional replica if miss.
• Hot-partition caching and adaptive TTLs; negative caching for misses.
• Read replicas colocated with edge POPs for very hot reads (read-through replication via apply workers).
• Client libraries implement stale-read tolerance (accept reads within X ms) or strong-read from local region if needed.

Failover & availability:

• Region failure: global control plane detects outage (healthchecks, BGP/anycast signals) and re-routes traffic to next-closest region automatically.
• Automated traffic shaping: gradually shift via weighted DNS/traffic manager with canary and circuit-breaker.
• For per-shard failures, elect regional leader from replicas via consensus (Raft) to serve reads/writes if strong writes required.
• Ensure WAL + multi-region durable storage to allow replay and recovery.

Operational considerations (SRE):

• SLOs: 99.95% availability, 95th read latency <50ms; error budget & alerting.
• Observability: distributed tracing (sampled), latency histograms per region/pop, replication lag metrics per shard, per-topic lag in Kafka, consumer group lag alerts.
• Autoscaling: read-replica pool autoscaled by RPS and CPU; backpressure to edge caches when overwhelmed.
• Chaos engineering: simulate region loss, network partitions, and replication lag to validate failover.
• Capacity planning: provision read cache hit target (e.g., 90% edge hit) so origin read QPS <20k; size replication bus for peak write throughput + buffer.

Trade-offs:

• Eventual consistency keeps writes fast but requires careful conflict logic.
• CRDTs increase complexity but reduce conflicts for many use-cases.
• Leaderless replication scales writes; leader-based eases conflict resolution for heavy-write keys.

This design achieves low-latency reads by serving from edge caches and regional replicas, provides durable multi-region writes via async replication, handles conflicts deterministically, and includes automated failover and observability so SREs can maintain SLOs.

Requirements & constraints:

• Provide observability (tracing, metrics, logs), automatic mTLS, and traffic control (routing, retries, circuit breaking) for containerized microservices on Kubernetes. Minimal app changes, incremental rollout, low latency/CPU overhead, multi-cluster support.

High-level architecture:

• Control plane (e.g., Istio/Consul Connect) manages policies, certs, config.
• Data plane: sidecar proxies (Envoy) injected per pod for mTLS, telemetry, and L7 controls.
• Telemetry backend: Prometheus + Grafana for metrics, Jaeger/Tempo for traces, ELK/Fluentd for logs.
• CA: integrated (Citadel) or external Vault for certs/rotation.
• Ingress/Egress gateways for north-south traffic.

Core components & responsibilities:

1. Sidecar proxy (Envoy): terminate TLS, collect metrics/traces, enforce policy.
2. Control plane: config distribution, RBAC, certificate issuance.
3. Observability pipeline: exporters, collectors, storage.

Trade-offs: Sidecar proxies vs library-based

• Sidecars (Envoy/Istio):
  • Pros: Transparent to apps, uniform policy, rich L7 features, consistent telemetry, centralized mTLS.
  • Cons: CPU/memory overhead per pod, complexity of control plane, debugging network-level issues, upgrade/ops burden.
• Library-based (e.g., in-app mTLS/metrics libs):
  • Pros: Lower per-pod overhead, simpler runtime, fewer network hops.
  • Cons: Requires code changes per service, inconsistent behavior, harder to enforce org-wide policies, varied instrumentation quality.

Performance considerations:

• Measure p99 latency added by Envoy; use sidecar resource limits, enable in-proxy pooling/keepalives, and offload heavy RBAC or policy checks to control plane caching.
• Use passthrough modes for latency-sensitive paths.

Incremental migration steps:

1. Pilot: pick non-critical namespace, enable automatic sidecar injection, deploy control plane in passive mode (metrics/tracing only).
2. Observability first: enable telemetry to validate dashboards and SLOs.
3. mTLS opt-in: enable mutual TLS per namespace with permissive mode to monitor failures.
4. Traffic control: add routing, retries, and canary tests for a few services.
5. Enforce mTLS and policies gradually; add ingress/egress gateways and implement global rate limits.
6. Expand rollout, automate injection via admission controllers, add CI checks for mesh compatibility.
7. Runbooks & rollback: prepare Canary rollback, monitor resource usage, and maintain capability to disable injection per deployment.

Why this approach:

• Sidecars give immediate, organization-wide security and observability without code changes—critical for SREs managing reliability. Incremental rollout minimizes blast radius and lets you tune performance and ops practices before full adoption.

Decision criteria between synchronous and asynchronous communication boils down to user-visible latency, reliability/SLO goals, and operational complexity.

When to choose synchronous:

• Use when callers need an immediate, deterministic response (e.g., payment authorization, auth token).
• SLO/latency: the end-to-end P99 must meet user-facing SLAs; include downstream latency in budget.
• Error handling: simpler in-flow retries and backpressure, but you must bound retries to avoid cascading failures (circuit breakers, timeouts, bulkheads).
• Example: HTTP REST call with 200–500ms P99 requirement → sync if downstream can meet that.

When to choose asynchronous:

• Use when work can be deferred (batch jobs, event processing, notifications) or to decouple availability between services.
• SLO/latency: user-facing latency improves because request completes quickly; define separate SLOs for ingestion (ack within tens of ms) and processing (eventual consistency SLA, e.g., process within 30s).
• Error handling complexity: higher — you need durable queues, idempotency, visibility into stuck messages, DLQs, and replay mechanisms.
• Retries and fallbacks:
  • Use exponential backoff with jitter for consumer retries; cap max attempts.
  • Persist failed messages to Dead Letter Queue (DLQ) after attempts; attach metadata for diagnosis.
  • Design idempotent consumers or store dedupe IDs to allow safe replays.
  • Provide user-facing fallback: synchronous ack + "processing pending" UI or a compensating action (e.g., optimistic update + eventual reconciliation).
  • Monitor processing lag and DLQ rate as SLO indicators; alert on growing lag or DLQ spikes.
• Example pattern: enqueue order, respond 202 with order-id, process asynchronously with retries up to N, move to DLQ, notify user if final failure.

Trade-offs summary:

• Sync: simpler semantics, tighter latency guarantees, higher coupling and risk of cascading failures.
• Async: better isolation and scalability, more complex operational tooling and eventual-consistency thinking; requires careful SLO decomposition, observability (ingest latency, processing latency, DLQ rate), and robust retry/fallback design.