Approach: Treat ADRs as the single source of truth for architectural decisions and their operational impact. For each decision include a dedicated "Observability" section that maps decision goals to SLOs, the exact metrics to collect, dashboards to build (with key visualizations), alert thresholds tied to business impact, and runbook links. This makes operational requirements explicit during design, sizing, and purchase decisions.Example ADR snippet (decision: introduce async order processing):yaml
Title: Introduce async order processing via OrderQueue service
Status: Accepted
Date: 2025-11-22
Decision: Replace sync order submit flow with async processing using a durable FIFO queue and worker pool to improve throughput/latency isolation.
Observability:
Objectives:
- Preserve user-perceived order submit availability >= 99.9 (SLO)
- Ensure end-to-end order processing within 60s for 95% of orders (SLO)
SLOs:
- Submit availability: successful enqueue / enqueue attempts >= 99.9% per 30d
- Processing latency: 95th percentile end-to-end processing <= 60s (1d/7d windows)
- Error rate: processed orders failing >= 0.5% over 1h rolling window triggers mitigation
Metrics (implement as labeled, high-cardinality-aware):
- queue.depth: current messages in OrderQueue (by topic, priority)
- queue.enqueue.rate: enqueues/sec
- queue.dequeue.rate: dequeues/sec
- processing.latency.ms: time from enqueue -> processed (histogram)
- processing.error.count: count of failed processing attempts, with error_type label
- worker.utilization: percent busy per worker instance
- dead_letter.count: messages moved to DLQ
Dashboards:
- "Order Queue Health" panel: time-series queue.depth, enqueue/dequeue rates, dead_letter.count
- "Processing Latency" panel: p50/p95/p99 of processing.latency.ms + histogram heatmap
- "Worker Pool" panel: worker.count, worker.utilization, dequeue.rate per worker
- "Errors & DLQ" panel: processing.error.count by error_type, recent DLQ samples (links)
Alerts (actionable, with severity & runbook):
- P0: queue.depth > 10k for 5m AND enqueue.rate - dequeue.rate > 0 => Alert: "Backlog growing" -> page on-call, runbook: scale workers, investigate poison messages
- P1: processing.p95 > 60s for 10m => Alert: "High processing latency" -> notify SRE, check downstream dependencies
- P1: processing.error.count rate > 0.5% over 1h => Alert: "Increased processing errors" -> trigger rollback or routing to degraded path
- P2: dead_letter.count increases by >100 in 15m => Alert: "DLQ spike" -> investigate message content, apply fixes
Instrumentation notes:
- Emit metrics with consistent labels: environment, region, sku, priority
- Use histograms for latency with fixed buckets (0-1s,1-5s,5-30s,30-120s,>120s)
- Trace sample when processing.latency > p95 and when errors occur; link traces in dashboards
Runbooks:
- Link: /runbooks/orderqueue/backlog-incident
- Link: /runbooks/orderqueue/error-investigation
Reviews:
- Operational acceptance required: SRE sign-off on metrics/dashboards before rollout
Why this works:- Ties architectural intent (throughput, UX) to measurable SLOs so decisions are validated in production.- Specifies concrete metrics, dashboard panels and alert thresholds making monitoring implementable and testable.- Ensures operational readiness via runbooks and SRE sign-off, reducing rollout friction and aligning stakeholders.