**Overview (role: Backend Developer)** I’d deliver an end-to-end observability plan covering SLIs/SLOs, metrics, tracing, logs, alerts, and stakeholder reporting so teams can detect, diagnose, and prioritize reliability work.**SLIs & SLOs**- Availability SLI: % of successful production requests (2xx/3xx) per service. SLO: 99.9% monthly.- Latency SLI: P95 end-to-end request latency. SLO: P95 < 300 ms.- Error-rate SLI: % of requests returning 5xx. SLO: < 0.1% daily.- Throughput SLI: requests/sec (capacity planning). SLO: sustain expected traffic ±10%.**Key metrics to collect**- Latency: P50, P95, P99, histogram buckets for request duration.- Error-rate: 4xx vs 5xx counts, exception classes.- Throughput: rps, requests per endpoint.- Saturation: CPU%, memory%, DB connection pool usage, queue lengths.**Tracing & Spans**Instrument:- Ingress API gateway span- Auth validation span- Handler/controller span per endpoint- DB query span (including SQL + rows returned)- External HTTP/gRPC call spans- Cache (Redis) get/set spansInclude tags: user_id, request_id, tenant_id, route, status_code, db.table.**Structured logging & correlation**Use JSON logs with consistent fields:json
{
"timestamp":"2026-01-01T12:00:00Z",
"level":"ERROR",
"service":"orders",
"env":"prod",
"request_id":"req-12345",
"user_id":"u-678",
"trace_id":"abcd-ef01",
"span_id":"span-02",
"msg":"failed to write order",
"error":"PG: duplicate key",
"latency_ms":420
}
Propagate request_id and trace_id across services; prefer W3C traceparent.**Alerting thresholds**- Page (urgent): Availability SLO burn rate > 4x for 1h OR errors >1% global 5xx for 5m.- Pager: P95 latency breach > 2x SLO sustained 10m.- Ops/Slack: DB connection pool > 80% for 10m, queue backlog > threshold.Use burn-rate alerts tied to SLO windows to avoid noisy symptom-only alerts.**Prioritization & workflow**- Triage by SLI impact: failures affecting SLOs (availability/latency) get high priority.- Use error budget: If error budget exhausted, freeze risky releases and prioritize fixes.- Root-cause via traces + logs: correlate trace_id to pull spans and logs; use histogram to find regressions.- Postmortem: quantify SLO impact (minutes of downtime, % error budget used), RCA, action items.**Communicating reliability**- Weekly dashboard for engineering: SLO status, burn rate, top offenders (endpoints/services).- Monthly summary for stakeholders: uptime %, incidents, business impact, roadmap items.- Present mitigation plan when SLOs are near breach and use error-budget-driven decisions for releases.