Observability and Monitoring for Distributed Systems Questions

Overview — goals
Minimize toil but enable fast RCA: capture latency/availability/error-rate, structured logs, traces for request flow, a few dashboards and noisy-resistant alerts.

Key metrics (SLI candidates)

• Request success rate (2xx / total)
• P99 / P95 latency for user API endpoints
• Saturation: CPU, memory, thread-pool/queue depth
• External dependency error rate & latency
• Deployment/healthcheck rate

Logging

• Structured JSON logs (example):

{
  "timestamp":"2026-03-01T12:00:00Z",
  "service":"orders",
  "env":"prod",
  "level":"error",
  "request_id":"abc123",
  "user_id":"u789",
  "msg":"payment failed",
  "error":"timeout",
  "duration_ms":1200
}

• Retention: hot logs 14d, cold archive 365d; index only necessary fields (service, level, request_id, user_id, timestamp).

Distributed tracing

• Instrument incoming HTTP entry spans + downstream RPC/db spans, propagate trace/request_id headers.
• Sample at 1% global; keep 100% of errors and traces above latency threshold.
• Use adaptive sampling (increase when error or high latency).

Dashboards (essential)

• Overview: success rate, latency p95/p99, error rate, system saturation
• Dependency map: external call latencies & errors
• Recent deployments and healthchecks
• Top slow endpoints and traces

Alerting & thresholds

• Alert types: P0 (page), P1 (notify)
• Examples:
  • Page: request success rate < 99% over 5m AND >1% user-impacting errors
  • Page: p99 latency > 2s for 5m
  • Notify: CPU > 85% for 15m
• Avoid noise: require sustained windows (e.g., 5–15m), combine signals (latency + error), use burn-rate alerts for SLOs.
• Set thresholds from baseline: set at 2–3 sigma from median, tune after 2 weeks.

Example SLOs for user-facing API

1. Availability: 99.95% successful requests per rolling 30d
2. Latency: 99% requests < 800 ms per rolling 7d
3. Error budget for external-dependency failures: external dependency success >= 99.5% per 30d (budget tied to incident playbooks)

Implementation notes: instrument early, automate dashboards/alerts as code, review thresholds quarterly and after major changes.

Context / goals
I would treat the authentication API as critical user-path: availability, correctness, and latency directly impact sign-ins, payments, and security flows. My SLIs/SLOs focus on user-observable outcomes, measurable from production ingress and client response.

Three SLIs (measurement method)

1. Successful authentication rate — % of auth requests that return HTTP 2xx and valid token (measured by sampling real traffic + synthetic probes; correlate with downstream token validation logs).
2. End-to-end auth latency (p95) — time from request arrival at edge to token issuance (measured by distributed tracing and edge latency histograms).
3. Error-rate for user-facing errors — % requests returning >=500 or security-blocking 4xx (measured at API gateway and enriched with user-impact tags).

SLO targets & alert thresholds

• Successful auth rate: SLO 99.95% over 30 days. Pages at 99.9% (P1), warns at 99.97% (P2).
• Latency p95: SLO 300 ms. Pages if p95 > 500 ms for 5m (P1); warn if p95 > 350 ms for 10m (P2).
• User-facing error-rate: SLO < 0.1% errors. Pages if > 0.5% for 5m (P1); warn if > 0.2% for 10m (P2).

On-call runbook actions

• P2 (warning): Check synthetic probe dashboards, recent deployments, config flags; review downstream dependency metrics (DB replica lag, IdP latency); roll forward telemetry tracing spans for sample requests; if likely transient, document and monitor 30–60m.
• P1 (page): Immediately acknowledge, run "isolation checklist": rollback recent deploys, toggle traffic to standby cluster, throttle non-essential flows, collect traces and request dumps, engage DB/identity vendor if dependency implicated; open incident with timeline, notify Product/Security if tokens possibly compromised.
• Post-incident: RCA, SLI/SLO adjustment if needed, playbook updates, and corrective automation.

Reducing false positives

• Use user-impacting composite SLIs (combine success + latency) rather than raw infra metrics.
• Require corroboration: alert only when both edge synthetic probes and a percentage of real-user samples show failure.
• Apply dynamic baselining (anomaly detection tuned to hourly patterns) and maintenance windows/deploy-exemptions.
• Implement alert deduplication and a short confirmation window (example: alert only if condition holds for 5m and affects >X users) to avoid noise while keeping real incidents visible.