Observability and Monitoring for Distributed Systems Questions

Situation & goals (brief)
I’d design an observability stack that preserves debuggability while cutting storage/query cost ~5–10x for a global microservice fleet producing PBs/month.

High-level design

• Central multi-tenant pipeline: local agents → regional Kafka → processing layer (Vector/Fluentd) → cold/hot tiers in object store + indexer.
• Use metadata catalog to map service/host/slo tags for selective retention.

Sampling & ingest controls

• Ingest-level adaptive sampling per-service: preserve 100% error/latency/failed requests; reservoir/sample ratios for normal traffic (e.g., 1:1000) tuned by traffic type.
• Rate-limits with fingerprinting to keep at-least-one-per-user/session.

Metric strategy

• Separate high-cardinality labels into sparse-indexed dimensions; drop low-value labels at ingest.
• Rollups: pre-aggregate to 1s/10s at hot tier, 1m/5m for medium, 1h for long-term.
• Use bloom filters + sketches (t-digest, HDR histograms) for distribution metrics to keep percentile accuracy.

Trace sampling

• Head-based sampling: keep all traces for errors/alerts; probabilistic for success traces (e.g., 0.1% baseline).
• Priority sampling: boost traces for unusual spans or anomalous latency signatures; keep sampled span trees for context.
• Link traces to sampled logs/metrics via stable trace IDs and sample-retention bookmarks.

Retention tiers & storage

• Hot (7–30d): full logs, full-resolution metrics, traces for retained samples — SSD/TSDB.
• Warm (30–90d): compressed logs (columnar/parquet), aggregated metrics (1m), sampled traces — object store with index shards.
• Cold (90d+): compressed rollups, sketches, and sparse indexes for on-demand retrieval; cost-based restore.

Maintaining debuggability

• Always keep: full traces for failures, top-N slow endpoints, and a per-user/session breadcrumb sample.
• On-demand fetch: keep small index to efficiently restore raw logs/traces from cold storage for specific trace IDs.
• Implement “forensic mode” to temporarily increase sampling for a service if an incident is declared.

Trade-offs & governance

• Accepts lower fidelity for normal traffic; requires robust alerting to detect undersampling issues.
• Provide SLO-driven default policies; allow teams overrides with quotas.
• KPIs: storage $/month, query latency, incident MTTI/MTTR to validate.

This balances cost and signal retention while giving teams control and fast incident workflows.

Definition & high-level goal
For a public search API we measure relevance + speed. SLIs quantify those dimensions, SLOs set targets over a 30‑day window, and the error budget = 1 - SLO allowable failures to drive trade-offs.

Proposed SLIs (measurable)

• Latency: p50, p95, p99 of end-to-end query response time (client-observed).
• Availability: % of successful HTTP 200 responses for valid queries (excluding client errors) over 30 days.
• Correctness (relevance): % of queries with relevance score ≥ threshold (A/B or human-labeled gold set) — or normalized click-through/CTR quality metric for production.

30-day SLO targets (example)

• Latency: p50 < 50 ms, p95 < 300 ms, p99 < 800 ms — 99% of queries must meet these.
• Availability: 99.95% successful responses.
• Correctness: 98% of sampled queries meet relevance threshold.

Error budget (30 days) = 1 - SLO (use the tightest SLO per product priority). Example: if availability SLO is 99.95%, budget = 0.05% of queries.

Alerting strategy when budget is depleted

• Tiered alerts:
  • Soft (warning): consumption > 50% — notify on-call and product manager.
  • Hard (critical): consumption > 90% or remaining budget < 10% — page SRE/engineering lead.
  • Emergency (budget exhausted): immediate incident channel, halt non-essential releases.
• Alerts include root metrics, recent deploys, traffic spikes, error types, and correlation with relevance metrics.

Prioritization & runbook when budgets exhausted

• Immediately: freeze risky launches and non-urgent feature work. Shift sprint capacity to reliability (hotfixes, rollbacks, capacity and cache tuning).
• Short-term: mitigate (traffic shaping, rate limits, circuit breakers, autoscale, degraded-but-safe mode for heavy queries).
• Mid-term: dedicate a reliability sprint (indexing improvements, query optimization, caching, instrumentation).
• Communication: notify stakeholders, adjust roadmap commitments, and document lessons with postmortem and updated SLOs if needed.

Rationale
This approach balances user experience (latency + relevance) with operational realism: precise SLIs, measurable SLOs, clear alerts and a simple, actionable error-budget-driven prioritization policy that engineering teams can operationalize.

Overview

As an Engineering Manager launching a new microservice, I’d ensure instrumentation across metrics, logs, traces, and alerts so the team can measure health, detect regressions, and respond quickly.

Essential Metrics

• Request rate (RPS) — reasons: capacity planning, traffic patterns.
• Error rate (4xx/5xx by endpoint) — reasons: user impact, bug detection.
• Latency percentiles (p50/p95/p99) per endpoint — reasons: UX & SLOs.
• Resource metrics (CPU, memory, thread pool/queue length) — reasons: detect saturation.
• DB/cache latency & error rates — reasons: downstream bottlenecks.

Essential Logs

• Structured request/response logs with request_id, user_id (if applicable), endpoint, status, latency, and key params — reasons: debugging, postmortem.
• Access logs + audit logs for security and compliance.
• Application error logs with stack traces and context.

Traces

• Distributed tracing with sampled spans across service boundaries (incoming request → downstream DB/HTTP calls) capturing span duration, annotations, and request_id — reasons: pinpoint latency sources and root cause.

Alerts

• High error rate alert: triggered when >1% 5xx over 5m (or +x sigma vs baseline).
• Latency alert: p95 > 300ms for 5m (example threshold) — triggers paging.
• Resource saturation: CPU > 85% for 10m or queue length increasing — auto-escalate.
• Downstream failures: DB error rate > 1% or cache unavailable — notify.
• Deployment/regression guard: spike in errors or latency within 15m post-deploy — immediate rollback runbook.

SLI / SLO (Public API endpoint)

• Availability SLI: successful responses (2xx/3xx) / total requests. SLO: 99.9% monthly.
• Latency SLI: fraction of requests below threshold. SLO: p95 < 300ms for 99% of requests per month.

Why these values: 99.9% availability balances user expectations and engineering cost for public APIs; p95 latency target reflects perceptible UI responsiveness while allowing tail variability. Adjust thresholds using real traffic and business impact.