Observability and Monitoring for Distributed Systems Questions

Clarify scope & goal

• Confirm affected region, time windows, percent of requests (5%), and exact user paths/endpoints. Goal: determine whether cause is network routing, CDN, load balancer (LB), or backend.

1) Collect baseline metrics

• Check SLOs, median/p95/p99 latencies from monitoring (Prometheus/Grafana, CloudWatch).
• Correlate by region, POP, endpoint, and backend instance IDs.

2) Traces & distributed tracing

• Pull traces for slow requests (Jaeger/NewRelic/Datadog). Look for where time is spent: client→edge, edge→origin, LB→backend, DB.
• If traces show long edge→client time → suspect network/CDN. If long backend spans → app.

3) Network-level tests (from affected region)

• Run traceroute/MTR to edge and origin to spot routing loops or packet loss:

mtr -c 100 -r -w <edge-ip-or-hostname>
traceroute -I <edge-ip-or-hostname>

• Test TCP connect and handshake latency:

tcptraceroute <host> 443
curl -w "@curl-format.txt" -o /dev/null -s "https://host/path"
# curl-format.txt example: time_namelookup: %{time_namelookup}\n...

• Use pings and packet capture:

ping -c 50 <edge-ip>
sudo tcpdump -i eth0 host <edge-ip> and port 443 -w capture.pcap

• Interpretation: high initial RTT or packet loss => routing/ISP.

4) CDN checks

• Validate CDN edge behavior: request with and without CDN bypass, inspect response headers (cache-status, Via):

curl -I -H "Host: example.com" https://cdn.example.com/path
curl -I --resolve example.com:443:<origin-ip> https://example.com/path

• Compare headers and times; use CDN logs/edge metrics for spikes. If origin is fine but edge shows delay → CDN issue or POP saturation.

5) Load balancer checks

• Check LB metrics: active connections, request queue length, target health, per-AZ distribution.
• Query LB logs (ELB/ALB/Nginx): look for 5xx, target response times.
• Reproduce with direct-to-backend bypassing LB (if safe) using --resolve to map DNS to a backend IP and compare latencies.

6) Backend application checks

• Correlate slow traces to backend spans (DB calls, GC, thread starvation).
• Check server metrics during incidents: cpu, iowait, load, disk, network:

top -b -n1
iostat -xz 1 3
sar -n DEV 1 3
ss -tna | wc -l

• Review application logs for timeouts, slow queries. Capture flamegraphs or profiling if necessary.

7) Narrow with synthetic tests and canary probes

• From multiple vantage points (affected region and control regions) run synthetic curl loops, mtr, and APM-injected traces to compare:

for i in {1..100}; do curl -s -w '%{time_total}\n' -o /dev/null https://example.com/path; done

• Run tests at varying times to align with observed spikes.

8) Packet-level for root cause

• If network suspected, compare captures from client-side and edge to find where latency adds (SYN->SYN/ACK delay, retransmits).
• Use Wireshark/tshark to inspect:

tshark -r capture.pcap -q -z io,phs

9) Hypotheses & quick fixes

• If CDN POP saturated → route traffic to healthier POPs or increase capacity; enable cache TTL tuning.
• If LB queueing → scale targets, adjust deregistration/draining, or change LB timeouts.
• If backend → scale, fix DB queries, increase connection pool, or patch slow code.

10) Document and escalate

• Produce timeline, correlated graphs (latency vs packet loss vs errors), and trace examples. If issue is ISP or CDN provider, open ticket with packet captures, traceroutes, and timestamps.

This plan ties metrics, logs, and traces to concrete tests/commands so you can isolate whether delays occur on the network path, CDN edge, LB layer, or backend service.

Problem clarification & goals
Detect slow endpoints, find root cause (service, DB, network, infra), minimize noise and cost, enable fast triage for on-call.

Instrumentation plan (spans & tags)

• Capture server-side root span per incoming request (HTTP/gRPC) with:
  • Tags: http.method, http.route, http.status_code, http.host, client.ip (dot-obfuscated), user.id (if low-card), tenant.id (if SSO/multi-tenant)
  • Custom tags: deploy.version, instance.id, AZ/region, container_id
• Child spans: auth, business logic, cache access, DB client, external HTTP, message queue produce/consume, filesystem I/O
• Each DB/external span: db.system, db.name, db.statement/hashed, db.rows_returned, peer.address, network.latency_ms
• Add error and retry tags: error=true, exception.type, retry.count

Metrics to emit

• Per-endpoint histograms (p95/p99 latency), counters (requests, errors), active requests gauge, DB latency histograms, CPU/mem/disk IO per instance
• Use Prometheus + exporters and link to tracing via same labels (service, instance, route)

Sampling strategy

• Default: probability sampling 5–10% for production to control volume
• Keep 100% sampling for:
  • Any span with error=true
  • Latency-based tail sampling: sample traces whose root span duration > p95
  • High-value tenants or endpoints via dynamic rules
• Use adaptive tail-sampling (OpenTelemetry Collector) with local buffering

Correlation (logs, metrics, traces)

• Inject trace_id and span_id into logs (structured JSON) and metrics labels where feasible (trace_id for error logs)
• Ensure timestamps use a common NTP-synced clock and ISO8601
• Store logs in ELK/Opensearch with trace_id and service tags; use Jaeger/Tempo linked-view
• In dashboards (Grafana) link from metric (p99 spike) to traces and related logs by trace_id and time window

Operational notes

• Limit high-card tags (user.id, request_id) on metrics; keep them on traces/logs only
• Retention: keep full traces 7–30 days depending on compliance; keep aggregated metrics longer
• Test by injecting latency and verifying end-to-end visibility (sample traces, logs, and metrics correlate)

This plan uses OpenTelemetry for instrumentation, Jaeger/Tempo for traces, Prometheus for metrics, and ELK for logs—balanced sampling and tail-sampling ensure fidelity where it matters while controlling cost.