Observability and Monitoring for Distributed Systems Questions

Situation: At my last company I owned observability for a production NLP inference service (bert-based intent classifier + reranker) serving 50k req/day.

What I implemented (concise list):

• Key metrics:
  • Latency percentiles: p50/p90/p99 for end-to-end inference and model-only (in ms)
  • Throughput: requests/sec, concurrent requests
  • Error rates: 4xx/5xx HTTP, model failures, timeouts
  • Model-quality: per-class precision/recall, prediction distribution, confidence histogram, population/stability drift (KL divergence) vs baseline, label-feedback mismatch rate
  • Resource metrics: GPU/CPU utilization, memory, queue length
• Logs and structured events:
  • JSON logs per request with request_id, user_id (hashed), model_version, input_size, latency_ms, top_k_preds, confidence, sampling_flag
  • Periodic batch events: daily labeled-sample ingestion, backfilled eval results
• Tracing & context propagation:
  • OpenTelemetry spans across API gateway → preprocessor → model server → postprocessor; propagate request_id, trace_id, model_version, sample_rate
  • Span annotations for cache hits, batched inference, and fallback model
• Dashboards & SLO-driven alerts:
  • Grafana dashboards: latency heatmaps, drift over time, precision-by-class
  • SLOs: 99th-percentile latency < 500ms (95% target), error rate < 0.5%, model F1 ≥ baseline ±1%
  • Alerts: PagerDuty for SLO breaches, Slack for warning thresholds, automated runbook links
• Automation:
  • Auto-rollback of model deployment when post-deploy F1 drops >2% or drift exceeds threshold for 1hr; can auto-scale GPU pool on p90 latency spikes

Real incident detected earlier:

• After a backend tokenizer library update, p99 latency rose and confidence histogram shifted low for a specific intent. Traces showed a 10x latency in preprocessing span and increased CPU on preprocessor pod. The drift metric crossed threshold within 15 minutes and triggered a P1 alert. Because structured logs included model_version and tokenizer version, we quickly pinpointed the change and rolled back the tokenizer rollout. Mean time to detection was ~12 minutes vs previous ~3 hours, preventing large-scale misclassification and reducing user complaints by ~90%.

What I learned:

• Instrument the entire ML pipeline (not just model) and include model-quality signals; correlate traces + drift metrics to triage root cause quickly.

Situation: I’d instrument services using OpenTelemetry (OTel) and OTLP export to a tracing backend (Jaeger/Tempo/Lightstep). Use language SDKs for Python, Java, Go so instrumentation is consistent.

Context propagation:

• Use W3C Trace-Context (traceparent/tracestate) as primary propagation for cross-language compatibility; fall back to B3 if any older components require it.
• Initialize automatic instrumentation (HTTP frameworks, gRPC, message queues) so the incoming trace context is extracted and attached to the current span; ensure outgoing HTTP/gRPC clients inject trace headers.

Span and attribute design (model metadata):

• Create spans for API request → orchestration → model inference. Attach attributes on the inference span:
  • model.id: "resnet50-v2"
  • model.version: "2025-03-12-finetune"
  • model.framework: "torch"
  • input.size: 1024 (bytes) or input.shape: [1,3,224,224]
  • batch.size: 8
  • device: "gpu:2"
  • model.stage: "preproc/infer/postproc"
• Record events for GPU queueing, model load, cache hits, and resource waits.

Sampling strategies:

• Default: probabilistic (e.g., 1-5%) to control volume.
• Adaptive tail-based sampling: keep full traces with high latency/error rates; important for catching expensive model calls. Configure rules: sample 100% when status >= 500 or latency > SLO threshold (e.g., >300ms).
• Head-based sampling for cost control; enrich sampled set using dynamic decisions (e.g., A/B groups, model versions under experiment).

How traces diagnose issues:

• High-latency model calls: follow trace tree to find whether latency is in network, preprocessing, GPU queue, model execution, or post-processing. Inference span split into child spans (queue_wait, model_load, forward_pass, memcpy) to localize.
• Resource contention: traces with device attribute show multiple long queue_wait durations and correlated high GPU utilization; correlation with host metrics (prometheus) and span tags (batch.size) reveals whether large batches cause queuing.
• Version regressions: filter traces by model.version to compare p50/p95 latency and error rates; identify which version introduced longer forward_pass times.
• Root cause: combine traces with flamegraphs and logs (link trace_id in logs) to reconstruct sequence and correlate with infra metrics.

Operational notes:

• Ensure low-overhead instrumentation: sampling, offload heavy attributes (full inputs) to object storage and include reference URL in span.
• Secure PII: avoid sending raw inputs; instead send hashed/summarized features where necessary.
• Test propagation end-to-end (client → API → inference workers → async storage) and validate trace continuity across language boundaries.

Metrics:

• Latency (p50/p90/p99 end-to-end and model inference), throughput (requests/sec), error rates (4xx/5xx), GPU/CPU/memory utilization, queue length, input/output feature distributions (histograms), model confidence/entropy, prediction drift metrics.

Traces:

• Distributed traces for request lifecycle (client → API gateway → inference), span timings for preprocessing, model load, inference, postprocessing to pinpoint bottlenecks.

Logs:

• Structured request/response logs (sampled), model version, input hashes, warnings, exceptions, resource OOMs; retain small samples for audit and root cause.

Synthetic tests:

• Canary requests, latency/accuracy probes, regression test suite with labeled inputs, data drift detectors run periodically.

SLOs & alerts:

• Example SLOs: 99% requests <200ms p95; error rate <0.5%; model accuracy ≥ baseline. Alert on SLO burn rate (fast), sustained p95/p99 breaches, sudden drift in feature distributions, rising low-confidence or bias metrics, infrastructure saturation.

How signals help:

• Metrics + traces detect perf regressions and bottlenecks; logs provide context for failures; synthetic tests and drift metrics detect model quality regressions before customer impact; SLO + burn-rate alerts prioritize incidents.

Observability plan — Machine-Learning Microservice

Overview: monitor three levels — infrastructure, application, and model — with clear dashboards, alerting tied to decision thresholds, retention policies, and runbooks for automated or manual remediation (partial rollback, emergency scaling).

Essential metrics

• Infrastructure
  • CPU utilization (per node / pod): warn 70%, critical 90%
  • GPU utilization / memory: warn 75%, critical 95%
  • Node memory, disk I/O, network throughput, pod restarts
  • Cluster-level pod replica counts and pending pods
• Application
  • Request rate (RPS), concurrent requests
  • Latency percentiles: p50, p95, p99 (SLO p95 < 300ms)
  • Error rate (4xx/5xx) and error types; retry counts
  • Throughput, queue length/backpressure metrics
  • Trace spans: request path durations, DB / model call durations
• Model
  • Prediction distribution and key feature distributions (per-feature histograms)
  • Model confidence / entropy, calibration metrics (ECE)
  • Label arrival rate (ground-truth), label latency
  • Data drift metrics (KL-divergence, population stability index) per feature
  • Concept drift detectors, prediction skew vs training baseline
  • Business KPIs: conversion rate, CTR, revenue per prediction

Dashboards (examples)

• Service Health: RPS, p95/p99 latency, error rate, instance count
• Infra Overview: CPU/GPU heatmap, memory, pending pods
• Model Quality: feature drift charts, calibration curve, confidence histogram, recent labeled vs predicted confusion matrix
• Traces: slow request waterfall, top N slow endpoints and span breakdown
• Business Impact: model-driven KPI trends vs baseline

Alerting rules & thresholds (with actions)

• Critical infra: GPU >95% for 3m or pod restarts >3/hr → Pager + auto-scale up policy (add nodes)
• Latency: p95 > 2x SLO for 5m OR p99 > 1s → page ops; trigger canary rollback if correlated with recent deployment
• Error rate: 5xx rate >1% for 5m or spike >5x baseline → page + circuit-break on ingress
• Model drift: feature KL > threshold (e.g., 0.2) for 24h OR ECE >0.1 → alert ML owner; consider triggering canary shadowing or partial rollback to previous model
• Calibration drop: sudden drop in accuracy or business KPI degradation >5% over 1 day → alert PM/ML lead

Log & trace retention

• Logs: service-level structured logs retained 30 days hot, 90 days cold; error-level logs retained 365 days
• Traces: full traces retained 7–14 days; sampling 10% for production long-term (retain 100% for errored traces for 30d)
• Metrics: high-resolution (1s-15s) retained 14 days; aggregated (1m) retained 90 days; long-term aggregated (1h) retained 2 years for trend analysis

How signals support decisions

• Emergency scaling: sustained high GPU/CPU utilization + increased queue length + rising latency → trigger automated horizontal/vertical scaling. If utilization is high but error rate low and model confidence stable, prefer adding capacity. If model confidence falls, scaling may hide model quality issues.
• Partial rollback / canary: when increased error rate, latency, or degradation in business KPIs correlates temporally with a new model/deployment, use canary analysis: roll back new model for a percentage of traffic (partial rollback) while keeping rest running. Use traces and logs to pinpoint failing code path; use prediction distribution and calibration to decide if issue is model-related.
• Uncertain conditions: combine signals — e.g., if latency increases but infra healthy, and model confidence drops while feature drift rises, prefer partial rollback of model; if infra unhealthy but model metrics stable, prefer scaling. Runbooks should specify exact thresholds and who approves partial rollback vs auto action.

Additional practices

• Implement canary deployments with automated metric comparison (SLOs + model quality)
• Use distributed tracing correlation IDs linking requests → model inputs → predictions → downstream outcomes
• Periodic data quality checks and synthetic traffic for smoke tests
• Blameless postmortems and continuous refinement of thresholds based on historical incidents

This plan ties observability to concrete thresholds and remediation paths so teams can act quickly and confidently under uncertainty.