Situation: After a deploy, tail latency (99th/99.9th) increased. Below is a practical, prioritized debugging playbook you can run as a data engineer to quickly triage whether CPU, GC, network, or downstream contention is responsible.
- Confirm & scope
- Verify signal: check p99/p999, percent of requests impacted, time window, and deploy metadata (git sha, config changes).
- Scope by host / service / endpoint using APM and logs.
- Quick-sensor telemetry (fast wins)
- CPU: host CPU% and per-process CPU (top, htop, CloudWatch/GCP metrics). Look for CPU saturation or steal.
- GC/JVM: JVM GC pause time, young/old gen sizes, pause percent (jvm_gc_pause_ms, GC count). For Spark, check executor GC metrics.
- Network: NIC throughput, error counts, retransmits, latency (if available from eBPF or cloud VPC flow logs).
- Downstream: DB/warehouse query latency, connection pool saturation, queue consumer lag (Kafka offsets), and error rates.
- Correlate with tracing
- Pull distributed traces for slow requests (tempo/Jaeger/Zipkin). Identify whether time is spent in our app CPU, GC, waiting on network I/O, or blocked on downstream RPCs.
- Lightweight experiments
- Canary rollback / feature-flag the deploy to confirm causality.
- Traffic-split: route a fraction to previous revision to compare latencies.
- Replay: run a representative replay against staging (same payloads at lower scale) to reproduce.
- Deep diagnostics (if suspected)
- CPU: capture pprof/strace/perf or async-profiler flamegraphs to find hotspots; check for spinning loops or lock contention.
- GC: trigger and capture GC logs (-Xlog:gc*), sample heap (jmap/jcmd), analyze with GCViewer; consider increasing heap or tuning GC.
- Network: run tcpdump/ss, measure RTT to downstream, check kernel networking queues; test with iperf between services.
- Downstream contention: review DB slow query logs, check connection pool metrics, scale consumers, and measure queue lag.
- Mitigations & monitoring
- Short-term: scale replicas, increase connection pools, circuit-break slow downstream, set timeouts/retries/backoffs.
- Long-term: fix hotspot code, tune JVM/GC, add caching, optimize DB queries, add bulkheads.
- Postmortem data to collect
- Deploy diff, traces, host metrics, GC logs, flamegraphs, DB metrics, and experiment outcomes.
This approach moves from quick detection to targeted experiments and deeper profiling, allowing you to identify whether CPU, GC, network, or downstream contention is the root cause and apply appropriate short- and long-term fixes.
