First, clarify scope and impact: which services, when did latency begin, is it client-to-service or service-to-DB? Confirm migration timeline and any simultaneous releases.
Investigation steps (methodical):
- Reproduce & baseline: capture current p50/p95/p99 for the service and DB before/after migration. Compare against historical baselines.
- Collect telemetry:
- APM traces (distributed traces showing service→DB spans) to find slow spans and percentiles.
- Metrics: request rate, error rate, latency histograms, CPU, memory, GC on app, thread-pool wait times.
- DB metrics: query latency, throughput, locks, CPU, IO, cache hit rate, active connections, wait events (e.g., pg_stat_activity, InnoDB metrics).
- Network metrics: RTT, packet loss, retransmits, interface errors, MTU mismatches.
- Driver/client logs: connection pool stats, DNS resolution times, SSL/TLS handshake times.
- Query-level diagnostics: slow query log, EXPLAIN plans, pg_stat_statements or equivalent.
- Narrow root-cause hypotheses and test:
- Config: compare connection string, TLS settings, timeouts, max connections, connection pool size.
- Network: run traceroute, mtr, tcpdump between app and DB; check private link/VPC peering, NAT gateways, cross-AZ traffic.
- Driver changes: verify driver version, behavior changes (sync vs async), default timeouts, new instrumentation.
- Queries/schema: run slow queries, check indexes, statistics refresh after migration, plan changes.
- Use targeted traces: synthetic transactions that isolate DB calls, run A/B tests (point some traffic to old DB if still available).
Short-term mitigations:
- Enable query/result caching (app-level or edge) for heavy read paths.
- Increase DB instance size or IOPS temporarily or promote read replica for reads.
- Tune connection pool: reduce churn by increasing max connections or reusing connections; enable keepalives.
- Adjust timeouts/retries and exponential backoff to avoid thundering herd.
- Add circuit breakers and degrade non-critical features to reduce load.
- Route a portion of traffic to fallback (previous DB or cached snapshot) while investigating.
Communicate findings and next steps to stakeholders, collect data from each mitigation to validate impact, then implement permanent fixes (config, driver upgrade/downgrade, query/index changes, network path optimization).
