Performance Troubleshooting & Incident Response Questions

Clarify scope & impact: confirm which endpoints and customers see p99 spikes, timeframe, recent deploys or config changes, and whether errors increased.

Telemetry to examine:

• Distributed traces (span durations, tail latency hotspots)
• Metrics: per-endpoint p50/p95/p99, QPS, concurrency, CPU/memory, GC pauses, thread pool queues, connection pools
• Logs: slow request traces, stack traces, timeouts
• Infra: network telemetry (RTT, retransmits), load balancer metrics, DB/queue latencies
• Change history/alerts and deployment timeline

Experiments to run:

• Load tests reproducing tail behavior (traffic shape, concurrency, burstiness)
• Profiling in staging under similar load (CPU sampling, allocation, async scheduling)
• Thread/heap analysis and GC logging
• Fault injection: latency k8s/iptables, DB slowdowns, circuit-breaker behavior
• A/B tests toggling suspect features or config flags

Potential fixes (short → long):

• Quick mitigations: throttle/buffer bursts, increase timeouts/retry jitter, enable request hedging, route heavy queries to dedicated paths, roll back recent deploy, increase instance autoscale thresholds
• Code: optimize hot paths, reduce sync/blocking work, batch/async I/O, avoid head-of-line blocking
• Resources: right-size CPU/memory, tune GC, increase connection pool sizes
• Network: tune keepalives, retry/backoff, colocate services or use faster routes
• Config: tune thread pools, queue limits, circuit-breakers, timeouts
• Long-term: split services, capacity planning, client-side backpressure, improved SLAs, rewrite hot modules, more observability (end-to-end tracing, SLO-based alerts)

Prioritization:

1. Contain impact fast (throttles, rollbacks, autoscale)
2. Diagnose root cause with reproducible tests and profiling
3. Implement targeted code/resource fixes
4. Plan architecture changes and preventative SLO-driven work with timelines and risk assessment

Communicate status, owner, and ETA throughout.

1. Clarify scope & baseline

• Confirm deployment time, services/versions changed, affected SLA (p95 +40%). Note rollout pattern (canary/global).

2. Rapid triage (5–15 min)

• Check high-level metrics: p95/p99 latency, error rate, throughput/requests/sec, CPU, memory, GC pause, thread counts, queue lengths.
• Tools: Grafana + Prometheus / Datadog / New Relic dashboards, Cloud provider metrics (CloudWatch / GCP Monitoring).

3. Narrow by dimension (15–30 min)

• Break down by endpoint/route, host/instance, container/pod, AZ, user region, client version, and time-series around deploy.
• Correlate with traces: look for increased span durations on specific services or downstream calls. Tools: Jaeger/Zipkin, Datadog APM, New Relic APM.

4. Pinpoint component

• If single endpoint: inspect handler, DB calls, external APIs, cache hits, serialization, network.
• If single host/container: check resource starvation, OOMs, CPU steal, noisy neighbor.
• If downstream dependency: see tail latencies on DB, queue, auth service.

5. Profiling (deeper)

• CPU: take production-safe CPU profiles (async-profiler / JFR for Java / py-spy for Python) during high-latency window; generate flamegraphs to find hotspots.
• Memory: heap dumps (heap dump + MAT/JProfiler) or allocation profiles to detect leaks or growth.
• Threads: thread dumps / stack traces to detect blocking/waits, thread pool saturation.
• I/O: trace syscalls / network latency (strace, eBPF tools like bpftrace, tcptracer).
• DB: slow query logs, EXPLAIN, connection pool exhaustion.

6. Hypothesis & short experiment

• Reproduce in staging or replay traffic if possible. Apply temporary mitigation: increase instance count, bump thread pools, circuit-break downstream, clear caches.
• Use feature-flag toggles or rollback single microservice version in canary.

7. Validate rollback/fix

• Deploy rollback or fix to a small subset (canary), monitor p95, errors, throughput, CPU, GC, and traces for regressions for several p95 windows.
• If fixed, promote rollout gradually; if not, revert and continue investigation.

8. Postmortem

• Root cause, timeline, action items (alerts, runbooks, tests). Add synthetic checks and run load tests to prevent regression.

Tools summary: Prometheus/Grafana, Datadog/New Relic, Jaeger/Zipkin, async-profiler/JFR/py-spy, flamegraphs, heap analyzers, CloudWatch/Stackdriver, kubectl / container logs, eBPF/tracing.

Situation: After a feature rollout the 95th percentile latency jumped from 300ms → 1.2s; team suspects an N+1 DB regression.

Detect & verify in production:

• Start with metrics: confirm spike correlates with rollout time using APM (New Relic, Datadog, Instana) and logs.
• Capture traces on high-latency requests (APM distributed traces). Look for many small DB calls inside a single HTTP trace.
• Run a lightweight sampling profiler or enable query logging for the service (with low overhead) to collect SQL counts per request ID. Example: log number of DB calls and total DB time in request context.
• If traces show repeated identical parameterized queries (e.g., SELECT user WHERE id=? repeated), that's strong evidence of N+1.
• Optionally reproduce with a production-like load on staging using the same dataset and request that triggers the path, and use query-count instrumentation.

Proposed fix:

• Eager-load / batch queries: change code to fetch related rows in one query (JOIN or WHERE id IN (...)) or use ORM prefetch/Include (e.g., Hibernate fetch join, Django select_related/prefetch_related).
• Add caching for hot reference data if appropriate.
• Add defensive instrumentation: assertions/metrics that count DB queries per HTTP request and alert on thresholds.

Estimate complexity:

• Small (1–2 days): If ORM supports simple prefetch and code paths are localized.
• Medium (3–7 days): If multiple call sites or complex object graphs require redesign and test coverage.
• Larger if pagination/consistency constraints or cross-service changes required.

Testing before deploy:

• Unit tests: verify replaced data access returns same results.
• Integration tests: on staging with production-sized data set, run representative requests and assert DB query counts reduced.
• Load test: run a load test comparing old vs. new implementation; measure P95 latency and DB QPS.
• Canary deploy: rollout to small percentage with request-level DB-query-count metric and alert if query counts or latency regress.
• Rollback plan: feature flag or quick revert if canary shows issues.

This approach verifies root cause, applies the minimal invasive fix, quantifies effort, and safely validates in production-like conditions.

Investigation (first 30–60 minutes)

• Verify signal: confirm the 30% increase at 95th using multiple sources (APM/dashboard, Prometheus, Datadog). Check service-level and downstream metrics (error rate, throughput, CPU, GC, thread pools).
• Scope blast radius: is regression in all regions/instances or only canary/after full rollout? Correlate deploy time with metric change.
• Pinpoint cause: enable/inspect distributed traces (Jaeger/NewRelic) to see which spans increased. Turn on verbose logs for a small sample of requests. Compare stack traces and timing pre/post-upgrade.
• Reproduce locally: run benchmark/load test with both old and new library versions to validate regression and measure which code paths are affected.
• Quick mitigations: if safe, reduce traffic to affected instances (circuit breaker, rate limit), or increase replica capacity to absorb latency while investigating.

Root-cause analysis (next 1–4 hours)

• If traces show specific new calls or serialization changes, dive into library changelog and diffs. Search for known issues in vendor repos.
• Profile CPU, memory, syscall waits, and GC to see if library changed resource usage.
• Run binary search: roll back library in a single region or subset to confirm causality.

Rollback vs patching — decision criteria

• Rollback if:
  • Clear causality: metric improves when reverting in canary
  • Impact severe (SLA breach, revenue/customer-visible errors)
  • Fix requires behavioral change in library that cannot be hot-fixed quickly
  • Rollback is low-risk and fast (automated deploy)
• Patch if:
  • Regression is minor and workaround exists (config toggle, timeout tuning)
  • Root cause is a code path in our code that can be patched and deployed faster than full rollback
  • Rollback risks losing other important fixes deployed after library change
• Always consider risk of partial rollback: ensure compatibility across services and schema.

Concrete plan

• If rollback chosen: execute rollback playbook in canary → monitor 95th latency and error rates → progressively roll back regions if improvement confirmed. Keep one rollback champion (owner) and an on-call for follow-ups.
• If patch chosen: implement minimal safe patch (e.g., reduce buffer size, change call pattern, add timeouts), validate in staging with load tests, deploy to canary, monitor, then full rollout.

Communication plan

• Internal (within 15–30 minutes): Post incident summary in incident channel with impact, affected services, initial mitigation, owner, and ETA for next update (every 30–60 minutes). Notify SRE, product, and customer-facing teams.
• Executives (30–60 minutes): Short email/Slack: severity, breach vs approaching SLA, customer impact estimate, chosen mitigation (rollback/patch), expected time to resolution.
• Customers (if visible): If user-facing impact or SLA risk, send a brief status update via status page and customer ops: what happened, who’s affected, immediate mitigation, and expected timeline for fix. Follow up with root-cause and permanent fix within 24–72 hours.
• Postmortem: within 48–72 hours publish timeline, root cause, action items (test coverage, canary policy, rollout automation), and owners for each follow-up.

What I’d deliver as engineer

• Repro script, trace snapshots, before/after dashboards, rollback/patch playbook, and a clear timeline for stakeholders. Emphasize measurable gates (95th latency threshold) for safe rollout.

Situation clarification: confirm scope (which endpoints/queries show high tail latency, recent deploys, traffic pattern changes, single region or global) and SLOs (p95/p99 targets), then run a systematic investigation.

1. Metrics to collect

• Latency percentiles: p50/p90/p95/p99/p999 plus histograms and latency by endpoint/query template.
• Throughput, QPS, error rate, retry counts.
• Resource: CPU, per-core utilization, load average, context switches.
• Memory: heap used/free, GC pause times (G1/Full GC counts).
• I/O: disk latency, IOPS, network RTT, socket queue lengths.
• Threads: thread pool queue sizes, active threads, blocked/waiting counts.
• OS metrics: file descriptors, TCP retransmits.
  Collect per-instance and aggregated metrics, and tag by tenant/route.

2. Tools to use

• Distributed tracing (Jaeger/Zipkin/OpenTelemetry) to see spans and hot paths across services.
• APM (New Relic/DataDog) for transaction breakdowns.
• Profilers: async-profiler / YourKit / perf for CPU/flamegraphs; async-profiler for Java lock/C-state stacks.
• Heap/GC tools: jstat, GC logs + GCViewer, jmap/jcmd for heap dumps.
• Thread dumps (jstack) and contention analysis.
• OS tools: iostat, sar, vmstat, tcpdump, perf, eBPF (bcc/tracee) for syscall waits.
• Load testing: replay with Gatling/Locust/Vegeta that matches production QPS and arrival patterns.

3. Likely culprits and how to detect

• GC pauses: spikes in p99 latency correlated with GC pause metrics / safepoint logs.
• Lock contention: profiler shows blocked threads, or thread dump shows many threads waiting on same monitor; async-profiler lock events.
• I/O bottlenecks: high disk latency, fsyncs, or network saturation; iostat / network metrics spike.
• Hot shards/hot keys: some queries touch larger posting lists causing variance; analyze query distribution and per-shard latency.
• Resource exhaustion: CPU-stealing, CPU frequency scaling, overloaded thread pools (queue growth).
• JVM safepoints or classloading during cold starts.
• External dependencies: slow downstream calls (logging, auth, metrics exporters).

4. Concrete optimizations

• Index/query tuning: add targeted filters, pre-compute facets, use doc-values, reduce stored fields, tune analyzers, use index sharding/replica placement to avoid hotspot.
• Batching & bulk processing: batch I/O, bulk fetches for documents, compress network calls.
• Asynchronous work: offload non-critical work (analytics, logging, metrics) to background queues; use async I/O for network/disk.
• Caching: result caches, LRU caches for expensive postings, query plan caching; use warmed caches on restart.
• Concurrency & pooling: increase worker threads or use bounded queues with backpressure; tune connection pools; switch to non-blocking IO where appropriate.
• Reduce GC pressure: object pooling, avoid excessive short-lived objects, use escape analysis-friendly patterns, tune JVM (G1/ZNGen parameters) or move to ZGC/Shenandoah for low-pause needs.
• Lock-free data structures or fine-grained locking: replace global locks with per-shard locks or concurrent collections.
• I/O optimizations: async filesystem calls, SSDs, avoid fsync per document, increase read-ahead, use memory-mapped indices if suitable.
• Query timeout/rate limiting: fail-fast for extremely expensive queries, provide degraded results.
• Architecture: add read replicas, re-balance shards, introduce query fanout limits.

5. Validation plan

• Reproduce: run load tests that mimic production percentiles and spike patterns; include warm-up to simulate cache state.
• Canary deploy: roll changes to subset, compare p50/p95/p99 and resource metrics against control.
• Use tracing to confirm reduced span durations and fewer blocked/waiting events.
• Automated benchmarks and regression suite to ensure no throughput regression.
• Monitor SLOs over several hours/days; use alerts for p99 regressions.
• Post-change postmortem: document root cause, applied fix, and monitoring added to prevent recurrence.

Example quick wins: enable query caching and result caching to reduce I/O; tune JVM heap to avoid frequent GCs; add timeouts and backpressure to prevent queue buildup. For persistent tail at p99, focus on root causes shown by tracing and flamegraphs — optimize hot functions or offload them.