Performance Troubleshooting & Incident Response Questions

Immediate conclusions:

• This is a latency/timeout issue: requests are exceeding configured upstream timeouts (database and cache).
• Two distinct upstreams implicated — database and cache — so cause may be upstream slowdowns, contention, or intermittent network/IO issues rather than only app-side logic.
• The durations (1.2s / 1.4s) exceed their respective timeout thresholds (1s / 0.5s), meaning clients saw or would have seen failures or retries, increasing user-visible latency and load.

Next steps to validate:

1. Correlate logs with tracing and metrics:
   • Look up the request_id traces to see where time was spent (DB queries, network, retries).
   • Check APM/spans for DB query times and cache hit/miss rates.
2. Aggregate and quantify:
   • Query centralized logs/metrics for frequency of timeout warnings (per minute/hour), percent of requests affected, and trend over recent hours/days.
   • Break down by endpoint, region, instance, and deployment version.
3. Check upstream health and capacity:
   • Database: slow queries, locks, CPU/I/O, connection pool saturation, recent schema changes or long-running maintenance.
   • Cache: eviction rate, miss rate, network errors, node load.
4. Reproduce and narrow:
   • Run synthetic load tests for the affected endpoints and simulate high load to reproduce latency.
   • Compare behavior from multiple app instances to rule out single-instance/network issues.
5. Inspect recent changes:
   • Review recent deploys, config changes (timeouts, pool sizes), schema or query changes, and traffic shifts.

Prioritization and remediation plan:

• Priority = impact on SLO + blast radius. If these endpoints are on critical request paths and affect many users, escalate to P0.
• Short-term mitigations:
  • Increase upstream timeouts and connection pools cautiously (temporary).
  • Add circuit breakers / rate limits to prevent cascading retries.
  • Route traffic away from unhealthy nodes or scale replicas.
• Medium/long-term fixes:
  • Optimize slow queries, add indexes, tune DB config, or add read replicas.
  • Tune cache configuration to reduce misses and network latency.
  • Add granular tracing and SLO-based alerts for regressions.
• Post-incident:
  • Create dashboards showing latency vs timeout, runbook for this alert, and a postmortem outlining root cause and action items.

What I would communicate:

• Immediate status (scope, user impact), temporary mitigations in-flight, and timeline for investigation and next deployment(s).

When diagnosing a CPU spike in a Go backend with pprof, I follow: capture a profile, inspect interactively (web), and generate a flamegraph. Steps and commands:

1. Capture a 30s CPU profile (app exposes net/http/pprof at /debug/pprof):
   curl -sS http://localhost:6060/debug/pprof/profile?seconds=30 > cpu.prof

Or if using runtime/pprof programmatically:
go tool pprof http://localhost:6060/debug/pprof/profile?seconds=30

2. Open interactive web UI (svg flame graph / call graph):
   go tool pprof -http=":8080" ./mybinary cpu.prof
   Then open http://localhost:8080 → explore “Top”, “Graph”, “Flame Graph”.

3. Generate an SVG flamegraph file:
   go tool pprof -svg ./mybinary cpu.prof > cpu.svg
   Or using Brendan Gregg-style flamegraph (if using github.com/google/pprof + flamegraph.pl):
   go tool pprof --raw ./mybinary cpu.prof | ./flamegraph.pl > flame.svg

Key inspection points:

• Look at “Top” to find functions with highest sample counts (percentage of CPU).
• Use “Flat” vs “Cum” (self time vs inclusive time).
• Expand callgraph to see callers of hot functions; check annotation to see source lines.

Example interpretation:
Top shows function database.QueryRows at 42% (cum) and 30% (flat). That indicates QueryRows itself spends significant CPU (not just callers). Investigate:

• Are rows being scanned with reflection (encoding/json / rows.Scan into interface{})? Replace with manual decoding or use preallocated structs.
• Add instrumentation: measure latency, DB query EXPLAIN, check connection pooling.
  Resulting action: changed Scan targets to concrete types and reduced allocations; CPU usage dropped 35%.

Notes & best practices:

• Ensure build has symbol info (no -ldflags "-s -w").
• For containers, port-forward or run pprof endpoint in sidecar.
• Capture profiles during the spike; short captures may miss intermittent issues.

Interpretation:

• This iostat sample shows almost no read traffic (r/s 0.1) and heavy write IO (w/s 120) with very large wsec/s (307,200). Since iostat reports sectors/sec (usually 512 bytes/sector), 307,200 sectors/s ≈ 157 MB/s of outgoing writes. await = 12.5 ms indicates average IO latency is elevated but not catastrophic. Pattern = write-dominated, high throughput, moderate latency — likely sequential large-block writes or sustained write-back activity (e.g., bulk log/backup, database checkpoints, RAID rebuild, or heavy fsyncs).

Possible problems:

• Device is saturated by sustained writes (throughput-bound) causing increased latency.
• Heavy writeback from memory (dirty pages) or a process doing large sequential writes.
• Underlying storage (RAID controller, SAN, multipath) bottleneck or degraded array.
• Mount/write options or kernel tunables (commit interval, barriers) causing more synchronous writes.

Next diagnostics (what and why):

1. iostat -x 1 5
   • get %util, svctm, avgqu-sz, await and confirm saturation and queue depth.
2. iotop -aoP (or iotop -o)
   • identify processes causing the high write throughput.
3. pidstat -d 1
   • per-PID IO stats over time for automation/scriptable reporting.
4. vmstat 1 5 and free -m
   • check memory pressure, swap, and dirty page counts (writeback behavior).
5. sar -d 1 5 or dstat -dny
   • longer historical IO patterns and device-level stats.
6. lsblk, cat /proc/mdstat, multipath -ll, dmsetup ls --tree
   • verify device topology: LVM, dm-multipath, RAID degradation.
7. smartctl -a /dev/sda (if local disk) and storcli/megacli (if hardware RAID)
   • check disk/raid health and errors.
8. ioping / fio (small tests) or blktrace + blkparse (deeper)
   • measure latency distribution and characterize IO size/seq vs random.
9. check application logs/config:
   • DB checkpoints, backup jobs, cron jobs, filesystem mount options (noatime, barriers), and kernel tunables (vm.dirty_ratio / dirty_background_ratio).

Actionable next steps:

• If a single process: throttle or tune it (ionice, nice, batching).
• If writeback/dirty pages: tune vm.dirty_* or adjust application fsync patterns.
• If storage degraded/saturated: investigate RAID rebuild, increase throughput capacity, or open vendor ticket.
• Add monitoring alerts on %util, await, and throughput to detect recurrence.

This sequence isolates whether the problem is application-driven, kernel/writeback-driven, or underlying storage hardware/network related and guides remediation.

Why throttling happens

• Linux CFS enforces container CPU limits (quota/period). When a container tries to use more CPU than its cpu limit (or when sum of container quotas exceeds node capacity), the kernel stops scheduling it until the next period — this increments cpu.cfs_throttled_periods. In short: the workload is capped by the container/node quota, not raw CPU availability.

How throttling affects latency

• Throttling reduces CPU time available to the process, causing increased request latency and higher tail latency (p99/p999) and greater jitter. Requests may queue, time out, or suffer increased service response times under bursts.

Three actions to investigate and mitigate

1. Investigate current state (diagnose)

• Commands/metrics: kubectl describe pod <pod>; kubectl top pod/node; kubectl exec into container + top/htop; inspect node and kubelet metrics: container_cpu_cfs_throttled_periods_total, container_cpu_usage_seconds_total (Prometheus), node_exporter/cAdvisor.
• Check events for OOM/throttling, compare cpu requests vs limits, check sum of limits on node vs capacity.
• Reproduce under load (ab) while observing metrics to correlate throttling and latency.

2. Immediate mitigation (short-term)

• If safe, increase the pod CPU limit (or remove the limit) so it can use more CPU: kubectl patch deployment ... to raise resources.limits.cpu. Alternatively lower concurrency or rate-limit incoming requests to reduce instantaneous CPU demand.
• If limits must stay, scale horizontally: increase replica count via HPA to spread load and lower per-pod CPU demand.

3. Long-term fixes / prevention

• Right-size resources: profile the app, set requests to baseline usage and limits to expected burst; for predictable CPU-intensive workloads, use Guaranteed QoS (requests == limits) or use CPU Manager static policy to pin CPUs for latency-sensitive services.
• Capacity planning: ensure node types/cluster autoscaler can add nodes before saturation; avoid overcommitting CPU limits that sum > node capacity.
• App changes: optimize hot paths, move heavy tasks to async workers, add backpressure/circuit-breaking so bursts don’t overload CPU.

Key trade-offs

• Removing or raising limits reduces throttling but risks node contention; scaling and right-sizing preserve stability but require profiling and cost trade-offs. Use metrics/alerts on throttled_periods and p99 latency to detect regressions.

Start with a hypothesis-driven, low-risk process: collect safe evidence, correlate signals, then narrow to reproducible inputs or code paths.

Safe core-dump collection

• Enable controlled core dumps (systemd or kernel) and limit size & rate. Example (systemd):

# enable persistent cores with size limits
mkdir -p /var/lib/systemd/coredump
echo 'Storage=external' > /etc/systemd/coredump.conf
echo 'ProcessSizeMax=200M' >> /etc/systemd/coredump.conf
systemctl restart systemd-coredump
# or for kernel
sysctl -w kernel.core_pattern=/var/crash/core-%e-%p-%t
ulimit -c 200M

• For containers: write cores to host-mounted directory, avoid setuid/binary permissions issues, run containers with SYS_PTRACE only when necessary. Use seccomp/profile changes sparingly.
• Sanitize PII: ensure core dumps do not violate policy—strip heap or encrypt at-rest if required.
• Automate secure upload and TTL: collect with crash reporter (apport/abrt/custom) that tags metadata and removes old cores.

Correlate core dumps with metrics/traces

• Tag dump metadata: service, version, git SHA, container id, node, timestamp, thread info, last GC/heap stats, config flags.
• Correlate with APM/tracing (Jaeger/Zipkin/OpenTelemetry): link trace IDs from request logs to crash time window. Capture last N spans for the crashing thread.
• Pull metrics around timestamp (CPU, memory, minor/major GC, file descriptors, network I/O). Use time-series queries to detect anomalies (spike in latency, memory growth, swap).
• Combine with structured logs: include line-buffered logs and high-cardinality tags (request id, user id). Use log tailing to capture pre-crash stack traces or OOM patterns.

Rapid triage workflow

• Extract stack traces from core via gdb or lldb non-interactively:

gdb -batch -ex "thread apply all bt full" -ex "quit" /path/to/binary /path/to/core > /var/log/crash-logs/core-${PID}.txt

• If symbols missing, fetch matching build artifacts (debuginfo) via git SHA. Maintain reproducible build artifacts and symbol server.

Finding reproducible test cases

• Enable sanitizers in dev builds: ASAN/UBSAN/TSAN and address/undefined behavior detectors catch corruption early. Run long-running fuzzing with libFuzzer/AFL on serialized inputs or RPC payloads.
• Heap/Memory tools: run with valgrind/helgrind/heaptrack in a staging environment, or use jemalloc's heap profiling and redzone options to detect corruption points.
• Deterministic replay: if using record-replay (RR) or system-level tracing (strace with -f and timestamps, or packet capture for network input), replay the exact sequence to reproduce. For RPC-heavy services capture request sequences with request IDs and replay harness.
• Binary instrumentation: use Electric Fence or compile with -fsanitize=address and fail fast to get precise stack at faulting allocation.
• Reduce test surface: perform delta-reduction on inputs and sequence of events using automated bisect (git bisect for code regressions, input reduction scripts for payloads).
• If intermittent and time-based, run targeted chaos: increase load, add GC pressure, run under resource constraints to stress the same code path.

Root-cause best practices

• Reproduce with same runtime flags, allocator, and kernel; ensure same library versions (use container images).
• If core shows corrupted heap but stack points elsewhere, instrument allocator to log free/alloc sites; enable canaries.
• Iterate: create minimal unit/integration test that reproduces; if impossible, add guardrails (validation, bounds checking, crash recovery) and deploy as mitigation while debugging.

Outcome: combine safe core collection, rich metadata, telemetry correlation and staged tooling (sanitizers, fuzzing, record/replay) to move from intermittent native crashes to a reproducible failure and a fix.