Performance Troubleshooting & Incident Response Questions

Situation & goal
Identify subtle lock contention causing p99 latency spikes at high concurrency in a C++ service, isolate the hot lock(s), validate root cause, and evaluate mitigations (sharding, lock-free, redesign).

Methodology — observation & profiling

• Start with reproducible load: run a controlled loadtest (constant QPS, varying threads) and capture metrics (p50/p95/p99, cpu, syscalls).
• System-wide sampling: use perf to capture stacks during spikes:
  bash

  perf record -F 99 -g --pid <pid> -- sleep 60
  perf report --sort comm,period
  

  Generate flamegraphs from perf script to see hotspots.
• Lock-specific profiling:
  • perf lock:
    bash

    perf lock record -p <pid> sleep 60
    perf lock report
    

    This shows contended mutexes, hold/wait times.
  • async-profiler or Intel VTune can show monitor/wait time and contention with call stacks.
• OS-level tracing: use eBPF (bcc/tools offcpu or runqlat) to see threads blocked in kernel or user-space.

Code instrumentation

• Add lightweight lock wrappers that record acquisition time and owner (atomic timestamp + histograms). Example data points: wait duration, hold duration, stack trace (sampled).
• Incrementally enable at runtime (config flag) to avoid adding overhead in normal runs.
• Log per-lock histogram (HdrHistogram) and tag by endpoint/operation.

Pinpointing

• Correlate perf flamegraphs + perf lock report + per-lock histograms to map contention to code paths.
• Confirm caller stacks: ensure stack traces from perf match instrumentation owner stacks.

Validation

• Microbenchmark the hot critical section in isolation under varying thread counts.
• Run A/B tests: baseline vs change (sharding/lock-free) under identical load; compare p99, throughput, CPU.
• Use regression tests to ensure correctness under concurrency.

Evaluating fixes

• Lock sharding: easiest—split global lock into N shards keyed by consistent hash. Pros: low code change, multiplies parallelism; Cons: increased complexity, potential imbalance.
• Lock-free queues/data structures: use atomics, MPMC ring buffers (boost::lockfree or Folly). Pros: lower latency under high contention; Cons: complex correctness, ABA, memory reclamation (hazard pointers/epoch).
• Redesign hot path: remove work under lock (defer via queues), reduce critical section size, cache immutable data. Often most effective.
• For each candidate:
  • Measure p99, CPU overhead, GC/alloc behavior.
  • Check fairness/starvation, memory growth.
  • Validate correctness with stress tests and fuzzed concurrency scenarios.

Trade-offs & final checks

• Prefer smallest invasive change that reduces contention (sharding or reducing critical section).
• Avoid premature lock-free unless measured need.
• After rollout, monitor real traffic p99, SLOs, and collect the same profiling artifacts to ensure spike elimination.

Quick diagnosis (what’s bottleneck and why)
The likely bottleneck is disk I/O. iostat shows high write throughput (wkB/s 8192) with %util 62.5 and avg service time (svctm) 2.50 ms and await 25 ms — await is much larger than svctm, indicating queueing. vmstat shows r=4 runnable tasks and wa=40% CPU waiting on I/O, which aligns with heavy IO wait. CPU idle (id=10) and user/sys (us+sy=50) don’t indicate CPU saturation — instead processes are blocked waiting for disk.

Immediate action

• Throttle/remove heavy writers or move batch jobs off the device; reduce write pressure (e.g., pause backup/cron jobs) to relieve queueing.

Commands / metrics to collect to confirm

• Live I/O per-process and latencies:

iotop -aoP

• Per-file or FS activity:

sudo perf trace -e sys_write -p <pid>

• Detailed block device stats:

iostat -x 1 10
sudo blktrace -d /dev/sda -o - | blkparse -i -

• Check kernel dirty settings and memory pressure:

cat /proc/sys/vm/dirty_ratio
vmstat 1 10
free -m

• Check filesystem and queue depths:

lsblk -o NAME,ROTA,TYPE,MOUNTPOINT
cat /sys/block/sda/queue/nr_requests

Collecting these confirms whether latency is device, filesystem, or process-driven and guides longer-term fixes (tuning, faster storage, RAID adjustments, or spreading I/O).

Alert Summary
Trigger: CPU Saturation — sustained high CPU on service hosts excluding scheduled batch windows. Purpose: detect real contention affecting user-facing workloads while minimizing noise from known batch jobs.

Alert Rule & Threshold Logic

• Query: evaluate per-host CPU user+system over past 5m.
• Aggregation: use 95th percentile across 5m per host to ignore spikes:
  • host_cpu = percentile(cpu.user + cpu.system, 95%, 5m) per host
• Threshold: fire when host_cpu > 85% for 3 consecutive 5m windows (i.e., sustained).
• Multi-host impact: page if >3 hosts in a service exceed threshold OR service-level 90th percentile > 80%.

Maintenance Window Handling

• Integrate with scheduled maintenance calendar (e.g., PagerDuty/CalDAV). Rule suppressed automatically during declared windows.
• For batch jobs, tag hosts with "batch=true" and apply separate rule with higher thresholds or mute during configured cron windows.

False-Positive Reduction Techniques

• Dependency-aware suppression: if downstream batch orchestrator (e.g., Airflow) reports RUNNING, suppress alerts on tagged hosts.
• Anomaly detection: baseline per-host CPU by time-of-day/week using rolling 28-day model; only alert if deviation > 30% from expected and above absolute threshold.
• Escalation policy: initial warning channel (Slack) for single-host; only page on multi-host or sustained anomalies.

Documentation for On-call

• One-page runbook: intent, exact query, thresholds, maintenance integration, run/stop commands, checklist: check scheduled jobs, check process list (top/ps), check I/O wait, correlate with orchestration UI, remediation steps (nice/kill, scale up, cordon & migrate), postmortem template and SLO impact calculation.
• Include playbook snippets and links to dashboards, runbooks, and owners.

Approach summary
Start by distinguishing managed-heap growth (JVM objects retained) vs native/C heap growth (malloc, native libs, JNI, thread stacks). Triage in stages: quick signals, targeted captures, then deeper analysis.

Quick checks (low impact)

• Check process RSS and threads:

ps -o pid,cmd,%mem,rss,vsz -p <pid>
top -p <pid>

• See mapped memory and anonymous allocations:

pmap -x <pid>   # summary per mapping
cat /proc/<pid>/smaps | egrep -A1 "Rss:|Private_Dirty:" | less

Capture artifacts to decide managed vs native

• JVM-managed heap evidence:
  • Trigger and inspect heap/GC stats:

jcmd <pid> GC.class_histogram        # quick heap histogram
jcmd <pid> GC.run                    # force full GC, low impact
jcmd <pid> GC.heap_info

• Create a heap dump for offline analysis:

jmap -dump:live,format=b,file=/tmp/heapdump.hprof <pid>

• Native memory evidence:
  • Ask JVM native memory summary:

jcmd <pid> VM.native_memory summary

• Capture OS-level snapshot / core if needed:

pmap -x <pid> > /tmp/pmap.txt
gcore -o /tmp/core.<pid> <pid>     # large file; use only if safe

• Use lsof / perf or malloc hooks for suspect native libs:

lsof -p <pid>
perf top -p <pid>   # CPU hotspots may point to allocators

What to analyze offline

• Heap dump: open in Eclipse MAT — look for growing dominator trees, leaking caches, classloader leaks.
• Native/core: use GDB, addr2line, and tools like valgrind/heaptrack (on a staging replica) to find malloc stacks.

Immediate low-disruption mitigation (production)

• Force a safe full GC to free managed heap retained by soft refs/caches:

jcmd <pid> GC.run

• Or apply throttling/graceful restart of a single instance behind the load‑balancer:
  • Remove instance from LB, restart JVM, re-add — avoids cluster-wide outage.

Notes and trade-offs

• Avoid heapdump/gcore on disk-constrained machines; they can be large and I/O heavy.
• gcore and valgrind are intrusive — prefer captures on a staging replica.
• If VM.native_memory shows native growth, plan a rolling restart and deeper native profiling.

Approach overview

I treat a regression like an incident: reproduce, narrow to change, validate fix, and safe rollout.

Reproduce & gather CI artifacts

• Re-run the failing benchmark from CI using the same build artifact (build ID, docker image, commit). Collect logs, perf profiles, flamegraphs, and system metrics (CPU, I/O, GC).
• Compare golden CI artifacts (baseline) to current run to quantify regression (e.g., p50 latency +20%, throughput −10%).

Narrow using benchmark suites + git bisection

• Run a targeted micro/benchmark suite that isolates subsystems (network, serialization, DB calls). If regression is deterministic, run git bisect across commits referenced by CI (use the same build environment) to find the first bad commit.
• During bisect, automate running a short, reliable perf test and fail/ok criteria (e.g., latency above threshold = bad). Record artifacts at each step.

Canary deployment to validate hypothesis

• Once a suspect change or fix exists, deploy to a small canary cohort (1–5% traffic) with feature flags and telemetry enabled.
• Monitor SLOs, error rates, resource usage, and application-level performance traces. Use automated rollback triggers (threshold breaches) and alerting.

Ensure fix validated before full rollout

• Run CI benchmark suite against the fixed commit and the original baseline; require statistical significance (multiple runs, trimmed means).
• Validate in staging under realistic load and in canary under production traffic. Keep rollback plan and runbook ready.
• After gradual ramp (5→25→100%) with health checks at each step, finalize rollout.

Example specifics

• If CPU spike traced to a changed serialization path, revert to previous serializer in a canary, verify p95 latency drop and CPU % back to baseline, then proceed with staged rollout.

This process emphasizes reproducibility (same artifacts), automated bisecting, measurable benchmarks, safe canaries, and guarded rollouts.