Start with evidence & isolation:- Check metrics/alerts (RSS, OOMs) and reproduce on a staging host if possible.- Identify PID and timeline: ps, top, pidstat, or grafana metrics.Quick OS-level inspection:- pmap -x <pid> to view mappings and RSS/Anon/Swap:pmap -x 1234 | sort -k3 -n
- /proc/<pid>/smaps gives per-mapping detailed RSS, PSS, private/clean/dirty — useful to find which mapped file or anonymous region grows. Inspect with:grep -A1 -E "Rss:|Pss:|Private_Clean:|Private_Dirty:" /proc/1234/smaps
or monitor deltas over time to see growing regions.Native (C/C++) debugging:- Run under valgrind memcheck (heavy, use in staging): valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes ./service
- Use massif for heap snapshots and visualize with massif-visualizer:valgrind --tool=massif ./service
ms_print massif.out.<pid>
- For less overhead and production sampling, use heaptrack (Linux, low overhead) to collect allocation stack traces and analyze with heaptrack_gui.Managed languages (Python, Java, Go):- Python: use tracemalloc to take snapshots and compare:import tracemalloc
tracemalloc.start()
# capture snapshot periodically
s1 = tracemalloc.take_snapshot()
# later:
s2 = tracemalloc.take_snapshot()
print(s2.compare_to(s1,'lineno')[:20])
- Java: use jmap/jcmd/jstack and heap dumps analyzed with Eclipse MAT.- Go: use pprof (net/http/pprof) and capture heap profiles.Methodology:1. Establish baseline and reproduce. Capture periodic RSS/PSS and smaps snapshots.2. Correlate mapping names (lib, anonymous, file-backed) with code areas.3. Collect heap profiles or allocation traces at intervals to see growth trends.4. Analyze flamegraphs/stack traces to find leaking allocation paths.5. Create minimal repro and fix (freeing, object lifecycle, caching bug).Mitigation without restart:- Reclaim caches: tune or clear internal caches, use eviction policies, or reduce cache sizes via feature flags/config.- Drop caches (temporary, affects performance): echo 3 > /proc/sys/vm/drop_caches (use cautiously).- Spawn a helper process to periodically restart worker processes (graceful rolling restart) or restart individual leaking worker without full service downtime.- Throttle input or shed load to limit allocations until patch deployed.- Use OOM score adjustments and cgroups to limit blast radius.Notes:- Prefer PSS from smaps for multi-process/shared memory attribution.- Valgrind unsuitable for production; use sampling/low-overhead profilers there.- Always reproduce, collect traces over time, and verify fix with tests and monitoring.