Memory Management & Garbage Collection Questions

Situation & goal
I’d treat this as a progressive memory leak incident: stop OOMs, find root cause, and deliver a safe fix path with clear trade-offs.

Metrics & tools

• Observe: RSS, heap usage, GC pause times, threads, file descriptors, swap, cgroup memory.stats.
• Tools: Prometheus/Grafana, node_exporter / cAdvisor, top/ps, pmap, smem, jemalloc/malloc_stats, perf, eBPF (bcc / bpftrace), heap profilers (Java Flight Recorder / jmap / MAT, Go pprof, Python tracemalloc), container logs, kubelet events.

Local reproduction

• Capture workload: replay production traffic (record HTTP traces or use synthetic load), use representative data sets and similar container limits.
• Run under profiler with same flags (GC tuning, LD_PRELOAD if using malloc hooks); run with lowered memory limit to speed reproducing.

Triage / analysis steps

1. Correlate timeline: map memory rises to requests/features/GC logs.
2. Heap snapshots at intervals; compare retained object graphs to find growth sources.
3. CPU/alloc flame graphs to see allocation hot paths.
4. Check native vs. managed memory (FD leaks, caches, thread stacks).
5. Instrument candidate modules, add alloc logging or metrics.

Quick production mitigations

• Increase memory limit or add swap as emergency stop-gap.
• Restart strategies: rolling restarts, OOMScore adjustments, lifecycle probes to auto-restart before OOM.
• Throttle or circuit-break poorly used endpoints; reduce cache sizes/configurable limits.
• Deploy feature flag to disable suspected feature.

Communication to PMs

• Plain summary: “Service leaking memory; OOM risk; root cause unknown yet.” Provide two-track plan:
  • Immediate (hours): mitigations to stop customer impact (increase limits, throttles).
  • Investigation (1–3 days): profiling, targeted fix, and test.
  • Fix & deploy (3–14 days): patch, validation, rollout plan.
    Explain trade-offs: raising limits buys time but increases cost and risk of bigger failures; rolling restarts risk short downtime; disabling features may reduce UX. Commit to specific checkpoints and metrics to decide next steps.