Memory Management & Garbage Collection Questions

Why the heap looks healthy while the container dies. A direct ByteBuffer (from ByteBuffer.allocateDirect) keeps its bytes in native memory outside the garbage-collected heap, so only a tiny wrapper object is on the heap. A container's memory limit (a Linux cgroup, a kernel feature that caps a group of processes) counts the whole process: heap + direct buffers + thread stacks + Metaspace (class metadata) + JIT code + GC bookkeeping. When the total passes the limit the kernel's OOM killer sends SIGKILL (a signal that terminates a process immediately and cannot be caught or handled), with no Java exception and no heap dump. Heap graphs and -Xmx never see it.

Evidence from a run (JDK 21 container, -Xmx512m -XX:NativeMemoryTracking=summary). The program allocates eight 32 MiB direct buffers and prints heap, the direct buffer pool and the process's resident memory (RSS, the physical memory the process occupies), then drops the references, calls System.gc(), and finally keeps allocating until the cap is hit:

import java.lang.management.BufferPoolMXBean;
import java.lang.management.ManagementFactory;
import java.nio.ByteBuffer;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.List;

public class DirectDemo {
    static long rssMiB() throws Exception {
        for (String line : Files.readAllLines(Path.of("/proc/self/status")))   // reading a file through NIO
            if (line.startsWith("VmRSS:")) return Long.parseLong(line.replaceAll("\\D+", "")) / 1024;
        return -1;
    }

    static void report(String label) throws Exception {
        BufferPoolMXBean direct = ManagementFactory.getPlatformMXBeans(BufferPoolMXBean.class).stream()
            .filter(p -> p.getName().equals("direct")).findFirst().get();
        Runtime rt = Runtime.getRuntime();
        System.out.printf("%-28s heapUsed=%4d MiB  maxHeap=%4d MiB  direct(count=%d, used=%4d MiB)  RSS=%4d MiB%n",
            label, (rt.totalMemory() - rt.freeMemory()) >> 20, rt.maxMemory() >> 20,
            direct.getCount(), direct.getMemoryUsed() >> 20, rssMiB());
    }

    public static void main(String[] args) throws Exception {
        report("start");
        List<ByteBuffer> bufs = new ArrayList<>();
        for (int i = 0; i < 8; i++) bufs.add(ByteBuffer.allocateDirect(32 << 20));   // 8 x 32 MiB, native memory
        report("after 8 x 32 MiB direct");
        new ProcessBuilder("jcmd", String.valueOf(ProcessHandle.current().pid()), "VM.native_memory", "summary", "scale=MB")
            .inheritIO().start().waitFor();
        bufs = null;                                                                  // drop every reference
        report("references dropped");
        System.gc();                                                                  // lets the Cleaner free the bytes
        Thread.sleep(500);
        report("after System.gc()");
        List<ByteBuffer> hold = new ArrayList<>();
        try {
            while (true) hold.add(ByteBuffer.allocateDirect(32 << 20));               // keep going until the cap
        } catch (OutOfMemoryError e) {
            System.out.println("OOM: " + e);
        }
    }
}

It printed (the Native Memory Tracking report is abridged to its total and its Other entry, with ... where lines are skipped):

start                        heapUsed=   2 MiB  maxHeap= 512 MiB  direct(count=0, used=   0 MiB)  RSS=  72 MiB
after 8 x 32 MiB direct      heapUsed=   3 MiB  maxHeap= 512 MiB  direct(count=9, used= 256 MiB)  RSS= 330 MiB
Total: reserved=2219MB, committed=597MB
...
-                     Other (reserved=256MB, committed=256MB)
                            (malloc=256MB #10) (at peak)
...
references dropped           heapUsed=   4 MiB  maxHeap= 512 MiB  direct(count=9, used= 256 MiB)  RSS= 335 MiB
after System.gc()            heapUsed=   1 MiB  maxHeap= 512 MiB  direct(count=1, used=   0 MiB)  RSS=  88 MiB
OOM: java.lang.OutOfMemoryError: Cannot reserve 33554432 bytes of direct buffer memory (allocated: 503324672, limit: 536870912)

Two counts need explaining. Eight buffers were allocated, yet count=9: reading /proc/self/status through Files.readAllLines makes the JDK's NIO layer allocate one small temporary direct buffer (8 KiB) and cache it for the thread, so it is the ninth. That cached buffer is also why count=1 and used=0 MiB remain after System.gc() (8 KiB rounds to 0 MiB): the eight 32 MiB buffers are gone and only the cached one is left. A variant that reads the file with a plain BufferedReader printed count=8 and then count=0. The last line comes from the final loop: it holds 15 buffers of 33,554,432 bytes (15 x 33,554,432 = 503,316,480) plus the 8,192-byte cached buffer, which is the allocated: 503,324,672; a 16th buffer would bring the total to 536,879,104, over the limit of 536,870,912 (512 MiB), so it throws.

What it shows: (1) 256 MiB of buffers moved RSS from 72 to 330 MiB while heap stayed at 3 MiB, which is the "healthy heap" illusion. (2) Dropping the references freed nothing (direct used still 256 MiB): the native memory is released only after a garbage collection finds the unreachable wrapper object and runs its cleanup. A service whose heap rarely collects can hold gigabytes of dead buffers. (3) After System.gc() it fell to 0 and RSS returned to 88 MiB. (4) The failed allocation message shows the cap: with no flag set, the limit in this run was 536,870,912 bytes, exactly the 512 MiB -Xmx. The JVM docs only say that if -XX:MaxDirectMemorySize is unset "the JVM chooses the size automatically"; in this run it equalled the maximum heap, so confirm it on your JVM version. The consequence is that a service can legally use heap + direct up to about twice the heap, before adding stacks and Metaspace, and a container limit sized as "heap + a bit" is too small. Note the NMT entry labelled Other held the direct memory (committed=256MB, allocated with malloc); the Total includes it.

Finding where the memory went, in order.

1. Confirm it is a kernel kill and not a Java OOM. dmesg / the orchestrator shows exit code 137 (a process killed by a signal exits with 128 plus the signal number, and SIGKILL is signal 9) and "OOMKilled". A Java OutOfMemoryError: ... direct buffer memory would be a normal exception with a stack trace instead.
2. Compare the pieces: container RSS (cgroup memory usage) vs Runtime heap used vs the BufferPoolMXBean values for "direct" and "mapped" (exposed by JMX, so you can chart them). If RSS is far above heap + direct pool, the memory is elsewhere.
3. Turn on Native Memory Tracking (-XX:NativeMemoryTracking=summary, or detail for call sites; Oracle documents about 5 to 10% overhead). Then jcmd <pid> VM.native_memory baseline, wait for growth, and jcmd <pid> VM.native_memory summary.diff scale=MB. The diff names the growing category: Java Heap, Class (Metaspace), Thread, Code, GC, Other (direct buffers appeared in Other above).
4. If NMT's total stays flat but RSS grows, the leak is in code NMT does not track (non-JVM code such as JNI and third-party native libraries, per Oracle's note that NMT does not track allocations made outside the JVM): use pmap -x <pid> (a Linux tool that lists the process's memory mappings with their sizes) to find large mappings, and a native profiler (for example heap profiling in jemalloc, an alternative malloc implementation that can record which call sites allocate; or async-profiler, a low-overhead sampling profiler with an allocation mode) to find the callers.
5. If the growth is the direct pool, take a heap dump and count java.nio.DirectByteBuffer instances and who holds them. A large number of small-looking wrappers with big capacities is the pattern.

What to change.

1. Reuse instead of allocating. Allocate a few large direct buffers once (or use a pool such as Netty's pooled allocator, the buffer pool of the Netty networking library) and recycle them. Allocating a direct buffer is slow partly because the bytes are zeroed (the JavaDoc guarantees each element is initialised to zero), so churning them is costly; and because the memory is released only after a garbage collection, churn also lets dead buffers pile up.
2. Release deterministically where you can. ByteBuffer has no public close method, so memory goes only when the wrapper is collected. For new code, on Java 22 and later the Foreign Function & Memory API gives Arena.ofConfined(): memory is deallocated the moment the try-with-resources block closes (JEP 454). On older versions, if you adopt a library with reference-counted buffers (Netty), release them in a finally. A leaked reference-counted buffer is a leak the GC cannot fix.
3. Cap it so the failure is a Java exception, not a SIGKILL. Set -XX:MaxDirectMemorySize explicitly and set the container limit above heap + direct cap + Metaspace + thread stacks + headroom (a budget you compute, not a fixed ratio). A JVM OutOfMemoryError with a stack trace is debuggable; an OOM kill is not.
4. Make collections happen. If direct buffers pile up because the heap rarely collects, a smaller young generation or a periodic cleanup is a workaround, not a fix. Check whether your service sets -XX:+DisableExplicitGC: it turns explicit System.gc() calls into no-ops, which removes one way of forcing the cleanup shown above. With that flag added to the demo's command line, direct used stayed at 256 MiB after System.gc() and the final loop hit the cap with the dead buffers still in place.

What would change my choice: if buffers are few and long-lived, none of this matters beyond setting the cap. If the service handles many short requests that each take a buffer, a pool is the fix, and the direct cap is the safety net.

Direct answer

Work from cheap and always-on to expensive and invasive, and match the tool to the question. First, counters and GC logs say whether memory or GC is a problem at all (heap size after each collection, GC time share, allocation rate). Second, an allocation profile says which code creates the garbage. Third, a heap snapshot or dump says what is being retained and who holds it, which is the tool for leaks. Fourth, operating-system numbers (resident memory against the container limit) catch memory the runtime does not count. Every runtime has a tool at each level; the names differ, the questions do not.

Tools by runtime and level of detail

The rows ask the same question in every column, so read down the column of the runtime you operate. A JVM class histogram sits in row 3, not row 2: it counts the objects alive at that moment, so growth between two histograms means retention, while it says nothing about who allocated them. jcmd rates its impact as high (by default it inspects only reachable objects, unless -all is given), so run it on one instance, not casually across a fleet. In Go's pprof, alloc_space is the total bytes each function has allocated since the program started (freed or not, so it tracks the allocation rate), while inuse_space is the bytes still live as of the most recent collection (so it points at retention). A heap profile is a sampled record of which call sites allocated memory.

Detail each gives, in order of increasing detail: counters give totals over time with no attribution; allocation profiles give call sites and types, sampled; a heap dump gives every live object and the reference path from a root to it, at one moment, with a pause and a large file.

Command shapes

Run in Linux containers (JDK 21.0.12, Go 1.23, Python 3.12, Node 22), these commands print the shapes described:

• java -Xlog:gc App prints lines such as Pause Young (Normal) (G1 Evacuation Pause) 24M->4M(64M).
• jstat -gcutil <pid> 500 2 prints a header row (columns including YGC, YGCT, FGC, GCT) and one row per sample.
• jcmd <pid> GC.class_histogram prints instance counts and bytes per class; jcmd <pid> GC.heap_dump -gz=1 /path/heap.hprof.gz wrote a compressed dump (the JDK documentation lists the impact as high and says it requests a full GC unless -all is given).
• jcmd <pid> JFR.start name=r duration=2s filename=r.jfr started a recording; java -XX:StartFlightRecording=filename=rec.jfr,settings=profile App followed by jfr print --events jdk.ObjectAllocationSample --stack-depth 10 rec.jfr printed sampled allocations, each with a weight and a stack.
• GODEBUG=gctrace=1 go run main.go printed lines of the form gc 1 @0.001s 2%: ..., where the Go runtime documentation defines the percentage as the share of time spent in GC since program start. go test -bench . -benchmem -memprofile mem.out printed B/op and allocs/op, and go tool pprof -sample_index=alloc_space -top mem.out ranked functions by bytes allocated.
• python -X tracemalloc or tracemalloc.start() then take_snapshot() and compare_to(old, "lineno") printed a per-line size and count difference, such as size=1065 KiB (+1065 KiB), count=2003 (+2003) for a line that built 1,001 one-kilobyte bytearray objects (the count is consistent with about two allocations per object, the object and its buffer, plus the list). The documentation notes that allocations made before tracing started are not included, and more stored frames cost more memory and CPU.
• node --trace-gc app.js printed Scavenge lines (the young-generation collection) with before and after sizes, for example Scavenge 4.2 (4.9) -> 3.3 (5.9) MB, pooled: 0 MB, 0.19 / 0.00 ms (average mu = 1.000 ...) allocation failure: heap in use before (and allocated in parentheses) -> after, then the pause in milliseconds; mu is the share of time the application rather than the collector was running, so 1.000 means no GC drag; allocation failure is the trigger, a full young space.

Reading one tool's output end to end

jstat -gcutil <pid> 500 3 samples a running JVM every 500 ms, three times. To reproduce it, run this small program, which keeps about 20 MB live and churns short-lived 4 KB arrays through a ring so the JIT cannot optimise the allocations away:

import java.util.ArrayList;
import java.util.List;

public class App {
    static final List<byte[]> keep = new ArrayList<>();
    static final byte[][] ring = new byte[256][];   // short-lived arrays escape into this ring

    public static void main(String[] args) throws Exception {
        for (int i = 0; i < 2000; i++) keep.add(new byte[10_000]);   // about 20 MB stays live
        long end = System.currentTimeMillis() + 15_000;
        long n = 0;
        while (System.currentTimeMillis() < end) {
            ring[(int) (n++ & 255)] = new byte[4096];                 // garbage soon after
            if ((n & 0x7FFF) == 0) Thread.sleep(10);                  // throttle so GC stays a small share
        }
        System.out.println(ring[0].length + keep.size());
    }
}

Start it with java -Xmx256m App & on JDK 21, wait a few seconds, then run jstat -gcutil <pid> 500 3. G1 is the default collector on a machine with at least two CPUs and a reasonable amount of memory; a container limited to one CPU selects the Serial collector instead, and its S0 and S1 columns look different, so add -XX:+UseG1GC to reproduce the table. The block below is one captured run. The absolute numbers differ from run to run and with machine load (on a busy shared machine the same program showed several times more GC time per second, still with no full collections), so read the structure, not the digits:

  S0     S1     E      O      M     CCS    YGC     YGCT     FGC    FGCT     CGC    CGCT       GCT
     -  50.20  69.18  21.66  25.10   5.00    200     0.143     0     0.000     0     0.000     0.143
     -  50.20  47.80  21.66  25.91   5.00    225     0.153     0     0.000     0     0.000     0.153
     -  50.20  24.53  21.66  25.91   5.00    250     0.165     0     0.000     0     0.000     0.165

The columns S0, S1, E, O, M and CCS are utilisation percentages of the survivor spaces, eden, the old generation, metaspace and the compressed class space (the JDK documentation defines them as percentages of the space's current capacity). YGC and YGCT are the count of young collections and their total time in seconds, FGC and FGCT the same for full collections, CGC and CGCT the same for G1's concurrent cycles, and GCT the total GC time. What to read: between samples 1 and 3 (one second), YGC rose from 200 to 250, so 50 young collections ran. GCT rose from 0.143 to 0.165, so in this captured run 0.022 s of that second was spent in GC (about 2%, or roughly 0.44 ms per collection; a quiet machine gives a figure of this kind, a loaded one a larger share). FGC stayed 0, so there were no full collections. E (eden) is only a snapshot at the instant of sampling, so it bounces anywhere between 0 and 100% as eden fills and is emptied; judge it over many samples, not one. O (old generation) stayed flat near 21%, which is the live data (about 20 MB) and the sign that nothing is accumulating. A healthy reading looks like this one: many cheap young collections, no full ones. A worrying one has FGC and FGCT climbing, or O staying high after collections.

Worked investigation: "memory keeps growing"

1. Look at heap size after each collection, not the peak. Flat after-GC occupancy with a high allocation rate means churn: take an allocation profile. After-GC occupancy rising over hours is retention: take two heap dumps or snapshots 30 minutes apart and compare, looking for the types whose counts only increase, then follow the reference path to the root (a static map, a listener list, a cache without a limit).
2. If the runtime heap is flat but the process grows, look off the heap: native memory tracking, direct buffers, thread stacks, memory fragmentation.
3. Fix, then repeat step 1 and confirm the curve is flat.

Pitfalls and judgment

• Dumps stop the world: a JVM heap dump requests a full GC, and dotnet-gcdump collect triggers a generation 2 collection that the documentation warns can suspend the runtime for a long time on a large heap. Do it on one instance taken out of rotation, or accept the pause deliberately.
• Dump size: a dump is about as large as the live heap, so plan disk space and who may see the data (heap dumps contain secrets and personal data).
• Sampling and overhead: allocation profilers sample, so rerun them; tracemalloc is only useful if it started early, and its cost grows with the number of frames stored.
• Start cheap: counters are safe to leave on in production; profiles and dumps are on-demand tools.

Recommendation

Keep GC logs and runtime counters permanently on and graphed, add an allocation profiler you can trigger in production (JFR, Go pprof endpoint), and keep heap dump or snapshot procedures rehearsed for leaks. Pick the next level only when the previous one points at it.

Direct answer

Split the case into three numbers: how much CPU the service spends in garbage collection (GC), how much of the allocation volume a specific fix removes, and what the CPU saved is worth in capacity or latency. Measure the first two, compute the third, then validate the estimate with a prototype before asking for a full commitment. For a product manager the pitch is: "this change frees about X% of our compute, equal to N servers or Y milliseconds off the 99th-percentile (p99) latency, for Z engineer-weeks, and we will know after a two-day prototype whether the number holds."

Step 1: quantify how much of the cost is allocation and GC

1. GC share of CPU. Read it from the runtime: GC logs on a JVM (-Xlog:gc; jstat -gcutil shows cumulative GC time), the percentage field of GODEBUG=gctrace=1 in Go (the Go runtime documentation defines it as the share of time spent in GC since program start), dotnet-counters in .NET. Pause time alone understates the cost of concurrent collectors, which burn CPU on background threads, so also compare per-thread CPU for the GC worker threads against total CPU.
2. Allocation rate and where it comes from. An allocation profile (JFR jdk.ObjectAllocationSample on the JVM, the alloc_space view of a Go heap profile, tracemalloc in Python) ranks allocating call sites by bytes. A "long tail of small allocations" means the ranking is flat, which decides the strategy below.
3. Allocation versus other costs. Compare the profile's allocation-heavy frames with the CPU profile. GC time is only the collector's share; the allocating code also pays for zeroing memory and cache misses, which show up as ordinary application time.

Step 2: estimate the payoff of each fix

For a tracing collector with a fixed heap size, collection frequency is proportional to the allocation rate, so GC CPU scales with the allocation volume you remove. If GC is a fraction g of CPU and the fix removes a fraction a of the bytes allocated, CPU per request goes from 1 to (1 - g) + g(1 - a), and capacity (requests per second at the same CPU) rises by the inverse of that. The alternative, spending memory instead of code, is also computable: GOGC is the percentage of live heap allowed as growth before the next collection, and the Go GC guide says doubling it will double heap memory overheads and roughly halve GC CPU cost. (On a JVM the matching lever is a larger maximum heap, -Xmx, and a larger young generation, so collections run less often.)

# Inputs measured from the service (the numbers below are an example, pinned for this calculation)
gc_cpu_share = 0.18          # fraction of total CPU spent in GC (from GC logs or profiler)
alloc_cut = 0.60             # fraction of allocation volume the fix removes (from an allocation profile)
fleet = 40                   # instances, sized so each runs near its CPU limit
price_per_instance_month = 250.0   # assumed example price in dollars

# With the heap size fixed, GC frequency is proportional to allocation rate, so GC CPU scales with it.
new_gc_share = gc_cpu_share * (1 - alloc_cut)
cpu_per_request_before = 1.0
cpu_per_request_after = (1 - gc_cpu_share) + new_gc_share      # app work unchanged, GC part shrinks
capacity_gain = cpu_per_request_before / cpu_per_request_after - 1
instances_needed = fleet * cpu_per_request_after
saved = fleet - int(instances_needed + 0.9999)                  # whole instances freed
print(f"GC share: {gc_cpu_share:.1%} -> {new_gc_share:.1%}")
print(f"CPU per request: {cpu_per_request_after:.3f} of before; capacity gain {capacity_gain:.1%}")
print(f"instances needed: {instances_needed:.1f} of {fleet}; whole instances saved: {saved}")
print(f"annual saving: ${saved * price_per_instance_month * 12:,.0f}")

# Alternative: spend memory instead of code. Doubling GOGC roughly halves GC CPU (Go GC guide).
gc_half = gc_cpu_share / 2
gain_half = 1 / ((1 - gc_cpu_share) + gc_half) - 1
live_heap_mb = 400
extra_mb = live_heap_mb * (200 - 100) / 100                    # target = live * (1 + GOGC/100)
print(f"GOGC 100->200: GC share {gc_half:.1%}, capacity gain {gain_half:.1%}, +{extra_mb:.0f} MB per instance")

GC share: 18.0% -> 7.2%
CPU per request: 0.892 of before; capacity gain 12.1%
instances needed: 35.7 of 40; whole instances saved: 4
annual saving: $12,000
GOGC 100->200: GC share 9.0%, capacity gain 9.9%, +400 MB per instance

The inputs (18% GC share, a fix that removes 60% of allocation volume, a 40-instance fleet, $250 per instance-month) are example numbers; replace them with your measurements. The heap formula ignores GC roots, which the full formula in the Go guide adds. Every line is checkable: 0.18 x 0.4 = 0.072; 0.82 + 0.072 = 0.892; 1 / 0.892 = 1.121; 40 x 0.892 = 35.7, which rounds up to 36 instances needed, so 4 are freed; 4 x $250 x 12 = $12,000.

Step 3: decide what is worth doing with a flat profile

The percentage cutoffs are rules of thumb, not measured constants. The reasoning behind them is the formula above: the gain cannot exceed g, and the effort scales with the number of sites touched.

Step 4: validate and make the case

Prototype the highest-value fix behind a flag, run the same load on two instances, and compare GC share, CPU per request and p99 latency. If the real GC share drops to about 7%, the linear model holds. If it drops much less, the live set rather than the allocation rate is driving cost, and the plan changes. Tell the product manager three things in business terms: capacity (instances saved, or headroom for the next traffic spike), user-visible latency (collection pauses show up in p99), and risk (the prototype is a two-day experiment with a rollback flag). The same example, written as the pitch itself: "About 18% of our CPU goes to garbage collection. A change that removes 60% of the memory we allocate per request cuts that to about 7%, so each request costs about 11% less CPU and each server handles about 12% more traffic. On our 40 servers that frees 4, roughly $12,000 a year at an assumed $250 per server-month, for an estimated two engineer-weeks (an assumed figure, replace it with the team's estimate). We will run a two-day prototype behind a flag before committing, and we can turn it off in minutes if the number does not hold." The latency clause needs different evidence: the model above predicts CPU, not pauses, so quote the p99 pause time that the GC log already shows as an upper bound on what pause-related latency could improve, and report the measured p99 change after the prototype rather than promising one.

Recommend heap headroom first when memory is cheap: it takes hours and delivers about 9.9% in this example, so it is the baseline any code change must beat; commit engineers to code fixes when the measured top sites would deliver the larger gain or when memory is the constrained resource.

Pitfalls

• Counting only pauses: for concurrent collectors the CPU burned in the background is real capacity.
• Assuming linearity: the model fails if survivors, not garbage, dominate (a large live set), so confirm with the prototype.
• Selling the estimate as a measurement: say which figures were measured (GC share, profile ranking) and which were modelled (the capacity gain).

Short answer. Commit to a plan that separates the two cases by the shape of the heap after full collections, then proves the retaining path with a heap dump, then fixes and verifies with a soak test. A cache converges (rises, then plateaus at its configured limit). A leak does not (the floor keeps rising for the same load).

Step 1: classify the growth from data you already have

• Look at heap after each collection (live data), not heap used. Used heap climbs naturally because the collector runs lazily. Plot old-generation occupancy after full or mixed collections (a mixed collection is a G1 young collection that also reclaims some old-generation regions) over weeks, next to traffic.
• Leak: the post-collection floor rises monotonically, independent of load, until collections run back to back and an OutOfMemoryError follows. Retained cache: floor rises, then flattens near the cache's maximum size.
• Direct test: if the service has a cache-clear or size-reduction switch, use it once. If the floor drops back and then regrows to a plateau, it is a cache. If it barely drops, something else holds the memory.
• In the GC log, watch for the pattern the HotSpot troubleshooting guide walks through: a Full GC that reclaims almost nothing (its example is Pause Full (Ergonomics) 45M->45M(48M)), which it says means either the heap is smaller than the application needs or there is a leak.

Step 2: compare heap contents at two points in time

For most leaks, Steps 1 and 2 (the post-collection floor, then a histogram diff and a heap dump) are what find the cause; JFR, NMT and a profiler are for the cases below where those leave a gap.

The cheapest capture: jcmd <pid> GC.class_histogram (instances and bytes per class). Take two, an hour apart, and diff. Whatever class grows with time is the suspect. Caveat shown in the demo below: the histogram forces a full collection (the log shows Pause Full (Heap Inspection Initiated GC)), so take it on one instance behind a load balancer, not on every replica at once.

For the retaining path you need a heap dump: jcmd <pid> GC.heap_dump filename=/path/heap.hprof, or -XX:+HeapDumpOnOutOfMemoryError set at startup so the dump is waiting after a crash. A dump pauses the application while it is written and produces a file on the order of the heap size, so confirm disk space and run it on a drained instance. Open it in Eclipse MAT or a similar analyzer, sort by retained size (memory freed if the object disappeared), and use dominator tree (what each object exclusively keeps alive) and path to GC roots (the reference chain from a root to the suspect). The chain ends at the thing to fix: a static map, a listener list, a ThreadLocal on a pooled thread, a queue with no consumer. (A ThreadLocal gives each thread its own private value. A pooled thread lives as long as the pool does, so a large value stored in a ThreadLocal and never removed stays reachable from that thread indefinitely.)

Java Flight Recorder (JFR) is the usual lower-impact option for production; the troubleshooting guide describes recording overhead as very low. The dump step that adds root paths is not cheap, though. The guide's recipe is -XX:StartFlightRecording=... to record, jcmd <pid> JFR.dump filename=recording.jfr path-to-gc-roots=true to dump, and jfr print --events OldObjectSample file.jfr to list sampled long-lived objects with their root paths. The guide notes that path-to-gc-roots=true creates overhead similar to a full garbage collection, so take that dump on one instance, as with a heap dump. Because it is sampled, measure its overhead on a canary before leaving it on fleet-wide. A sampling profiler can attribute allocations to call sites: async-profiler (an agent that runs inside the target JVM) does this with its --alloc option, for example asprof -e cpu --alloc 2m -d 30 -f profile.jfr <pid>, which takes one allocation sample per 2 MB allocated on average and writes a recording you can open in a JFR viewer.

Step 3: rule out memory the heap does not hold

If container memory (RSS, the resident set size the OS sees) grows while heap after collection is flat, the leak is off-heap: direct ByteBuffers (buffers whose bytes live in native memory outside the heap), thread stacks, metaspace (the native memory holding loaded class definitions), native libraries. Start the JVM with Native Memory Tracking (NMT, the JVM's own accounting of its native allocations) switched on, -XX:NativeMemoryTracking=summary, then jcmd <pid> VM.native_memory baseline and later VM.native_memory summary.diff to see which category grew. Check the BufferPoolMXBean "direct" pool for direct-buffer counts. Heap size plus those categories should account for RSS; the gap is native code.

Step 4: what to turn on in staging and production

-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=<volume with space>, GC logging (-Xlog:gc*:file=gc.log, rotated), NMT summary mode, JFR available on demand, and the JVM's memory metrics exported to your metrics system (for example through Micrometer to Prometheus), with alerts on the post-collection floor rather than on raw heap used (which is noisy by design).

A leak reproduced and found

import java.util.*;

public class GrowingMapLeak {
    // A static field is a GC root: everything reachable from it stays alive.
    static final Map<Long, byte[]> sessionPayloads = new HashMap<>();
    public static void main(String[] args) throws Exception {
        for (long id = 0; id < 400_000; id++) {
            sessionPayloads.put(id, new byte[8 * 1024]);   // never removed
            if (id % 100 == 0) Thread.sleep(20);
        }
        System.out.println("done: " + sessionPayloads.size());
    }
}

Run: javac -d /tmp/o GrowingMapLeak.java, then java -cp /tmp/o -Xmx256m -Xlog:gc -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/o/leak.hprof GrowingMapLeak (OpenJDK 21.0.12 in a container), plus two jcmd <pid> GC.class_histogram calls about one second apart, give these excerpts (G1 is the default collector on a machine with at least 2 CPUs and roughly 1.8 GB of memory; in a one-CPU or smaller container the JVM picks Serial, and the late full collections then print as Pause Full (Allocation Failure) instead of G1 Compaction Pause, while the histogram-triggered line is the same):

   1:         21413      110345968  [B (java.base@21.0.12.1)     <- first histogram
   1:         26313      150565168  [B (java.base@21.0.12.1)     <- one second later
[3.236s][info][gc] GC(4) Pause Full (Heap Inspection Initiated GC) 109M->107M(254M) 9.178ms
[7.358s][info][gc] GC(15) Pause Full (G1 Compaction Pause) 255M->246M(256M) 5.198ms
java.lang.OutOfMemoryError: Java heap space
Dumping heap to /tmp/o/leak.hprof ...

[B is byte[]: instance count and bytes both grow between snapshots and the full collections reclaim almost nothing (255M to 246M). The dump's path to GC roots would end at the static sessionPayloads field of GrowingMapLeak, which is a root.

Step 5: fix and verify

Fix by cause: bound the cache (maximum entries or bytes plus expiry, so it has a ceiling), remove listeners or registrations on close, clear ThreadLocals in pooled threads, close consumers and queues. Verify: deploy the fix to one instance, replay or mirror production load for several multiples of the time it took the leak to show (hours, not minutes), and confirm the post-collection floor goes flat while the class histogram no longer shows the suspect growing. Keep the dump-on-OOM flag on so a regression leaves evidence.

If the result is a cache that is merely too large for the pod, the fix is a smaller limit or a bigger heap, and the verification is the plateau sitting comfortably below the heap limit. Remember that jcmd, jmap, JFR and NMT are JDK tools; on other runtimes the same sequence applies with that runtime's heap snapshot tool.

Short answer. Tell it as one specific incident in the order situation, the actions you personally took, the root cause, the fix and its measured result, and the guard that stopped a repeat, and be honest about what you got wrong. The example below is a model you can adapt; replace every detail and number with your own, because an interviewer will probe them.

What a strong story contains

1. Situation with a measurable symptom. Not "memory was high", but a number and a consequence.
2. Your actions in sequence, with the tool and what each told you. "We looked at it" is weak; "I took two heap histograms ten minutes apart" is strong.
3. A root cause stated precisely: which object, held by which reference, added by which change.
4. The fix, plus how you proved it worked on the same metric that showed the problem.
5. Prevention and a reflection: a detection gap closed, and one thing you would do differently.

Example answer (first person)

Situation. I was on call for a Java order-API service. After a Tuesday release its instances were being restarted by the orchestrator about every nine hours, and each restart dropped in-flight requests, which showed up as a burst of 5xx errors. The JVM heap was capped at 4 GB inside a 6 GB container, so the limit that was hit was the Java heap, not the kernel's container limit: the JVM threw OutOfMemoryError: Java heap space and the instance was restarted. On the dashboard, the heap after each full collection climbed from about 1.2 GB to the 4 GB ceiling over those nine hours. That is a rise of about 2.8 GB, roughly 310 MB per hour. A rising post-collection floor, as opposed to a rising raw number, is what told me this was a leak and not just a busy period.

Actions.

1. Contained it. I did not start with the cause. We shortened the interval between planned rolling restarts to six hours (more frequent than the nine hours it took a leaking instance to run out), so each instance was restarted on our schedule before it reached the ceiling, which stopped the unplanned crashes and the 5xx bursts while I investigated.
2. Narrowed it to one object type. On an instance I had taken out of the load balancer, I ran jcmd <pid> GC.class_histogram twice, ten minutes apart. One entry, a metrics class (Micrometer's timer, the object that records how long an operation took), had grown by tens of thousands of instances while everything else was flat. This histogram reports live objects by class, so growth in it is retention, not just allocation.
3. Found who was holding them. I took a heap dump with jcmd <pid> GC.heap_dump from that isolated instance (the jcmd docs list the impact as high and say it triggers a full collection unless -all is given, which is why I used an instance with no traffic) and opened it in Eclipse MAT, a heap-dump analyser. Its dominator tree lists objects by retained size, the memory that would be freed if that object disappeared (an object dominates another when every reference path from a GC root to the second goes through the first); it showed the metrics registry, the in-memory object that holds every meter the application has created, retaining most of the heap. "Path to GC roots" shows the chain of references that keeps one object alive, and it ended at the registry's internal map of meters.
4. Tied it to a code change. Reading the diff of the release, a new timer had been created with a tag built from the order ID. A tag is a label on a metric, and each distinct combination of metric name and tag values is its own meter series, which Micrometer keeps as a meter object in the registry for the life of the process. Cardinality is the number of distinct series, and an order ID has one value per order, so the cardinality grows without bound. In the dump those meters were all still registered and referenced by the registry, so every order added an entry that was never released. Micrometer's own documentation warns that tag values from user-supplied sources can blow up a metric's cardinality and should be bounded. The numbers fit: the service handled about 50 orders per second, which is 180,000 an hour and 30,000 new meters in each ten-minute histogram interval (the "tens of thousands" above), and 310 MB per hour divided by 180,000 orders is about 1.7 KB retained per order, a plausible size for a timer with its tags and registry entry. (These figures are illustrative; use your own.)

I should say that, after containing it and before the histogram in step 2, I spent a while on a flame graph (a chart of the call stacks sampled over time, where wider boxes mean more samples), and it did not help: an allocation flame graph shows where memory is allocated, which was a busy, unremarkable code path, not what stays retained. The histogram and the dominator tree were what answered the question.

Fix. I changed the tag to a bounded value (the order's status and the customer tier, a few dozen combinations at most) and moved the order ID to the log line, where it belongs. I also added a Micrometer meter filter (MeterFilter.maximumAllowableMetrics, which denies any meter after the registry reaches a set count) so a future mistake would be capped instead of fatal.

Result and monitoring. I deployed it to one canary instance first and watched the post-collection heap floor. It stayed near 1.3 GB over 72 hours of normal traffic where the old build would have crashed about eight times (72 hours / 9 hours per leak cycle) without our six-hour restarts. I then rolled it out and put the restart interval back to normal. To catch this class earlier I added an alert on the slope of the post-collection heap floor (not on the usage percentage) and a code-review checklist line: "does any new metric tag or map key have unbounded values?".

What I would do differently. The release had no soak test, so a nine-hour leak could not show in our twenty-minute pre-release load test. I proposed and later ran a four-hour soak stage on every release candidate. I would also have pulled the heap dump as soon as the rising floor appeared on the dashboard, which was hours before the first restart, instead of detouring through the flame graph.

Common mistakes when telling this story

• Claiming a result with no metric ("it got better").
• Saying "we" for every step so your own contribution is invisible.
• Naming tools without saying what each one told you.
• A neat story with no mistake in it, which reads as rehearsed.
• Skipping detection and prevention, and ending at "I fixed the bug".

If you have no production leak story

Use the closest real one (a staging or test-environment leak, a leak you found in a colleague's code review, or a memory problem in a side project) and say plainly where it ran. A smaller true story with real steps beats a large invented one.