Memory Management & Garbage Collection Questions

Short answer. Yes. A garbage collector frees objects the program can no longer reach; it cannot free objects the program still references but will never use again. A leak in a managed language is therefore unintended reachability: something with a long life (a static field, a global cache, a listener list, a thread) keeps a reference to something that should have been dropped.

How it happens

• Unbounded cache or map. Entries are added per request and never evicted.
• Registrations never removed. A long-lived publisher holds a listener, which holds a whole screen or request context (on mobile, a leaked screen object can pin its entire view tree: see the mobile section below).
• Pooled threads with thread-local data. The thread is a root and lives as long as the pool.
• Native resources. Memory behind a handle (direct buffers, which are Java byte buffers whose memory lives outside the garbage-collected heap, plus files and sockets) the GC does not see or only frees late.

CPython (the standard Python interpreter) frees mostly by reference counting: each object keeps a count of the references to it and is freed the moment the count reaches zero. The same rule applies: a reference held from a module-level object keeps the object alive, and reference cycles (objects that point at each other, so neither count reaches zero) wait for the cyclic collector, a separate periodic pass that finds and frees such groups.

Real leak versus normal growth

Normal growth: heap used climbs between collections and falls when one runs. Collectors take their time, so a process can look large and be healthy. The signal is the low point after collection:

• Real leak: the low points keep rising, with the same load, until collections thrash.
• Normal growth: low points settle on a plateau (a warm cache, a loaded dataset).
  Then confirm by comparing two snapshots and reading which allocation site or class grows.

A Python demonstration of the compare-two-snapshots method with tracemalloc (standard library):

import tracemalloc

_cache = {}          # module-level: reachable for the life of the process


def handle_request(i):
    _cache[i] = bytearray(10_000)       # added per request, never evicted


tracemalloc.start()
for i in range(1_000):
    handle_request(i)
snap1 = tracemalloc.take_snapshot()
for i in range(1_000, 3_000):
    handle_request(i)
snap2 = tracemalloc.take_snapshot()

top = snap2.compare_to(snap1, "lineno")[0]
print("biggest growth between snapshots:")
print(" ", top.traceback[0].filename.split("/")[-1], "line", top.traceback[0].lineno,
      "size_diff_MB=%.1f" % (top.size_diff / 1e6))

Run with python -I leak_vs_growth.py on python:3.12-slim (Python 3.12.15). Output:

biggest growth between snapshots:
  leak_vs_growth.py line 7 size_diff_MB=20.2

Two thousand more requests added 20.2 MB at line 7 of the listing (counting import tracemalloc as line 1), the _cache[i] = ... store. That line is the leak because nothing removes entries; the same code with an eviction limit would show a size_diff that stops growing. The size matches the arithmetic: 2,000 requests x 10,000 bytes is 20.0 MB, plus a little container overhead.

On the JVM the equivalent tools are jcmd <pid> GC.class_histogram and heap dumps (with -XX:+HeapDumpOnOutOfMemoryError), per the JDK troubleshooting guide.

The C++ contrast

C and C++ leak differently: an allocation with no matching free. new int[1000] with no delete[] loses the only pointer, and nothing ever reclaims it, even though the program can no longer reach it. A garbage collector would reclaim that case; the managed-language leak is the opposite case, where a pointer is still held.

#include <cstdio>

int main() {
    int* buf = new int[1000];   // never deleted
    buf[0] = 1;
    std::printf("%d\n", buf[0]);
    return 0;
}

Built with g++ -g -fsanitize=address in gcc:14 and run, AddressSanitizer's leak checker (LeakSanitizer) reports at exit:

==15==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 4000 byte(s) in 1 object(s) allocated from:
    #0 0xffffb0c94b9c in operator new[](unsigned long) (/usr/local/lib64/libasan.so.8+0xd4b9c)
    #1 0x400910 in main /w/cpp_missing_delete.cpp:4
    (frames #2 to #4, in libc and the program entry point, omitted)

SUMMARY: AddressSanitizer: 4000 byte(s) leaked in 1 allocation(s).

The addresses and process id differ between runs and machines; the 4000 bytes (1000 ints of 4 bytes) and the file and line do not.
Valgrind's memcheck (a tool that runs the program on a simulated CPU and reports leaks) serves the same purpose. These are optional tools you can use instead of or alongside the compiler flag; both find unreachable-but-allocated memory, which is a different test from the heap-snapshot method for reachable growth. In modern C++ the usual fix is to stop calling new and delete by hand and use smart pointers (objects such as std::unique_ptr that call delete automatically when they go out of scope) and containers.

The same leak on a phone

Mobile apps follow the same rule: a long-lived object holds a reference to a short-lived screen.

• Android (Kotlin or Java). A destroyed Activity (one screen) stays in memory if a static field, a singleton, a registered listener or a running background task still references it, and with it its entire view tree and bitmaps. LeakCanary, a library added as a debugImplementation dependency so it only exists in debug builds, watches destroyed Activities and Fragments and reports the reference chain that keeps one alive. Android Studio's Memory Profiler can capture a heap dump and its class filter "Show activity/fragment leaks" lists instances that should be gone.
• iOS (Swift). Swift uses automatic reference counting, the same count-the-references idea as CPython. Two objects that hold strong references to each other, or a closure stored on an object that captures self strongly, form a retain cycle and are never freed; the usual fix is a weak capture such as [weak self]. Xcode's Memory Graph Debugger shows live objects and the references between them and flags suspected leaks, so you can look for leaked instances in the object list and follow their reference arrows.
• Without a production heap dump, reproduce the leak in a debug build: open and close the suspect screen about ten times, force a collection or take a heap snapshot, and count how many instances of that screen class are still alive. Zero or one is healthy; one per visit is the leak, and the reference chain from the tool names the holder.

Takeaway

GC removes the unreachable-leak class and keeps the reachable-leak class. Decide with post-collection heap over time, then name the retaining reference with a snapshot diff or heap dump.

Short answer. Memory that keeps climbing in a Go service has three look-alike causes, and one tool, the pprof profiler, separates them if you ask three different questions: how many goroutines exist and where are they parked (goroutine leak), what is live on the heap right now and who allocated it (retention, including slices that pin big arrays), and how much is allocated in total versus how much stays live (plain allocation churn, which is usually not a leak at all). Three Go basics matter throughout. A goroutine is Go's lightweight thread, started with go f(); each one holds a stack (small at first, it grows) and keeps alive everything its stack references. A channel is a typed queue goroutines use to pass values: an unbuffered channel (capacity 0) makes a send wait until another goroutine receives, while a buffered channel accepts sends until its capacity is full. A slice is a small header (a pointer, a length and a capacity) describing a window onto a backing array; a slice of 1 KB taken from a 1 MiB array still points into that whole array, so the array cannot be freed. The garbage collector (GC) only frees unreachable memory, and a goroutine blocked forever is never unreachable, so its stack and everything it points to stay.

Step 1: a quick triage that points at the right suspect

Export runtime.NumGoroutine() and the runtime's memory stats as metrics (the runtime/metrics package or your metrics library), then compare three signals over an hour:

If the live heap after GC is flat and the process memory only looks high, you may have a GC pacing question (see the pacing bullet in Step 3), not a leak.

Step 2: reproduce the three causes and read the profiles

This program creates all three on purpose, then prints the numbers a metrics dashboard would show and writes a heap profile and a goroutine profile:

package main

import (
	"fmt"
	"os"
	"runtime"
	"runtime/pprof"
	"strings"
)

var sink []byte
var retained [][]byte // slice retention: keeps 1 KB slices, each pinning its 1 MB backing array

// goroutine leak: the sender gives up, nobody ever receives, the goroutine blocks forever.
func leakyLookup(i int) {
	size := 0
	if os.Getenv("FIXED") == "1" {
		size = 1 // buffered: the send completes even if nobody receives
	}
	results := make(chan int, size)
	go func() {
		results <- i * 2 // unbuffered: blocks forever if the caller has returned
	}()
	// caller "times out" and returns without reading results
}

// slice retention: we only need 1 KB but the sub-slice pins the whole 1 MB array.
func keepHeader(i int) {
	big := make([]byte, 1<<20)
	big[0] = byte(i)
	if os.Getenv("FIXED") == "1" {
		retained = append(retained, append([]byte(nil), big[:1024]...)) // copy 1 KB, release the 1 MB array
		return
	}
	retained = append(retained, big[:1024])
}

// churn: allocates a lot, retains nothing.
func churn() int {
	n := 0
	for i := 0; i < 20000; i++ {
		b := make([]byte, 10_000)
		b[0] = 1
		sink = b // overwritten every iteration; only the latest survives
		n += len(b)
	}
	return n
}

func stats(label string) {
	runtime.GC()
	var m runtime.MemStats
	runtime.ReadMemStats(&m)
	fmt.Printf("%-18s goroutines=%-5d HeapInuse=%6.1f MiB TotalAlloc=%7.1f MiB NumGC=%d\n", label, runtime.NumGoroutine(), float64(m.HeapInuse)/(1<<20), float64(m.TotalAlloc)/(1<<20), m.NumGC)
}

func main() {
	stats("start")
	for i := 0; i < 5000; i++ {
		leakyLookup(i)
	}
	stats("after goroutine leak")
	for i := 0; i < 100; i++ {
		keepHeader(i)
	}
	stats("after slice retain")
	for i := 0; i < 5; i++ {
		churn()
	}
	stats("after churn")

	f, _ := os.Create("heap.pb.gz")
	pprof.Lookup("heap").WriteTo(f, 0)
	f.Close()
	var sb strings.Builder
	pprof.Lookup("goroutine").WriteTo(&sb, 1)
	fmt.Println(strings.Join(strings.Split(sb.String(), "\n")[:6], "\n"))
}

Run with go run . in the golang:1.23 container (the program passes go vet), it prints lines like these:

start              goroutines=1     HeapInuse=   0.4 MiB TotalAlloc=    0.2 MiB NumGC=1
after goroutine leak goroutines=5001  HeapInuse=   3.7 MiB TotalAlloc=    3.6 MiB NumGC=3
after slice retain goroutines=5001  HeapInuse= 103.8 MiB TotalAlloc=  103.6 MiB NumGC=8
after churn        goroutines=5001  HeapInuse= 103.8 MiB TotalAlloc= 1080.2 MiB NumGC=16

(The program calls runtime.GC() before each reading, so HeapInuse is the live heap, and the NumGC column includes those forced cycles. TotalAlloc is stable run to run; HeapInuse moves by a few tenths of a MiB and NumGC by a few, so read the pattern, not the digits.) HeapInuse is the bytes in heap spans that hold at least one object, and TotalAlloc is the cumulative bytes ever allocated, so it only grows. Reading it as an operator would:

1. Goroutine leak. goroutines jumped 1 to 5,001 after 5,000 calls. Each caller returned without reading its unbuffered channel, so each helper goroutine is stuck on results <- i * 2 forever. The goroutine profile (pprof.Lookup("goroutine") written with debug=1, or /debug/pprof/goroutine?debug=1 from net/http/pprof) groups goroutines with identical stacks and prints a count for each, whereas debug=2 prints every goroutine's full stack separately. The program's own debug=1 dump prints six lines; the first three are shown here (the rest is the profiler's own goroutine):

   goroutine profile: total 5001
   5000 @ 0x7a108 0x14dc0 0x149f8 0xd7760 0x81414
   #	0xd775f	main.leakyLookup.func1+0x2f	/w/main.go:22
   

   That is 5,000 goroutines sharing one stack, parked at line 22, the blocked send (addresses and the path depend on the build). The heap barely moved (0.4 to 3.7 MiB) because a goroutine's stack is accounted separately from HeapInuse: the runtime.MemStats documentation lists StackInuse as its own figure for stack spans. With real payloads captured in the closure (the function value plus the variables it captured) the heap would grow too.
2. Slice retention. 100 calls kept big[:1024], a 1 KB view of a 1 MiB array, and HeapInuse rose by about 100 MiB: the sub-slice holds a pointer into the backing array, so the whole array stays live. The heap profile with -sample_index=inuse_space (live bytes at the last GC; alloc_space would be cumulative) pointed straight at it. go tool pprof -sample_index=inuse_space -top heap.pb.gz printed a short File and Time header and then rows like these (excerpt; the megabytes and percentages shift a little between runs, and further rows follow, including one for main.leakyLookup itself):

   Type: inuse_space
   Showing nodes accounting for 101.15MB, 99.02% of 102.15MB total
         flat  flat%   sum%        cum   cum%
      97.15MB 95.10% 95.10%    97.15MB 95.10%  main.keepHeader
          4MB  3.92% 99.02%        4MB  3.92%  runtime.malg
   

   flat is the bytes attributed to the function itself, cum includes what it called, and sum% is the running total of flat%. One function holds nearly all of the live heap (about 95% or more; the share varies slightly between runs). (runtime.malg is the runtime allocating goroutine structures for the 5,000 leaked goroutines.) The Go GC guide describes exactly these four heap-profile views: inuse_space, inuse_objects, alloc_space, alloc_objects.
3. Churn. Five calls of churn() allocated about 977 MiB (TotalAlloc went 103.6 to 1,080.2) yet HeapInuse stayed at 103.8 MiB. Nothing leaked: the cost is GC cycles and CPU. This is the case where alloc_space or a CPU profile shows the hot allocator and the fix is to allocate less (reuse buffers, preallocate slice capacity, avoid conversions in hot loops), not to hunt for a holder.

Running the same program with FIXED=1 go run . switches on both fixes (a buffered channel of size 1, and a 1 KB copy instead of a sub-slice). A typical run:

start              goroutines=1     HeapInuse=   0.4 MiB TotalAlloc=    0.2 MiB NumGC=1
after goroutine leak goroutines=1     HeapInuse=   2.0 MiB TotalAlloc=    2.0 MiB NumGC=2
after slice retain goroutines=1     HeapInuse=   1.8 MiB TotalAlloc=  102.1 MiB NumGC=16
after churn        goroutines=1     HeapInuse=   1.8 MiB TotalAlloc= 1078.7 MiB NumGC=67

What is stable is goroutines=1 and HeapInuse of a few MiB instead of about 104. HeapInuse ends at a few MiB or less and the final NumGC ranges from several dozen to over a hundred between runs, because the forced collections and the timing of the background ones differ.

Step 3: confirm on the real service

• Expose net/http/pprof on an internal-only port (it registers handlers under /debug/pprof/; never on the public listener, because profiles reveal internals).
• Take two goroutine and two heap profiles ten minutes apart and diff them: in go tool pprof -base first.pb.gz second.pb.gz, -base subtracts the first profile from the second, so only what grew is shown. Whatever grows between them is the leak. Start with goroutine?debug=1, which groups goroutines by identical stack with a count on each (use debug=2 when you need every goroutine's full stack, laid out like a panic dump), and look for thousands of goroutines sharing one stack that is blocked on a channel operation, a lock, or a network read with no deadline.
• If goroutines and live heap are both flat but the container still hits its limit, look at pacing: Go's GOGC defaults to 100 (collect when new allocation equals the live data after the last collection), so the heap can reach about twice the live size, and GOMEMLIMIT is a soft limit that makes the collector work harder near a ceiling without guaranteeing it, per the Go docs.

Step 4: what I would change

• Goroutine leak: give every goroutine a way out. Pass a context.Context (a value carrying a cancellation signal and deadline) and select (wait on several channel operations at once) on ctx.Done() (a channel that closes when the caller cancels) when sending or receiving, or use a buffered channel of size 1 so a lone send always completes. In the program, setting FIXED=1 makes the channel buffered, and the goroutine count read 1 after the leak loop (a goroutine whose send has just completed may still be exiting when the count is read, so the count can briefly read slightly higher).
• Slice retention: copy what you keep: append([]byte(nil), big[:1024]...). With FIXED=1 the same program ends with HeapInuse of a few MiB at most instead of about 104 MiB.
• Churn: reduce allocation on the hot path with sync.Pool (a cache of reusable objects the runtime may empty at any collection) or reused buffers only where the profile shows it matters, and measure the change in TotalAlloc per request, since a pool adds complexity and can hold memory longer.
• Prevent recurrence: a test that asserts the goroutine count returns to its baseline after a request burst (or a goroutine-leak checker such as the goleak library, which fails a test if goroutines are still running when it ends) and an alert on goroutine count slope.

Recommendation

Start with the goroutine count, because it is one number and it is cheap: if it climbs, fix that first, since each leaked goroutine also pins whatever its closure captured. If it is flat, diff two inuse_space heap profiles. Only when live heap is flat do you treat the problem as churn and tune allocation or GOGC and GOMEMLIMIT. What would change the order: a service that already exports goroutine count and shows it flat should skip straight to the heap diff.

Short answer. I would build it on Java 21 with the generational ZGC collector, a 6 GiB heap inside the 8 GiB container (75%), a hot path that allocates close to nothing per request, and large payload buffers either reused or held off-heap. The 10 ms p99.9 target means 999 of every 1,000 requests must finish within 10 ms, and only the slowest one in 1,000 is allowed to take longer, so a single long pause or a burst of allocation stalls breaks it. Everything below follows from three questions: what is the memory budget, what does a request allocate, and what does the collector do while requests are running. p99.9 is the latency that 99.9% of requests beat. Allocation pressure is the rate at which the program creates objects the collector must later reclaim. Off-heap memory is memory the program uses directly (for example Java direct buffers) that the garbage collector (GC) does not scan or move.

1. The memory budget: split 8 GiB before choosing a heap

The container limit covers more than the heap, and when the total passes the limit the kernel kills the process (an out-of-memory kill), which is far worse than a pause. Here is the budget I would start from. The non-heap numbers are assumptions to replace with measurements of the real process under load:

The five rows add up to 8,192 MiB = 8 GiB. In a container the JVM's default maximum heap is only 25% of available memory (-XX:MaxRAMPercentage, default 25 per the java command documentation), which would give a 2 GiB heap here, so set the heap explicitly. I run -Xms equal to -Xmx so the heap does not resize under load, and -XX:+AlwaysPreTouch so the operating system's page-faulting for heap memory happens at start-up and not during requests (by default pages are committed as the application uses them, per the documentation). Cost: slower start and the full 6 GiB is resident from the beginning.

How I size the heap itself. Measure the live set, the memory still reachable after a full collection, under peak load. Suppose it is 1.5 GiB. Then a 6 GiB heap is 25% occupied after collection, which leaves a concurrent collector plenty of room to work while requests keep allocating. The rule I would apply: the post-collection floor stays well under half the heap (a starting rule to verify in the GC log, not a standard); if live data grows toward that, add memory or cut the live set instead of squeezing the heap.

2. Choose the collector for the 10 ms target

A stop-the-world pause halts all request threads. The default collector in HotSpot on server-class machines (at least 2 CPUs and about 1.8 GB; a container with one CPU gets Serial), G1, has a pause-time goal whose default Oracle documents as about 200 ms. That is a target, not a guarantee, and it is 20 times the whole 10 ms budget, so I would not rely on G1 here without evidence. ZGC, per Oracle's collector guide, provides maximum pause times under a millisecond independent of heap size, and on JDK 21 it is enabled with -XX:+UseZGC -XX:+ZGenerational. The cost is more CPU spent on concurrent work and a need for spare heap, so I would verify under realistic load that the log shows no allocation stalls (periods when the application must wait for the collector to free memory) and that throughput is acceptable. If the team cannot move off G1, the fallback is to keep the heap well above the live set, keep allocation low (below) and verify the pause list against the budget.

Go version of the same design. If the service is in Go, the knobs differ. GOGC defaults to 100 (collect when new allocation equals the live data after the last collection), so the heap can reach about twice the live set. GOMEMLIMIT is a soft memory limit that covers the Go heap and other runtime-managed memory but excludes memory held in other languages (for example via cgo), per the Go runtime documentation, and the runtime makes no guarantee it will hold it. I would set it with margin below the container limit, for example 7,000 MiB for the 8 GiB container. The arithmetic: 8 GiB is 8,192 MiB, so 7,000 MiB is 7,000 / 8,192 = 85.4%, which leaves 1,192 MiB for what the limit does not govern: the Go binary's own space, operating-system kernel memory held on behalf of the process, and memory allocated by C code called through cgo (Go's mechanism for calling C libraries). Goroutine stacks are managed by the Go runtime, so they are counted inside the limit. Verify against the container's real memory use.

3. Keep allocation overhead in check

Every byte allocated on the request path becomes work for the collector. The effect is easy to measure. This program handles requests that need an 8 KB scratch buffer, once allocating a new buffer per request and once reusing a per-thread buffer, and reads the thread's allocated bytes from the JVM:

import com.sun.management.ThreadMXBean;
import java.lang.management.GarbageCollectorMXBean;
import java.lang.management.ManagementFactory;

public class BufferReuseDemo {
    static final int BODY = 8_192;
    static final ThreadLocal<byte[]> REUSED = ThreadLocal.withInitial(() -> new byte[BODY]);
    static long checksum;

    // Naive: a fresh 8 KB buffer for every request.
    static void handleNaive(int seed) {
        byte[] buf = new byte[BODY];
        fill(buf, seed);
        checksum += sum(buf);
    }

    // Reused: one buffer per worker thread, cleared by overwrite. Safe only because
    // the buffer never escapes the request and is never handed to another thread.
    static void handleReused(int seed) {
        byte[] buf = REUSED.get();
        fill(buf, seed);
        checksum += sum(buf);
    }

    static void fill(byte[] b, int seed) { for (int i = 0; i < b.length; i++) b[i] = (byte) (seed + i); }
    static long sum(byte[] b) { long s = 0; for (byte x : b) s += x; return s; }

    static void run(String name, boolean reuse) {
        ThreadMXBean tm = (ThreadMXBean) ManagementFactory.getThreadMXBean();
        long tid = Thread.currentThread().getId();
        long gcBefore = ManagementFactory.getGarbageCollectorMXBeans().stream().mapToLong(GarbageCollectorMXBean::getCollectionCount).sum();
        long allocBefore = tm.getThreadAllocatedBytes(tid);
        int n = 300_000;
        for (int i = 0; i < n; i++) { if (reuse) handleReused(i); else handleNaive(i); }
        long allocated = tm.getThreadAllocatedBytes(tid) - allocBefore;
        long gcs = ManagementFactory.getGarbageCollectorMXBeans().stream().mapToLong(GarbageCollectorMXBean::getCollectionCount).sum() - gcBefore;
        System.out.printf("%-7s requests=%d allocated_per_request=%d bytes collections=%d%n", name, n, allocated / n, gcs);
    }

    public static void main(String[] args) {
        run("warmup", false); run("warmup", true);   // let the JIT compile first
        run("naive", false);
        run("reused", true);
        System.out.println("checksum " + checksum);
    }
}

In the demo, a ThreadLocal is a Java object that gives each thread its own private copy of a value, here one 8 KB buffer per thread, so no two threads ever share it. Run in eclipse-temurin:21-jdk with java -Xmx64m BufferReuseDemo (the JIT-warmup lines come first; ignore them):

naive   requests=300000 allocated_per_request=8208 bytes collections=63
reused  requests=300000 allocated_per_request=0 bytes collections=0

The naive path allocated 8,208 bytes per request (the 8,192-byte array plus a 16-byte header), about 2.46 GB over the 300,000 requests, which cost 63 collections in the small 64 MB heap on a multi-CPU machine where the JVM picked G1 (the collection count depends on the collector and heap, and a container with a single CPU or a small memory limit, where the JVM selects the Serial collector, prints a different number; the 8,208 bytes per request did not change); the reused path allocated nothing and triggered none. Your real service allocates more than a loop does, so run the same measurement on the real handler with an allocation profiler. The mechanics and trade-offs:

• Buffer reuse removes the allocation, but a reused buffer must never escape the request or be shared across threads, and a bug here corrupts data silently. Keep the reuse local to one function and test it.
• On-heap versus off-heap. Large, long-lived or I/O buffers can live off-heap as direct buffers, which the collector does not scan. The cost is that you manage lifetimes yourself and cap them (-XX:MaxDirectMemorySize), because off-heap memory does not count against -Xmx but does count against the 8 GiB container limit. Keep small, short-lived objects on-heap, where allocation is cheap.
• Warm pools. A freshly started instance runs slower until the JIT (just-in-time compiler) has compiled the hot paths and caches are filled, so the first requests can miss the 10 ms budget. Keep a new instance out of the load balancer until a warm-up script of synthetic requests has run, and size the connection and thread pools at start-up so they are not growing under load.
• Cut the live set: bound caches, avoid per-request object graphs held by long-lived structures, and avoid very large arrays on the hot path.

4. Verify the design

1. Load-test at peak plus margin for long enough to see the tail: to observe about 100 slower-than-p99.9 requests you need 100 / 0.001 = 100,000 requests, which is about 3.3 minutes at 500 requests per second per instance.
2. Read the GC log (-Xlog:gc:file=gc.log): longest pause, any allocation stalls, the post-collection heap floor.
3. Check the container's total memory (not just the heap) against 8 GiB at the end of a soak test.
4. Alert on p99.9 and on the heap floor in production.

Recommendation

Generational ZGC with a 6 GiB heap, a measured non-heap budget and near-zero allocation on the hot path. What would flip it: if the live set exceeds about 3 GiB, either the container must grow or the data must move off-heap or to a cache tier; if ZGC's CPU overhead breaks the cost budget, accept G1 with a larger heap and a stricter allocation diet and re-measure the tail.

What tracemalloc is. It is a module in the Python standard library that records, for every block of memory Python's own allocator hands out, the call stack that allocated it. You take two snapshots (frozen copies of that record) at different times and subtract them: the lines whose allocations grew between the snapshots are your suspects.

How to turn it on. Three equivalent ways, from the Python docs: the environment variable PYTHONTRACEMALLOC=25, the command line python -X tracemalloc=25, or tracemalloc.start(25) in code. The number is how many stack frames to keep per allocation (default 1; more frames name the caller, not just the allocating line, at a higher cost). Starting it from the environment captures allocations made during imports; starting it later captures only what happens afterwards.

A runnable example (the service leaks by appending to a module-level list on every request):

import tracemalloc

_audit_log = []            # BUG: grows forever, one entry per request


def build_response(user_id):
    payload = {"user": user_id, "items": list(range(50))}
    return payload


def handle_request(user_id):
    response = build_response(user_id)
    _audit_log.append(dict(response))     # the allocation site we want tracemalloc to name
    return response


tracemalloc.start(10)                       # keep 10 frames per allocation

for i in range(1_000):                      # warm-up so caches and imports are not in the diff
    handle_request(i)
before = tracemalloc.take_snapshot()

for i in range(20_000):                     # the traffic window we are investigating
    handle_request(i)
after = tracemalloc.take_snapshot()

print("== top 3 by lineno (size_diff) ==")
for stat in after.compare_to(before, "lineno")[:3]:
    print(stat)

print("\n== same growth with a call stack ==")
top = after.compare_to(before, "traceback")[0]
print(f"{top.count_diff:+d} blocks, {top.size_diff / 1024:+.0f} KiB")
for line in top.traceback.format():
    print(line)

current, peak = tracemalloc.get_traced_memory()
print(f"\ntraced now = {current / 1e6:.1f} MB, peak = {peak / 1e6:.1f} MB, "
      f"tracemalloc's own overhead = {tracemalloc.get_tracemalloc_memory() / 1e6:.1f} MB")

Output on python:3.12-slim (Python 3.12.15):

== top 3 by lineno (size_diff) ==
/w/tracemalloc_snapshot_diff.py:7: size=9352 KiB (+8906 KiB), count=42000 (+40000), average=228 B
/w/tracemalloc_snapshot_diff.py:13: size=3942 KiB (+3754 KiB), count=42001 (+40000), average=96 B
/w/tracemalloc_snapshot_diff.py:23: size=617 KiB (+617 KiB), count=19743 (+19743), average=32 B

== same growth with a call stack ==
+40000 blocks, +8906 KiB
  File "/w/tracemalloc_snapshot_diff.py", line 24
    handle_request(i)
  File "/w/tracemalloc_snapshot_diff.py", line 12
    response = build_response(user_id)
  File "/w/tracemalloc_snapshot_diff.py", line 7
    payload = {"user": user_id, "items": list(range(50))}

traced now = 14.3 MB, peak = 14.3 MB, tracemalloc's own overhead = 5.4 MB

How to read a diff line. file:line: size=<now> (+<change>), count=<blocks now> (+<change>), average=<bytes per block>. A block is one allocation handed out by the allocator (the part of the runtime that hands out memory): an object can need one block or several. The list is sorted by the size of the change, largest first, so you read from the top; a negative change means that line's memory shrank. Here, 20,000 requests added 40,000 blocks at line 7 (two per request: the list object and the separate array that holds its 50 element slots; the per-request payload dict itself is freed and does not appear) totalling 8,906 KiB (about 8.7 MiB), at about 228 bytes per block on average. The line 13 row works the same way: dict(response) makes a new dict, which is two blocks (the dict object and its key table), so 20,000 requests add 40,000 blocks of about 96 bytes each. The line 23 row is the for i in range(20_000) loop itself. Each i is an int object of 32 bytes, and each one is kept alive because it is stored as user inside the dict that _audit_log retains. That is 19,743 blocks rather than 20,000 because CPython pre-allocates and shares the integers from -5 to 256, so those 257 values (0 to 256) were already in existence and add no new block; 20,000 - 257 = 19,743. A row like this is real growth (the objects are retained), but it is a small one beside the 8.7 MiB at line 7, so the top of the sorted list is where to start. The count tells you how it is growing: blocks per request. 40,000 / 20,000 = 2 per request is a leak that scales with traffic, while a one-off jump that does not grow between a second and third snapshot is a cache warming up.

Reading the traceback: Traceback.format() prints the outermost caller first and the allocating line last. In the output above the first frame is line 24 (the loop calling handle_request), then line 12 (the call to build_response), and the last frame, line 7, is the line that allocated.

Two traps in this very output.

1. It names where memory was allocated, not who is holding it. The biggest growth is on line 7 (build_response), which looks innocent. The retention is on line 13 (_audit_log.append(dict(response))), which appears second: dict(response) copies only the outer dict, so the original items list stays alive through the copy. Follow the traceback to the allocator, then ask what references the objects. gc.get_referrers(obj) on a sample object, or a heap-graph tool such as objgraph (a third-party library that counts objects by type and draws the chain of references holding a given object), answers that.
2. Group by the right key. compare_to(before, 'lineno') groups by file and line; 'filename' is coarser (find the module first); 'traceback' groups by full call stack, which separates two callers of the same helper (needs start(n) with n above 1).

Using it on a real service after a release.

1. Reproduce the growth under load first (or in a canary, a single instance that receives the new release and a share of real traffic while the rest of the fleet stays unchanged), and take the first snapshot after warm-up so one-time caches and imports are not in the diff.
2. Take a second snapshot after the memory has grown, and a third later. A suspect line that grows in both diffs is a leak; one that grows once and stops is a cache.
3. Compare with the previous release: run the same traffic window on both builds and diff the diffs. The release's change set tells you which lines to suspect.
4. Filter noise: snapshot.filter_traces([...]) takes a list of tracemalloc.Filter(inclusive, filename_pattern) objects; tracemalloc.Filter(False, "<frozen importlib._bootstrap>") excludes allocations made by the import machinery, and Filter(False, tracemalloc.__file__) excludes tracemalloc's own.
5. Expose it safely: trigger the snapshot from an admin endpoint or a signal handler rather than leaving per-request prints, and write the top lines to a log.

Cost. The docs say storing more frames increases memory and CPU overhead and give tracemalloc.get_tracemalloc_memory() to measure it. In the example above tracing added 5.4 MB on top of 14.3 MB traced (38% for this small-object workload), so run it on a canary or a single replica, not the whole fleet, and with a small frame count first. Measure the slowdown on your own workload before enabling it in production.

Limits. tracemalloc sees memory allocated through Python's allocators, so a C extension that calls plain malloc is invisible to it. If RSS (resident set size, the memory the OS has given the process) grows while tracemalloc's total stays flat, the problem is not in Python objects.

Short answer. A garbage collector (GC) is the part of a managed runtime (the JVM, the .NET CLR, Go, V8, CPython) that finds heap memory the program can no longer reach and hands it back for reuse, so developers do not call free themselves. The HotSpot tuning guide describes it as the component that "automatically manages the application's dynamic memory allocation requests": it gets memory from the operating system, hands it out, works out which parts are still in use, and reclaims the rest.

Why a managed runtime needs one

Objects in these languages are created with new (or a literal) and live on the heap, the region for data whose lifetime is not tied to one function call. Nothing in the language says when the last user of an object is done with it. Manual freeing gets that wrong in two ways: freeing too early gives a use-after-free (a dangling pointer to memory that now holds something else), and forgetting gives a leak. The runtime removes both classes of bug by owning the decision.

How the collector decides what to free

It does not ask "is this object used?". It asks "can the program still reach it?" (reachability).

1. Roots. Start from the references the running program holds directly: local variables and arguments on each thread's stack, static (global) fields, and references held by native code (code outside the runtime, such as a C library that Java calls through JNI, the Java Native Interface). These are the GC roots.
2. Trace (mark). Follow every reference from the roots, then every reference from those objects, and so on, marking each object visited.
3. Reclaim. Anything not marked is garbage. Collectors reclaim it in different ways: sweep (free it in place; Go's collector is concurrent, mark-sweep and non-moving, meaning live objects stay at their addresses, per the Go GC guide), copy (move the live objects into a fresh region and discard the old one wholesale, which is how HotSpot's young generation works), or compact (slide live objects together to remove gaps).

An object that is reachable but never used again is still live, which is why leaks exist in garbage-collected languages. A stop-the-world (STW) pause is a stretch where the collector halts all application threads so the object graph cannot change while it works. Concurrent collectors do most marking while the application runs and keep the STW part short.

What it looks like in a run

This program keeps 8 MB reachable and allocates 4 GB of 16 KB arrays that each die immediately:

import java.lang.management.*;
import java.util.*;

public class ShortLivedGarbage {
    static byte[] sink;
    public static void main(String[] args) {
        // Small long-lived set: about 8 MB stays reachable the whole run.
        List<byte[]> live = new ArrayList<>();
        for (int i = 0; i < 8; i++) live.add(new byte[1024 * 1024]);
        // Heavy churn: 4 GB of 16 KB arrays, each dead right after the store.
        for (int i = 0; i < 262_144; i++) sink = new byte[16 * 1024];
        long count = 0, ms = 0;
        for (GarbageCollectorMXBean b : ManagementFactory.getGarbageCollectorMXBeans()) {
            System.out.println(b.getName() + ": collections=" + b.getCollectionCount());
            count += b.getCollectionCount(); ms += b.getCollectionTime();
        }
        System.out.println("total collections=" + count + ", live set still reachable: " + live.size() + " MB");
    }
}

Run on eclipse-temurin:21-jdk (OpenJDK 21.0.12) with java -Xmx64m -Xlog:gc ShortLivedGarbage.java, excerpt (the first pause, the collection count and the timings vary a little from run to run):

[0.001s][info][gc] Using G1
[0.214s][info][gc] GC(0) Pause Young (Normal) (G1 Evacuation Pause) 24M->5M(64M) 4.404ms
[0.228s][info][gc] GC(1) Pause Young (Normal) (G1 Evacuation Pause) 50M->22M(64M) 2.357ms
[0.229s][info][gc] GC(2) Pause Young (Normal) (G1 Evacuation Pause) 49M->22M(64M) 0.375ms
...
[0.317s][info][gc] GC(119) Pause Young (Normal) (G1 Evacuation Pause) 57M->22M(64M) 0.176ms
G1 Young Generation: collections=120
G1 Concurrent GC: collections=0
G1 Old Generation: collections=0
total collections=120, live set still reachable: 8 MB

Read a line as heap in use before -> after (total heap) pause. A young collection cleans only the recently allocated part of the heap (the young generation), where most objects die quickly, and G1 calls the pause an evacuation pause because it copies (evacuates) the survivors out of the regions it is clearing.

Why does the heap after collection show 22M when only 8 MB of arrays are reachable? Adding -Xlog:gc,gc+heap=debug to the same command shows Humongous regions: 16->16. With this 64 MB heap G1 uses 1 MB regions, and an object of half a region or more is humongous and gets whole regions to itself. Each 1 MB array plus its header needs 2 regions, so the 8 reachable arrays occupy 16 MB. The other roughly 6 MB is not the program's own data: java ShortLivedGarbage.java is source-file mode, where the launcher first compiles the program in memory, and that compiler data was already live after GC(0), which ran before the arrays existed (the log shows 0 humongous regions there). Compiling with javac first and running java -cp removes it: every collection then ends at 17M (16 MB of rounded-up arrays plus about 1 MB), with pauses mostly well under a millisecond (an occasional pause of a few milliseconds can still appear). So the floor of the sawtooth is live data plus this kind of overhead (22M in source-file mode, 17M precompiled), and what to watch is whether the floor holds steady. About 120 collections ran in roughly 0.1 s of wall time, each under a few milliseconds here (timings will differ on your machine). G1 is the default collector in this run only because the JVM treats a machine as server-class when it sees two or more processors and at least 1792 MB of memory (the JDK 21 ergonomics page lists G1 on server-class machines and Serial otherwise), so the same command in a one-CPU container picks the Serial collector and prints different lines and a different count. The other collectors named near the end (ZGC, Parallel) are alternatives to G1; the JDK 21 guide says G1 is selected by default "on most hardware and operating system configurations".

What an application developer notices in production

• Latency spikes. Pauses show up as tail latency, the slowest requests, measured as percentiles (p99 is the latency that 99% of requests beat, p999 is 99.9%), not averages. Compare request-latency spikes against the GC log timestamps.
• GC CPU and allocation rate. The more you allocate per request, the more often young collections run (about 120 in the demo). Rising CPU with flat traffic often means more allocation, not more work.
• A heap sawtooth with a floor. Usage climbs, drops after each collection, and the low points (live data) are the number that matters. A floor that keeps rising points at a leak.
• Footprint larger than live data. Collectors need headroom. Go's guide gives the target as live heap + (live heap + roots) x GOGC/100, where GOGC is Go's setting for how far, in percent, the heap may grow past live data before the next cycle (default 100) and roots here means the goroutine stacks and global variables the collector also scans; with its example of 8 MiB live and 2 MiB of roots at the default GOGC=100, the heap can grow by 10 MiB before the next cycle, to about 18 MiB.
• Thrashing and OutOfMemoryError. When live data nears the heap limit the collector runs back to back and reclaims almost nothing (thrashing), then the runtime throws OutOfMemoryError (or the container is killed).

What to do about it

Turn on GC logging first (-Xlog:gc* on HotSpot, and the GC trace option of whichever runtime you use), because every tuning decision needs the pause and frequency numbers. Then fix in this order: reduce allocation in hot paths, size the heap so live data uses a modest share of it, and only then change collectors (for example ZGC, a collector built for very short pauses, when pause time is the priority, or Parallel, which uses many threads to maximise throughput; the guide lists ZGC for low latency and Parallel for peak throughput). Tune from the log, not from folklore.