Direct answer. A memory leak that shows up as restarts creeping up over weeks needs an investigation on that same timescale: the goal is to find what's accumulating and never getting released, using data collected over the leak's actual lifetime, not a single snapshot.
Structured elaboration.
- Immediate mitigations. Scheduled or threshold-triggered restarts (restarting an instance once its memory crosses a safe ceiling, before it OOMs (runs out of memory) and restarts uncontrolled) buy time without fixing anything, and are a reasonable stopgap while you investigate, since an OOM kill mid-request is worse for users than a controlled, load-balanced restart.
- Data to collect. A memory-usage-over-time metric per instance, ideally with enough resolution and history to see the actual growth RATE and whether it's linear (suggesting something growing proportional to time or request count) or has a different shape. Periodic heap dumps or heap-profiling snapshots (not just at the moment of failure, but at intervals across the leak's timeline) let you compare what's IN memory at, say, hour 1 versus hour 100 and see what grew. Traces or logs correlating memory growth against specific request types or code paths, if the leak is triggered by a subset of traffic rather than uniformly.
- Instrumentation to add if not already present. Object-count or memory-usage metrics broken down by type or subsystem (many runtimes support this, at some overhead), so you're not limited to a single aggregate number when trying to localize where growth is happening. A way to trigger an on-demand heap snapshot without a full restart, so you can capture state right when memory looks unusually high rather than waiting for a scheduled dump.
- Root-cause analysis without disrupting production. Compare heap snapshots from two points in time (early versus late in an instance's uptime) and look specifically for object types or data structures whose COUNT grew disproportionately relative to request volume over the same window; this is usually the fastest way to localize a leak to a specific subsystem or cache. Once localized, review that code path specifically for an unbounded collection (a cache with no eviction, a list that's appended to but never trimmed, a subscription or listener that's registered but never unregistered) since these are the most common leak patterns in long-running services.
- Validate the fix on the same timescale as the leak. Because this leak takes weeks to become visible, confirming a fix worked needs a similarly long observation window (watching the memory-over-time trend stay flat for a comparable period), not just a quick sanity check right after deploying the fix.
Worked example. Suppose comparing heap snapshots from early and late in an instance's lifetime shows one specific object type, a per-request context object, growing in count roughly linearly with total requests served, while other object types stay roughly constant. Tracing that object type's lifecycle in the code reveals it's stored in a dictionary keyed by request ID for the duration of the request, intended to be removed once the request completes, but a specific error path (triggered by a small fraction of requests) returns early without reaching the cleanup code, leaving that entry in the dictionary forever. If that error path fires on, say, 0.1% of requests, and the service handles a few million requests per week, that's still thousands of leaked entries per week, small enough per-request to be invisible immediately but large enough in aggregate to explain a multi-week creep toward OOM. The fix is moving the cleanup into a finally-equivalent block (or whatever the language's guaranteed-cleanup construct is) so it runs regardless of which path the request takes, and validating means confirming the dictionary's size metric (once added) stays flat under normal error-path traffic instead of growing without bound.
Trade-offs and pitfalls. A single heap snapshot at the moment of failure is much less useful than a COMPARISON between two points in time, because a snapshot alone shows you what's in memory but not what's actually GROWING; always try to capture at least two, meaningfully separated in time. It's also worth being suspicious of any code path that stores something keyed by a per-request identifier without an equally certain, guaranteed removal path, since that pattern is one of the most common sources of exactly this class of leak.
