Brief approach: the leak is caused by a global container accumulating session objects with no eviction. Fix by changing the container to an eviction-capable structure (LRU/TTL) or weak references and adding explicit lifecycle management. Detect with tracemalloc/objgraph and mitigate in prod via memory limits, rolling restarts, and live profiling.Code fixes (examples):Use collections.OrderedDict TTL/LRU:python
from collections import OrderedDict
import time
class TTLCache:
def __init__(self, max_size=10000, ttl=3600):
self.store = OrderedDict()
self.max_size = max_size
self.ttl = ttl
def set(self, key, value):
now = time.time()
if key in self.store:
self.store.pop(key)
self.store[key] = (value, now)
if len(self.store) > self.max_size:
self.store.popitem(last=False) # evict oldest
def get(self, key):
val = self.store.get(key)
if not val:
return None
value, created = val
if time.time() - created > self.ttl:
self.store.pop(key, None)
return None
return value
Use weakref for session-like objects if nothing else holds strong refs:python
import weakref
_session_map = weakref.WeakValueDictionary() # entries auto-removed when object dies
Detection steps (no deploy):- Start tracemalloc snapshot: python -m tracemalloc to compare growth.- Use objgraph: identify most-growing types and show referrers (objgraph.show_most_common_types(), objgraph.show_backrefs()).- Heap dump with heapy/gc.get_objects() and analyze on host.- Export memory metrics (RSS) and sample detailed profiles periodically.Operational mitigations (immediate, no-code deploy):- Enforce memory caps via cgroups/kubernetes OOMLimit and restart policy to bound impact.- Trigger rolling restart of processes/containers during low traffic windows.- Add runtime configuration to enable periodic eviction if service already reads config (e.g., via feature flag or env var) — prefer short rollout.- Attach a temporary sidecar or debugging pod to run the profiler and stream heap dumps.- Create alerts on sustained growth: RSS increasing > X% over Y hours, or large number of session objects (custom metric).Why this works:- Global strong refs prevent GC from reclaiming objects; TTL/LRU/weakref breaks that chain and bounds memory.- Operational controls limit blast radius while you deploy a permanent fix.Edge cases & considerations:- Ensure concurrency safety (use locks or thread-safe caches like cachetools.TTLCache).- Verify eviction semantics won’t drop active sessions unexpectedly; consider graceful eviction (mark expired then notify).- Test GC/refgraph after change to confirm leak resolved.Longer-term:- Add unit/integration tests for cache eviction, instrumentation to count cache size, and SLOs for memory usage.