Object pooling and reusing large buffers reduce GC pressure by avoiding frequent allocation/deallocation of big Python objects (bytes, bytearray, large numpy arrays). For a JSON-to-numpy ingestion pipeline this matters: repeatedly creating 10s–100s MB objects triggers the allocator and GC, causing pauses and memory churn. Reuse keeps large buffers alive and repopulated, reducing allocator work and fragmentation.Concrete techniques and libraries- Pre-allocate numpy arrays and overwrite them in-place instead of allocating new arrays per message (use numpy.resize/[:n] or create fixed-shape arrays and fill slices).- Use bytearray or memoryview for parsing raw JSON bytes so you can refill buffers without reallocating.- Use mmap for very large datasets or files to avoid copying into Python heaps.- Consider PyArrow for zero-copy parsing/serialization and Feather/IPC formats; its buffers are C-owned and reduce Python-level churn.- Use multiprocessing.shared_memory (Python 3.8+) to share large buffers across processes without copies.- Use a simple object pool (queue of ready buffers) with a max pool size to bound memory.Example pool pattern (python):python
import queue
import numpy as np
class NumpyPool:
def __init__(self, shape, dtype, maxsize=10):
self.pool = queue.LifoQueue(maxsize)
for _ in range(maxsize):
self.pool.put(np.empty(shape, dtype=dtype))
def acquire(self, timeout=None):
return self.pool.get(timeout=timeout)
def release(self, arr):
# optionally zero or validate before returning
self.pool.put(arr)
# usage
pool = NumpyPool((1000000,), np.float32, maxsize=5)
arr = pool.acquire()
# parse JSON values into arr[:n] in-place
# ... processing ...
pool.release(arr)
