System Resource & I/O Optimization Questions

Design principle: shared-nothing per node, not one big shared heap

NUMA (Non-Uniform Memory Access, where each CPU socket has its own local memory and reaching another socket's memory costs extra latency over an inter-socket interconnect) rewards a design that partitions work by node rather than one that lets any thread touch any memory freely. The strongest NUMA-friendly pattern is to shard the service's data and threads one partition per node, so each node's threads mostly touch that node's own memory by construction, instead of trying to retrofit locality onto a shared-everything design after the fact.

Thread and CPU affinity

Pin worker threads to the CPUs of a single node at startup, using pthread_setaffinity_np() in C/C++ or an equivalent affinity API in the runtime, or by launching the process under numactl --physcpubind. Without pinning, the OS scheduler can migrate a thread to a core on a different node mid-run; each such migration risks a burst of remote memory accesses until the thread's working set effectively 'catches up' by being re-touched (and thus reallocated) on the new node.

Allocator choice

• Use a NUMA-aware allocator, for example jemalloc or tcmalloc configured with one arena per node (or per thread, matched to node count), so allocations naturally land on the calling thread's local node instead of a single global arena that anyone can touch.
• Linux uses a first-touch allocation policy for anonymous memory: a page is not actually assigned to a physical NUMA node until the first time something writes to it, regardless of which node's allocator called malloc(). For large, latency-sensitive structures, allocate explicitly with numa_alloc_onnode() or mbind() and make sure the FIRST touch genuinely happens on the intended node, since a naive 'warm-up' loop run on the wrong thread can silently place memory on the wrong node before the real workload ever runs.

Testing approach

• Before investing in NUMA-aware code, measure whether the penalty is real on this specific workload and hardware: run the same benchmark under numactl --membind forced to the local node versus forced to a remote node, and compare. The gap varies a lot by platform and workload, so guessing is a bad use of engineering time; an A/B measurement settles it.
• During a real load test, check numastat deltas per process to confirm the design's INTENDED locality is what actually happens at runtime, since fragmentation or an allocator falling back under pressure can quietly break locality even when the code was written correctly.

Approach

Stream in fixed-size chunks straight from the request body to the destination (disk or object storage) instead of buffering the whole file, checksum incrementally as bytes flow through rather than in a separate pass, track how many bytes have landed so an interrupted transfer can resume from that offset instead of restarting, and let a bounded concurrency limit (not an unbounded goroutine or thread per upload) be the backpressure mechanism under load.

Design

Never load the whole file into memory: read a fixed-size chunk (a few megabytes is typical), write it through immediately, and discard it before reading the next chunk, so peak memory use is bounded by the chunk size, not the file size, regardless of whether the file is 10MB or 10GB. Chunked or multipart upload straight to disk or object storage: for object storage specifically, most providers expose a multipart upload API (initiate, upload part N with its own ETag, complete) that maps naturally onto this same fixed-chunk read loop, each chunk becomes one part. Resumability: persist the highest successfully-written offset (or the list of successfully completed part numbers for multipart object storage) somewhere durable (a small metadata record, not just in-process memory), so a retry after a dropped connection can query that offset and resume from there instead of re-uploading bytes already safely stored. Backpressure: the destination write call itself provides natural backpressure for a single stream, if the write blocks because the disk or object store is slow, the next read from the source simply waits, so the service does not keep pulling data faster than it can be persisted; for many concurrent uploads, a bounded worker pool (a fixed-size semaphore or channel) caps how many uploads are in flight at once, so total memory use stays bounded even under a burst of concurrent requests. Integrity verification: compute a checksum (SHA-256 or a similar strong hash) incrementally as each chunk streams through, feeding the same bytes into both the destination writer and the hash function in one pass, rather than re-reading the whole file afterward to verify it, which would defeat the memory-bounded design.

Worked example (Go)

const chunkSize = 4 << 20 // 4 MiB per chunk: never buffer the whole object

// streamChunks reads src in chunkSize pieces and writes each straight
// through to dst while it reads, returning a hex sha256 of the whole
// stream. Because writes happen inline with reads, a slow or blocked
// destination stalls the next Read call automatically: that stall IS
// the backpressure, no separate signalling channel needed for one stream.
func streamChunks(dst io.Writer, src io.Reader) (string, int64, error) {
    h := sha256.New()
    w := io.MultiWriter(dst, h)
    buf := make([]byte, chunkSize)
    var total int64
    for {
        n, rerr := src.Read(buf)
        if n > 0 {
            written, werr := w.Write(buf[:n])
            total += int64(written)
            if werr != nil {
                return "", total, fmt.Errorf("write at offset %d: %w", total, werr)
            }
        }
        if rerr == io.EOF {
            break
        }
        if rerr != nil {
            return "", total, fmt.Errorf("read at offset %d: %w", total, rerr)
        }
    }
    return hex.EncodeToString(h.Sum(nil)), total, nil
}

// uploadConcurrent fans independent files out to a bounded worker pool.
// maxInFlight is the concurrency-level backpressure knob for the
// multi-upload case: unbounded goroutines-per-upload wins a
// microbenchmark and then exhausts memory or file descriptors under
// real concurrent load.
func uploadConcurrent(paths []string, destDir string, maxInFlight int) map[string]string {
    sem := make(chan struct{}, maxInFlight)
    var wg sync.WaitGroup
    var mu sync.Mutex
    results := make(map[string]string)
    for _, p := range paths {
        wg.Add(1)
        go func(p string) {
            defer wg.Done()
            sem <- struct{}{}        // blocks once maxInFlight are in flight
            defer func() { <-sem }()
            src, err := os.Open(p)
            if err != nil {
                return
            }
            defer src.Close()
            dst, err := os.Create(filepath.Join(destDir, filepath.Base(p)))
            if err != nil {
                return
            }
            bw := bufio.NewWriterSize(dst, chunkSize)
            sum, _, err := streamChunks(bw, src)
            if err == nil {
                bw.Flush()
                mu.Lock()
                results[p] = sum
                mu.Unlock()
            }
            dst.Close()
        }(p)
    }
    wg.Wait()
    return results
}

Run against six ~9MB source files with maxInFlight=3, each result's SHA-256 checksum was independently recomputed by reading the destination file back and compared to what streamChunks produced inline:

file0.bin: checksum_match=true
file1.bin: checksum_match=true
file2.bin: checksum_match=true
file3.bin: checksum_match=true
file4.bin: checksum_match=true
file5.bin: checksum_match=true
all_checksums_match: true

What changes for a Go service maximizing throughput under concurrent load

maxInFlight becomes the primary tuning knob: too low under-utilizes available disk/network bandwidth, too high causes goroutines to pile up waiting on a slow destination and inflates memory (each blocked goroutine still holds its chunk buffer); size it from measured throughput per upload and the destination's actual concurrent-write capacity, not a guess. Reuse chunk buffers across uploads with sync.Pool rather than allocating a fresh chunkSize buffer per upload, since allocation and garbage collection of large buffers under high concurrency becomes a real throughput cost at scale (garbage collection, the Go runtime's automatic memory reclamation, has more work to do the more short-lived large objects it must track). For multipart uploads to object storage specifically, upload multiple parts of the SAME file concurrently (not just multiple files concurrently), since a single large file's throughput is otherwise capped by one sequential part-upload stream even while overall service concurrency looks high. Use io.CopyBuffer or the pattern above rather than ioutil.ReadAll anywhere in the path, since the latter reintroduces full in-memory buffering and defeats the design's whole point.

Trade-offs and pitfalls

Incremental checksumming means the destination receives unverified bytes as they stream; if integrity must be confirmed before the object is considered durable, either use a storage API that verifies a provided checksum against what it received (many object stores support this natively as part of the multipart-complete call) or accept a brief window where a corrupted upload is only caught after the fact and must be re-uploaded. Persisting resume state adds its own reliability requirement: if the resume-offset record itself is lost or inconsistent with what actually landed at the destination, resuming from a stale offset can either skip bytes or duplicate them, so treat the offset record with the same durability care as the data itself. Bounding concurrency for throughput requires real measurement (vary maxInFlight and measure actual aggregate throughput and destination-side error rate) since the optimal value depends on the destination's own concurrency limits, not just the source service's own capacity.