System Resource & I/O Optimization Questions

Design principle: shared-nothing per node, not one big shared heap

NUMA (Non-Uniform Memory Access, where each CPU socket has its own local memory and reaching another socket's memory costs extra latency over an inter-socket interconnect) rewards a design that partitions work by node rather than one that lets any thread touch any memory freely. The strongest NUMA-friendly pattern is to shard the service's data and threads one partition per node, so each node's threads mostly touch that node's own memory by construction, instead of trying to retrofit locality onto a shared-everything design after the fact.

Thread and CPU affinity

Pin worker threads to the CPUs of a single node at startup, using pthread_setaffinity_np() in C/C++ or an equivalent affinity API in the runtime, or by launching the process under numactl --physcpubind. Without pinning, the OS scheduler can migrate a thread to a core on a different node mid-run; each such migration risks a burst of remote memory accesses until the thread's working set effectively 'catches up' by being re-touched (and thus reallocated) on the new node.

Allocator choice

• Use a NUMA-aware allocator, for example jemalloc or tcmalloc configured with one arena per node (or per thread, matched to node count), so allocations naturally land on the calling thread's local node instead of a single global arena that anyone can touch.
• Linux uses a first-touch allocation policy for anonymous memory: a page is not actually assigned to a physical NUMA node until the first time something writes to it, regardless of which node's allocator called malloc(). For large, latency-sensitive structures, allocate explicitly with numa_alloc_onnode() or mbind() and make sure the FIRST touch genuinely happens on the intended node, since a naive 'warm-up' loop run on the wrong thread can silently place memory on the wrong node before the real workload ever runs.

Testing approach

• Before investing in NUMA-aware code, measure whether the penalty is real on this specific workload and hardware: run the same benchmark under numactl --membind forced to the local node versus forced to a remote node, and compare. The gap varies a lot by platform and workload, so guessing is a bad use of engineering time; an A/B measurement settles it.
• During a real load test, check numastat deltas per process to confirm the design's INTENDED locality is what actually happens at runtime, since fragmentation or an allocator falling back under pressure can quietly break locality even when the code was written correctly.

Direct answer

A buffer temporarily holds data in transit between two things operating at different speeds, and once that data is consumed it's typically gone; it exists to smooth throughput and absorb bursts. A cache holds a copy of data specifically so it can be reused later, trading memory for avoiding a repeated expensive operation; it exists to cut latency and reduce load on whatever it's standing in front of. In a backend service you generally need both, sized for their different jobs, rather than choosing one over the other.

Structured elaboration

Buffers, and what they're for. A socket buffer (configured via SO_RCVBUF/SO_SNDBUF) holds bytes that have arrived from the network before the application reads them, or bytes the application has written before the kernel has finished sending them. It decouples the producer's rate from the consumer's rate, letting each side proceed at its own pace within the buffer's capacity. The data isn't kept for later reuse; once consumed, it's gone. Effect on throughput and latency: a buffer that's too small causes stalls or drops under bursts (the sender must wait or the packet is lost); a buffer that's too large causes "bufferbloat" (data sits queued for a long time before being processed, adding latency even though nothing is technically being dropped).

Caches, and what they're for. The OS page cache (kernel-managed, holds recently accessed disk blocks) and application-level caches (an in-process LRU, least-recently-used, cache in front of a database, or an external cache like Redis) both exist to serve a repeat request faster than redoing the original work, and secondarily to reduce load on the backing system, fewer disk reads, fewer database queries, which indirectly helps everyone else sharing that backend. Unlike a buffer, the data is deliberately kept around for reuse, governed by an eviction policy (commonly LRU) or a TTL (time-to-live, an expiry after which an entry is considered stale and discarded).

How each affects latency and throughput, side by side. A buffer's main effect is on throughput smoothness and burst tolerance: sized correctly, it prevents drops and stalls from rate mismatches; sized wrong in either direction, it either drops data or adds unnecessary queuing delay. A cache's main effect is on latency for repeat access: a cache hit skips the original expensive operation entirely, and its secondary effect is throughput for the whole system, since offloading repeat requests from the backend frees capacity for everyone else's requests too.

When to prefer one over the other in a backend service. In practice you don't choose between them, you apply each to the problem it actually solves. Size buffers (socket buffers, request queues) to absorb realistic burst and jitter without excessive queuing delay, neither so small that normal bursts cause drops nor so large that queued data sits stale by the time it's processed. Add a cache specifically for data that's read far more often than it changes, where some staleness within your TTL window is acceptable. A common design mistake is using a cache where you actually need a buffer, for example trying to "cache" write-once event data that should really just be queued and drained, or the reverse, failing to add a cache in front of a backend that genuinely can't sustain the real read rate without one.

Worked example

A backend service accepting webhook events over a socket needs an appropriately sized SO_RCVBUF (a buffer) so a burst of simultaneously arriving events doesn't get dropped by the kernel before the application has a chance to read them; that data is processed once and discarded, no reuse involved. The same service, when it needs to look up a customer's plan tier to decide how to route each event, benefits from an application-level cache (an in-process map or an external cache like Redis) in front of the database, since the same customer's plan tier is looked up repeatedly across many events and rarely changes, making it a textbook case for reuse-oriented caching with a short TTL (a few minutes is often enough to keep the data reasonably fresh while eliminating the vast majority of repeat database lookups).

Trade-offs and pitfalls

Oversizing a buffer to "be safe" is a common mistake with a real cost: it doesn't prevent drops any better once the true burst exceeds even the larger buffer, but it does add latency to everything that queues behind a large backlog (bufferbloat), which can make a service feel sluggish under load even though nothing is technically failing. On the cache side, the common mistake is caching data that changes on every access (defeating any reuse benefit while adding cache-management overhead for nothing) or setting a TTL so long that stale data causes real correctness problems downstream.

Direct answer

Read-ahead is the kernel (or an application) speculatively fetching data beyond what was explicitly requested, betting that a sequential access pattern will keep asking for the next chunk. It helps when access really is sequential and predictable, because it hides storage latency by having data ready before it's asked for. It hurts on random-access workloads, where it wastes I/O bandwidth and evicts genuinely useful cached pages fetching data that's never used. For a media streaming service serving large sequential reads, you'd deliberately increase the block-device readahead window well beyond the default.

Structured elaboration

OS-level read-ahead. The kernel detects a sequential access pattern to a file (successive reads at increasing offsets) and issues additional reads ahead of what the application asked for, populating the page cache before those blocks are requested. This is controlled per block device via the readahead setting, viewable and settable with blockdev --getra/--setra /dev/sdX (values are in 512-byte sectors), and persisted across reboots via a udev rule that reapplies the setting when the device is detected.

Application-level prefetching. Applications can give the kernel explicit hints rather than relying purely on heuristic detection: posix_fadvise(fd, ..., POSIX_FADV_SEQUENTIAL) tells the kernel to be more aggressive about read-ahead for this specific file descriptor, and POSIX_FADV_WILLNEED explicitly requests the kernel start prefetching a specific range immediately. Databases commonly do their own index-page prefetching above the OS layer, and media servers or download managers may issue explicit range-request prefetching over the network, a similar idea one layer up the stack.

When it helps. Any workload with a genuinely predictable, sequential access pattern: streaming a large media file, a full table scan, sequential log processing. The core mechanism is hiding storage latency by overlapping "fetch the next chunk" with "process the chunk you already have," so by the time the application asks for the next piece, it's already in cache.

When it hurts. Random-access workloads, most notably databases doing index lookups scattered across a large file, where the "next" blocks read-ahead fetches speculatively are usually never actually requested. This wastes I/O bandwidth on the storage device (competing with real requests) and pollutes the page cache by evicting genuinely hot pages to make room for speculative ones that go unused, hurting both throughput (wasted device bandwidth) and latency (real requests now queue behind speculative ones, and useful cache entries get evicted).

Tuning for a media streaming service. Increase the block device's readahead window well above the default (commonly 128 KB) to something sized for the streaming bitrate and access pattern, for example:

blockdev --setra 8192 /dev/nvme0n1

8192 sectors of 512 bytes each is 4 MB, a much larger anticipatory read window suited to sustaining large sequential streams at high bitrate with fewer, larger I/O operations rather than many small ones. Pairing this with an explicit posix_fadvise(fd, 0, 0, POSIX_FADV_SEQUENTIAL) call from the serving application lets the kernel be aggressive about read-ahead for that file from the very first read, rather than waiting for its own heuristics to detect the sequential pattern after a few reads have already happened. Because this trades page cache capacity for prefetch depth, validate against your actual cache hit ratio on a multi-tenant streaming box: a readahead window large enough to help one stream but big enough to evict other tenants' hot content is a net loss.

Worked example

Default readahead (blockdev --getra typically reporting 256 sectors, 128 KB) on a media server sustaining a 20 Mbps stream means the kernel issues many small anticipatory reads to keep up, each one a separate I/O operation with its own queueing and completion overhead. Increasing that to 8192 sectors (4 MB) means far fewer, larger I/O operations are needed to sustain the same stream, which reduces per-operation overhead and gives the kernel a much bigger buffer of already-fetched data to hide any transient storage latency behind. The trade-off becomes visible on a multi-tenant box: if 200 concurrent streams each get the configured 4 MB working set of read-ahead data cached (matching the 8192-sector/4 MB setting above), that's 200 x 4 MB = 0.8 GB of cache committed to prefetch alone, worth explicitly sizing against total available cache rather than assuming a larger number is free.

Trade-offs and pitfalls

Applying a large readahead window globally to a device that serves both sequential (streaming) and random (metadata lookups, small file serving) workloads is the most common mistake: the random-access traffic pays the read-ahead tax (wasted bandwidth, cache pollution) without getting any of the sequential-access benefit. If the same device serves mixed workloads, prefer per-file posix_fadvise hints from the application over a single global blockdev setting, so the tuning only applies to the traffic that actually benefits from it.

Approach

Serving a static file the ordinary way (read() into a userspace buffer, then write() to the socket) costs two data copies (disk cache to userspace buffer, userspace buffer to socket buffer) and at least two user/kernel context switches. sendfile() asks the kernel to copy the file's page-cache pages straight into the socket's send buffer without ever handing the bytes to userspace, cutting the copy count and the syscall count.

Demonstration (real, executed)

The following Python code uses os.sendfile(), the direct syscall wrapper, to serve a file over a loopback TCP socket. It was run as written; the printed output below is the actual result of that run, not a hypothetical.

import os, socket, threading, tempfile

data = b"X" * (64 * 1024)  # a 64 KiB "static file"
with tempfile.NamedTemporaryFile(delete=False) as f:
    f.write(data)
    fname = f.name

def server(holder):
    srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    srv.bind(("127.0.0.1", 0))
    srv.listen(1)
    holder["port"] = srv.getsockname()[1]
    holder["ready"].set()
    conn, _ = srv.accept()
    with open(fname, "rb") as fh:
        filesize = os.fstat(fh.fileno()).st_size
        offset, sent_total = 0, 0
        while sent_total < filesize:
            sent = os.sendfile(conn.fileno(), fh.fileno(), offset, filesize - sent_total)
            if sent == 0:
                break
            offset += sent
            sent_total += sent
    conn.close(); srv.close()
    holder["sent"] = sent_total

holder = {"ready": threading.Event()}
t = threading.Thread(target=server, args=(holder,)); t.start()
holder["ready"].wait()

client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect(("127.0.0.1", holder["port"]))
received = b""
while len(received) < len(data):
    chunk = client.recv(65536)
    if not chunk:
        break
    received += chunk
client.close(); t.join()

print("bytes sent via os.sendfile():", holder["sent"])
print("bytes received by client:", len(received))
print("payload matches:", received == data)
os.unlink(fname)

Output:

bytes sent via os.sendfile(): 65536
bytes received by client: 65536
payload matches: True

os.sendfile(out_fd, in_fd, offset, count) mirrors the Linux syscall signature exactly: it copies count bytes from in_fd (a regular file, must be mmap-able) to out_fd starting at offset, and returns how many bytes it actually moved so the loop above can resume on a short transfer. out_fd has always been allowed to be a socket, which is the case this demo uses; the version boundary actually runs the other way. Before Linux 2.6.33, out_fd was required to be a socket; since 2.6.33 it can be any file, including a regular file, which is what additionally enables file-to-file zero-copy transfers, not the socket case this demo already relied on.

Key points

• in_fd must be a regular file (or a file the kernel can page-cache), not a pipe or arbitrary character device: the kernel is copying page-cache pages, not streaming arbitrary bytes.
• On modern NICs, the copy can be a scatter-gather DMA (Direct Memory Access: hardware moves the data without the CPU copying it byte by byte) straight from page cache to the NIC, so there may be zero CPU-driven copies at all, only DMA.

Limitations

• File descriptor limits: sendfile() needs valid, seekable file descriptors on both ends; it cannot serve from a socket-to-socket relay directly.
• TLS: sendfile() hands raw file bytes to the kernel network stack. TLS encryption normally happens in userspace (OpenSSL, etc.), which requires touching the plaintext, defeating the zero-copy path entirely, so a plain TLS-terminating server usually cannot use sendfile() for the encrypted response body.
• kTLS (kernel TLS) is the fix: once the TLS session keys are handed to the kernel, the kernel encrypts the data during the same copy sendfile() already does, restoring the zero-copy path for encrypted traffic.
• splice() is the more general alternative: it moves data between two file descriptors via an intermediate kernel pipe buffer, so it also covers socket-to-socket proxying, a case plain sendfile() does not handle well.

Approach

Stream in fixed-size chunks straight from the request body to the destination (disk or object storage) instead of buffering the whole file, checksum incrementally as bytes flow through rather than in a separate pass, track how many bytes have landed so an interrupted transfer can resume from that offset instead of restarting, and let a bounded concurrency limit (not an unbounded goroutine or thread per upload) be the backpressure mechanism under load.

Design

Never load the whole file into memory: read a fixed-size chunk (a few megabytes is typical), write it through immediately, and discard it before reading the next chunk, so peak memory use is bounded by the chunk size, not the file size, regardless of whether the file is 10MB or 10GB. Chunked or multipart upload straight to disk or object storage: for object storage specifically, most providers expose a multipart upload API (initiate, upload part N with its own ETag, complete) that maps naturally onto this same fixed-chunk read loop, each chunk becomes one part. Resumability: persist the highest successfully-written offset (or the list of successfully completed part numbers for multipart object storage) somewhere durable (a small metadata record, not just in-process memory), so a retry after a dropped connection can query that offset and resume from there instead of re-uploading bytes already safely stored. Backpressure: the destination write call itself provides natural backpressure for a single stream, if the write blocks because the disk or object store is slow, the next read from the source simply waits, so the service does not keep pulling data faster than it can be persisted; for many concurrent uploads, a bounded worker pool (a fixed-size semaphore or channel) caps how many uploads are in flight at once, so total memory use stays bounded even under a burst of concurrent requests. Integrity verification: compute a checksum (SHA-256 or a similar strong hash) incrementally as each chunk streams through, feeding the same bytes into both the destination writer and the hash function in one pass, rather than re-reading the whole file afterward to verify it, which would defeat the memory-bounded design.

Worked example (Go)

const chunkSize = 4 << 20 // 4 MiB per chunk: never buffer the whole object

// streamChunks reads src in chunkSize pieces and writes each straight
// through to dst while it reads, returning a hex sha256 of the whole
// stream. Because writes happen inline with reads, a slow or blocked
// destination stalls the next Read call automatically: that stall IS
// the backpressure, no separate signalling channel needed for one stream.
func streamChunks(dst io.Writer, src io.Reader) (string, int64, error) {
    h := sha256.New()
    w := io.MultiWriter(dst, h)
    buf := make([]byte, chunkSize)
    var total int64
    for {
        n, rerr := src.Read(buf)
        if n > 0 {
            written, werr := w.Write(buf[:n])
            total += int64(written)
            if werr != nil {
                return "", total, fmt.Errorf("write at offset %d: %w", total, werr)
            }
        }
        if rerr == io.EOF {
            break
        }
        if rerr != nil {
            return "", total, fmt.Errorf("read at offset %d: %w", total, rerr)
        }
    }
    return hex.EncodeToString(h.Sum(nil)), total, nil
}

// uploadConcurrent fans independent files out to a bounded worker pool.
// maxInFlight is the concurrency-level backpressure knob for the
// multi-upload case: unbounded goroutines-per-upload wins a
// microbenchmark and then exhausts memory or file descriptors under
// real concurrent load.
func uploadConcurrent(paths []string, destDir string, maxInFlight int) map[string]string {
    sem := make(chan struct{}, maxInFlight)
    var wg sync.WaitGroup
    var mu sync.Mutex
    results := make(map[string]string)
    for _, p := range paths {
        wg.Add(1)
        go func(p string) {
            defer wg.Done()
            sem <- struct{}{}        // blocks once maxInFlight are in flight
            defer func() { <-sem }()
            src, err := os.Open(p)
            if err != nil {
                return
            }
            defer src.Close()
            dst, err := os.Create(filepath.Join(destDir, filepath.Base(p)))
            if err != nil {
                return
            }
            bw := bufio.NewWriterSize(dst, chunkSize)
            sum, _, err := streamChunks(bw, src)
            if err == nil {
                bw.Flush()
                mu.Lock()
                results[p] = sum
                mu.Unlock()
            }
            dst.Close()
        }(p)
    }
    wg.Wait()
    return results
}

Run against six ~9MB source files with maxInFlight=3, each result's SHA-256 checksum was independently recomputed by reading the destination file back and compared to what streamChunks produced inline:

file0.bin: checksum_match=true
file1.bin: checksum_match=true
file2.bin: checksum_match=true
file3.bin: checksum_match=true
file4.bin: checksum_match=true
file5.bin: checksum_match=true
all_checksums_match: true

What changes for a Go service maximizing throughput under concurrent load

maxInFlight becomes the primary tuning knob: too low under-utilizes available disk/network bandwidth, too high causes goroutines to pile up waiting on a slow destination and inflates memory (each blocked goroutine still holds its chunk buffer); size it from measured throughput per upload and the destination's actual concurrent-write capacity, not a guess. Reuse chunk buffers across uploads with sync.Pool rather than allocating a fresh chunkSize buffer per upload, since allocation and garbage collection of large buffers under high concurrency becomes a real throughput cost at scale (garbage collection, the Go runtime's automatic memory reclamation, has more work to do the more short-lived large objects it must track). For multipart uploads to object storage specifically, upload multiple parts of the SAME file concurrently (not just multiple files concurrently), since a single large file's throughput is otherwise capped by one sequential part-upload stream even while overall service concurrency looks high. Use io.CopyBuffer or the pattern above rather than ioutil.ReadAll anywhere in the path, since the latter reintroduces full in-memory buffering and defeats the design's whole point.

Trade-offs and pitfalls

Incremental checksumming means the destination receives unverified bytes as they stream; if integrity must be confirmed before the object is considered durable, either use a storage API that verifies a provided checksum against what it received (many object stores support this natively as part of the multipart-complete call) or accept a brief window where a corrupted upload is only caught after the fact and must be re-uploaded. Persisting resume state adds its own reliability requirement: if the resume-offset record itself is lost or inconsistent with what actually landed at the destination, resuming from a stale offset can either skip bytes or duplicate them, so treat the offset record with the same durability care as the data itself. Bounding concurrency for throughput requires real measurement (vary maxInFlight and measure actual aggregate throughput and destination-side error rate) since the optimal value depends on the destination's own concurrency limits, not just the source service's own capacity.