System Resource & I/O Optimization Questions

Direct answer

Choose NVMe (Non-Volatile Memory Express, a solid-state storage interface built for low-latency, high-parallelism flash access) for the online transaction processing (OLTP, transactional database work with many small concurrent random reads/writes) workload and the time-series-with-compaction workload, both are dominated by random I/O and latency sensitivity where NVMe's advantage is largest; a traditional hard disk drive (HDD, spinning-platter mechanical storage) is a reasonable and often more cost-effective choice for large sequential archival writes, where NVMe's latency advantage matters least.

(a) Large sequential media archival writes: HDD

Archival writes are large, sequential, and not latency-sensitive (nobody is waiting on a single archival write to complete within milliseconds); HDDs achieve respectable sequential throughput (commonly in the 150-250MB/s range for modern drives) at a fraction of NVMe's cost per gigabyte, and archival workloads care about cost-per-terabyte far more than latency or IOPS. NVMe's main advantages, low random-access latency and high IOPS, are simply not exercised by a sequential write pattern, so paying its cost premium here buys nothing the workload actually uses.

(b) OLTP with small random writes: NVMe

OLTP workloads are dominated by many small, concurrent, random reads and writes with strict latency expectations (users and transactions waiting in real time); an HDD's mechanical seek time (the physical head movement between non-sequential locations on the platter, typically several milliseconds per seek) makes random I/O dramatically slower than sequential I/O on the same device, often by one to two orders of magnitude in achievable IOPS (I/O operations per second, how many individual read/write operations a device can complete each second). NVMe has no mechanical seek at all and is built specifically for high queue-depth, low-latency random access, making it the clear choice whenever the access pattern is random and latency-sensitive, which OLTP always is by nature.

(c) Time-series ingestion combined with compaction: NVMe

Time-series ingestion itself is often sequential-write-friendly (append-mostly), which might suggest HDD, but compaction (the background process many time-series and log-structured storage engines run to merge and reorganize on-disk data files) generates substantial additional random and mixed read/write I/O on top of the ingestion writes, and that background I/O directly competes with foreground query and ingestion latency. NVMe's high IOPS and low latency absorb this compaction overhead without degrading foreground performance nearly as much as an HDD would, where compaction I/O would visibly compete with and slow down both ingestion and queries.

Trade-offs: cost, latency, IOPS, throughput, write amplification

Cost: HDD wins decisively on cost per gigabyte, which is why it remains the right choice for (a) and for any workload where capacity, not speed, is the binding constraint. NVMe costs meaningfully more per gigabyte but that premium buys latency and IOPS that (b) and (c) cannot function well without. Latency: NVMe's random-access latency is typically in the tens-of-microseconds range versus an HDD's single-digit-to-double-digit milliseconds for a random seek, a difference of roughly two to three orders of magnitude that matters enormously for (b) and (c) and not at all for (a). IOPS: NVMe sustains vastly higher random IOPS than HDD because it has no mechanical seek penalty and supports much deeper command queuing (NVMe's queue depth can reach thousands versus a traditional SATA/SAS HDD's much shallower queue). Throughput: for pure large sequential transfers, the throughput gap between a fast HDD and NVMe is much smaller proportionally than the IOPS/latency gap, which is exactly why (a) tolerates HDD well. Write amplification (the effect where the physical bytes written to flash exceed the logical bytes the application asked to write, because flash must erase and rewrite in larger blocks than it programs in) is an NVMe/SSD-specific concern with no HDD equivalent; a write-heavy random workload like (b) and the compaction-heavy workload in (c) both increase write amplification, which affects the SSD's endurance/lifespan and can reduce effective write throughput as the drive fills, an over-provisioning margin and a filesystem/database tuned to reduce small random writes (batching, larger block sizes where the workload allows it) both help manage this.

Trade-offs and pitfalls

The most common mistake is picking storage based on throughput or capacity alone and ignoring latency and IOPS, which are what actually determine OLTP and compaction-heavy performance; a spec sheet's MB/s number is close to irrelevant for workload (b). The second common mistake is under-provisioning NVMe capacity for a write-heavy workload without accounting for write amplification, which can quietly erode both endurance and effective throughput well before raw capacity is exhausted.

The command

ps -eo pid,user,rss,comm --sort=-rss | \
  awk 'NR==1{printf "%-8s %-10s %10s  %s\n","PID","USER","RSS(MiB)","COMMAND"}
       NR>1 && NR<=6{printf "%-8s %-10s %10.1f  %s\n",$1,$2,$3/1024,$4}'

ps -eo pid,user,rss,comm --sort=-rss lists every process with its PID, owning user, RSS (Resident Set Size, the memory currently held in physical RAM by that process, in kilobytes) and command name, sorted descending by RSS. The awk pass prints a header row, then formats rows 2 through 6 (the top 5 processes after the header), converting RSS from kilobytes to mebibytes by dividing by 1024 for readability. Running this on a real host will print the top 5 by RSS at that moment; the exact PIDs, users, and values are entirely host- and moment-specific, so no fabricated sample output is shown here.

Caveats when using RSS to estimate true memory usage

• RSS counts shared pages, shared libraries, memory-mapped files, or copy-on-write pages inherited from a fork, fully against EVERY process that maps them. Summing RSS across several processes double- or triple-counts that shared memory, overstating how much RAM would actually be freed by killing all of them.
• RSS does not reflect memory that has been swapped out: a process using a large amount of memory that is currently sitting in swap shows a LOW RSS, understating its true footprint rather than overstating it.
• Neither ps column alone answers 'how much RAM would be freed if I killed this process': VSZ (virtual memory size) overstates it by counting reserved-but-never-touched address space, while RSS understates the true incremental cost of any one process due to sharing. PSS (Proportional Set Size, from /proc/<pid>/smaps_rollup, which divides shared pages fairly across the processes mapping them) or USS (Unique Set Size, private pages only) give a more accurate answer, but neither is exposed by plain ps.

Direct answer

Use the cgroup v2 io controller (the cgroup v1 equivalent was called blkio; both throttle block-layer I/O per cgroup) to either give each container a relative weight (fair-share) or a hard cap on IOPS (input/output operations per second) and throughput (io.max). Weights are fairer under contention and waste less capacity when a container is idle; hard caps give predictable, guaranteed ceilings but can leave a device under-utilized when only one tenant is active.

Discussion

Implementation: io.weight (default 100, range 1-10000) sets a relative share of disk time per device, visible per cgroup at /sys/fs/cgroup/<cgroup>/io.weight. io.max sets hard per-device caps in four independent dimensions: rbps/wbps (read/write bytes per second) and riops/wiops (read/write operations per second), e.g. echo "8:0 rbps=50000000 wbps=20000000" > io.max caps device major:minor 8:0 to 50MB/s reads and 20MB/s writes. Container runtimes (Docker, containerd, Kubernetes via the kubelet) expose both as --blkio-weight / io.weight and per-device IOPS/bandwidth limits in the container spec, which map straight onto these cgroup files.

Fairness: weights only redistribute contention, they do not reserve anything; a cgroup with weight 500 gets roughly 5x the disk-time share of one at weight 100, but only when both are actively issuing I/O. On an idle device a low-weight cgroup can still use 100% of the available throughput, which is usually desirable (unused capacity should not go to waste) but means weight alone does not guarantee a worst-case bound.

Burst behavior: io.max has no built-in token-bucket burst allowance the way, say, a network traffic shaper does; there's no mechanism to bank unused I/O capacity while idle and spend it later. What looks like burst tolerance is really an artifact of enforcement granularity: the block-layer throttle evaluates the configured rbps/riops/wbps/wiops caps over a short rolling accounting window rather than instantaneously, so a workload can briefly exceed the nominal cap within that window before the throttle catches up and starts delaying requests. io.latency is a separate, unrelated cgroup v2 controller and does not give the cgroup it's applied to any burst allowance for io.max; it works the other direction, protecting one cgroup's I/O latency by throttling other, lower-priority cgroups on the same device once the protected cgroup's measured latency exceeds its configured target. If you actually need deliberate burst-then-throttle behavior (accumulate credit while idle, spend it under load, similar to cloud burstable-volume IOPS credits), the kernel's io controller doesn't provide that natively for io.max; it has to be built above the kernel, in the application or a sidecar. Weight-based fairness (io.weight) has no burst concept either; it is purely proportional at whatever instant contention exists.

Latency impact: hard IOPS/bandwidth caps convert what would be a fast I/O completion into an artificially delayed one once the cap is hit, which shows up as increased read/write latency inside the throttled container even though the underlying device has spare capacity, because the kernel is intentionally queuing or rejecting requests to enforce the cap. This is the most common surprise: a team sees their own container's latency spike after ops adds an I/O limit and assumes the disk is failing, when actually the limit is working as designed.

Verification: read back the cgroup's own I/O accounting at /sys/fs/cgroup/<cgroup>/io.stat, which reports rbytes, wbytes, rios, wios per device and lets you confirm actual throughput matches the configured cap under load; cross-check with iostat -x 1 on the host to see aggregate device utilization, and with the application's own latency metrics to see the throttling's user-visible cost. Load-test with fio (Flexible I/O Tester, a standard Linux disk I/O benchmarking tool) targeting the container specifically (not the host) to validate the cap holds under the container's real I/O pattern (random 4K vs sequential 1M writes throttle very differently even at the same nominal cap).

Limitations: the io controller only throttles requests that go through the block layer with a cgroup context attached, so buffered writes that hit the page cache first and get written back later by a kernel writeback thread can, on older kernels, lose their cgroup association and evade the cap (this is largely fixed on modern kernels with cgroup-aware writeback, but worth verifying on the specific kernel version in use). Writes issued with O_DIRECT (an open flag that bypasses the page cache so I/O goes straight from the application's buffer to the block layer) sidestep that specific writeback-attribution problem, since there is no intermediate writeback thread to lose the association, but O_DIRECT I/O is just as subject to io.max/io.weight caps as any other block-layer request; it does not grant an exemption. Caps are per-device, so a container spanning multiple physical devices or a logical volume striped across several disks needs the limit applied at the right layer to actually constrain aggregate I/O. Network-attached storage (NFS, iSCSI) is not always cleanly covered by the block-layer io controller in the same way local block devices are.

Trade-offs and pitfalls

Weights are the right default for fair-sharing a busy shared disk; hard caps (io.max) are the right tool when you need a guaranteed, predictable ceiling for capacity planning or SLA (service-level agreement, a commitment about the level of service provided) purposes, at the cost of wasting device capacity when the capped container is the only active tenant. The single biggest pitfall is setting a cap without first measuring the container's actual I/O pattern with a representative benchmark; a cap that looks generous for sequential throughput can be crushing for a workload that is actually IOPS-bound on small random reads, since bytes/sec and ops/sec are independent dimensions.

Direct answer

Use fio (Flexible I/O tester, the standard Linux disk-benchmarking tool) with direct=1 to bypass the page cache (the kernel's in-memory cache of recently read/written disk blocks, which would otherwise let a small working set serve reads from RAM and make the disk look far faster than it is), sweep a small matrix of block sizes and queue depths (iodepth), run each combination long enough to reach steady state after a discarded warmup period, and report IOPS (I/O operations per second), throughput, and latency percentiles (not just the average) for each cell. The result you present is not a single number but a table (or a latency-versus-throughput curve) showing where the device's latency starts climbing sharply as queue depth increases, which is the actionable signal for capacity planning.

Structured elaboration

Test matrix. Sweep two dimensions independently, holding the other fixed at a representative default, then run the full cross product if time allows:

• Block sizes: 4KiB, 16KiB, 64KiB, 256KiB, 1MiB (spans small-random-IO-dominated to large-sequential-dominated workloads).
• Queue depths (iodepth): 1, 4, 8, 16, 32, 64 (spans "nothing in flight, latency-bound" to "deeply queued, throughput-bound").
• Access pattern: run randread, randwrite, read (sequential), and write (sequential) as separate jobs; random and sequential performance can differ by an order of magnitude on the same device, especially spinning disks, and conflating them hides that.

Exact fio parameters, and why each one is there.

[job]
ioengine=libaio        ; async I/O engine so iodepth actually queues multiple requests
direct=1                ; O_DIRECT: bypass the page cache, talk straight to the block device
rw=randread              ; swept: randread, randwrite, read, write
bs=4k                    ; swept: 4k, 16k, 64k, 256k, 1m
iodepth=16               ; swept: 1, 4, 8, 16, 32, 64
numjobs=1                ; keep at 1 for a single-queue characterization; raise separately to test concurrency scaling
size=20g                 ; working-set size, chosen larger than the box's RAM (see cache note below)
runtime=60               ; steady-state measurement window
ramp_time=10             ; warmup, EXCLUDED from reported stats
time_based=1             ; keep running for the full runtime even if size is read/written before then
group_reporting=1        ; one aggregated report per job, not per-thread
invalidate=1             ; (fio default) drop cached pages for the target before the run starts
output-format=json       ; machine-parseable, needed to pull clat percentiles programmatically

• direct=1 is the primary page-cache bypass; it forces every I/O through O_DIRECT, which requires the buffer, offset, and length to be aligned to the device's logical block size (typically 512B or 4096B) or the I/O fails, which is why block sizes below 4KiB are usually skipped on modern devices.
• size deliberately larger than system RAM (or run against a raw block device with no filesystem at all, when that is safe to do) is a second, independent safeguard against page-cache effects: even with direct=1, some filesystems and layers can still exhibit caching behavior for metadata or via a volume manager layer, so making the dataset not fit in RAM removes any ambiguity.
• ramp_time=10 runs and discards 10 seconds before fio starts counting, so the numbers reflect steady-state device behavior, not the initial burst while queues are filling up from empty.
• For a batch of these jobs run back to back in one job file (all block-size/iodepth combinations in sequence), add stonewall between job stanzas so they run strictly one after another rather than fio trying to run them concurrently, which would make every cell's numbers reflect contention with its neighbors instead of the device alone.

Metrics to capture, per cell. IOPS, throughput (MB/s), and latency (clat, completion latency) at p50, p95, p99 (the 50th/95th/99th percentile, the value below which that percentage of requests complete; p99 is roughly the slowest 1%), and max, not just the mean: a device can have a fine average latency and a terrible p99 under load, and average-only reporting hides exactly the tail behavior most latency-sensitive services care about. fio's JSON output includes a clat_ns.percentile block when output-format=json and appropriate percentile_list is set (e.g. percentile_list=50:95:99:99.9).

Presenting results meaningfully. A flat table of (block size x queue depth) cells with IOPS/throughput/p99 is the raw data; the meaningful presentation on top of it is a latency-versus-achieved-throughput curve per block size, with queue depth as the swept parameter along the curve. As queue depth increases from 1 upward, throughput climbs roughly linearly at first while latency stays low and flat; past some point (the "knee"), additional queue depth buys little or no extra throughput while latency grows sharply, because the device is now saturated and requests are simply queueing. That knee, not the peak IOPS number alone, is the number worth reporting for capacity planning: it tells you the queue depth beyond which you are trading latency for no real throughput gain.

Worked example

Illustrative shape of the result, with plausible but non-measured numbers to show the pattern a real run would reveal, on a hypothetical NVMe device, random 4KiB reads:

Reading this: IOPS scales close to linearly with queue depth through iodepth 16, then flattens sharply between 32 and 64 while p99 latency roughly triples over the same range (1.40ms to 4.80ms, about a 3.4x increase). The knee sits around iodepth 32: that is the depth to design an application's connection pool or async I/O concurrency around, not 64, because past 32 the device gives back almost nothing in throughput while candidate requests wait far longer.

Trade-offs and pitfalls

• Testing against a file inside a filesystem (even with direct=1) versus a raw block device can differ because of filesystem-level overhead (allocation, journaling, metadata updates on writes); state explicitly which one was tested, since the two are not interchangeable results.
• Write tests are destructive and, on flash-based devices, contribute to wear; run write benchmarks against scratch/disposable devices or partitions, never against a device holding data you care about, and be aware that a fresh flash device can show artificially high write performance until it has been "conditioned" by prior writes (SSD steady-state behavior differs from a factory-fresh drive).
• A single short run can be misleading on devices with large internal caches (some SSDs/HBAs (HBA: host bus adapter, the controller card connecting the drive) have their own DRAM or SLC cache (a small, fast-but-expensive flash tier some SSDs use as a write buffer) ahead of the slower backing media): a short runtime can measure the cache's speed, not the device's sustained speed. Extend runtime (minutes, not seconds) when characterizing sustained rather than burst performance.
• numjobs=1 deliberately isolates single-queue behavior from application-level concurrency; if the real workload issues I/O from many threads/processes concurrently, also run a numjobs sweep, since aggregate throughput and per-request latency under many concurrent issuers do not simply multiply from the single-job numbers.
• Comparing results across different runs or machines only makes sense if every parameter that affects the outcome (block size, iodepth, engine, direct flag, runtime, filesystem vs raw device) is held identical and stated alongside the numbers; a throughput number with no stated block size or queue depth is not a meaningful benchmark result.

Mechanism: how a kernel upgrade breaks NUMA placement

NUMA (Non-Uniform Memory Access: each CPU socket has its own local memory, and reaching another socket's memory over the interconnect costs extra latency) placement is governed by kernel defaults that can change silently across an upgrade. Two common culprits: the default value of vm.zone_reclaim_mode (historically some distributions shipped it on by default on certain kernel versions, which prefers reclaiming local memory over allocating remote memory, and can end up thrashing the page cache instead of simply placing the allocation remotely) and automatic NUMA balancing (kernel.numa_balancing), the kernel's own background feature that migrates pages and threads toward each other over time, which can become more or less aggressive between kernel releases and add scanning overhead the database was not tuned to expect. Either change can quietly break a database's previously stable locality assumptions without anyone touching the database's own configuration.

Stepwise detection and remediation

1. Confirm the regression is actually NUMA-related before chasing it: check numastat -p <db_pid> now, and if a historical baseline exists, compare numa_hit/numa_miss/other_node before and after the upgrade.
2. Rule out a hardware or firmware change at the same maintenance window: numactl --hardware should show the same topology as before; if BIOS NUMA interleaving settings were also touched during the same maintenance, that is a competing explanation worth ruling out first.
3. Diff the relevant sysctls against a known-good baseline: sysctl kernel.numa_balancing, sysctl vm.zone_reclaim_mode, and cat /sys/kernel/mm/numa_balancing. A silent default change here is the most common root cause for this exact symptom (regression right after an otherwise routine kernel upgrade).
4. Confirm the database's CPU affinity mask is still actually applied: taskset -pc <pid>. An upgrade to the init system or container runtime can drop an affinity setting that used to be set at boot without any visible error.
5. Run a controlled experiment: force strict locality on a canary instance with numactl --cpunodebind=N --membind=N around the database's startup, and measure whether latency recovers. This isolates whether placement is genuinely the cause versus something else that coincidentally changed with the kernel.
6. If automatic NUMA balancing overhead is the culprit, look for elevated numa_pages_migrated in /proc/vmstat alongside CPU time in kernel migration threads; disable it (echo 0 > /proc/sys/kernel/numa_balancing) and re-pin statically instead, trading the kernel's adaptive behavior for a known-good fixed layout.
7. Make the fix durable: pin the database explicitly through the systemd unit (AllowedCPUs= or a numactl wrapper) rather than relying on kernel defaults, and record the required sysctl values in configuration management so drift-detection catches the next kernel upgrade's default change before it pages someone.