System Resource & I/O Optimization Questions

Direct answer

Use the cgroup v2 io controller (the cgroup v1 equivalent was called blkio; both throttle block-layer I/O per cgroup) to either give each container a relative weight (fair-share) or a hard cap on IOPS (input/output operations per second) and throughput (io.max). Weights are fairer under contention and waste less capacity when a container is idle; hard caps give predictable, guaranteed ceilings but can leave a device under-utilized when only one tenant is active.

Discussion

Implementation: io.weight (default 100, range 1-10000) sets a relative share of disk time per device, visible per cgroup at /sys/fs/cgroup/<cgroup>/io.weight. io.max sets hard per-device caps in four independent dimensions: rbps/wbps (read/write bytes per second) and riops/wiops (read/write operations per second), e.g. echo "8:0 rbps=50000000 wbps=20000000" > io.max caps device major:minor 8:0 to 50MB/s reads and 20MB/s writes. Container runtimes (Docker, containerd, Kubernetes via the kubelet) expose both as --blkio-weight / io.weight and per-device IOPS/bandwidth limits in the container spec, which map straight onto these cgroup files.

Fairness: weights only redistribute contention, they do not reserve anything; a cgroup with weight 500 gets roughly 5x the disk-time share of one at weight 100, but only when both are actively issuing I/O. On an idle device a low-weight cgroup can still use 100% of the available throughput, which is usually desirable (unused capacity should not go to waste) but means weight alone does not guarantee a worst-case bound.

Burst behavior: io.max has no built-in token-bucket burst allowance the way, say, a network traffic shaper does; there's no mechanism to bank unused I/O capacity while idle and spend it later. What looks like burst tolerance is really an artifact of enforcement granularity: the block-layer throttle evaluates the configured rbps/riops/wbps/wiops caps over a short rolling accounting window rather than instantaneously, so a workload can briefly exceed the nominal cap within that window before the throttle catches up and starts delaying requests. io.latency is a separate, unrelated cgroup v2 controller and does not give the cgroup it's applied to any burst allowance for io.max; it works the other direction, protecting one cgroup's I/O latency by throttling other, lower-priority cgroups on the same device once the protected cgroup's measured latency exceeds its configured target. If you actually need deliberate burst-then-throttle behavior (accumulate credit while idle, spend it under load, similar to cloud burstable-volume IOPS credits), the kernel's io controller doesn't provide that natively for io.max; it has to be built above the kernel, in the application or a sidecar. Weight-based fairness (io.weight) has no burst concept either; it is purely proportional at whatever instant contention exists.

Latency impact: hard IOPS/bandwidth caps convert what would be a fast I/O completion into an artificially delayed one once the cap is hit, which shows up as increased read/write latency inside the throttled container even though the underlying device has spare capacity, because the kernel is intentionally queuing or rejecting requests to enforce the cap. This is the most common surprise: a team sees their own container's latency spike after ops adds an I/O limit and assumes the disk is failing, when actually the limit is working as designed.

Verification: read back the cgroup's own I/O accounting at /sys/fs/cgroup/<cgroup>/io.stat, which reports rbytes, wbytes, rios, wios per device and lets you confirm actual throughput matches the configured cap under load; cross-check with iostat -x 1 on the host to see aggregate device utilization, and with the application's own latency metrics to see the throttling's user-visible cost. Load-test with fio (Flexible I/O Tester, a standard Linux disk I/O benchmarking tool) targeting the container specifically (not the host) to validate the cap holds under the container's real I/O pattern (random 4K vs sequential 1M writes throttle very differently even at the same nominal cap).

Limitations: the io controller only throttles requests that go through the block layer with a cgroup context attached, so buffered writes that hit the page cache first and get written back later by a kernel writeback thread can, on older kernels, lose their cgroup association and evade the cap (this is largely fixed on modern kernels with cgroup-aware writeback, but worth verifying on the specific kernel version in use). Writes issued with O_DIRECT (an open flag that bypasses the page cache so I/O goes straight from the application's buffer to the block layer) sidestep that specific writeback-attribution problem, since there is no intermediate writeback thread to lose the association, but O_DIRECT I/O is just as subject to io.max/io.weight caps as any other block-layer request; it does not grant an exemption. Caps are per-device, so a container spanning multiple physical devices or a logical volume striped across several disks needs the limit applied at the right layer to actually constrain aggregate I/O. Network-attached storage (NFS, iSCSI) is not always cleanly covered by the block-layer io controller in the same way local block devices are.

Trade-offs and pitfalls

Weights are the right default for fair-sharing a busy shared disk; hard caps (io.max) are the right tool when you need a guaranteed, predictable ceiling for capacity planning or SLA (service-level agreement, a commitment about the level of service provided) purposes, at the cost of wasting device capacity when the capped container is the only active tenant. The single biggest pitfall is setting a cap without first measuring the container's actual I/O pattern with a representative benchmark; a cap that looks generous for sequential throughput can be crushing for a workload that is actually IOPS-bound on small random reads, since bytes/sec and ops/sec are independent dimensions.

Framing

A write-ahead log (WAL, the append-only durability record a database or distributed store writes before applying a change) needs every committed write to survive a crash. Turning off the disk's write-back cache is the blunt way to get that guarantee: every write physically lands on media before the drive reports success. But it also means every write now waits for a real rotate/seek or NAND program cycle, which is why latency jumps. The better answer is not 'cache on or off' but a stack of independent techniques that each buy back some latency without giving up durability.

Kernel level

• Use fdatasync() instead of fsync() for WAL segments: it flushes data and only the metadata needed to read it back (not access-time/size-unrelated metadata), which is one fewer metadata write per commit.
• Batch flushes with io_uring or libaio (Linux APIs that let an application submit many I/O requests without blocking on each one individually): submit several WAL writes and one fsync per batch instead of a syscall round trip per write.

Filesystem level

• Put the WAL on its own partition/filesystem (ext4 or XFS with data=writeback journal mode) so WAL fsyncs are not serialized behind unrelated journal commits (a filesystem journal is a log of pending metadata/data changes the filesystem writes and commits before applying them, so it can recover cleanly after a crash; committing it costs its own I/O) from other files on the same volume.
• Pre-allocate and pre-zero WAL segment files (fallocate) so a commit-time write is a pure in-place overwrite, not a filesystem metadata operation (extent allocation) that adds its own journal commit.

Hardware level (the actual fix for the trade-off in the question)

• Prefer a drive with power-loss protection (PLP): enterprise SSDs with PLP hold enough capacitor charge to flush the on-drive cache to flash during a power failure, so the write-back cache can stay ON (fast) while still being durable, because 'cache' no longer means 'volatile'. This removes the false dilemma the question poses.
• If PLP hardware is not an option, use per-I/O Force Unit Access (FUA) instead of a global cache-disable: FUA tells the drive to bypass its cache for that specific write only, so the WAL fsync path is durable while unrelated reads/writes on the same device still benefit from caching.

Application level

• Group commit: instead of one fsync per transaction, batch transactions that arrive within a small window (for example 1 to 5 ms) into a single fsync. If a workload does roughly 5,000 individual commits per second and each fsync costs about 1 ms on a non-PLP disk, per-transaction fsyncs cap throughput near 1,000 tx/s; batching 10 transactions per fsync raises that ceiling roughly 10x at the cost of up to one batching window of added latency per transaction, a bounded and tunable trade rather than an unbounded one.
• Parallel WAL streams across multiple independent devices so one commit's fsync latency does not serialize every other stream's commits.

None of these individually removes the durability guarantee; together they turn 'cache off' from the only lever into one of several, most of which cost far less latency.

Direct answer

cgroups (control groups) are a Linux kernel mechanism for grouping processes together and then measuring, limiting, and prioritizing their collective use of resources like CPU, memory, and I/O (input/output). cgroups v2 replaced v1's design of separate, independently-mounted hierarchies per resource controller with a single unified hierarchy where every controller applies to the same process tree. For new deployments, v2 is the right default: it's simpler, it's what systemd and modern container runtimes actually assume, and it exposes better signals like pressure stall information.

Structured elaboration

cgroups v1's design and its problems. In v1, each controller (cpu, memory, blkio, and others) is mounted as its own separate cgroup filesystem, and a process can in principle belong to different groups in different controller hierarchies at the same time. That flexibility sounds useful but in practice made delegation (letting something like a container runtime manage a sub-tree of cgroups safely) fragile, made accounting inconsistent between controllers (the cpu and cpuacct split is a well-known wart), and meant tooling had to reason about multiple independent trees instead of one coherent picture of "what resources does this group of processes use."

cgroups v2's unified model. v2 uses a single hierarchy: every process belongs to exactly one cgroup, and all controllers (cpu, memory, io, pids, and others) apply consistently to that same tree. This makes delegation to a container runtime or to systemd itself much cleaner (a runtime can safely manage a sub-tree without controller mismatches), and it enabled per-cgroup PSI (pressure stall information: the cpu.pressure, memory.pressure and io.pressure files, which report how much time the processes in that specific group actually spent stalled waiting for a resource, a far more actionable signal than a raw utilization percentage). Be precise about which half of PSI is the v2 feature: the host-wide files under /proc/pressure/ are a separate kernel feature (added in 4.20, gated on CONFIG_PSI) and are present regardless of which cgroup version the host boots, so a pure v1 host still has them. What v1 never got is the per-cgroup breakdown, which is the half that lets you attribute a stall to one specific service instead of to the box as a whole. v2 also introduced memory.high, a soft, reclaim-triggering ceiling below the hard memory.max limit, letting the kernel apply graceful backpressure to a group before resorting to killing a process.

Why an SRE (site reliability engineer) would prefer v2 for new deployments. systemd has used cgroups v2 as its default resource-control mechanism for several years now, and container runtimes (containerd, Docker, CRI-O) have followed. Building on v2 means your systemd unit MemoryMax=/CPUQuota= directives, your container runtime's resource limits, and your own monitoring all agree on the same underlying accounting, instead of straddling two incompatible models. It also gives you PSI natively, which is a materially better early-warning signal for "this group is starting to suffer resource contention" than watching raw CPU or memory percentages, since PSI tells you about actual stalled time, not just busy time.

Worked example

Checking which version a host is running: mount | grep cgroup. On a v2-only host you'll see a single line for cgroup2 mounted at /sys/fs/cgroup. On a hybrid or v1 host you'll see several separate cgroup mounts, one per controller (/sys/fs/cgroup/cpu, /sys/fs/cgroup/memory, and so on). Reading a v2 cgroup's memory pressure directly: cat /sys/fs/cgroup/system.slice/myservice.service/memory.pressure gives you PSI output like:

some avg10=0.00 avg60=0.00 avg300=0.00 total=0
full avg10=0.00 avg60=0.00 avg300=0.00 total=0

Non-zero avg10 on the full line for that specific cgroup means processes in that cgroup are being actively stalled by memory pressure right now, a signal v1 simply doesn't expose per-group.

Trade-offs and pitfalls

Some older container images or orchestration tooling still assume v1 semantics and can misbehave (or refuse to start) on a v2-only host if they haven't been updated; check your runtime and orchestrator versions before flipping a fleet over, rather than assuming compatibility. On a genuinely mixed environment you may need to run in "hybrid" mode (some controllers on v1, the rest on v2) as a migration step, but that's a transitional state, not a target architecture, since it reintroduces some of v1's split-hierarchy fragility.

Direct answer

Use fio (Flexible I/O tester, the standard Linux disk-benchmarking tool) with direct=1 to bypass the page cache (the kernel's in-memory cache of recently read/written disk blocks, which would otherwise let a small working set serve reads from RAM and make the disk look far faster than it is), sweep a small matrix of block sizes and queue depths (iodepth), run each combination long enough to reach steady state after a discarded warmup period, and report IOPS (I/O operations per second), throughput, and latency percentiles (not just the average) for each cell. The result you present is not a single number but a table (or a latency-versus-throughput curve) showing where the device's latency starts climbing sharply as queue depth increases, which is the actionable signal for capacity planning.

Structured elaboration

Test matrix. Sweep two dimensions independently, holding the other fixed at a representative default, then run the full cross product if time allows:

• Block sizes: 4KiB, 16KiB, 64KiB, 256KiB, 1MiB (spans small-random-IO-dominated to large-sequential-dominated workloads).
• Queue depths (iodepth): 1, 4, 8, 16, 32, 64 (spans "nothing in flight, latency-bound" to "deeply queued, throughput-bound").
• Access pattern: run randread, randwrite, read (sequential), and write (sequential) as separate jobs; random and sequential performance can differ by an order of magnitude on the same device, especially spinning disks, and conflating them hides that.

Exact fio parameters, and why each one is there.

[job]
ioengine=libaio        ; async I/O engine so iodepth actually queues multiple requests
direct=1                ; O_DIRECT: bypass the page cache, talk straight to the block device
rw=randread              ; swept: randread, randwrite, read, write
bs=4k                    ; swept: 4k, 16k, 64k, 256k, 1m
iodepth=16               ; swept: 1, 4, 8, 16, 32, 64
numjobs=1                ; keep at 1 for a single-queue characterization; raise separately to test concurrency scaling
size=20g                 ; working-set size, chosen larger than the box's RAM (see cache note below)
runtime=60               ; steady-state measurement window
ramp_time=10             ; warmup, EXCLUDED from reported stats
time_based=1             ; keep running for the full runtime even if size is read/written before then
group_reporting=1        ; one aggregated report per job, not per-thread
invalidate=1             ; (fio default) drop cached pages for the target before the run starts
output-format=json       ; machine-parseable, needed to pull clat percentiles programmatically

• direct=1 is the primary page-cache bypass; it forces every I/O through O_DIRECT, which requires the buffer, offset, and length to be aligned to the device's logical block size (typically 512B or 4096B) or the I/O fails, which is why block sizes below 4KiB are usually skipped on modern devices.
• size deliberately larger than system RAM (or run against a raw block device with no filesystem at all, when that is safe to do) is a second, independent safeguard against page-cache effects: even with direct=1, some filesystems and layers can still exhibit caching behavior for metadata or via a volume manager layer, so making the dataset not fit in RAM removes any ambiguity.
• ramp_time=10 runs and discards 10 seconds before fio starts counting, so the numbers reflect steady-state device behavior, not the initial burst while queues are filling up from empty.
• For a batch of these jobs run back to back in one job file (all block-size/iodepth combinations in sequence), add stonewall between job stanzas so they run strictly one after another rather than fio trying to run them concurrently, which would make every cell's numbers reflect contention with its neighbors instead of the device alone.

Metrics to capture, per cell. IOPS, throughput (MB/s), and latency (clat, completion latency) at p50, p95, p99 (the 50th/95th/99th percentile, the value below which that percentage of requests complete; p99 is roughly the slowest 1%), and max, not just the mean: a device can have a fine average latency and a terrible p99 under load, and average-only reporting hides exactly the tail behavior most latency-sensitive services care about. fio's JSON output includes a clat_ns.percentile block when output-format=json and appropriate percentile_list is set (e.g. percentile_list=50:95:99:99.9).

Presenting results meaningfully. A flat table of (block size x queue depth) cells with IOPS/throughput/p99 is the raw data; the meaningful presentation on top of it is a latency-versus-achieved-throughput curve per block size, with queue depth as the swept parameter along the curve. As queue depth increases from 1 upward, throughput climbs roughly linearly at first while latency stays low and flat; past some point (the "knee"), additional queue depth buys little or no extra throughput while latency grows sharply, because the device is now saturated and requests are simply queueing. That knee, not the peak IOPS number alone, is the number worth reporting for capacity planning: it tells you the queue depth beyond which you are trading latency for no real throughput gain.

Worked example

Illustrative shape of the result, with plausible but non-measured numbers to show the pattern a real run would reveal, on a hypothetical NVMe device, random 4KiB reads:

Reading this: IOPS scales close to linearly with queue depth through iodepth 16, then flattens sharply between 32 and 64 while p99 latency roughly triples over the same range (1.40ms to 4.80ms, about a 3.4x increase). The knee sits around iodepth 32: that is the depth to design an application's connection pool or async I/O concurrency around, not 64, because past 32 the device gives back almost nothing in throughput while candidate requests wait far longer.

Trade-offs and pitfalls

• Testing against a file inside a filesystem (even with direct=1) versus a raw block device can differ because of filesystem-level overhead (allocation, journaling, metadata updates on writes); state explicitly which one was tested, since the two are not interchangeable results.
• Write tests are destructive and, on flash-based devices, contribute to wear; run write benchmarks against scratch/disposable devices or partitions, never against a device holding data you care about, and be aware that a fresh flash device can show artificially high write performance until it has been "conditioned" by prior writes (SSD steady-state behavior differs from a factory-fresh drive).
• A single short run can be misleading on devices with large internal caches (some SSDs/HBAs (HBA: host bus adapter, the controller card connecting the drive) have their own DRAM or SLC cache (a small, fast-but-expensive flash tier some SSDs use as a write buffer) ahead of the slower backing media): a short runtime can measure the cache's speed, not the device's sustained speed. Extend runtime (minutes, not seconds) when characterizing sustained rather than burst performance.
• numjobs=1 deliberately isolates single-queue behavior from application-level concurrency; if the real workload issues I/O from many threads/processes concurrently, also run a numjobs sweep, since aggregate throughput and per-request latency under many concurrent issuers do not simply multiply from the single-job numbers.
• Comparing results across different runs or machines only makes sense if every parameter that affects the outcome (block size, iodepth, engine, direct flag, runtime, filesystem vs raw device) is held identical and stated alongside the numbers; a throughput number with no stated block size or queue depth is not a meaningful benchmark result.

Why load average can spike without CPU or disk moving

On Linux, load average counts both runnable (state R) processes and processes in uninterruptible sleep (state D), unlike some other Unix systems that count only runnable ones. A process in D-state is blocked waiting on something the kernel considers uninterruptible, usually I/O of some kind, and it inflates load average even though it is doing zero CPU work and may not even be driving visible disk throughput if what it is actually waiting on is a hung remote resource rather than a busy local disk.

Kernel-level causes to consider

• An unresponsive or hung NFS (Network File System) server: a client thread doing a synchronous NFS RPC (remote procedure call) under a hard mount option blocks in D-state indefinitely by design, since hard mounts are meant to retry forever rather than fail; the hard versus soft mount trade-off is exactly this durability-versus-availability choice.
• A hung or failing-over storage path: a multipath LUN (logical unit, a block storage target) stuck mid-failover can put processes doing I/O to that specific device in D-state even while iostat's aggregate view across all disks looks perfectly healthy.
• A stuck or misbehaving IRQ (interrupt request) handler or driver holding a kernel spinlock, or a deadlocked kernel mutex inside a filesystem, can produce the same symptom without any of the usual CPU or disk utilization metrics moving.

Systematic debugging plan and evidence to capture

1. ps -eo stat,pid,comm | grep ' D' (or ps aux | awk '$8 ~ /D/') to find which processes are in D-state and what commands they are running.
2. cat /proc/<pid>/stack (requires root and a kernel built with stack-trace support) or cat /proc/<pid>/wchan to see exactly which kernel function the process is blocked inside; a name like an NFS wait function or an XFS journal wait function immediately tells you which subsystem to investigate further, rather than guessing.
3. For NFS specifically, nfsstat -c and nfsiostat show RPC retransmits and timeouts against the specific server, letting you check that server's own health independently instead of only looking at the client symptom.
4. watch -d cat /proc/interrupts to spot an interrupt counter that has stopped incrementing entirely (a stuck handler) or one counter spiking disproportionately on a single CPU.
5. dmesg | grep -i 'hung task': the kernel's own hung-task detector logs a full stack trace automatically for any task that stays in D-state longer than kernel.hung_task_timeout_secs (120 seconds by default), which is often the fastest single path to root cause, since it has already done the stack-inspection work for you.

Summary of the approach

Do not trust CPU- or disk-utilization dashboards alone when load average spikes without a matching resource signal; go straight to ps for D-state processes, dmesg for the kernel's own hung-task log, and /proc/<pid>/stack for the specific blocking point, because Linux's runnable-plus-uninterruptible-sleep definition of load average captures waiters that never register as 'busy' on a standard resource dashboard.