System Resource & I/O Optimization Questions

Direct answer

Layer isolation at three points: admission (reject bad requests before they hit the scheduler), scheduling (requests/limits and Quality of Service, QoS, classes that determine who gets evicted first), and the node's cgroups (control groups, the Linux kernel mechanism that actually enforces CPU, memory, and I/O caps). Kubernetes wires the first two together well by default; the third, especially I/O isolation, has to be deliberately configured because the default cgroup setup only guarantees CPU and memory isolation well.

Design

Admission controls: a ResourceQuota per tenant namespace caps aggregate CPU, memory, and pod count for that tenant, so one team cannot schedule enough pods to starve everyone else regardless of per-pod limits. A LimitRange sets default and max/min requests-limits per container so a pod with no explicit resources still gets something sane. A validating/mutating admission webhook (built with Kyverno or OPA Gatekeeper, two common Kubernetes policy-engine tools that let you write custom admission rules without hand-rolling a webhook server; either works, pick whichever your team already runs, or a small custom one) enforces policy ResourceQuota cannot express: reject any pod with no memory limit at all, require requests == limits for anything labeled tier: critical, and block hostPath or privileged mounts that could let a tenant reach outside its cgroup.

Default requests/limits: set per-namespace defaults via LimitRange so a forgotten resource block does not silently become BestEffort. Size Guaranteed-tier requests near actual peak usage (not headroom), because Guaranteed requires requests == limits, so over-provisioning there is pure waste.

QoS classes: Kubernetes derives three classes from what you set. Guaranteed (every container's CPU and memory request equals its limit) is evicted last and gets the highest OOM-score protection. Burstable (requests below limits) is evicted before Guaranteed under memory pressure. BestEffort (no requests or limits set) is evicted first. Map workload tiers directly: latency-sensitive critical services get Guaranteed, standard services get Burstable, batch/background jobs get BestEffort or a low-priority Burstable capped hard by the batch namespace's ResourceQuota.

Eviction thresholds: the kubelet's --eviction-hard (e.g. memory.available<5%, nodefs.available<10%) triggers immediate, ungraceful eviction; --eviction-soft paired with --eviction-soft-grace-period gives pods time to shut down cleanly before the hard threshold is reached. Under pressure the kubelet evicts BestEffort pods first, then Burstable pods using more than their request, then Guaranteed pods only as a last resort. Pair this with PodDisruptionBudgets on critical services so an eviction sweep cannot take out every replica at once.

Node-level cgroup settings: run the kubelet with --cgroup-driver=systemd on cgroup v2 hosts so kubelet-managed limits (memory.max, memory.high, cpu.weight) land in the same unified hierarchy systemd itself uses, avoiding the split-brain that cgroup v1's cgroupfs/systemd mismatch used to cause. Enable --cpu-manager-policy=static and --memory-manager-policy=Static for Guaranteed, latency-sensitive pods that need exclusive, NUMA (non-uniform memory access: memory attached to one CPU socket is faster for that socket to reach than memory attached to another)-local cores. Kubernetes does not set I/O weights per QoS class by default, so add a small DaemonSet or node-init step that writes io.weight (or io.max for a hard IOPS, input/output operations per second, and throughput cap) under the BestEffort and Burstable kubepods.slice paths, so a batch job doing heavy disk writes cannot starve a critical pod's reads on the same disk.

Custom controllers/webhooks: the admission webhook above for policy enforcement, plus an optional lightweight controller that watches node-pressure metrics (node_memory_MemAvailable_bytes, node_disk_io_time_seconds_total) and proactively taints or cordons a node before the kubelet's own eviction kicks in, giving the scheduler time to steer new pods elsewhere instead of reacting after the fact.

Worked example

A 16-vCPU / 64GiB node runs three tenants. tier: critical gets Guaranteed pods at cpu: 2, memory: 4Gi (request equals limit). tier: standard gets Burstable pods at requests: {cpu: 250m, memory: 256Mi}, limits: {cpu: 1, memory: 512Mi}. tier: batch is capped at the namespace level by a ResourceQuota of limits.cpu: 8, limits.memory: 16Gi across the whole namespace, so even a misconfigured batch pod requesting cpu: 20 is rejected at admission, before the scheduler ever sees it. Eviction is set to --eviction-hard=memory.available<5% (about 3.2GiB free) and --eviction-soft=memory.available<10% with a 90-second grace period, so a batch memory leak gets 90 seconds to be killed gracefully before the node starts hard-evicting.

Trade-offs and pitfalls

Guaranteed QoS is expensive: since requests must equal limits, you provision for peak rather than average, which lowers bin-packing efficiency across the cluster. Static CPU/memory manager policies reserve whole cores per pod, which is even less flexible. Admission webhooks add request latency and become a new availability dependency for the cluster's control plane: decide explicitly whether the webhook fails open (a webhook outage lets unpoliced pods through) or fails closed (an outage blocks all pod creation), and monitor the webhook itself. The single most common gap is assuming requests/limits solve I/O isolation the same way they solve CPU/memory: cgroup v1's blkio controller is weak and cgroup v2's io controller must be explicitly wired to the kubepods hierarchy, or a BestEffort batch job doing sequential writes will still starve a Guaranteed pod's disk reads on the same device.

Root cause: TIME_WAIT, not a systemd bug

After a TCP connection closes, the side that closed first holds that connection's 4-tuple (local IP/port, remote IP/port) in TIME_WAIT for 2*MSL (Maximum Segment Lifetime, roughly 60 seconds total on Linux by default) to guarantee any duplicate or delayed packets from the old connection cannot be mistaken for a new one. If the LISTENING socket itself was bound without SO_REUSEADDR, a fast restart can hit 'Address already in use' because the kernel refuses to bind a new listening socket to a port that still has any sockets, including TIME_WAIT ones, associated with it.

Application-level fix

Set SO_REUSEADDR on the listening socket before calling bind(). Most mature server frameworks already do this by default; a hand-rolled or minimal server might not, and that omission is the actual bug here, not TCP itself.

Unit file changes

• Add RestartSec=<a few seconds> to Restart=on-failure as a crude mitigation: it gives the old socket time to fully release before the new process tries to bind.
• The real system-level mitigation is socket activation: pair the service with a .socket unit. systemd itself creates and binds the listening socket once, then hands it to the service process over a file descriptor on every restart. The listening socket now outlives individual restarts of the service entirely, so the bind race disappears rather than being merely reduced.
• Ensure the unit has a real graceful shutdown path (a correct ExecStop or signal handling that lets the process close its socket cleanly) rather than being SIGKILLed, since an abrupt kill can leave lingering state that makes the race more likely, not less.

TIME_WAIT implications to know

• TIME_WAIT is a correctness mechanism, not a bug to be tuned away in general. It prevents old duplicate segments from corrupting a new connection reusing the same address/port pair.
• net.ipv4.tcp_tw_reuse=1 is commonly confused with a fix for this: it allows outbound (client-side) connections to reuse a TIME_WAIT 4-tuple faster, but it has no effect on the ability to rebind a LISTENING socket, which is governed only by SO_REUSEADDR. Reaching for tcp_tw_reuse here is a common wrong turn worth explicitly ruling out in the answer.

Direct answer

The page cache is RAM the kernel uses to hold copies of disk blocks so repeat reads are served from memory instead of storage, and so writes can return quickly while the kernel flushes them to disk in the background. A write() call normally just copies data into the page cache and marks those pages dirty; it does not wait for the disk. fsync() is the call that forces those dirty pages (and, unlike fdatasync(), the file's metadata) to actually reach the storage device before it returns. Whether the page cache is helping or just eating memory comes down to one field: MemAvailable, not MemFree.

Structured elaboration

Buffered writes and write-back. By default, file I/O in Linux is buffered: write() copies your data into page-cache pages and returns immediately. Those pages are marked "dirty" (modified but not yet on disk). Kernel writeback threads (one per backing device) flush dirty pages to disk asynchronously, governed by two sysctls: vm.dirty_background_ratio (the percentage at which background flushing starts quietly) and vm.dirty_ratio (the percentage at which the writing process itself gets blocked until writeback catches up). Both percentages are taken against dirtyable memory, meaning free plus reclaimable pages, not against total RAM, so the byte ceiling they imply shrinks as anonymous memory grows; vm.dirty_bytes/vm.dirty_background_bytes set the same thresholds as absolute byte counts when you need a fixed bound. This is why a burst of writes can feel instant at first and then suddenly stall: you've crossed dirty_ratio and the kernel is now throttling you.

What fsync actually guarantees. fsync(fd) flushes all dirty pages belonging to that file descriptor to the underlying block device, and issues a cache-flush/FUA (force unit access) command so the write is committed past any volatile cache on the device itself, not just handed to the disk controller. Once fsync() returns successfully, that data survives a kernel crash or power loss, assuming the device honors flush commands correctly (a device with a broken or disabled write-cache flush can lie about this, which is a real failure mode to check for, not a theoretical one). fdatasync() gives the same data guarantee but skips flushing metadata that isn't needed to read the data back (e.g. mtime), which is cheaper when you don't need it.

Where O_DIRECT fits. O_DIRECT is an open flag that bypasses the page cache entirely for that file, transferring data close to directly between your buffer and the device. It exists for applications, mostly databases, that manage their own buffer pool and don't want the OS silently double-caching (and double-copying) the same pages. The trade-off: you give up the free page cache readahead and write coalescing the kernel would otherwise do, and you take on the responsibility for your own caching and I/O scheduling. O_DIRECT alone still does not guarantee durability: you generally still need fsync or O_DSYNC (an open flag that folds an fdatasync-equivalent durability guarantee into every write() call, syncing the data plus only the metadata needed to read it back; O_SYNC is the stronger sibling that folds in a full fsync-equivalent guarantee, including metadata like mtime that isn't needed to retrieve the data), because the storage device can have its own volatile write cache sitting below O_DIRECT.

Genuine benefit versus memory pressure. Don't judge this from free memory alone; idle RAM used for cache is not a problem by design. Instead:

• Check MemAvailable in /proc/meminfo (or the available column of free -h). This is the kernel's own estimate of how much memory could be freed for new allocations without swapping, after accounting for reclaimable cache. High MemAvailable relative to total RAM means the cache is healthy and reclaimable on demand.
• Check Dirty and Writeback in /proc/meminfo. If Dirty is climbing toward your dirty_ratio threshold, you're accumulating unflushed writes, not free performance.
• Watch vmstat 1 for si/so (swap in/out). Any sustained non-zero value means real pressure is forcing anonymous memory to disk, which is a different and much worse problem than "a lot of cache."
• Look at /proc/pressure/memory (pressure stall information, a kernel-reported metric of how much time processes actually stalled waiting on memory). Near-zero avg10 means processes aren't stalling; rising values mean the cache size is a symptom of real contention, not a free lunch.

Worked example

On a 64 GB box that's been serving reads for a while, free -h might show:

              total        used        free      shared  buff/cache   available
Mem:            62Gi        11Gi       1.2Gi       450Mi        49Gi        48Gi
Swap:            0B          0B          0B

49 GiB is sitting in buff/cache, but available is 48 GiB, almost the same. That tells you the kernel considers nearly all of that cache reclaimable right now: it's genuinely free capacity being used productively, not memory the system is straining to hold onto. Contrast that with a box where available has dropped to 4 GiB despite 49 GiB of buff/cache: that gap means most of that "cache" is actually dirty pages waiting on writeback, or the kernel is close to needing to reclaim aggressively, which is your early warning to check Dirty in /proc/meminfo and vmstat's si/so before it turns into swapping or an OOM (out-of-memory) kill, where the kernel terminates a process to free memory it can't reclaim any other way.

Trade-offs and pitfalls

The most common mistake is alerting on low MemFree directly, which pages nearly every healthy Linux box in existence, since the kernel is designed to use spare RAM for cache rather than leave it idle. Alert on MemAvailable instead. The second common mistake is assuming fsync guarantees durability against every failure mode: a storage controller with a volatile write cache that ignores flush commands, or a filesystem mounted with barriers disabled, can make fsync return success while data is still one power outage away from being lost. On database servers this is worth verifying explicitly rather than assuming the defaults are safe.

Approach

/proc/meminfo is a Linux virtual file (generated by the kernel on read, not stored on disk) with one Key: value kB pair per line; parse it into a dictionary keyed by field name rather than by fixed line position, since the kernel does not guarantee field order is stable across kernel versions, then derive used memory and percent used from the four fields the question names, treating any missing field as an explicit None rather than crashing or silently defaulting to zero.

Implementation (Python)

def parse_meminfo(text):
    """Parse /proc/meminfo text into a dict with derived used-memory fields.

    Handles missing fields (returns None for anything not present) and
    does not assume any particular line order, since /proc/meminfo's
    field order is not a guaranteed kernel ABI.
    """
    wanted = {"MemTotal", "MemFree", "Buffers", "Cached", "SwapTotal", "SwapFree"}
    values = {}
    for line in text.splitlines():
        if ":" not in line:
            continue
        key, _, rest = line.partition(":")
        key = key.strip()
        if key not in wanted:
            continue
        # Values look like "   16345600 kB"; keep the number, kB is the base unit.
        parts = rest.strip().split()
        if not parts:
            continue
        try:
            values[key] = int(parts[0])
        except ValueError:
            continue

    result = {k: values.get(k) for k in wanted}

    mem_total = result["MemTotal"]
    mem_free = result["MemFree"]
    buffers = result["Buffers"]
    cached = result["Cached"]

    if None in (mem_total, mem_free, buffers, cached):
        result["used_kb"] = None
        result["percent_used"] = None
    else:
        used = mem_total - mem_free - buffers - cached
        result["used_kb"] = used
        result["percent_used"] = round((used / mem_total) * 100, 2) if mem_total else None

    return result

Output

Ran against a normal sample (all six fields present, in a different order than the code checks them in) and a partial sample (SwapTotal/SwapFree lines entirely absent, which some kernels omit when swap is disabled):

{
  "Cached": 4271208,
  "MemFree": 6531220,
  "SwapFree": 2097148,
  "SwapTotal": 2097148,
  "Buffers": 412872,
  "MemTotal": 16336984,
  "used_kb": 5121684,
  "percent_used": 31.35
}
{
  "Cached": 4271208,
  "MemFree": 6531220,
  "SwapFree": null,
  "SwapTotal": null,
  "Buffers": 412872,
  "MemTotal": 16336984,
  "used_kb": 5121684,
  "percent_used": 31.35
}

Hand-checked: 16336984 - 6531220 - 412872 - 4271208 = 5121684, and 5121684 / 16336984 * 100 = 31.354..., which rounds to 31.35, matching the printed output; the partial-sample case correctly still computes used_kb/percent_used (they only depend on the four non-swap fields) while reporting SwapTotal/SwapFree as null since those lines were genuinely absent from the input.

Key points

Order-independence: the parser reads every line and keys results by field name (key.strip() from before the colon), so it does not matter whether MemTotal is line 1 or line 12, unlike a parser that assumes a fixed line index. Missing fields: each wanted field defaults to None via values.get(key) rather than 0, which matters because a missing SwapTotal (swap disabled) is a meaningfully different fact than SwapTotal: 0 (swap enabled with zero configured, unusual but different), and silently treating "absent" as "zero" would hide that distinction from anyone consuming the JSON. Derived fields only compute when all four required inputs are present; the arithmetic itself (MemTotal - MemFree - Buffers - Cached) uses the exact formula the question specifies, and both derived fields fall back to None together rather than partially computing on incomplete data.

Trade-offs and pitfalls

This definition of "used" memory is the classic (and slightly pessimistic) one that treats Buffers and Cached as fully reclaimable, which is close to true but not exact on modern kernels (some cached memory is not trivially reclaimable, and there is a more accurate MemAvailable field the kernel exposes directly for this purpose since Linux 3.14). For a production monitoring script, prefer reading MemAvailable directly over recomputing this classic formula, since the kernel's own estimate accounts for reclaim nuances this simple subtraction does not; the formula in this answer follows the question's explicit specification. All units here stay in kB (the native unit /proc/meminfo reports) throughout, converting to MB or GB and mixing units partway through the calculation is a common source of off-by-1024x bugs in scripts like this.

Direct answer

Use the cgroup v2 io controller (the cgroup v1 equivalent was called blkio; both throttle block-layer I/O per cgroup) to either give each container a relative weight (fair-share) or a hard cap on IOPS (input/output operations per second) and throughput (io.max). Weights are fairer under contention and waste less capacity when a container is idle; hard caps give predictable, guaranteed ceilings but can leave a device under-utilized when only one tenant is active.

Discussion

Implementation: io.weight (default 100, range 1-10000) sets a relative share of disk time per device, visible per cgroup at /sys/fs/cgroup/<cgroup>/io.weight. io.max sets hard per-device caps in four independent dimensions: rbps/wbps (read/write bytes per second) and riops/wiops (read/write operations per second), e.g. echo "8:0 rbps=50000000 wbps=20000000" > io.max caps device major:minor 8:0 to 50MB/s reads and 20MB/s writes. Container runtimes (Docker, containerd, Kubernetes via the kubelet) expose both as --blkio-weight / io.weight and per-device IOPS/bandwidth limits in the container spec, which map straight onto these cgroup files.

Fairness: weights only redistribute contention, they do not reserve anything; a cgroup with weight 500 gets roughly 5x the disk-time share of one at weight 100, but only when both are actively issuing I/O. On an idle device a low-weight cgroup can still use 100% of the available throughput, which is usually desirable (unused capacity should not go to waste) but means weight alone does not guarantee a worst-case bound.

Burst behavior: io.max has no built-in token-bucket burst allowance the way, say, a network traffic shaper does; there's no mechanism to bank unused I/O capacity while idle and spend it later. What looks like burst tolerance is really an artifact of enforcement granularity: the block-layer throttle evaluates the configured rbps/riops/wbps/wiops caps over a short rolling accounting window rather than instantaneously, so a workload can briefly exceed the nominal cap within that window before the throttle catches up and starts delaying requests. io.latency is a separate, unrelated cgroup v2 controller and does not give the cgroup it's applied to any burst allowance for io.max; it works the other direction, protecting one cgroup's I/O latency by throttling other, lower-priority cgroups on the same device once the protected cgroup's measured latency exceeds its configured target. If you actually need deliberate burst-then-throttle behavior (accumulate credit while idle, spend it under load, similar to cloud burstable-volume IOPS credits), the kernel's io controller doesn't provide that natively for io.max; it has to be built above the kernel, in the application or a sidecar. Weight-based fairness (io.weight) has no burst concept either; it is purely proportional at whatever instant contention exists.

Latency impact: hard IOPS/bandwidth caps convert what would be a fast I/O completion into an artificially delayed one once the cap is hit, which shows up as increased read/write latency inside the throttled container even though the underlying device has spare capacity, because the kernel is intentionally queuing or rejecting requests to enforce the cap. This is the most common surprise: a team sees their own container's latency spike after ops adds an I/O limit and assumes the disk is failing, when actually the limit is working as designed.

Verification: read back the cgroup's own I/O accounting at /sys/fs/cgroup/<cgroup>/io.stat, which reports rbytes, wbytes, rios, wios per device and lets you confirm actual throughput matches the configured cap under load; cross-check with iostat -x 1 on the host to see aggregate device utilization, and with the application's own latency metrics to see the throttling's user-visible cost. Load-test with fio (Flexible I/O Tester, a standard Linux disk I/O benchmarking tool) targeting the container specifically (not the host) to validate the cap holds under the container's real I/O pattern (random 4K vs sequential 1M writes throttle very differently even at the same nominal cap).

Limitations: the io controller only throttles requests that go through the block layer with a cgroup context attached, so buffered writes that hit the page cache first and get written back later by a kernel writeback thread can, on older kernels, lose their cgroup association and evade the cap (this is largely fixed on modern kernels with cgroup-aware writeback, but worth verifying on the specific kernel version in use). Writes issued with O_DIRECT (an open flag that bypasses the page cache so I/O goes straight from the application's buffer to the block layer) sidestep that specific writeback-attribution problem, since there is no intermediate writeback thread to lose the association, but O_DIRECT I/O is just as subject to io.max/io.weight caps as any other block-layer request; it does not grant an exemption. Caps are per-device, so a container spanning multiple physical devices or a logical volume striped across several disks needs the limit applied at the right layer to actually constrain aggregate I/O. Network-attached storage (NFS, iSCSI) is not always cleanly covered by the block-layer io controller in the same way local block devices are.

Trade-offs and pitfalls

Weights are the right default for fair-sharing a busy shared disk; hard caps (io.max) are the right tool when you need a guaranteed, predictable ceiling for capacity planning or SLA (service-level agreement, a commitment about the level of service provided) purposes, at the cost of wasting device capacity when the capped container is the only active tenant. The single biggest pitfall is setting a cap without first measuring the container's actual I/O pattern with a representative benchmark; a cap that looks generous for sequential throughput can be crushing for a workload that is actually IOPS-bound on small random reads, since bytes/sec and ops/sec are independent dimensions.