System Resource & I/O Optimization Questions

Direct answer

O_DIRECT and fsync solve different problems and are typically used together, not as alternatives. O_DIRECT is an open flag that bypasses the OS page cache so reads and writes go close to directly between the application's buffer and the storage device. fsync (and fdatasync) is a durability guarantee: it forces dirty data to actually reach persistent storage, including flushing past any volatile write cache the device itself maintains, regardless of whether O_DIRECT was used to get there. For database systems, both matter, and getting the interaction wrong is a real correctness risk, not just a performance nuance.

Structured elaboration

What each one actually does. Without O_DIRECT, a normal buffered write() call copies data into the page cache and returns immediately; the kernel's writeback threads flush it to the device asynchronously later. That write is not durable the instant it returns, only fsync provides that guarantee. O_DIRECT changes where the write goes, pushing it toward the device directly instead of parking it in the page cache first, but it does not by itself guarantee durability: the storage device can have its own volatile write cache that acknowledges the write as complete before the data has actually reached durable NAND or platter. fsync is what closes that gap: it issues a cache-flush (or FUA, force unit access, a write mode that bypasses the device's write cache entirely) command to the device, so that even data sitting in a volatile on-device cache gets committed to durable media before fsync returns successfully.

Interaction with filesystem buffering and device write caches. Filesystem-level buffering (the page cache) is what O_DIRECT bypasses. Device-level write caches are a separate layer below that, and they come in two flavors that behave very differently: battery-backed write caches (common on enterprise RAID controllers) can safely acknowledge a write as durable before it physically reaches the storage medium, because the battery guarantees that cached data survives a power loss and gets flushed once power returns, so fsync against such a controller can return quickly without sacrificing durability. Volatile write caches with no battery backing are a correctness hazard if they don't honor flush commands properly: fsync can appear to succeed while the data is still sitting in a cache that a power loss would wipe out, which is why production database deployments either disable volatile write caches outright, rely on FUA writes, or explicitly verify that the storage stack correctly honors flush/barrier commands rather than assuming it does.

Performance and correctness trade-offs for database systems. O_DIRECT's benefit is avoiding double buffering: without it, the same data can end up cached twice, once in the OS page cache and again in the database's own buffer pool, wasting RAM and an extra memory copy for no benefit, since the database's own cache eviction policy is usually better tuned to its actual access patterns than the kernel's generic one. The cost is that the database now has to implement its own readahead, write coalescing, and I/O scheduling, work the OS page cache would otherwise have done for free. fsync, separately, is what actually provides the durability guarantee (the "D" in ACID, atomicity, consistency, isolation, durability, the standard correctness properties a transactional database aims to guarantee) regardless of whether O_DIRECT is in play. The trade-off there is pure latency versus durability risk: calling fsync on every single transaction commit is safe but bounded by the device's flush latency, so many systems batch multiple transaction commits together and call fsync once per batch (commonly called group commit) to amortize that cost across several transactions, and some deployments knowingly relax this further (an async or deferred commit mode) to gain throughput at the explicit cost of being able to lose the most recent commits if the system crashes before the next flush.

Worked example

A database using O_DIRECT for its data files avoids caching the same pages twice, but if its write-ahead log (WAL, a sequential record of every change written before the change is applied, used to recover after a crash) is written with O_DIRECT alone and no fsync at commit time, a crash immediately after a transaction reports success can still lose that transaction, because O_DIRECT only changed where the write went, not whether it durably landed past every cache layer. Adding fsync (or opening the WAL with O_DSYNC, which folds an fdatasync-equivalent guarantee, the data plus only the metadata needed to read it back, into every write call; the stronger O_SYNC is the one that matches fsync's full guarantee, including metadata like mtime that a WAL commit doesn't actually need) at each commit point closes that gap. If commit-time fsync latency becomes the throughput bottleneck, group commit, batching several transactions into one fsync call, is the standard fix that keeps the same durability guarantee per committed batch while amortizing the flush cost across more transactions than committing one at a time would allow.

Trade-offs and pitfalls

The most consequential mistake is assuming O_DIRECT alone provides durability; it does not, it only changes where data goes, not whether it has actually reached persistent storage past every cache layer. A second common mistake is assuming any hardware write cache is safe to leave enabled without verifying it's actually battery-backed and correctly honoring flush commands; a volatile cache that silently ignores flushes will pass every functional test and then lose data on the first real power loss, precisely because the failure only shows up under a condition testing doesn't normally exercise.

Approach

Serving a static file the ordinary way (read() into a userspace buffer, then write() to the socket) costs two data copies (disk cache to userspace buffer, userspace buffer to socket buffer) and at least two user/kernel context switches. sendfile() asks the kernel to copy the file's page-cache pages straight into the socket's send buffer without ever handing the bytes to userspace, cutting the copy count and the syscall count.

Demonstration (real, executed)

The following Python code uses os.sendfile(), the direct syscall wrapper, to serve a file over a loopback TCP socket. It was run as written; the printed output below is the actual result of that run, not a hypothetical.

import os, socket, threading, tempfile

data = b"X" * (64 * 1024)  # a 64 KiB "static file"
with tempfile.NamedTemporaryFile(delete=False) as f:
    f.write(data)
    fname = f.name

def server(holder):
    srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    srv.bind(("127.0.0.1", 0))
    srv.listen(1)
    holder["port"] = srv.getsockname()[1]
    holder["ready"].set()
    conn, _ = srv.accept()
    with open(fname, "rb") as fh:
        filesize = os.fstat(fh.fileno()).st_size
        offset, sent_total = 0, 0
        while sent_total < filesize:
            sent = os.sendfile(conn.fileno(), fh.fileno(), offset, filesize - sent_total)
            if sent == 0:
                break
            offset += sent
            sent_total += sent
    conn.close(); srv.close()
    holder["sent"] = sent_total

holder = {"ready": threading.Event()}
t = threading.Thread(target=server, args=(holder,)); t.start()
holder["ready"].wait()

client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect(("127.0.0.1", holder["port"]))
received = b""
while len(received) < len(data):
    chunk = client.recv(65536)
    if not chunk:
        break
    received += chunk
client.close(); t.join()

print("bytes sent via os.sendfile():", holder["sent"])
print("bytes received by client:", len(received))
print("payload matches:", received == data)
os.unlink(fname)

Output:

bytes sent via os.sendfile(): 65536
bytes received by client: 65536
payload matches: True

os.sendfile(out_fd, in_fd, offset, count) mirrors the Linux syscall signature exactly: it copies count bytes from in_fd (a regular file, must be mmap-able) to out_fd starting at offset, and returns how many bytes it actually moved so the loop above can resume on a short transfer. out_fd has always been allowed to be a socket, which is the case this demo uses; the version boundary actually runs the other way. Before Linux 2.6.33, out_fd was required to be a socket; since 2.6.33 it can be any file, including a regular file, which is what additionally enables file-to-file zero-copy transfers, not the socket case this demo already relied on.

Key points

• in_fd must be a regular file (or a file the kernel can page-cache), not a pipe or arbitrary character device: the kernel is copying page-cache pages, not streaming arbitrary bytes.
• On modern NICs, the copy can be a scatter-gather DMA (Direct Memory Access: hardware moves the data without the CPU copying it byte by byte) straight from page cache to the NIC, so there may be zero CPU-driven copies at all, only DMA.

Limitations

• File descriptor limits: sendfile() needs valid, seekable file descriptors on both ends; it cannot serve from a socket-to-socket relay directly.
• TLS: sendfile() hands raw file bytes to the kernel network stack. TLS encryption normally happens in userspace (OpenSSL, etc.), which requires touching the plaintext, defeating the zero-copy path entirely, so a plain TLS-terminating server usually cannot use sendfile() for the encrypted response body.
• kTLS (kernel TLS) is the fix: once the TLS session keys are handed to the kernel, the kernel encrypts the data during the same copy sendfile() already does, restoring the zero-copy path for encrypted traffic.
• splice() is the more general alternative: it moves data between two file descriptors via an intermediate kernel pipe buffer, so it also covers socket-to-socket proxying, a case plain sendfile() does not handle well.

Why swap is uniquely harmful for latency-sensitive services

A normal memory access completes in nanoseconds. When the touched page has been swapped out, that same access becomes a synchronous disk read, milliseconds on flash, tens of milliseconds on spinning disk: a five-to-six order of magnitude latency cliff. Worse, it is essentially random which page gets swapped and when it gets touched again, so it does not show up as a steady average-latency cost; it shows up as unpredictable p99/p999 tail-latency spikes (the slowest 1-in-100 or 1-in-1000 requests), which is exactly what latency SLAs (Service Level Agreements, the promised performance targets a service commits to) are built to protect and exactly what swap defeats. It is also invisible to the application: nothing in the code path 'knows' it just paid a disk-I/O cost for what looked like a memory read.

Techniques to avoid swap-induced latency

• Strict memory limits: set a cgroup memory.max hard cap comfortably under available RAM, and set memory.swap.max=0 (cgroups v2, the second-generation Linux resource-control hierarchy) so that cgroup can never swap at all; it gets OOM-killed instead. For a latency SLA, a fast, clean failure the orchestrator can restart is usually preferable to slow degradation nobody notices until it is too late.
• Overcommit tuning: vm.overcommit_memory=2 makes the kernel refuse to promise more memory than it can actually back (paired with vm.overcommit_ratio), preventing a host-wide memory crunch that would otherwise force swapping across every process on the box, not just the one that overcommitted.
• mlock: mlock()/mlockall() pin a process's critical working set in RAM so the kernel is not permitted to swap those pages out, at the cost of needing CAP_IPC_LOCK (the Linux capability that permits a process to lock memory) and a raised RLIMIT_MEMLOCK (the per-process limit on how much memory it may lock).
• cgroup settings: memory.low and memory.high are soft thresholds that trigger earlier, gentler reclaim before the hard memory.max is hit; disabling swap host-wide (swapoff -a) is a blunt but common choice on pure latency-sensitive fleets that accept OOM-kill as the failure mode.
• Memory reservation: capacity-plan so peak resident set size plus headroom fits comfortably in RAM; in Kubernetes terms, setting requests equal to limits for latency-critical pods avoids the scheduler bin-packing them too tightly.
• Admission control: shed or queue new work before memory pressure builds, using a memory high-watermark or PSI (Pressure Stall Information, a kernel metric for how much time processes spend blocked on a resource) signal as the trigger, so the service degrades on its own terms instead of the kernel's.
• Application-level fallbacks: circuit breakers that react to rising allocation/GC latency, backpressure on incoming requests, and exporting a memory-pressure health signal to the load balancer so an instance is pulled from rotation before it starts swapping, not after.

The common thread: prefer a controlled, fast failure (reject work, OOM-kill, fail over) to letting the kernel silently trade throughput and predictability for staying alive via swap.

Approach

/proc/stat exposes cumulative kernel counters in units of jiffies (kernel clock ticks, usually 100 per second, i.e. USER_HZ=100) accumulated since boot. Each cpu/cpuN line has ten space-separated fields in a fixed order: user, nice, system, idle, iowait, irq (hardware interrupt handling time), softirq (deferred, software-handled interrupt time), steal, guest (CPU time spent running a virtual machine's code on behalf of this host), guest_nice (the same, for a niced/low-priority guest). Because these are running totals, a single snapshot tells you nothing about "current" utilization: you need two snapshots and the interval between them, then divide the delta of interest by the total delta across all ten fields for that window.

The core steps: parse each snapshot into {label: [ten integers]}, keep only labels present in both snapshots, compute per-field deltas with wraparound correction, sum the deltas for the total elapsed work, then each requested percentage is the matching field's delta divided by the total delta, times 100.

Wraparound handling: these are unsigned kernel counters (64-bit on modern kernels; some older/32-bit kernels expose them as 32-bit unsigned long). Subtracting a later value from an earlier one after the counter has wrapped past its maximum produces a negative raw delta. The fix is to add the modulus back whenever the raw delta is negative, which is standard unsigned-arithmetic correction, not a special case.

import json

FIELDS = ["user", "nice", "system", "idle", "iowait", "irq", "softirq", "steal", "guest", "guest_nice"]
WRAP_MOD = 2 ** 64  # modern kernels store these as unsigned 64-bit; use 2**32 if targeting a 32-bit kernel


def _parse_snapshot(text):
    rows = {}
    for line in text.strip().splitlines():
        parts = line.split()
        if not parts or not parts[0].startswith("cpu"):
            continue
        label = parts[0]
        values = [int(v) for v in parts[1:]]
        values += [0] * (len(FIELDS) - len(values))  # pad pre-guest-field kernels
        rows[label] = values[: len(FIELDS)]
    return rows


def _delta(before, after):
    d = after - before
    if d < 0:
        d += WRAP_MOD  # unsigned counter wrapped between snapshots
    return d


def cpu_utilization(snapshot1, snapshot2):
    before = _parse_snapshot(snapshot1)
    after = _parse_snapshot(snapshot2)
    result = {}
    for label in after:
        if label not in before:
            continue  # core only present in snapshot2 (hot-added): no valid delta yet
        deltas = [_delta(b, a) for b, a in zip(before[label], after[label])]
        total = sum(deltas)
        if total <= 0:
            result[label] = {k: 0.0 for k in ("user", "system", "idle", "iowait")}
            continue
        by_field = dict(zip(FIELDS, deltas))
        result[label] = {
            "user": round((by_field["user"] + by_field["nice"]) / total * 100, 2),
            "system": round((by_field["system"] + by_field["irq"] + by_field["softirq"]) / total * 100, 2),
            "idle": round(by_field["idle"] / total * 100, 2),
            "iowait": round(by_field["iowait"] / total * 100, 2),
        }
    return result

Key points

• This is a windowed measurement, not a point reading: both snapshots and the elapsed time between them matter.
• nice is folded into user, and irq/softirq into system, matching how most dashboards summarize CPU time into a small bucket set. That is a documented design choice here, not hidden behavior: keep them separate in by_field if a caller needs finer breakdown.
• iowait counts time a CPU was idle while at least one process was blocked on disk I/O. It is still technically idle CPU time (the core did no work), but it is broken out separately because a high iowait percentage points at storage latency, not compute demand, and the two need different fixes.
• The total divisor always sums all ten raw fields (including steal, guest, guest_nice), because the four reported percentages are shares of the full accounting window, not just of the four buckets being reported.

Complexity

Both parsing and the utilization computation are O(number of CPU lines), i.e. linear in core count, with O(1) work per field. No sorting or nested scans; memory use is a small fixed-size list per core.

Edge cases

• CPU hotplug: a cpuN label present in only one snapshot is skipped rather than crashing or reporting a bogus delta.
• Zero elapsed time (or a counter set that round-trips through a full wrap so every field's corrected delta happens to sum to zero): guarded to return zeros instead of dividing by zero.
• Pre-guest/guest_nice kernels (eight-field lines): padded to the fixed ten-field width.
• Counter wraparound: corrected via the modulus, shown in the worked example below.
• Malformed input is intentionally not swallowed here (int() will raise on a corrupt line): in a production collector you would wrap the parse in a try/except per line so one bad read does not take down the whole scrape.

Worked example

Two snapshots, roughly one second apart, two cores:

snapshot1:
cpu  3357 0 4313 1362393 234 0 0 0 0 0
cpu0 1679 0 2156 681000 117 0 0 0 0 0
cpu1 1678 0 2157 681393 117 0 0 0 0 0

snapshot2:
cpu  3427 0 4358 1362473 334 0 0 0 0 0
cpu0 1739 0 2186 681010 217 0 0 0 0 0
cpu1 1688 0 2172 681463 117 0 0 0 0 0

By hand for the aggregate cpu line: user delta 70, system delta 45, idle delta 80, iowait delta 100, total 295. user% = 70/295100 = 23.73, system% = 45/295100 = 15.25, idle% = 80/295100 = 27.12, iowait% = 100/295100 = 33.90. Running cpu_utilization(snapshot1, snapshot2) and printing the result confirms this exactly:

{
  "cpu":  {"user": 23.73, "system": 15.25, "idle": 27.12, "iowait": 33.9},
  "cpu0": {"user": 30.0,  "system": 15.0,  "idle": 5.0,   "iowait": 50.0},
  "cpu1": {"user": 10.53, "system": 15.79, "idle": 73.68, "iowait": 0.0}
}

cpu0 is doing a real burst of both compute and disk-wait work (50% iowait, only 5% idle) while cpu1 is mostly idle, which the aggregate cpu line alone would have hidden by averaging the two cores together.

For the wraparound path, feeding a synthetic pair where the counter has rolled past its modulus (before=18446744073709500000, after=200) still yields a sane, bounded percentage (100% user, since only that field moved) instead of the huge negative-delta garbage a naive subtraction would produce.

Trade-offs and pitfalls

• The sampling interval is a smoothing choice: a sub-second window is noisy and amplifies short bursts; production agents commonly sample every 5 to 15 seconds and rely on a dashboard's own rate/window function for further smoothing.
• Reading only the aggregate cpu line hides per-core imbalance (a single hot core in an otherwise idle box), which matters for diagnosing single-threaded bottlenecks; always keep the per-cpuN breakdown available for that kind of alerting.
• Do not confuse this with per-process accounting: /proc/stat is whole-system; a specific process's CPU time lives in the utime/stime fields of /proc/<pid>/stat, which is a different counter space entirely.
• On a virtualized host, a nonzero and growing steal share (time the hypervisor gave to other tenants instead of this VM) is itself an important capacity signal even though the question's four requested buckets do not surface it directly; it still counts toward the total divisor here, which is correct, but a monitoring agent should usually report it too.

Approach

/proc/meminfo is a Linux virtual file (generated by the kernel on read, not stored on disk) with one Key: value kB pair per line; parse it into a dictionary keyed by field name rather than by fixed line position, since the kernel does not guarantee field order is stable across kernel versions, then derive used memory and percent used from the four fields the question names, treating any missing field as an explicit None rather than crashing or silently defaulting to zero.

Implementation (Python)

def parse_meminfo(text):
    """Parse /proc/meminfo text into a dict with derived used-memory fields.

    Handles missing fields (returns None for anything not present) and
    does not assume any particular line order, since /proc/meminfo's
    field order is not a guaranteed kernel ABI.
    """
    wanted = {"MemTotal", "MemFree", "Buffers", "Cached", "SwapTotal", "SwapFree"}
    values = {}
    for line in text.splitlines():
        if ":" not in line:
            continue
        key, _, rest = line.partition(":")
        key = key.strip()
        if key not in wanted:
            continue
        # Values look like "   16345600 kB"; keep the number, kB is the base unit.
        parts = rest.strip().split()
        if not parts:
            continue
        try:
            values[key] = int(parts[0])
        except ValueError:
            continue

    result = {k: values.get(k) for k in wanted}

    mem_total = result["MemTotal"]
    mem_free = result["MemFree"]
    buffers = result["Buffers"]
    cached = result["Cached"]

    if None in (mem_total, mem_free, buffers, cached):
        result["used_kb"] = None
        result["percent_used"] = None
    else:
        used = mem_total - mem_free - buffers - cached
        result["used_kb"] = used
        result["percent_used"] = round((used / mem_total) * 100, 2) if mem_total else None

    return result

Output

Ran against a normal sample (all six fields present, in a different order than the code checks them in) and a partial sample (SwapTotal/SwapFree lines entirely absent, which some kernels omit when swap is disabled):

{
  "Cached": 4271208,
  "MemFree": 6531220,
  "SwapFree": 2097148,
  "SwapTotal": 2097148,
  "Buffers": 412872,
  "MemTotal": 16336984,
  "used_kb": 5121684,
  "percent_used": 31.35
}
{
  "Cached": 4271208,
  "MemFree": 6531220,
  "SwapFree": null,
  "SwapTotal": null,
  "Buffers": 412872,
  "MemTotal": 16336984,
  "used_kb": 5121684,
  "percent_used": 31.35
}

Hand-checked: 16336984 - 6531220 - 412872 - 4271208 = 5121684, and 5121684 / 16336984 * 100 = 31.354..., which rounds to 31.35, matching the printed output; the partial-sample case correctly still computes used_kb/percent_used (they only depend on the four non-swap fields) while reporting SwapTotal/SwapFree as null since those lines were genuinely absent from the input.

Key points

Order-independence: the parser reads every line and keys results by field name (key.strip() from before the colon), so it does not matter whether MemTotal is line 1 or line 12, unlike a parser that assumes a fixed line index. Missing fields: each wanted field defaults to None via values.get(key) rather than 0, which matters because a missing SwapTotal (swap disabled) is a meaningfully different fact than SwapTotal: 0 (swap enabled with zero configured, unusual but different), and silently treating "absent" as "zero" would hide that distinction from anyone consuming the JSON. Derived fields only compute when all four required inputs are present; the arithmetic itself (MemTotal - MemFree - Buffers - Cached) uses the exact formula the question specifies, and both derived fields fall back to None together rather than partially computing on incomplete data.

Trade-offs and pitfalls

This definition of "used" memory is the classic (and slightly pessimistic) one that treats Buffers and Cached as fully reclaimable, which is close to true but not exact on modern kernels (some cached memory is not trivially reclaimable, and there is a more accurate MemAvailable field the kernel exposes directly for this purpose since Linux 3.14). For a production monitoring script, prefer reading MemAvailable directly over recomputing this classic formula, since the kernel's own estimate accounts for reclaim nuances this simple subtraction does not; the formula in this answer follows the question's explicit specification. All units here stay in kB (the native unit /proc/meminfo reports) throughout, converting to MB or GB and mixing units partway through the calculation is a common source of off-by-1024x bugs in scripts like this.