System Resource & I/O Optimization Questions

What commonly causes context-switch spikes

A context switch is the kernel saving one thread's execution state and loading another's onto the CPU. Spikes fall into two broad categories with different root causes.

Voluntary switches happen when a thread blocks itself, most often on I/O or a lock. A spike here usually means increased lock contention: many threads repeatedly fighting over the same mutex, each one blocking (switching out) and later waking (switching back in) as the lock changes hands, rather than any kernel defect.

Involuntary switches happen when the scheduler preempts a running thread because its timeslice expired or a higher-priority task became runnable. A spike here usually means CPU oversubscription, for example a thread pool sized far above the actual core count, or a burst of higher-priority work (interrupts, real-time tasks) repeatedly preempting normal ones.

futex and lock contention specifically

futex (fast userspace mutex, the primitive underneath most language runtimes' Mutex/Lock types on Linux) has a fast path that never touches the kernel at all: an uncontended lock/unlock is a pure userspace atomic compare-and-swap, no context switch involved. Contention is what turns it expensive: the losing thread calls into the futex() syscall to sleep (one context switch out) and is later woken (another context switch in). A high context-switch count alongside elevated sys CPU time and a rising syscalls:sys_enter_futex rate (visible via perf stat -e syscalls:sys_enter_futex or strace -c) is a strong, specific signal that the spike is lock contention, not scheduler churn from something else.

Thread models

A thread-per-connection or thread-per-request model creates far more runnable threads than available cores under real concurrency, forcing heavy scheduler churn by design. Event-loop or async models use far fewer OS threads for the same concurrency, which reduces this class of context-switch pressure structurally rather than through any kernel tunable; that is a design decision, not a sysctl.

Kernel parameters and user-space strategies

• Bound thread-pool size near the core count (nproc) for CPU-bound work instead of spawning an unbounded thread per request.
• kernel.sched_min_granularity_ns and kernel.sched_wakeup_granularity_ns (CFS scheduler tunables) control how eagerly the scheduler preempts a running task in favor of one that just woke up; raising them reduces preemption frequency at a real cost: added latency for whatever thread just became runnable and now waits longer.
• CPU pinning and isolation (isolcpus, taskset) give latency-critical threads dedicated cores with nothing else scheduled on them, eliminating involuntary switches from that core entirely.
• Reducing lock granularity in the application itself (finer-grained locks, lock-free data structures, sharding a single contended structure into several) addresses the actual cause of voluntary-switch spikes and is usually the highest-leverage fix, since it removes the contention rather than just changing how the scheduler responds to it.

Side effects of scheduler changes

Any global sysctl change (widening scheduler granularity, for example) affects every workload on the host, not just the one you are trying to fix. On a multi-tenant host, reducing context switches for the target service by making the scheduler less preemptive can starve or add latency to OTHER services sharing that host. Always canary such a change and measure the OTHER workloads' latency, not just whether the target's context-switch count went down.

Direct answer

Start by counting and categorizing which file descriptors a suspect process is actually holding, using /proc/<pid>/fd and lsof, to find out whether it's leaking sockets, regular files, or pipes. Mitigate immediately by raising that process's file descriptor limit without a restart (prlimit) and, if the global system table is also near its ceiling, raising fs.file-max. Fix the leak permanently by finding the code path that skips closing a handle, usually on an error branch, and add fd-count monitoring so the next leak pages someone before it becomes an outage.

Structured elaboration

Gathering evidence.

• ls -la /proc/<pid>/fd | wc -l gives a quick count of currently open descriptors for the suspect process; sample this every few seconds in a loop to see if it's actively growing (a real leak) or just high but stable (a legitimately fd-heavy but healthy process).
• cat /proc/<pid>/limits | grep "Max open files" shows the soft and hard ulimit that process is bound by, so you know how much headroom is left before EMFILE (a per-process "too many open files" error) hits.
• lsof -p <pid> gives a human-readable listing with the type of each descriptor (regular file, socket, pipe, directory) and, for files, the actual path. This is where you find out what class of resource is leaking.
• For sockets specifically, ss -tnp state close-wait lists sockets stuck in the CLOSE_WAIT TCP state, meaning the remote peer closed the connection but the application never called close() on its end. A pile of these is the single most common EMFILE root cause in networked services: a connection-handling code path that isn't releasing sockets on some error or timeout branch.
• System-wide, lsof | awk '{print $2}' | sort | uniq -c | sort -rn | head ranks every PID by descriptor count, useful when you don't yet know which process is the culprit, and cat /proc/sys/fs/file-nr shows the system-wide allocated/free/max descriptor counts (fs.file-max), which tells you whether this is a single-process EMFILE or you're also close to the global ceiling.

Short-term mitigations, in rough order of speed.

1. Raise the process's own limit live, without a restart: prlimit --pid <pid> --nofile=<new-soft>:<new-hard>. This buys immediate headroom for the running process.
2. If the global table (fs.file-nr's first field approaching its third field, fs.file-max) is also close to exhausted, raise fs.file-max via sysctl -w fs.file-max=<new-value>.
3. Roll a staggered restart of the affected service's instances behind the load balancer to reset fd counts on each one while the underlying fix ships, spreading the restarts to avoid a thundering herd of reconnects hitting downstream dependencies at once.
4. If you can identify the specific endpoint or client driving the leak, apply targeted rate limiting or backpressure there as a stopgap while a code fix is prepared.

Long-term fix. The root cause is almost always a code path, frequently an exception or error-handling branch, that returns or throws before a close() (or the language's equivalent, a try-with-resources block, a context manager, or a defer/finally cleanup) runs. Fix the leak at the source, then add regression protection: instrument fd count as a metric with alerting thresholds well below the ulimit (so you get paged before the next EMFILE, not after), and add a soak test that asserts the descriptor count is stable under sustained load rather than monotonically climbing. If the workload naturally opens and closes many connections, consider a connection pool with an idle timeout instead of ad hoc open/close per request, since pooling bounds the working set explicitly rather than relying on every code path remembering to clean up.

Worked example

Sampling ls /proc/12345/fd | wc -l every 10 seconds shows 4,200, then 4,850, then 5,600, a clear upward trend rather than a stable high number. lsof -p 12345 | awk '{print $5}' | sort | uniq -c shows the overwhelming majority are type sock. ss -tnp state close-wait | grep 12345 | wc -l returns a similarly large number, confirming the pattern: sockets where the remote side closed the connection but the process's own close() never fired, almost certainly a connection-handling error path that isn't cleaning up. That combination (growing fd count, dominated by sockets, dominated by CLOSE_WAIT) is specific enough to hand directly to the team owning that code as "your connection cleanup on the error path is leaking sockets," rather than a vague "check your file handles" report.

Trade-offs and pitfalls

Raising fs.file-max or a process's ulimit treats the symptom, not the leak; without also finding and fixing the code path, the same failure recurs on a longer timer with a higher ceiling. Restarting affected instances resets the count but loses any in-flight connections on those instances, so stagger it rather than restarting everything at once, especially behind a load balancer where a synchronized restart of every backend can itself cause a brief capacity dip.

Direct answer

Read-ahead is the kernel (or an application) speculatively fetching data beyond what was explicitly requested, betting that a sequential access pattern will keep asking for the next chunk. It helps when access really is sequential and predictable, because it hides storage latency by having data ready before it's asked for. It hurts on random-access workloads, where it wastes I/O bandwidth and evicts genuinely useful cached pages fetching data that's never used. For a media streaming service serving large sequential reads, you'd deliberately increase the block-device readahead window well beyond the default.

Structured elaboration

OS-level read-ahead. The kernel detects a sequential access pattern to a file (successive reads at increasing offsets) and issues additional reads ahead of what the application asked for, populating the page cache before those blocks are requested. This is controlled per block device via the readahead setting, viewable and settable with blockdev --getra/--setra /dev/sdX (values are in 512-byte sectors), and persisted across reboots via a udev rule that reapplies the setting when the device is detected.

Application-level prefetching. Applications can give the kernel explicit hints rather than relying purely on heuristic detection: posix_fadvise(fd, ..., POSIX_FADV_SEQUENTIAL) tells the kernel to be more aggressive about read-ahead for this specific file descriptor, and POSIX_FADV_WILLNEED explicitly requests the kernel start prefetching a specific range immediately. Databases commonly do their own index-page prefetching above the OS layer, and media servers or download managers may issue explicit range-request prefetching over the network, a similar idea one layer up the stack.

When it helps. Any workload with a genuinely predictable, sequential access pattern: streaming a large media file, a full table scan, sequential log processing. The core mechanism is hiding storage latency by overlapping "fetch the next chunk" with "process the chunk you already have," so by the time the application asks for the next piece, it's already in cache.

When it hurts. Random-access workloads, most notably databases doing index lookups scattered across a large file, where the "next" blocks read-ahead fetches speculatively are usually never actually requested. This wastes I/O bandwidth on the storage device (competing with real requests) and pollutes the page cache by evicting genuinely hot pages to make room for speculative ones that go unused, hurting both throughput (wasted device bandwidth) and latency (real requests now queue behind speculative ones, and useful cache entries get evicted).

Tuning for a media streaming service. Increase the block device's readahead window well above the default (commonly 128 KB) to something sized for the streaming bitrate and access pattern, for example:

blockdev --setra 8192 /dev/nvme0n1

8192 sectors of 512 bytes each is 4 MB, a much larger anticipatory read window suited to sustaining large sequential streams at high bitrate with fewer, larger I/O operations rather than many small ones. Pairing this with an explicit posix_fadvise(fd, 0, 0, POSIX_FADV_SEQUENTIAL) call from the serving application lets the kernel be aggressive about read-ahead for that file from the very first read, rather than waiting for its own heuristics to detect the sequential pattern after a few reads have already happened. Because this trades page cache capacity for prefetch depth, validate against your actual cache hit ratio on a multi-tenant streaming box: a readahead window large enough to help one stream but big enough to evict other tenants' hot content is a net loss.

Worked example

Default readahead (blockdev --getra typically reporting 256 sectors, 128 KB) on a media server sustaining a 20 Mbps stream means the kernel issues many small anticipatory reads to keep up, each one a separate I/O operation with its own queueing and completion overhead. Increasing that to 8192 sectors (4 MB) means far fewer, larger I/O operations are needed to sustain the same stream, which reduces per-operation overhead and gives the kernel a much bigger buffer of already-fetched data to hide any transient storage latency behind. The trade-off becomes visible on a multi-tenant box: if 200 concurrent streams each get the configured 4 MB working set of read-ahead data cached (matching the 8192-sector/4 MB setting above), that's 200 x 4 MB = 0.8 GB of cache committed to prefetch alone, worth explicitly sizing against total available cache rather than assuming a larger number is free.

Trade-offs and pitfalls

Applying a large readahead window globally to a device that serves both sequential (streaming) and random (metadata lookups, small file serving) workloads is the most common mistake: the random-access traffic pays the read-ahead tax (wasted bandwidth, cache pollution) without getting any of the sequential-access benefit. If the same device serves mixed workloads, prefer per-file posix_fadvise hints from the application over a single global blockdev setting, so the tuning only applies to the traffic that actually benefits from it.

Direct answer

The page cache is RAM the kernel uses to hold copies of disk blocks so repeat reads are served from memory instead of storage, and so writes can return quickly while the kernel flushes them to disk in the background. A write() call normally just copies data into the page cache and marks those pages dirty; it does not wait for the disk. fsync() is the call that forces those dirty pages (and, unlike fdatasync(), the file's metadata) to actually reach the storage device before it returns. Whether the page cache is helping or just eating memory comes down to one field: MemAvailable, not MemFree.

Structured elaboration

Buffered writes and write-back. By default, file I/O in Linux is buffered: write() copies your data into page-cache pages and returns immediately. Those pages are marked "dirty" (modified but not yet on disk). Kernel writeback threads (one per backing device) flush dirty pages to disk asynchronously, governed by two sysctls: vm.dirty_background_ratio (the percentage at which background flushing starts quietly) and vm.dirty_ratio (the percentage at which the writing process itself gets blocked until writeback catches up). Both percentages are taken against dirtyable memory, meaning free plus reclaimable pages, not against total RAM, so the byte ceiling they imply shrinks as anonymous memory grows; vm.dirty_bytes/vm.dirty_background_bytes set the same thresholds as absolute byte counts when you need a fixed bound. This is why a burst of writes can feel instant at first and then suddenly stall: you've crossed dirty_ratio and the kernel is now throttling you.

What fsync actually guarantees. fsync(fd) flushes all dirty pages belonging to that file descriptor to the underlying block device, and issues a cache-flush/FUA (force unit access) command so the write is committed past any volatile cache on the device itself, not just handed to the disk controller. Once fsync() returns successfully, that data survives a kernel crash or power loss, assuming the device honors flush commands correctly (a device with a broken or disabled write-cache flush can lie about this, which is a real failure mode to check for, not a theoretical one). fdatasync() gives the same data guarantee but skips flushing metadata that isn't needed to read the data back (e.g. mtime), which is cheaper when you don't need it.

Where O_DIRECT fits. O_DIRECT is an open flag that bypasses the page cache entirely for that file, transferring data close to directly between your buffer and the device. It exists for applications, mostly databases, that manage their own buffer pool and don't want the OS silently double-caching (and double-copying) the same pages. The trade-off: you give up the free page cache readahead and write coalescing the kernel would otherwise do, and you take on the responsibility for your own caching and I/O scheduling. O_DIRECT alone still does not guarantee durability: you generally still need fsync or O_DSYNC (an open flag that folds an fdatasync-equivalent durability guarantee into every write() call, syncing the data plus only the metadata needed to read it back; O_SYNC is the stronger sibling that folds in a full fsync-equivalent guarantee, including metadata like mtime that isn't needed to retrieve the data), because the storage device can have its own volatile write cache sitting below O_DIRECT.

Genuine benefit versus memory pressure. Don't judge this from free memory alone; idle RAM used for cache is not a problem by design. Instead:

• Check MemAvailable in /proc/meminfo (or the available column of free -h). This is the kernel's own estimate of how much memory could be freed for new allocations without swapping, after accounting for reclaimable cache. High MemAvailable relative to total RAM means the cache is healthy and reclaimable on demand.
• Check Dirty and Writeback in /proc/meminfo. If Dirty is climbing toward your dirty_ratio threshold, you're accumulating unflushed writes, not free performance.
• Watch vmstat 1 for si/so (swap in/out). Any sustained non-zero value means real pressure is forcing anonymous memory to disk, which is a different and much worse problem than "a lot of cache."
• Look at /proc/pressure/memory (pressure stall information, a kernel-reported metric of how much time processes actually stalled waiting on memory). Near-zero avg10 means processes aren't stalling; rising values mean the cache size is a symptom of real contention, not a free lunch.

Worked example

On a 64 GB box that's been serving reads for a while, free -h might show:

              total        used        free      shared  buff/cache   available
Mem:            62Gi        11Gi       1.2Gi       450Mi        49Gi        48Gi
Swap:            0B          0B          0B

49 GiB is sitting in buff/cache, but available is 48 GiB, almost the same. That tells you the kernel considers nearly all of that cache reclaimable right now: it's genuinely free capacity being used productively, not memory the system is straining to hold onto. Contrast that with a box where available has dropped to 4 GiB despite 49 GiB of buff/cache: that gap means most of that "cache" is actually dirty pages waiting on writeback, or the kernel is close to needing to reclaim aggressively, which is your early warning to check Dirty in /proc/meminfo and vmstat's si/so before it turns into swapping or an OOM (out-of-memory) kill, where the kernel terminates a process to free memory it can't reclaim any other way.

Trade-offs and pitfalls

The most common mistake is alerting on low MemFree directly, which pages nearly every healthy Linux box in existence, since the kernel is designed to use spare RAM for cache rather than leave it idle. Alert on MemAvailable instead. The second common mistake is assuming fsync guarantees durability against every failure mode: a storage controller with a volatile write cache that ignores flush commands, or a filesystem mounted with barriers disabled, can make fsync return success while data is still one power outage away from being lost. On database servers this is worth verifying explicitly rather than assuming the defaults are safe.

Approach

/proc/meminfo is a Linux virtual file (generated by the kernel on read, not stored on disk) with one Key: value kB pair per line; parse it into a dictionary keyed by field name rather than by fixed line position, since the kernel does not guarantee field order is stable across kernel versions, then derive used memory and percent used from the four fields the question names, treating any missing field as an explicit None rather than crashing or silently defaulting to zero.

Implementation (Python)

def parse_meminfo(text):
    """Parse /proc/meminfo text into a dict with derived used-memory fields.

    Handles missing fields (returns None for anything not present) and
    does not assume any particular line order, since /proc/meminfo's
    field order is not a guaranteed kernel ABI.
    """
    wanted = {"MemTotal", "MemFree", "Buffers", "Cached", "SwapTotal", "SwapFree"}
    values = {}
    for line in text.splitlines():
        if ":" not in line:
            continue
        key, _, rest = line.partition(":")
        key = key.strip()
        if key not in wanted:
            continue
        # Values look like "   16345600 kB"; keep the number, kB is the base unit.
        parts = rest.strip().split()
        if not parts:
            continue
        try:
            values[key] = int(parts[0])
        except ValueError:
            continue

    result = {k: values.get(k) for k in wanted}

    mem_total = result["MemTotal"]
    mem_free = result["MemFree"]
    buffers = result["Buffers"]
    cached = result["Cached"]

    if None in (mem_total, mem_free, buffers, cached):
        result["used_kb"] = None
        result["percent_used"] = None
    else:
        used = mem_total - mem_free - buffers - cached
        result["used_kb"] = used
        result["percent_used"] = round((used / mem_total) * 100, 2) if mem_total else None

    return result

Output

Ran against a normal sample (all six fields present, in a different order than the code checks them in) and a partial sample (SwapTotal/SwapFree lines entirely absent, which some kernels omit when swap is disabled):

{
  "Cached": 4271208,
  "MemFree": 6531220,
  "SwapFree": 2097148,
  "SwapTotal": 2097148,
  "Buffers": 412872,
  "MemTotal": 16336984,
  "used_kb": 5121684,
  "percent_used": 31.35
}
{
  "Cached": 4271208,
  "MemFree": 6531220,
  "SwapFree": null,
  "SwapTotal": null,
  "Buffers": 412872,
  "MemTotal": 16336984,
  "used_kb": 5121684,
  "percent_used": 31.35
}

Hand-checked: 16336984 - 6531220 - 412872 - 4271208 = 5121684, and 5121684 / 16336984 * 100 = 31.354..., which rounds to 31.35, matching the printed output; the partial-sample case correctly still computes used_kb/percent_used (they only depend on the four non-swap fields) while reporting SwapTotal/SwapFree as null since those lines were genuinely absent from the input.

Key points

Order-independence: the parser reads every line and keys results by field name (key.strip() from before the colon), so it does not matter whether MemTotal is line 1 or line 12, unlike a parser that assumes a fixed line index. Missing fields: each wanted field defaults to None via values.get(key) rather than 0, which matters because a missing SwapTotal (swap disabled) is a meaningfully different fact than SwapTotal: 0 (swap enabled with zero configured, unusual but different), and silently treating "absent" as "zero" would hide that distinction from anyone consuming the JSON. Derived fields only compute when all four required inputs are present; the arithmetic itself (MemTotal - MemFree - Buffers - Cached) uses the exact formula the question specifies, and both derived fields fall back to None together rather than partially computing on incomplete data.

Trade-offs and pitfalls

This definition of "used" memory is the classic (and slightly pessimistic) one that treats Buffers and Cached as fully reclaimable, which is close to true but not exact on modern kernels (some cached memory is not trivially reclaimable, and there is a more accurate MemAvailable field the kernel exposes directly for this purpose since Linux 3.14). For a production monitoring script, prefer reading MemAvailable directly over recomputing this classic formula, since the kernel's own estimate accounts for reclaim nuances this simple subtraction does not; the formula in this answer follows the question's explicit specification. All units here stay in kB (the native unit /proc/meminfo reports) throughout, converting to MB or GB and mixing units partway through the calculation is a common source of off-by-1024x bugs in scripts like this.