Linux System Administration Questions

Overview

As a systems administrator I monitor these primary metrics to assess immediate health and trends over time: CPU, memory, disk I/O & latency, network bandwidth & packet loss, process counts, and system load. Below each metric I state what a high value usually means and one Linux diagnostic command.

CPU usage

• High value indicates CPU saturation, runaway processes, or inadequate vCPU for workload.
• Command:

top    # or: mpstat -P ALL 1

Memory utilization

• High used memory can mean memory leaks, insufficient RAM, or heavy caching; watch swap use (bad).
• Command:

free -h

Disk I/O and latency

• High I/O or latency suggests disk contention, slow storage, or misbehaving processes causing waits.
• Command:

iostat -x 1 3    # from sysstat

Network bandwidth and packet loss

• High bandwidth usage may saturate links; packet loss indicates network congestion, bad NICs, or MTU mismatch.
• Command:

iftop -P    # or: ip -s link; ping -c 10 <peer>

Process counts

• Very high process count can signal fork bombs, runaway services, or resource exhaustion.
• Command:

ps -e --no-headers | wc -l

System load (load average)

• High load relative to CPU cores indicates CPU, IO wait, or blocked processes; compare to core count.
• Command:

cat /proc/loadavg

Each metric should be trended (Prometheus/Graphite) and correlated—e.g., high load with high iowait points to storage problems, high load with high CPU% to compute limits.

Situation / brief context
As a Systems Administrator tuning Linux for high-throughput network + storage, I'd adjust memory, CPU/IRQ handling, and I/O paths to reduce latency and increase throughput while balancing stability.

Hugepages

• Purpose/effect: Reduce TLB pressure by using large (e.g., 2MB/1GB) contiguous pages; lowers page-fault and mapping overhead for high-memory, high-I/O apps (databases, DPDK, VM guests).
• Helps: High-memory DBs, user-space networking stacks, VMs with large mallocs.
• Risks: Wastes memory (internal fragmentation), reduces RAM available for page cache, requires preallocation (reboot or hugetlbfs setup), complex with overcommit.

irqbalance

• Purpose/effect: Distributes hardware interrupts across CPUs to avoid hot spots.
• Helps: Multi-queue NICs/RDMA servers where IRQs should be spread to match CPU topology.
• Risks: Default balancing can conflict with manual CPU isolation/affinity; may move IRQs away from CPUs pinned to network threads causing cache churn. Use irqbalance with tuned config or set affinities manually (ethtool -C/echo > /proc/irq/.../smp_affinity).

nohz_full / isolcpus

• Purpose/effect: Reduce scheduler ticks and kernel interference on isolated CPUs for latency-sensitive threads.
• Helps: Pin high-throughput packet-processing or storage I/O threads to isolated CPUs for predictable latency.
• Risks: Starves system tasks if misconfigured, complicates load balancing, requires careful IRQ and affinity management; kernel threads and kswapd impacts.

I/O scheduler

• Choices: noop, mq-deadline, bfq, none (for NVMe).
• Helps:
  • SSD/NVMe: noop or none to avoid unnecessary merging/queuing.
  • Spinning disks: mq-deadline or bfq to improve fairness and reduce latency under mixed workloads.
• Risks: Wrong scheduler causes increased latency, throughput drops, or unfairness. Test with fio and real workload.

Practical approach

• Baseline metrics, change one variable at a time, use fio/ipERF, perf/top, and observe CPU, IRQ distribution, page-faults, and cache miss rates. Keep rollback plan and monitor memory pressure and latencies.

Approach

• Read /proc/stat twice a short interval apart to compute instant CPU busy/total delta.
• Read /proc/meminfo once to compute used = total - available.
• Print Prometheus-style metric lines with label host set to the local hostname.
• Use only Python stdlib.

Code

#!/usr/bin/env python3
import time, socket

def read_cpu():
    with open('/proc/stat','r') as f:
        line = f.readline()
    parts = line.split()
    # fields: cpu user nice system idle iowait irq softirq steal guest guest_nice
    vals = list(map(int, parts[1:]))
    idle = vals[3] + vals[4]  # idle + iowait
    total = sum(vals)
    return idle, total

def read_mem():
    mem = {}
    with open('/proc/meminfo','r') as f:
        for l in f:
            k,v = l.split(':',1)
            mem[k.strip()] = int(v.split()[0])  # kB
    total = mem.get('MemTotal',0)
    available = mem.get('MemAvailable', mem.get('MemFree',0) + mem.get('Buffers',0) + mem.get('Cached',0))
    used_pct = 0.0 if total==0 else (total - available) / total * 100.0
    return used_pct

def main():
    host = socket.gethostname()
    idle1, total1 = read_cpu()
    time.sleep(0.1)
    idle2, total2 = read_cpu()
    idle_delta = idle2 - idle1
    total_delta = total2 - total1
    cpu_pct = 0.0
    if total_delta > 0:
        cpu_pct = (1.0 - idle_delta / total_delta) * 100.0
    mem_pct = read_mem()
    print(f'node_cpu_percent{{host="{host}"}} {cpu_pct:.2f}')
    print(f'node_memory_percent{{host="{host}"}} {mem_pct:.2f}')

if __name__ == '__main__':
    main()

Notes

• Short sleep (0.1s) gives near-instant utilization; increase for smoother average.
• Memory values are in kB; percentage calculation is unit-neutral.
• Edge cases: missing /proc entries (containers), zero total delta checked.

Linux — CPU, Memory, Disk I/O, Network

• top / htop
  • Inspect: %CPU (per-process) — shows which process consumes CPU; key for isolating spikes.
  • Example:

top

• vmstat
  • Inspect: id (idle %) and wa (I/O wait) — high wa indicates CPU idle waiting on disk I/O.

vmstat 1 5

• free / -m
  • Inspect: available — indicates memory usable by apps without swapping.

free -m

• iostat (sysstat)
  • Inspect: await (ms) and %util — await = avg IO latency; %util ~100% means device saturated.

iostat -x 1 3

• sar / dstat

  • Inspect: tps / await (for trends) — historical view of IO or CPU over time.

• iftop / iperf3 / ss / nload

  • Inspect: bandwidth (bits/sec) or retransmits — shows throughput and network issues.

iperf3 -c server

Windows — CPU, Memory, Disk I/O, Network

• Task Manager / Resource Monitor

  • Inspect: CPU % (per process) or Disk Queue Length — quick GUI for hotspots.

• typeperf (PerfMon counters)

  • Inspect: \Processor(_Total)% Processor Time — measured CPU utilization over interval.

typeperf "\Processor(_Total)\% Processor Time" -si 1 -sc 5

• Get-Counter (PowerShell)
  • Inspect: \Memory\Available MBytes — low value signals memory pressure.

Get-Counter "\Memory\Available MBytes"

• diskperf / diskspd / Get-Counter disk counters
  • Inspect: Avg. Disk sec/Transfer or Current Disk Queue Length — latency and queueing indicate I/O bottleneck.

diskspd -c1G -d10 -r -w40 \\path\testfile

• Performance Monitor / Network Monitor / iperf3
  • Inspect: Bytes Total/sec or TCP Retransmissions — measures throughput and packet loss impact.

Notes: I use short sampling intervals and correlate CPU/memory/disk/network metrics together to find root cause (e.g., high wa + high %util → disk-bound; high CPU % with low iowait → CPU-bound).

Approach (short)
I’ll list key kernel/sysctl knobs I would tune for thousands/sec short-lived TCP connections, explain what each does, why it helps, and the risks on a live system.

1. net.ipv4.tcp_tw_reuse / net.ipv4.tcp_tw_recycle

• Effect: allow reuse/recycling of TIME_WAIT sockets so ephemeral ports are freed faster.
• Why: reduces TIME_WAIT buildup when many short connections from same client IPs.
• Risk: tcp_tw_recycle breaks NAT/clients with changing timestamps; can cause hard-to-diagnose connectivity issues. tcp_tw_reuse is safer; change carefully and test.

2. net.ipv4.tcp_fin_timeout

• Effect: how long sockets in FIN-WAIT/closing states linger.
• Why: lower value frees kernel memory and ephemeral ports faster for servers dominating short connections.
• Risk: too low can drop connections during slow close or retransmits, causing application-level errors.

3. net.ipv4.ip_local_port_range

• Effect: range of ephemeral ports available for outbound connections.
• Why: expanding the range (e.g., 1024–65535) increases available ports to support many concurrent short-lived client connections.
• Risk: larger range slightly increases port-space for scans; ensure no port collisions with services.

4. net.core.somaxconn and net.ipv4.tcp_max_syn_backlog

• Effect: backlog queue sizes for listen()/SYN queue respectively.
• Why: raise to handle bursty accept() and SYN arrival rates without dropping new connection requests.
• Risk: increasing uses more kernel memory and masks slow-accept bugs in app; excessive values won’t help if app can’t accept fast enough.

5. net.core.netdev_max_backlog and net.core.rmem_max/wmem_max

• Effect: packet receive queue length and socket buffer maxima.
• Why: larger backlog and buffers prevent packet drops under receive bursts and allow kernel to queue more data while application processes connections.
• Risk: higher buffers consume RAM; if set too high on many servers, can exhaust memory and cause OOM.

6. net.ipv4.tcp_max_tw_buckets / net.ipv4.tcp_max_syn_backlog (extra)

• Effect: caps for TIME_WAIT and SYN backlog; useful to raise if system hits limits.
• Risk: raising limits masks root cause and increases memory usage.

Operational notes: test changes on staging, incrementally apply, monitor netstat/ss, /proc/net/sockstat, memory and accept rates. Use connection pooling or keepalives at application layer where feasible to reduce connection churn.