System Resource & I/O Optimization Questions

Requirements & constraints:

• Detect sustained cgroup CPU quota breach per service instance (not transient spikes).
• Remediate safely: reduce priority (move to lower slice) or scale out replicas.
• Avoid cascading remediation (throttling, oscillation, resource starvation).

High-level design:

• Metrics pipeline: node agent (e.g., systemd/cgroup reader or cadvisor) -> metrics store (Prometheus) -> alerting & decision engine (remediator service).
• Remediator has a state store for actions, cooldowns, and global view (service-level replicas, node capacity).

Detection logic:

• Sample cgroup cpu.stat and cpu.max (or cpu.cfs_quota* & cpu.cfs_period*).
• Compute utilization = used_quota / allowed_quota over sliding windows.
• Trigger if utilization > 95% averaged over 3 consecutive windows (e.g., 30s windows => 90s sustained) AND tail-latency or error-rate SLOs degrade.
• Cross-check with node-level metrics (overall load, other pods) to avoid misattribution.

Safe remediation steps (ordered, incremental, idempotent):

1. Mark instance as “candidate” and annotate event store.
2. Soft remediation: throttle affinity/CPU shares — move to lower systemd slice (nice/CPUshares) for 30s.
3. If still high after cooldown, request orchestration action:
   • If stateless and autoscalable: scale out one replica (k8s HPA/CustomController).
   • If stateful: reduce CPU limit only after validation/coordination with owner.
4. If scaling out, wait for new pod readiness and re-evaluate load distribution before further action.
5. If remediation fails or causes errors, rollback by restoring original slice/limits.

Avoid cascading & instability:

• Global rate limit: X remediations per minute per cluster/team.
• Circuit breaker per service: if >N remediations in T minutes, pause automated actions and notify on-call.
• Dependency-aware checks: don’t move many replicas of same service on same node—spread decisions across nodes.
• Cooldown windows and exponential backoff per instance.
• Simulation/impact check: before moving slices, ensure node has spare CPU shares; before scaling, ensure cluster has capacity (or schedule to other nodes).
• Safe defaults and human-in-loop for high-risk services (stateful DBs).

Observability & safety:

• Auditable events, playbooks triggered, runbook links in alerts.
• Canary: start with low-risk namespaces/services, collect metrics and tweak thresholds.
• Unit + integration tests: simulate cgroup metrics and validate remediator behavior.

Trade-offs:

• Conservative thresholds reduce false positives but lengthen remediation. Aggressive actions reduce outage time but risk cascading; use multi-step incremental approach to balance.

This design balances automated corrective action with safeguards to prevent bulk destabilization while remaining observable and reversible.

NUMA (Non-Uniform Memory Access) is a hardware topology where each CPU socket has local memory (NUMA node) with lower latency/better bandwidth than remote memory attached to other sockets. On multi-socket servers, memory access time varies by node; poor locality causes higher latency, increased interconnect (e.g., QPI/UPI) traffic, cache-coherence overhead and degraded throughput—especially for latency-sensitive or memory-bandwidth-bound workloads.

Detection of NUMA imbalance

• numastat: shows per-node page allocation and remote hits. Look for high "numa_hit" vs "numa_miss" and many "numa_foreign" pages.

numastat -p <pid>

• /proc/zoneinfo: Inspect per-node zones for uneven free pages or fragmentation.

grep -A4 Node /proc/zoneinfo

• perf: hardware/events to measure remote memory latency or interconnect traffic, e.g., use perf mem or custom events to correlate stalls with remote accesses.

perf mem record -e mem_load_uops_retired.l1_miss -p <pid> -- sleep 10
perf mem report

Binding processes and memory: use numactl to enforce affinity and allocation policy so threads allocate local memory.

• Bind process and memory to node 0:

numactl --cpunodebind=0 --membind=0 /path/to/app

• Run with interleaved allocation across nodes (good for evenly threaded, memory-uniform apps):

numactl --interleave=all /path/to/app

• For containers, set CPU affinity (cgroups or taskset) and use numactl inside privileged containers or systemd slice with AllowedCPUs/NUMA bindings.

When to use hugepages or CPU pinning

• Hugepages (2MB/1GB): reduce TLB pressure and page walking; valuable for large-memory, high-throughput apps (DBs, JVMs, DPDK). Allocate hugepages per NUMA node to keep translations local:
  • Reserve per-node hugepages via sysfs (/sys/devices/system/node/nodeX/hugepages/hugepages-2048kB/nr_hugepages) and bind allocations (mmap with MAP_HUGETLB or libhugetlbfs).
• CPU pinning: pin worker threads to CPUs on the same NUMA node as their memory to avoid cross-node traffic. Use taskset or sched_setaffinity:

taskset -c 0-7 /path/to/worker

Combine CPU pinning + membind to maximize locality.

Operational guidance

• Measure first: add numastat and perf counters to monitoring and create alerts for rising remote accesses or interconnect utilization.
• Start with numactl --membind per service; use interleave for scalable, NUMA-agnostic workloads.
• Use hugepages for predictable memory-heavy apps after testing (watch fragmentation).
• Prefer pinning when you control thread-to-work mapping (low-latency SLOs). Avoid hard pinning for transient, multi-tenant workloads unless scheduler guarantees.

This approach minimizes latency, reduces interconnect saturation, and improves predictability—key for SRE-managed latency and availability targets.

cpu-clock and cycles are both perf sampling events but measure different things and lead to different interpretations of “hot” code.

Definitions

• perf record -e cpu-clock samples based on elapsed CPU-time (a software event; typically CLOCK_MONOTONIC). It fires at regular wall/CPU-time intervals (e.g., every X microseconds of CPU time consumed) and attributes samples to whatever instruction was executing at that time.
• perf record -e cycles uses the CPU performance-monitoring unit (PMU) hardware cycle counter to sample when a certain number of CPU clock cycles have elapsed. This is a hardware event sensitive to actual CPU cycles executed.

When to prefer each

• Use cycles when you want true cycle-weighted hotspots: CPU-bound code that consumes many cycles (including due to inefficient instructions, pipeline stalls, cache-misses) will show up prominently. Good for low-level optimization and understanding CPU micro-architectural costs.
• Use cpu-clock when you care about wall/CPU time or want a sampling that is independent of CPU frequency scaling, hyperthreading artifacts, or when PMU is restricted (containers/VMs with no PMU access). cpu-clock gives a more portable view of where time is spent from a scheduler/latency perspective.

How basis affects interpreting hotspots

• Frequency scaling and turbo: cycles reflect actual cycles at current frequency; if turbo boosts some cores, cycles may exaggerate hotspots on boosted cores. cpu-clock normalizes to elapsed time, so it's less sensitive to frequency variance.
• Sleep/IO vs busy-wait: code blocked in syscalls won't accumulate cycles but may still account for cpu-clock when accounting thread CPU time; however blocked threads usually don't consume CPU time—cpu-clock typically measures CPU time, so blocked time won’t show—be careful which clock (task-clock vs cpu-clock) you're using.
• Sampling bias: hardware counters may be biased by event overflow handling, PMU multiplexing, or kernel filtering. Software cpu-clock sampling is regular and less susceptible to PMU multiplexing artifacts.
• Attributing hotspots: cycles emphasize instruction-level cost; cpu-clock emphasizes where CPU time is scheduled. For SRE work: use cpu-clock to find latency/CPU-time hotspots affecting service SLA; use cycles when tuning hot functions for throughput and when micro-optimizations matter.

Practical tips

• Correlate both: run both and compare—functions high in cycles but low in cpu-clock might be CPU-inefficient but infrequent; high in cpu-clock but low in cycles could indicate waiting or spinning behavior impacted by scheduler.
• Account for virtualization/containers: PMU may be virtualized—validate counters. Use task-clock for per-process CPU time if that matches your SLA.
• Always complement sampling with flame graphs, perf report, and context (cache-misses, branch-misses) to explain why a hotspot exists.

Situation: On-call for an SRE team; alert: high disk latency observed on one or more hosts/volumes. Use this runbook to triage, mitigate, communicate, and follow up.

1. Immediate safety checks (don’t change state yet)

• Confirm alert and scope: which hosts/volumes, services, and time window.
• Note timestamps, alert IDs, and on-call handover.

2. Initial evidence-gathering commands (run on affected hosts and control plane)

• Disk metrics:
  • iostat -x 1 5
  • iotop -o -b -n 5
  • ioping -c 10 /path/to/device
  • fio --name=smoke --size=16M --rw=read --iodepth=1 --bs=4k --runtime=10 --time_based
• OS metrics:
  • vmstat 1 5
  • sar -d 1 3
  • pidstat -d 1 5
  • atop (if installed)
• Filesystem / capacity:
  • df -h
  • lsblk -o NAME,SIZE,ROTA,MOUNTPOINT
  • du -sh /var/* (if suspect)
• Processes / locks:
  • lsof +D /path/to/mount
  • fuser -v /path
• Kernel / hardware logs:
  • dmesg | tail -n 200
  • journalctl -u <service> --since "10 minutes ago"
  • smartctl -a /dev/sdX (careful on cloud VMs)
• Container / orchestration:
  • docker stats / pod metrics
  • kubectl get pods -o wide; kubectl top pod/node
  • describe pods for Evicted/IO errors
• Network/storage backend:
  • ping / traceroute to storage gateway
  • check SAN/NFS mount options and mount status
• Application signals:
  • check SLO/SLI dashboards, error rates (5xx), latency percentiles
  • check request queues, worker backlog

3. Criteria for service-impacting

• Any of:
  • SLO/alert for client-facing latency or error-rate exceeded
  • Application request latency P99 increased beyond threshold
  • Increased 5xx or timeouts affecting >X% of requests (e.g., >1% of traffic)
  • Worker queues/backpressure leading to failures or cascading delays
  • Essential control-plane components degraded (datastore masters, queue brokers)
    If criteria met -> declare incident SEV and follow Major Incident protocol.

4. Immediate mitigations (apply quickly, document each step)

• Throttle noisy IO:
  • Identify heavy processes (iotop, pidstat) and reduce priority: ionice -c2 -n7 -p <pid>
  • Use cgroups / systemd BlockIO or blkio controller to limit IO to noisy containers
• Move/migrate workloads:
  • Evict pods from affected node: kubectl drain <node> --ignore-daemonsets --delete-emptydir-data (if safe)
  • Scale out read replicas or redirect traffic to healthy instances
  • Use orchestrator migration (live migrate VMs if supported)
• Pause/stop IO-heavy background jobs:
  • Pause backups, compactions, bulk jobs, snapshot operations
  • Disable cron jobs or lower retention compaction rates
• Failover to replicas or degraded mode:
  • Promote healthy replica if primary's storage impacted
  • Switch read traffic to replicas
• Storage-level actions:
  • Remount with async options only if safe and approved
  • Contact cloud provider / storage team if backend shows faults; do not run destructive ops without approval
• If hardware failure suspected: cordon node, remove from service, reattach/replace disks via runbook

5. Communication steps

• Initial page: short summary (what, where, impact, first mitigation), declare SEV if applicable
• Status updates: every 15 minutes or per SLA; include scope, impact, actions, next ETA
• Notify stakeholders: Product owner, PIC on affected services, on-call manager, storage team, customers (per comms plan)
• Post to incident channel with timestamps of actions and links to dashboards/logs
• If public/externally visible, coordinate customer-facing status page messages

6. Post-incident actions (assign owners, timeline)

• Gather artifacts: logs, iostat outputs, fio results, dmesg, orchestration events, timestamps
• Run a blameless postmortem within 3-5 business days:
  • Root cause analysis (hardware, kernel, config, noisy job, capacity)
  • Contributing factors and detection gaps
  • Action items with owners and deadlines
• Prevent recurrence:
  • Implement automated mitigations: auto-throttle policies, CI changes to limit background IO, alert tuning for early IO queue growth
  • Capacity/architecture changes: add performance headroom, separate IO-heavy workloads to dedicated volumes/nodes, use faster disks or storage tiering
  • Add synthetic IO SLO checks and dashboards (iostat trend, ioping P99)
  • Harden backup/compaction scheduling and add circuit-breakers
• Update runbooks and playbooks with exact commands, thresholds, and contact list
• Close incident once actions are tracked and stakeholders informed

7. Checklist / quick reference

• Document affected hosts, services, timestamps
• Attach collected evidence
• Save command outputs to central incident artifacts storage
• Ensure rollback path exists for any mitigation that risks data loss

This runbook assumes safe coordination with ops/storage teams for destructive actions. When in doubt, prioritize read-only diagnostics and stakeholder communication.

Cgroups provide kernel-level resource accounting and enforcement (CPU, memory, IO). ulimit (RLIMITs) is a per-process userspace interface via the shell/exec to limit things like address space (AS) and CPU time. As an SRE you use cgroups for robust container/service limits and ulimits for lightweight per-process caps.

Systemd (cgroup v2) examples — set MemoryMax and CPUQuota in a unit:

[Service]
MemoryMax=512M        # cap resident + anon memory for the unit
CPUQuota=50%          # allow 50% of one CPU (or 50% of total CPU capacity)

Under the hood CPUQuota sets cpu.max (cfs_quota_us / cfs_period_us). MemoryMax maps to memory.max and triggers cgroup accounting.

Docker examples:

# memory limit and CPU share/quota via high-level flags
docker run --name app --memory=512m --cpus="0.5" myimage

# lower-level equivalent using quota/period
docker run --memory=512m --cpu-quota=50000 --cpu-period=100000 myimage

ulimit examples (bash):

ulimit -v 524288   # limit virtual memory (KB)
ulimit -t 60       # CPU time (seconds)

OOM killer interaction:

• When a cgroup's MemoryMax is exceeded, the kernel records an OOM event and will select processes to kill from that cgroup (or system-wide) based on oom_score and oom_score_adj. In cgroup v2 the cgroup that triggered the pressure is considered; systemd exposes memory.events and logs.
• For processes hitting RLIMIT_AS, allocations may fail (malloc returns NULL / ENOMEM) rather than immediately invoking the global OOM killer. But if overall system memory pressure is high, the kernel OOM killer can still choose targets across cgroups—properly sized per-cgroup limits help contain and make kills predictable.
  Best practices:
• Prefer cgroups/systemd for containers and services.
• Set oom_score_adj for critical services to avoid being killed.
• Monitor memory.events / OOM logs and use graceful OOM handling (OOM notifications, oom_kill_disable where appropriate) and safe defaults to avoid cascading failures.