Memory Management, Paging & Virtual Address Space Questions

Situation: Production C service intermittently crashes with SIGSEGV at varying user-space addresses, risking availability and data loss.

Immediate priorities (first 0–15 minutes)

• Preserve evidence without disrupting traffic: do not restart the process; if using containers, pause/unfreeze the container or take a process snapshot (CRIU) if supported. Route new traffic to healthy instances and keep the crashed host in maintenance mode.
• Collect volatile state:
  • Enable and collect a core dump (see kernel settings below) or copy any existing core files.
  • Gather process info: pid, cmdline, open fds (ls -l /proc/<pid>/fd), /proc/<pid>/maps, /proc/<pid>/stack, /proc/<pid>/status.
  • Capture running threads: gcore or perf record if safe.
  • Collect logs (stdout/stderr, app logs, systemd journal) and recent metrics/traces.
  • Snapshot binary and all shared libs (ldd output) and related debug symbols.

Kernel/settings to adjust (apply carefully)

• ulimit -c unlimited for the service user and systemd unit CoreDumpSize.
• echo "|/usr/bin/systemd-coredump %P %u %g %s %t %e" > /proc/sys/kernel/core_pattern or point to a safe core collector; ensure permissions.
• Check /proc/sys/kernel/yama/ptrace_scope: set to 0 temporarily if you need to attach debuggers across users.
• Ensure /proc/sys/vm/oom_kill_allocating_task not interfering.

Tools & steps (prioritized)

1. gdb: open core with exact binary + symbols: gdb /path/to/bin /path/to/core; bt full, info threads, thread apply all bt.
2. addr2line / eu-addr2line: translate addresses to file:line using exact binary and .debug files.
3. perf or pstack: if reproducible under load, collect perf record -g to see stacks leading to crash.
4. ASAN/UBSAN/LSAN build: run a debug build with ASAN in staging to catch heap/UB issues (not on prod at full traffic).
5. Valgrind/Memcheck: use in a mirrored environment; very slow—use for isolated reproductions.
6. sanitizers in production: consider AddressSanitizer with relaxed flags on a canary if safe.
7. Static tools: clang-tidy, coverity, or compiler sanitizers on CI.
8. strace/ltrace only if low-overhead acceptable; they add noise.

Investigation timeline

• Short term: analyze existing core and maps to find consistent frames or library.
• Mid term: deploy extra logging, increase sampling profilers, run builds with sanitizers against representative workloads.
• Long term: add continuous fuzzing/memory sanitizer in CI, enforce debug-symbol retention and automated core collection, add watchdog/auto-restart with rate limiting.

Preserve chain of custody

• Hash core, binaries, and libs (sha256) and store in immutable artifact store.
• Document timestamps, host, and commands used; avoid binary rebuilds that change addresses.

Key cautions and trade-offs

• Don’t turn off core collection silently—ensure storage/rotation to avoid disk fill.
• Avoid heavy instrumentation on production traffic; use canaries or traffic mirroring.
• ptrace_scope relax is temporary—restore after debugging.

This plan balances immediate evidence capture, non-disruptive investigation, tooling for root cause identification, and longer-term mitigation to prevent recurrence.

Page coloring (cache coloring) is an OS-level allocation technique that reduces cache-set conflicts that arise from arbitrary virtual-to-physical page mapping. It exploits the fact that low-order bits of the physical address determine which cache set (or line in direct-mapped caches) a physical address maps to. By grouping physical frames into “colors” (each color = set of frames that map to the same cache set index), the allocator can assign physical pages so that frequently used pages are spread across different colors, avoiding hot collisions.

Direct-mapped caches:

• Each cache index holds exactly one line; two virtual pages whose physical frames share the same color will always conflict.
• The allocator must ensure hot pages get different colors; simple policy: round-robin allocate across colors or reserve colors for kernel/user buffers to eliminate persistent thrashing.

Set-associative caches:

• Each index has N ways; conflicts occur only when >N pages map to same color+index. Coloring still matters—if the working set concentrated on fewer colors than associativity supports, performance degrades.
• Allocator can pack pages to keep per-index pressure below associativity; for high associativity, coloring requirements are looser, but L3 caches with large slices and complex indexing still benefit.

Effects on allocator behavior and fragmentation:

• Pros: reduces conflict misses, improves throughput and latency for cache-sensitive workloads (databases, in-memory caches).
• Cons: increases allocator complexity and external fragmentation because allocation must satisfy color constraints; worst-case some colors may fill while others are free, causing inability to satisfy contiguous multi-page allocations (important for hugepages).
• Mitigations: use color-aware freelists per color, allow color-aware coalescing and migration, use relaxed policies (best-effort coloring) to balance fragmentation vs. cache friendliness.

Practical SRE guidance:

• Measure: use perf (cache-misses, LLC-load-misses) before/after; profile with representative workloads.
• Apply selectively: enable coloring for cache-sensitive services or for hugepage pools; avoid global strict coloring that increases memory pressure.
• Combine with other techniques: page migration, NUMA-aware allocation, and application-level layout (malloc arenas) to get predictable performance.

Multi-level page tables partition the virtual-to-physical mapping into a hierarchy of smaller tables to avoid allocating one huge flat table for every process. They trade a few pointer lookups for large memory savings because intermediate page-table pages are allocated only for address ranges a process actually uses.

x86-64 (with 4KB pages and a 48-bit canonical virtual space) typically uses 4 levels:

• PML4 (level 4)
• PDPT (level 3)
• PD (level 2)
• PT (level 1)
  Bit layout (48-bit VA): [9-bit PML4][9-bit PDPT][9-bit PD][9-bit PT][12-bit page offset]

How this reduces memory vs single-level:

• Single-level: need one entry per virtual page => for 2^36 pages (48-12=36), a table with 2^36 entries (impractically large).
• 4-level: each table page contains 512 entries (2^9). You only allocate table pages for branches that contain mapped pages. Sparse address spaces (typical processes) therefore use far less RAM.

Steps to translate a virtual address to a physical address:

1. Split the VA into indices and offset: i4,i3,i2,i1,offset (each i = 9 bits; offset = 12 bits).
2. Read CR3 to get physical base of PML4.
3. Index PML4 at i4 to get PDPT physical base (if entry invalid -> page fault).
4. Index PDPT at i3 to get PD base (or huge-page mapping).
5. Index PD at i2 to get PT base (or huge-page mapping).
6. Index PT at i1 to get physical frame base (or fault if not present).
7. Physical address = frame_base + offset.

Notes relevant to SREs:

• Multilevel tables enable efficient memory usage for many processes and containers.
• TLBs and huge pages (2MB/1GB) reduce the number of lookups and page-table memory pressure; monitor page-fault/major-fault rates and consider large pages for high-performance services.

Situation: Production node repeatedly logs OOM-killer events and services are killed intermittently. As on‑call SRE I need to quickly stop customer impact, gather forensic evidence (pre-change and post‑change), find root cause, and restore safe operating conditions while preserving observability.

Immediate triage & evidence collection (before changes)

• Capture kernel OOM logs and context:
  • dmesg --ctime | tail -n 200
  • journalctl -k --since "10 minutes ago"
  • /var/log/kern.log or /var/log/messages
• Correlate with container/k8s events:
  • kubectl get events --all-namespaces --sort-by='.lastTimestamp'
  • kubectl describe node <node> and kubectl describe pod <victim-pod> to see OOMKilled reason
• Snapshot metrics and system state:
  • vmstat 1 5; free -h; cat /proc/meminfo
  • ps aux --sort=-rss | head -n 20 ; top / htop snapshot
  • crictl ps / docker ps & crictl stats (or docker stats)
• Preserve monitoring data (Prometheus/Grafana dashboards) and export current time-range graphs; take screenshots and export raw metrics (Prometheus API query).
• Collect kubelet and kube-proxy logs, node-exporter, cadvisor, and application logs; save copies centrally (scp to artifact store).
• Create a timeline (timestamped) of OOM events, pod restarts, deployments, traffic spikes, and recent config changes.

Immediate mitigations (minimize impact, keep observability)

• Avoid disruptive wide changes. Apply targeted mitigation on the affected node:
  • Cordon the node: kubectl cordon <node>
  • Drain if safe, but preserve monitoring/infra daemons: kubectl drain <node> --ignore-daemonsets --delete-emptydir-data --force (ensure Prometheus, logging agents run as DaemonSets with podAntiAffinity as needed)
  • If draining risks losing observability, instead scale down/evict only noisy/low‑priority pods: kubectl delete pod <noisy-pod> or change QoS.
• If pods are OOMKilled due to too-low container memoryLimits, temporarily increase limits for the affected deployment (small incremental bump), or change requests/limits using kubectl patch or new rollout; prefer patching a single replica to verify.
• If host memory exhausted by non-container processes, identify and kill runaway process with highest RSS after recording pid and stack dump if possible.
• If immediate pressure persists, add capacity: cordon + provision new node(s) or scale node group (cloud autoscaling) and move workloads; keep node cordoned until root cause found.
• Preserve observability agents: ensure node-exporter, Prometheus node exporters, logging and traces remain running or run them on control plane; if draining would stop them, run a monitoring pod on another node or push metrics snapshot to long-term store.

Root-cause diagnosis

• Correlate time-series: memory used (node_total - free - cache), OOM events, pod memory usage (cAdvisor / kubelet), GC timings or memory growth in app logs, recent deploys/config changes, cron jobs/backups.
• Check container QoS: BestEffort/Guaranteed/Burstable; BestEffort most likely to be killed.
• Inspect application: heap dumps (Java), pprof (Go), core dump (with kernel.core_pattern to capture).
• Verify kernel parameters and host config: vm.overcommit_memory, swap status (k8s recommends swap off), oom_adj/oom_score.
• Reproduce in staging if possible.

Evidence collection after changes

• Keep detailed audit trail: what commands/patches were applied, diffs of manifests, timestamps.
• Export Prometheus metrics for pre/post windows; save screenshots and raw query results.
• Re-run vmstat/free/top and collect logs showing stabilization.
• If you increased limits or added nodes, record observed reduction in OOM events and improvement in pod restarts and latency/error SLOs.

Post-incident actions

• Rollback risky temporary fixes to permanent safe solution: tune memory limits/requests, implement liveness/readiness probes, enforce resource requests with admission controller, set proper QoS.
• Add alerts and dashboards: alert on node memory pressure, OOM events metric (node_oom_events_total), container memory RSS approaching limit.
• Create runbook: step-by-step for similar OOM incidents (how to capture evidence, safe mitigations).
• Postmortem with timeline, root cause, permanent fixes, and follow-ups (capacity planning, code fixes, configuration enforcement).

Why this approach

• Collecting logs and metrics before changing state preserves forensic evidence.
• Targeted, minimally disruptive mitigations protect customer traffic and maintain observability.
• Correlating system and application telemetry finds whether cause is infra (node-level leak), misconfigured container requests/limits, or application memory leak.
• Postmortem and automation prevent recurrence.

Interpret low-order bits of the 64-bit PTE (bit0 = P, bit1 = RW, bit2 = US(User), bit5 = A, bit6 = D, bit63 = NX). The example PTE: 0x0000ff81234567f3. Expand low bytes: 0x67f3 -> 0xF3 (low byte) = 11110011b, 0x67 (next) = 01100111b.

Decoded relevant bits:

• bit 0 (Present, P): 1 (0xF3 bit0 = 1) → page is present
• bit 1 (Read/Write, RW): 1 (0xF3 bit1 = 1) → writable
• bit 2 (User, US): 0 (0xF3 bit2 = 0) → supervisor-only (not user)
• bit 5 (Accessed, A): 1 (0xF3 bit5 = 1) → CPU has accessed the page
• bit 6 (Dirty, D): 1 (0xF3 bit6 = 1) → page has been written
• bit 63 (NX): MSB byte is 0x00 → bit63 = 0 → NX not set (page is executable)

Summary answer:

• Present: yes
• RW (writable): yes
• User: no (supervisor only)
• Accessed: yes
• Dirty: yes
• NX (no-execute): no (executable)

You can also note other low flags visible here: PCD (bit4) = 1 and Global (bit8) = 1 from the same bytes, but those weren’t requested.