Linux System Administration Questions

systemd Service Type controls how systemd considers a unit "started" and how it tracks the process. Key types:

• simple

  • Behavior: systemd runs ExecStart and immediately considers the service started (no forking expected).
  • PID tracking: systemd tracks the main process = the PID of ExecStart child (cgroup-based).
  • Restart: restarts the single process; good for foreground daemons.
  • Use when: the binary stays in foreground and does not daemonize.

• forking

  • Behavior: ExecStart is expected to fork: parent exits after spawning a child; systemd waits for parent exit and then assumes the service started.
  • PID tracking: traditionally reads PIDFile or uses the child PID if discovered; better to set PIDFile= for reliability.
  • Restart: restarts the process by running ExecStart again; systemd may lose child PID if PIDFile not provided.
  • Use when: wrapping a legacy SysV-style daemon that double-forks.

• oneshot

  • Behavior: runs ExecStart and waits for it to finish; unit is considered started only after command exits successfully.
  • PID tracking: no long-running process tracked; useful for one-off tasks (setup scripts).
  • Restart: Restart= is typically unused; RemainAfterExit=yes can make it “active” after exit.
  • Use when: init tasks, configuration steps, or short-lived jobs.

• notify

  • Behavior: service must send readiness notification to systemd via sd_notify("READY=1").
  • PID tracking: systemd tracks the service process (foreground); until sd_notify is received, service is not "started".
  • Restart: strong for complex startups where readiness is asynchronous.
  • Use when: long startup where readiness depends on internal checks (e.g., server listening on sockets after initialization).

• dbus

  • Behavior: unit is started when the service acquires a well-known D-Bus name.
  • PID tracking: systemd associates the service with the process that owns the D-Bus name.
  • Restart: lifecycle tied to D-Bus name ownership.
  • Use when: services that register on the system bus (desktop/system services).

Guidance:

• Classic daemon that can run foreground: prefer simple (or convert to foreground mode).
• Forking SysV-style: use forking plus PIDFile= to ensure correct PID tracking; consider wrapping with systemd-sysv-generator or rewrite to systemd-native.
• Modern systemd-native service: prefer notify (or simple) and use sd_notify and socket activation where possible. For new services, avoid forking; use cgroup-aware foreground processes so systemd has accurate control, health checks, and restarts.

File descriptors (FDs) are small integer handles the Linux kernel gives a process to access resources (files, sockets, pipes, devices). They index entries in the process's file descriptor table which point to kernel objects (file descriptions) that reference an inode for regular files or a socket struct for network endpoints.

Difference between FD and inode:

• FD: per-process integer handle (e.g., 3,4,5) — ephemeral and local to a process.
• inode: filesystem metadata object identifying a file on disk (permissions, links, disk blocks) — global to the filesystem, independent of which process opened it.
  A single inode can be opened multiple times by many processes; each open yields a separate FD.

Conventional standard descriptors:

• 0: stdin
• 1: stdout
• 2: stderr

Inspect open descriptors:

• /proc: list a process’s FDs (symlinks to targets)

ls -l /proc/<pid>/fd
readlink /proc/<pid>/fd/3

• lsof: human-friendly listing

lsof -p <pid>
lsof +L1   # list deleted-but-open files

Increase allowed open file descriptors:

• Temporary (current shell/process): use ulimit

ulimit -n 65536

or for a single command:

ulimit -n 65536; ./your_service

• Permanent (system-wide or per-user):
  • /etc/security/limits.conf or a file in /etc/security/limits.d/:

    youruser soft nofile 65536
    youruser hard nofile 131072
    

  • For systemd services, set in the unit (e.g., /etc/systemd/system/your.service.d/limits.conf):

    [Service]
    LimitNOFILE=65536
    

  then reload daemon and restart service:

systemctl daemon-reload
systemctl restart your.service

Also verify with:

cat /proc/<pid>/limits | grep "Max open files"

Best practice: set both soft and hard limits and configure systemd for services; monitor with lsof/ss and alert on fd exhaustion.

Approach: explain Linux memory overcommit policy, sysctls, OOM selection, and SRE mitigations.
How overcommit works:

• Linux can allow processes to allocate more virtual memory than physical RAM+swap. Behavior controlled by vm.overcommit_memory:
  • 0 (heuristic): kernel uses heuristic based on RAM and allocations.
  • 1 (always): allow all allocations (no probing).
  • 2 (strict): enforce limit based on vm.overcommit_ratio.
• vm.overcommit_ratio: when mode=2, allowed = swap + RAM * (ratio/100).
  OOM killer decision:
• When system runs out of memory, the OOM killer scores processes using oom_score_adj and heuristics (memory usage, runtime, process privilege, badness). Highest badness killed. You can view /proc/<pid>/oom_score and oom_score_adj to influence.
  SRE strategies to avoid OOMs:
• Use memory limits at cgroup/container level to bound processes and avoid system-wide OOMs.
• Prefer vm.overcommit_memory=2 in memory-constrained critical hosts; tune vm.overcommit_ratio conservatively.
• Enable/size swap to provide headroom; monitor swapin penalties.
• Configure OOM_adj or systemd OOMPolicy for critical services; set oom_score_adj for low-score services.
• Use ulimit -v to constrain per-process virtual memory for riskier workloads.
• Instrument and alert: export node memory metrics (available, committed_as, slab, swap) and per-container RSS; set alerts for high commit_ratio and low available memory.
• Proactive testing: run fault-injection (stress-ng) in stage to validate OOM policies and graceful degradation.
  Why: these controls let SREs trade availability vs. utilization and prevent a single runaway allocation from taking down the host.

Immediate containment (minutes):

• Limit per-service processes via systemd: edit the unit or drop-in:

# /etc/systemd/system/myservice.service.d/limit.conf
[Service]
LimitNPROC=200
TasksMax=200

then reload and restart:

sudo systemctl daemon-reload
sudo systemctl restart myservice

• If using cgroup v2, enable pids controller and set per-slice limit:

# set for a non-root slice at runtime (example)
echo 200 > /sys/fs/cgroup/myservice.slice/pids.max

or via systemd unit: add TasksMax=200 (cgroup v2 uses TasksMax).

Short-term isolation (hours):

• Move service into its own PID namespace (containerize or run with systemd-nspawn/docker) so host PID exhaustion cannot occur for other services.
• Apply fork-rate limiting using a token-bucket wrapper (example using systemd token-based throttle is not native) — instead throttle at app level (see below) or use eBPF to monitor fork() rates and kill runaway processes.

Code & systemic fixes (days-weeks):

• Replace uncontrolled fork-per-request with a worker pool or thread pool; use a process pool (prefork) to bound concurrency.
• Reuse processes: implement long-lived workers handling multiple tasks, avoid exec/fork per short job.
• Add guarded retry/backoff and circuit-breaker logic when the service is overloaded.
• Add runtime checks: before forking, check current children count (getpid/getpgrp / proc) and refuse or queue work if near limit.
• Add observability: emit metrics (fork rate, children alive, pids.available) and alerts for thresholds.

Prevent recurrence / infra changes:

• Enforce per-service pids limits in orchestration (Kubernetes: pod-level pids cgroup via runtimeClass or host pid namespace off; set resourceQuota and limitRange).
• Use system-level eBPF-based protections to detect fork-bomb patterns and auto-throttle.
• CI checks and code reviews to prevent regressions.

Commands & checks:

# Check available PIDs
cat /proc/sys/kernel/pid_max
ps -eo pid,cmd | wc -l
# Check per-cgroup pids
cat /sys/fs/cgroup/myservice.slice/pids.current

Trade-offs:

• Aggressive LimitNPROC can cause user-visible failures; prefer graceful queuing + alerts.
• PID namespaces isolate but hide host-level visibility — combine with metrics and logging.

This plan gives immediate protection, short-term isolation, and durable code/infra fixes to prevent PID exhaustion.

Requirements & constraints:

• Run arbitrary user workloads as services per-tenant with strong isolation to prevent noisy neighbors.
• Keep per-tenant management and observability feasible for SRE team.
• Balance resource guarantees (isolation vs utilization), security hardening, and operational complexity.

High-level approach:

• Use systemd as the service manager, leveraging cgroups v2 and Linux namespaces per tenant service. Model each tenant as a systemd slice + one or more templated systemd units (services) that run in dedicated namespaces.

Core components & configuration:

1. cgroups (resource control)

   • Create a slice per-tenant: tenant-<id>.slice.
   • Configure cgroups v2 settings on the slice:
     • CPU: cpu.max = "quota period" or use cpu.weight for proportional shares. Example: cpu.max=200000 1000000 for 20% hard cap.
     • Memory: memory.max (hard limit) + memory.high (soft limit) for graceful throttling. Example: memory.high=1G, memory.max=1.2G.
     • IO: io.weight or io.max to limit block device bandwidth.
   • Use systemd directives: CPUQuota=, MemoryMax=, IOWeight= in unit templates.

2. Namespaces & capability hardening

   • Launch services with systemd's PrivateTmp=, ProtectSystem=full, ProtectHome=yes, PrivateDevices=yes.
   • Use User= and DynamicUser= or run under per-tenant user via user namespaces (systemd supports User= + PrivateUsers=) to map UID ranges and reduce host UID exposure.
   • Drop capabilities with CapabilityBoundingSet=, AmbientCapabilities= to minimal set.
   • Add seccomp filters (SystemCallFilter=) and AppArmor/SELinux profiles per tenant.

3. PID & proc isolation

   • Limit PIDs with PIDsMax= to prevent fork bombs.
   • PrivateNetwork= can give network namespace per-tenant; combine with veth + macvlan or CNI to control bandwidth and ACLs.

4. Management & automation

   • Use templated systemd unit files: /etc/systemd/system/tenant@.service and tenant@.slice for consistent policy.
   • Central tooling: orchestration scripts (Ansible/Go/Python) to provision units, set per-tenant resource configs, and rotate policies.
   • Integrate with Prometheus: cgroup metrics (cpu.stat, memory.current), systemd unit state, and per-tenant network/IO telemetry.
   • Alerts: noisy-neighbor detection (sustained CPU throttle, memory.high events, IO saturation).

Operational behaviors and policies:

• Guarantee model: mix of hard and soft controls.
  • Hard caps (memory.max, cpu.max) for strict isolation when tenants pay for reserved resources.
  • Soft limits (memory.high, cpu.weight) for burstable tenants; system can reclaim via throttling/oom_kill policies.
• OOM handling: set KillMode=process vs control-group; prefer OOM killer fallback but monitor OOM events and surface to tenant owner.
• Overcommit policy: maintain reserved capacity for SLO-critical tenants; allow best-effort tenants to use spare capacity.

Trade-offs:

• Resource guarantees
  • Hard limits provide strong isolation but reduce overall utilization and can lead to hard failures (OOM kills).
  • Soft limits and weighted shares maximize utilization but allow noisy neighbors to affect latency for low-priority tenants.
• Security
  • User namespaces + capability drops + seccomp greatly reduce escalation risk. But user namespaces complicate UID mapping (esp. NFS) and some workloads (setuid) may break.
  • AppArmor/SELinux profiles give stronger confinement but require policy maintenance and testing.
• Operational complexity
  • systemd + cgroups + namespaces keeps everything on a single host with well-understood tooling—lower infra stack than full container orchestration. Templated units and automation reduce per-tenant toil.
  • However, per-tenant namespace networking, seccomp filters, and per-tenant profiles increase testing surface and debugging complexity.
  • Debugging noisy-neighbor sources requires good observability across cgroup metrics and kernel events.

Practical recommendations:

• Start with a conservative default slice template: CPUQuota for critical tiers, cpu.weight for shared tier, MemoryMax for safety, PIDsMax conservative.
• Enforce capability drops and PrivateTmp/ProtectSystem as defaults; add stricter policies for untrusted tenants (seccomp + AppArmor).
• Implement telemetry: collect cgroup v2 stats, systemd unit logs, kernel oom events; build dashboards/alerts for throttle %, memory reclaim, and IO saturation.
• Provide self-service quotas via automation and clear tenant tiers (guaranteed, burstable, best-effort).
• Regularly run chaos tests (CPU/memory/IO stress) to validate isolation settings and incident runbooks.

This strategy gives strong, kernel-backed isolation (cgroups v2 + namespaces) while keeping management feasible via systemd templates and automation; trade-offs are explicit—choose hard vs soft limits and security strictness based on tenant trust and SLO priorities.