Linux System Administration Questions

Zombie vs orphan, precisely

A zombie process has already finished executing (it called exit() or was killed) but still has an entry in the process table because its parent has not yet called wait()/waitpid() to collect its exit status; it shows up as state Z and is sometimes labeled <defunct> in ps output. An orphan process is the opposite situation: it is still running, but its original parent has exited (or died) before it did. Orphans are not a problem by themselves: the kernel automatically re-parents an orphan to init/PID 1 (or, inside a container, whatever process occupies PID 1), which is specifically responsible for reaping any child, including one it didn't originally start. A zombie only becomes a lingering problem if its parent is itself broken and never calls wait(), or, in a container, if PID 1 isn't a real init system that reaps at all.

Detecting zombies

ps -eo pid,ppid,stat,cmd | awk '$3 ~ /Z/'

The STAT column showing Z (often rendered with a trailing + if in the foreground process group) identifies zombies directly; ps aux output labels the same processes' COMMAND column with <defunct>. The PPID column in the filtered output tells you exactly which process is failing to reap its children, which is the actual thing to go fix.

Safe remediation, and what not to do

• You cannot kill a zombie into disappearing. It has no running code left to terminate; sending it any signal, including SIGKILL, does nothing, because there is nothing left to receive or act on the signal.
• The correct fix is making the parent call wait(). If the parent is a long-running application with a bug (it fork()s child processes and never reaps them, or fails to handle SIGCHLD), that's an application-level fix: add a SIGCHLD handler that calls waitpid(), or use a process-management library that reaps for you.
• If the parent process itself can be safely restarted (e.g. a supervisor daemon with a known reaping bug, and restarting it won't disrupt in-flight work it's tracking), restarting it causes all its zombie children to be re-parented to init, which reaps them immediately. This is "safe" specifically because it doesn't touch any live data: the zombies are already-dead processes with no state left to corrupt, so re-parenting and reaping is purely a bookkeeping cleanup, not an operation on live application data.
• If the parent cannot be restarted (it's a critical, hard-to-restart process, or restarting would drop in-flight connections), the pragmatic short-term step is monitoring: a handful of zombies consumes a spare process-table slot each but no CPU or memory, so it's tolerable in the short term while a fix is scheduled, as long as the count is bounded and not still climbing.
• What not to do: don't try to "force remove" a zombie via any filesystem or /proc manipulation. There is no file to delete that represents the zombie; its entry is purely a kernel process-table record, and the only correct exit is the parent's wait() call (or re-parenting plus init's automatic reap).

Worked example

A supervisor process forks worker subprocesses per incoming job and, due to a missing SIGCHLD handler, never reaps them once they finish. Over a week, ps -eo stat | grep -c Z climbs from 0 to several hundred. The safe remediation path: confirm via PPID that all the zombies share the same parent (the supervisor), confirm the supervisor has a known, low-risk restart procedure (it's stateless between requests, or checkpoints in-flight work elsewhere), schedule a restart during a low-traffic window, and separately file a code fix to add proper SIGCHLD handling so the zombies stop accumulating going forward. The restart alone clears the existing backlog (all zombies re-parent to init and get reaped); it does not, by itself, prevent recurrence.

Trade-offs and pitfalls

• Treating an orphan process as if it were a problem to "fix" is itself a mistake: orphans are a completely normal, self-resolving situation the kernel already handles by re-parenting, and they carry no accumulation risk the way unreaped zombies do.
• A large, unbounded zombie count can eventually exhaust the system's maximum PID count (/proc/sys/kernel/pid_max), at which point no new process at all can start, including the tools you'd want to use to diagnose the problem; this is why zombie accumulation deserves proactive alerting rather than being dismissed as "harmless" indefinitely.
• Restarting a parent to clear zombies is safe with respect to the zombies themselves (no live data involved) but is not automatically safe with respect to whatever else that parent process was doing; confirm the parent can tolerate a restart before treating this as a routine cleanup step.

Direct answer

Billions of small files exhausting inodes at a growing write rate is a capacity problem that keeps getting worse, so the immediate mitigation buys time (freeing inodes and slowing the growth) while the real fix changes the STORAGE MODEL itself, since no amount of periodic cleanup keeps pace with a write rate that keeps growing. I'd pack or archive existing small files into fewer, larger units immediately to reclaim inode headroom, then move the architecture toward batching writes into larger objects (or a filesystem or storage system that doesn't tie capacity to a fixed inode count the way ext4 does), validating the new write path on a subset of traffic with an easy rollback before cutting everything over.

Structured elaboration

Immediate mitigation, to reclaim inode headroom and slow the bleeding:

• Identify and archive or delete the oldest, least-accessed files first, the same "delete frees inodes, nothing else does" principle as any inode exhaustion, but at this scale the practical mechanism matters: packing many small files that must be retained into a single archive (a tar file, for instance) collapses millions of inodes into one, satisfying both "keep the data" and "free the inodes" at once, rather than forcing a choice between them.
• If write rate is actively growing and outpacing cleanup, consider a temporary write-side throttle or buffer (queueing new records rather than writing each one as an individual file immediately) purely to prevent the emergency from getting worse while the real fix is built, understanding this is a stopgap, not a solution.

Longer-term architectural fix, addressing why this keeps recurring:

• Batch small records into larger files: instead of one file per telemetry record, buffer records for a time window (per minute or per hour, depending on freshness needs) and write them as one larger file (a columnar format like Parquet, or even a simple concatenated batch), which is often the single highest-leverage fix, since it directly attacks the file-count growth rate rather than just cleaning up after it.
• Move to object storage for this class of data: an S3-compatible object store has no inode concept at all in the traditional filesystem sense and is purpose-built for very large numbers of individually-addressable objects, removing this entire failure class rather than managing around it, at the cost of different latency and consistency characteristics than a local filesystem (network round-trip per access instead of local disk access, eventual consistency semantics depending on the specific store).
• Or, if a local filesystem is a hard requirement, choose one built for dynamic inode allocation rather than a fixed count set at format time (XFS allocates inodes as needed, growing with actual usage rather than being capped by a decision made once at mkfs time), which removes this specific failure mode without changing the application's write pattern at all, though it doesn't address the operational cost of managing billions of small files (backup time, find/directory traversal cost, and so on) the way batching does.

Validating the fix is safe before rolling it out: run the new write path (batching, or the object-store write path) alongside the existing one on a subset of traffic first (a percentage of incoming telemetry, or a single non-critical source), comparing data completeness and correctness against the old path over a representative time window (spanning normal daily/weekly traffic patterns, not just a quiet hour) before trusting it as the sole path. Confirm downstream consumers of this data (anything reading these files today) can actually read the new format or location before cutting them over, since a storage-format change is also a contract change with every reader.

Rollback plan if it isn't safe: keep the existing write path fully intact and simply route ALL new writes back to it (a feature flag on the write path, not a destructive migration), and don't delete or migrate historical data out of its original format until the new path has been validated in production for a meaningful stretch of time; a migration that deletes the old format as it goes has no rollback at all once it's underway.

Worked example

A telemetry system currently writes one file per record, generating roughly 50 million new files per day and growing 5% month over month as more sources are onboarded, on an ext4 filesystem nearing its fixed inode ceiling. Immediate mitigation: identify the oldest 30 days of records (well past any need for per-record access, since queries against this data are always aggregated by hour or day, never by individual record) and pack each day into a single compressed archive, collapsing roughly 1.5 billion individual small-file inodes into 30 archive files, immediately restoring substantial inode headroom. Longer-term fix: change the write path to buffer records in memory and flush one Parquet file per source per hour instead of one file per record, cutting file creation from 50 million per day to roughly 480 per day (20 sources times 24 hours) for the same data volume, a reduction of about five orders of magnitude in file count for equivalent information content. This new hourly-batch write path runs in parallel with the old per-record path for two sources for one week, with a validation job confirming every record present in the old path's output also appears correctly in the new path's Parquet files for that same window, before the remaining sources are cut over; the old per-record write path is disabled only after every source has been validated and the downstream query layer confirmed working against the new format, with the old path's code left in place, dormant, as the rollback option for that transition period.

Trade-offs & pitfalls

• Packing existing files into archives trades immediate inode relief for a different access cost: reading one record out of an archive now requires extracting or seeking within it rather than opening it directly, which is fine for cold, rarely-accessed historical data and a poor fit if that data still needs frequent per-record access.
• Switching to object storage removes the inode ceiling entirely but changes the access latency and consistency model; validate that downstream consumers can tolerate that before treating it as a drop-in replacement.
• A migration that deletes source-format data as it migrates has no rollback path if a problem is discovered partway through; always validate the new path fully before removing the option to fall back to the old one, not simultaneously with the cutover.

Direct answer

Recommendation: one systemd slice per tenant (tenant-<id>.slice), with each tenant's services
placed Slice=tenant-<id>.slice and resource limits (CPUQuota=, MemoryMax=, IOWeight=) set
on the slice rather than per-service, plus per-service PrivateTmp=yes,
NetworkNamespacePath=/PrivateNetwork=yes, and ProtectSystem=strict for namespace-level
isolation. This gets meaningful resource fairness and blast-radius containment from mechanisms
already built into systemd, without standing up a full container runtime, at the cost of weaker
isolation than actual containers or VMs for a genuinely adversarial multi-tenant workload.

Structured elaboration

cgroups: resource guarantees

A cgroup (control group) is the kernel mechanism that groups processes and enforces resource
limits and accounting on the group as a whole; systemd creates and manages one automatically per
unit and per slice. Nesting each tenant's services under their own slice means CPUQuota=,
MemoryMax=, and IOWeight= set on the slice apply to the tenant's combined workload, not just
one service, which is what actually prevents one tenant's burst from starving another's, the
"noisy neighbor" problem this design exists to solve.

Namespaces: isolation boundary

A namespace is the kernel mechanism that gives a process its own view of some global resource
(process IDs, network interfaces, mounts, hostname) instead of sharing the host's. Per tenant
service:

• PrivateTmp=yes: separate /tmp and /var/tmp per service, so tenants cannot see or collide
  on temp files.
• ProtectSystem=strict / ProtectHome=yes: mount namespace changes making most of the
  filesystem read-only to the service, sharply reducing what a compromised tenant process can
  tamper with.
• PrivateNetwork=yes (own network namespace, no interfaces except loopback unless explicitly
  configured) or NetworkNamespacePath= (share a specific pre-created namespace) if tenants need
  real network isolation from each other, not just from the host.
• DynamicUser=yes (a fresh, unprivileged UID/GID allocated per service invocation) avoids the
  operational overhead of provisioning a real user account per tenant while still ensuring no two
  tenants' services share a UID.
• NoNewPrivileges=yes: sets the kernel's no_new_privs bit on the service's process, so it can
  never gain more privileges than it started with for the rest of its life, no setuid/setgid
  binaries, no file capabilities, no escalating via sudo. It costs nothing for a service that
  never legitimately needs to escalate, and closes off a whole class of privilege-escalation exploit
  if the service is ever compromised.

graph TD
    A[system.slice] --> B[tenant-a.slice]
    A --> C[tenant-b.slice]
    B --> B1[tenant-a-app.service]
    B --> B2[tenant-a-worker.service]
    C --> C1[tenant-b-app.service]
    B1 -. private mount namespace + PrivateTmp .-> B1
    B1 -. private net namespace via NetworkNamespacePath .-> B1
    B --> BLIM[CPUQuota / MemoryMax / IOWeight on tenant-a.slice]
    C --> CLIM[CPUQuota / MemoryMax / IOWeight on tenant-b.slice]

Worked example

Directives verified as syntactically valid against a live systemd 255 via systemd-analyze verify:

# tenant-a.slice
[Slice]
CPUQuota=200%
MemoryMax=4G
IOWeight=100

# tenant-a-app.service
[Service]
Slice=tenant-a.slice
DynamicUser=yes
PrivateTmp=yes
ProtectSystem=strict
NoNewPrivileges=yes
ExecStart=/opt/app/bin/worker

Every tenant-a service placed under Slice=tenant-a.slice is accounted and limited as one pool:
if tenant-a-app.service and tenant-a-worker.service together try to exceed CPUQuota=200%,
the kernel throttles the combined group, not just one of the two services individually, which is
the actual guarantee a tenant is being sold ("your workloads collectively get up to 2 cores," not
"each of your processes independently gets 2 cores").

Trade-offs and pitfalls

Resource guarantees: slice-level limits give fairness against other tenants but not
isolation from noisy processes within the same tenant's own slice; a tenant with two services
sharing one slice's quota can still starve each other, size limits at whichever granularity
actually needs the guarantee.

Security: namespaces plus DynamicUser/ProtectSystem meaningfully raise the bar against a
compromised tenant process reading another tenant's files or the host's, but they share the same
kernel; a kernel-level exploit (a namespace escape, a cgroup-accounting bug) affects every tenant
on the host simultaneously. This is a materially weaker boundary than separate VMs, appropriate
for cooperating internal tenants or a trusted-but-imperfect multi-tenant SaaS control plane, not
for running arbitrary untrusted third-party code.

Operational complexity: systemd slices and per-service hardening directives are configuration,
not new infrastructure, they are the cheapest option to operate (no container runtime, no
orchestrator) and the easiest to reason about with existing systemctl/journalctl tooling. That
simplicity is the actual trade-off being made against the stronger, more complex isolation a
container runtime or firecracker-style microVM per tenant would provide.

Recommendation, stated as a choice: for internal or semi-trusted tenants where operational
simplicity matters and workloads are already systemd-managed services, slices plus namespace
hardening is the right default; move to real containers or VMs the moment "adversarial tenant" or
"regulatory isolation requirement" enters the picture, since no amount of systemd configuration
changes that they share one kernel.

Direct answer

"Unable to find root" dropping to an initramfs shell almost always means the boot loader found and loaded the kernel fine, but the early boot environment cannot locate or mount the real root filesystem, most often because the initramfs (the temporary root filesystem loaded into memory that contains just enough drivers and tools to find and mount the real one) is missing the driver for whatever changed, or /etc/fstab/the kernel's root parameter is pointing at a UUID or device that no longer matches after the kernel update. The fix is always the same shape: boot rescue media, mount the real filesystems, chroot into them so you are running the installed system's own tools against its own files, then regenerate whatever is stale (initramfs, GRUB config, or occasionally the boot loader itself) before rebooting.

Step by step recovery

1. Boot a rescue or live USB matching the installed distribution's family (a Debian live image to rescue a Debian/Ubuntu system, a Rocky/RHEL rescue image for that family), since the chroot step later needs compatible tools and library versions.
2. Identify the real root partition from the rescue environment: lsblk -f shows every partition with its filesystem type and label; the previously failing UUID (visible in the initramfs error message, or in the rescue system's own view of /etc/fstab on the mounted partition) should match one of them.
3. Mount the real filesystem tree under a temporary mountpoint: mount /dev/sda2 /mnt, then mount anything that was a separate partition (/boot is commonly separate: mount /dev/sda1 /mnt/boot).
4. Bind mount the virtual kernel filesystems the chroot needs to behave like a real running system, not just a static file tree: for d in dev proc sys run; do mount --bind /$d /mnt/$d; done.
5. chroot into the installed system: chroot /mnt /bin/bash. From this point every command runs using the installed system's own binaries and libraries, which matters because a rescue environment's grub2-install or mkinitrd can be a different version than what the installed system expects.
6. Diagnose and fix the actual cause, which is usually one of:
   • A stale or missing initramfs for the newly installed kernel, most often because a driver needed to find the root device (an NVMe, RAID, or LVM driver, where LVM, Logical Volume Manager, is a layer that lets storage be resized, snapshotted, and organized into logical volumes independently of the underlying physical partitions) is not included in it. Regenerate it: update-initramfs -u -k all on Debian/Ubuntu, or dracut -f --regenerate-all on RHEL/Fedora, which rebuilds the initramfs for every installed kernel using the currently installed set of modules and rules.
   • A /etc/fstab entry (or the kernel command line's root= parameter in GRUB's config) referencing a UUID that no longer exists, for example after a disk was reformatted or an LVM volume renamed. Fix /etc/fstab directly with the correct current UUID from blkid, and if the kernel command line itself is wrong, that lives in GRUB's generated config, fixed by regenerating it (next step) after correcting whatever GRUB reads to build the root= line, typically /etc/default/grub's GRUB_CMDLINE_LINUX or the installed kernel's own defaults.
   • Occasionally the boot loader's own installation is damaged (rare after a kernel update alone, more common after a disk swap or partition table change): reinstall it with grub-install /dev/sda (Debian/Ubuntu) or grub2-install /dev/sda (RHEL/Fedora), pointed at the whole disk device, not a partition.
7. Regenerate the boot loader's configuration so it actually reflects any fix above: update-grub (Debian/Ubuntu, which wraps grub-mkconfig -o /boot/grub/grub.cfg) or grub2-mkconfig -o /boot/grub2/grub.cfg (RHEL/Fedora BIOS; /boot/efi/EFI/<distro>/grub.cfg on UEFI systems, so confirm which boot mode is in use with [ -d /sys/firmware/efi ] before targeting the config path).
8. Exit the chroot, unmount everything in reverse order, and reboot: exit, then umount -R /mnt, then reboot without the rescue media attached.

Worked example (representative command sequence)

# from the rescue/live environment
lsblk -f
mount /dev/sda2 /mnt
mount /dev/sda1 /mnt/boot
for d in dev proc sys run; do mount --bind /$d /mnt/$d; done
chroot /mnt /bin/bash

# inside the chroot, Debian/Ubuntu family
update-initramfs -u -k all
update-grub

# inside the chroot, RHEL/Fedora family, if that is the installed distro instead
dracut -f --regenerate-all
grub2-mkconfig -o /boot/grub2/grub.cfg

exit
umount -R /mnt
reboot

This procedure is a well established, standard recovery sequence rather than something to execute end to end in a sandboxed container: a real boot loader install and a real initramfs regeneration require an actual bootable disk and firmware environment (BIOS or UEFI), neither of which exists inside this environment's containers, so nothing here claims to have been run and captured as live output; every command above is the exact, current syntax for each package (verified as the currently documented options for update-initramfs, dracut, grub-install/grub2-install, and update-grub/grub2-mkconfig), not folklore.

Trade-offs and pitfalls

The most common mistake is regenerating GRUB's config without first fixing the underlying cause (a missing driver in initramfs, or a stale UUID): update-grub/grub2-mkconfig only rewrites the boot menu from whatever is currently true about the system, it does not fix a wrong fstab entry or rebuild a driver-incomplete initramfs on its own. A second mistake is running grub-install against a partition (/dev/sda1) instead of the whole disk device (/dev/sda) on a BIOS/MBR system, which installs the boot loader in the wrong location entirely. Third, forgetting the bind mounts for /dev, /proc, /sys before chrooting means tools like dracut or grub2-mkconfig inside the chroot cannot see real devices or the running kernel's state, and will either fail outright or silently generate an incorrect configuration based on an incomplete view of the system.

Direct answer

On a modern systemd-based Linux distribution the network stack itself is normally managed by one
of two systemd-adjacent tools: Netplan (Ubuntu, rendering to systemd-networkd or
NetworkManager) or NetworkManager directly via nmcli (RHEL/Fedora/most other current
distributions). Both persist a static IP and a route as a declarative config file that survives
reboot; neither uses the legacy ifconfig/route add commands, which do not persist at all.

Structured elaboration

Ubuntu: Netplan

Write a YAML file under /etc/netplan/ (mode 600, root-only, since it can contain credentials
for other network types):

# /etc/netplan/01-static.yaml
network:
  version: 2
  renderer: networkd
  ethernets:
    eth0:
      dhcp4: false
      addresses:
        - 192.168.1.50/24
      routes:
        - to: default
          via: 192.168.1.1
      nameservers:
        addresses: [1.1.1.1, 8.8.8.8]

Apply with sudo netplan apply (or netplan try, which auto-rolls-back if you do not confirm
within a timeout, the safer choice over a remote SSH session where a bad config could lock you
out). Netplan is a generator: it translates this YAML into the actual backend config (a
systemd-networkd .network file, or NetworkManager keyfiles, depending on renderer:) and
that generated config, not the YAML directly, is what persists the setting across reboots.

RHEL 8+/current: NetworkManager via nmcli

nmcli connection add type ethernet con-name static-eth0 ifname eth0 \
  ipv4.method manual ipv4.addresses 192.168.1.50/24 \
  ipv4.gateway 192.168.1.1 ipv4.dns "1.1.1.1 8.8.8.8"
nmcli connection up static-eth0

This writes a persistent connection profile (a keyfile under
/etc/NetworkManager/system-connections/) that NetworkManager reactivates automatically on every
boot; there is no separate "make it permanent" step, creating the connection profile is the
persistence mechanism.

Verification, either path

• ip addr show <iface> confirms the address is actually bound.
• ip route show confirms the route table, in particular the default route via the gateway you
  set.
• resolvectl status (networkd) or nmcli device show <iface> (NetworkManager) confirms DNS
  servers took effect.
• journalctl -u systemd-networkd or journalctl -u NetworkManager for anything that failed to
  apply.
• Reboot (or at minimum, systemctl restart the relevant service) once and re-check the same
  three commands, since a config that "looks right" in the file but was never actually re-applied
  is a common false confidence trap.

Worked example

Both were verified for real. The Netplan YAML above was fed to netplan generate (the step
netplan apply runs internally before touching the live interface), which produced the exact
systemd-networkd unit it is documented to produce:

$ netplan generate && cat /run/systemd/network/10-netplan-eth0.network
[Match]
Name=eth0

[Network]
LinkLocalAddressing=ipv6
Address=192.168.1.50/24
DNS=1.1.1.1
DNS=8.8.8.8

[Route]
Destination=0.0.0.0/0
Gateway=192.168.1.1

The nmcli command above was run against a live NetworkManager instance and produced the
persisted keyfile exactly as documented:

$ cat /etc/NetworkManager/system-connections/static-eth0.nmconnection
[connection]
id=static-eth0
type=ethernet
interface-name=eth0

[ipv4]
address1=192.168.1.50/24
dns=1.1.1.1;8.8.8.8;
gateway=192.168.1.1
method=manual

One caveat worth calling out: this specific container's interface came up as plain eth0
(confirmed with ip -o link show), because Docker's bridge networking assigns veth interfaces
eth0 by convention and there is no PCI bus path for udev's predictable-naming rules to key off
inside a container network namespace. That does not generalize to real hardware or a VM: there,
with systemd-udev predictable network interface naming active (the default on virtually every
current distribution), the interface is almost never called eth0, it is something like
enp1s0, ens192, or enp0s3 depending on the bus and slot. Never hardcode eth0 in either
config on a real host; always confirm the actual name with ip -o link show first, since a
config written against a name the interface does not have will silently fail to apply to
anything.

Trade-offs and pitfalls

• Applying a bad static config over SSH can cut your own access; netplan try (auto-rollback on
  timeout) or bringing the change up on a console/out-of-band path first are the safe patterns,
  applying blind over the connection you are about to change is not.
• Mixing tools is a real footgun: if NetworkManager is active and you also hand-edit
  /etc/network/interfaces (the old Debian ifupdown format) or vice versa, the two can fight over
  the same interface. Check which manager actually owns the interface (nmcli device status,
  or networkctl status for networkd) before adding a second config source.
• A route added with the old ip route add ... command (no persistence file behind it) vanishes
  on reboot; this is the single most common "it worked, then broke after a routine reboot"
  incident on freshly built hosts, always encode the route in the same persistent config as the
  address, never as an ad hoc ip route add.
• DNS set via nameservers:/ipv4.dns only takes effect if nothing else (like a DHCP client on
  a different interface, or a stale /etc/resolv.conf not managed by resolvectl) is overriding
  it; resolvectl status showing the expected servers is the actual proof, not just the config
  file content.