Linux System Administration Questions

Direct answer

Count file descriptors (FDs, the small integers a process uses to refer to open files, sockets, and pipes)
over time via /proc/<pid>/fd, resolve what each one actually points at with readlink, and cross-check
names with lsof; a leak is a specific target that keeps reappearing without ever closing, not just a
rising raw count.

Command-line recipe

Detect growth over time: a simple watch loop is often enough for a single host:

watch -n5 'ls /proc/<pid>/fd | wc -l'

For a trend you can graph rather than eyeball, append timestamped rows instead:

while true; do
  echo "$(date +%s),$(ls /proc/<pid>/fd 2>/dev/null | wc -l)"
  sleep 5
done >> fd-count.csv

Correlate FDs to file paths or sockets: raw counts alone don't tell you what's leaking, resolve the
targets and rank by how often each one repeats:

for fd in /proc/<pid>/fd/*; do readlink "$fd"; done | sort | uniq -c | sort -rn | head

Many identical entries (the same file path, or many socket:[nnnn] entries with no corresponding closes)
is the actual leak signal. lsof -p <pid> gives the same information in a more readable form, including the
type (regular file, socket, pipe) and, for sockets, the remote peer.
Aggregate results across a fleet: run the same one-liner via your existing fan-out mechanism (an SSH
loop over hosts, or your configuration-management tool's ad-hoc exec) and collect hostname,pid,fd_count
rows into one place, so you can rank hosts by growth RATE rather than manually checking one host at a time.
An early, cheaper warning sign worth watching independently of the count itself is /proc/<pid>/limits's
"Max open files" row: a slow leak typically first shows up in application logs as EMFILE (too many open
files) errors right around when the FD count approaches that limit.

Worked example

A reproducible demonstration: a process opens 30 files over about a second and a half without closing any
of them, standing in for a real leak.

import time
fds = []
for i in range(30):
    fds.append(open(f"/tmp/leak_{i}.txt", "w"))
    time.sleep(0.05)
time.sleep(5)

Sampling the FD count twice, two seconds apart, while this runs:

--- snapshot 1 ---
22
--- snapshot 2 (after 2s) ---
33

(the exact counts will vary run to run by a file or two depending on scheduling, the upward trend across
samples is the reproducible signal, not the precise numbers). Resolving the targets shows the leak clearly:

/proc/<pid>/fd/5 -> /tmp/leak_2.txt
/proc/<pid>/fd/6 -> /tmp/leak_3.txt
...

and lsof -p <pid> cross-checks the same file handles by name, confirming they are ordinary open files that
were never closed, not, for example, sockets still legitimately in use.

Trade-offs and pitfalls

A raw FD count can't distinguish a genuine leak from a busy process that legitimately holds many
connections open at once, such as a connection-pooling server; the readlink-and-aggregate step is what
turns "the count is high" into "these 400 descriptors are all sockets to the same peer that should plausibly
have been closed by now", the actual actionable signal. Also, /proc/<pid>/fd only shows a complete picture
when you're reading it as root or as the process's own owning user, a monitoring agent running under a
different, unprivileged account will silently undercount, which is easy to miss when you're comparing
numbers across a fleet running under different service accounts.

Direct answer

find /var/logs/app -maxdepth 1 -type f -mtime -1 -print0 | xargs -0 -I{} mv {} /tmp/recent/ does this safely in one pipeline: find ... -print0 emits each matching filename terminated by a NUL byte instead of a newline, and xargs -0 reads that same NUL-delimited stream, so a filename containing a space or even an embedded newline is never misread as two separate arguments, which is exactly the failure mode a plain find ... | xargs (newline delimited) has.

Why each part is there

• -maxdepth 1: restricts the search to files directly inside /var/logs/app, not subdirectories, matching "all regular files in /var/logs/app."
• -type f: only regular files, not directories or symlinks that happen to also match the time window.
• -mtime -1: modified within the last 24 hours. find's -mtime counts in whole 24 hour periods from now; -mtime -1 means "modification time is less than 1 day ago," which is the standard, correct way to express "within the last 24 hours," as opposed to -mtime 1 (exactly between 24 and 48 hours ago) or -mtime +1 (more than 48 hours ago), which are easy to confuse with the intended meaning if you have not internalized the off by one convention here.
• -print0: the safety critical flag. Ordinary find output (or -print, its default) separates results with a newline, which is indistinguishable from a filename that legitimately contains a newline, and a plain space in a filename already breaks a naive for f in $(find ...) loop by splitting one filename into two words. -print0 uses the NUL byte instead, which cannot appear inside a POSIX filename, making it an unambiguous delimiter no matter what characters the filename itself contains.
• xargs -0: the matching consumer for NUL delimited input; without it, xargs would still assume newline/whitespace splitting and the safety -print0 provides on the producing side would be thrown away on the consuming side.
• -I{}: substitutes each individual filename into the {} placeholder in the mv {} /tmp/recent/ command, run once per file, which is what you want here since you are moving files individually into a common destination directory rather than passing them all as one argument list to a single invocation.

Worked example (executed, including the actual hostile filenames this pattern is meant to survive)

Created files with a space and with an embedded literal newline in the name specifically to prove the safety claim rather than assert it:

cd /var/logs/app
touch "app one.log" "app two.log" $'weird\nname.log'
touch -d "3 days ago" old.log
find /var/logs/app -maxdepth 1 -type f -mtime -1 -print0 | xargs -0 -I{} mv {} /tmp/recent/

Real captured result, listing what actually ended up in the destination:

-rw-r--r-- 1 root root 0 ... app one.log
-rw-r--r-- 1 root root 0 ... app two.log
-rw-r--r-- 1 root root 0 ... weird
name.log

All three files, including the one whose name contains a literal newline, moved correctly and intact as single files, and the fourth file (old.log, backdated three days) was correctly left behind since it fell outside the -mtime -1 window. The command handled exactly the two hostile cases (embedded space, embedded newline) that a naive unquoted find | xargs pipeline would have split incorrectly.

Trade-offs and pitfalls

The single most common mistake is writing find /var/logs/app -mtime -1 | xargs mv -t /tmp/recent/ without -print0/-0: it looks identical for ordinary filenames and silently breaks the moment a filename contains a space, which is common enough in real log directories (application names, dates, or hostnames embedded in filenames) that this is not a hypothetical edge case. A second, subtler issue: -I{} runs mv once per file, which is correct here but slower than batching for a very large number of files; xargs -0 -t mv -- {} /tmp/recent/ versus a batched form (xargs -0 mv -t /tmp/recent/ without -I, letting xargs pack multiple filenames into fewer mv invocations) is worth knowing as an alternative when moving thousands of files, though the batched form only works because GNU mv -t accepts multiple sources, which not every command does. Finally, always sanity check with a dry run before deleting or moving anything destructively: swap mv for echo first (... | xargs -0 -I{} echo mv {} /tmp/recent/) to see exactly what would run.

Where to look first, and why

When investigating an issue on a Linux server, the first log to check is whichever one is closest to the layer where the symptom is showing up, because logs under /var/log are organized by subsystem, not chronologically merged, and reading the wrong one wastes time. A practical default order: check the application's own log first (it usually has the most specific, actionable detail), then the system/kernel-level logs if the application log shows nothing unusual or the symptom looks infrastructure-related (a crash, an OOM kill, a hardware issue), then authentication logs if the issue involves access or a login failure.

Three (and a few more) log files under /var/log, and what each contains

• /var/log/syslog (Debian/Ubuntu) or /var/log/messages (RHEL/CentOS-family): the general-purpose system log, aggregating messages from the kernel and most system daemons that haven't been routed elsewhere. This is the right first stop for "something went wrong and I don't yet know which subsystem," since it's the broadest catch-all.
• /var/log/auth.log (Debian/Ubuntu) or /var/log/secure (RHEL/CentOS-family): authentication and authorization events: SSH logins (successful and failed), sudo invocations, su, and PAM module activity. This is the first place to look for a suspected brute-force attempt, an unexpected login, or a permission-related failure.
• /var/log/kern.log: kernel-ring-buffer messages specifically (a subset of what also appears in syslog on many distributions), including hardware errors, driver messages, and out-of-memory (OOM) killer activity. Relevant when a process died with no application-level explanation at all, since the OOM killer terminates processes silently from the application's own point of view.
• /var/log/<application-name>/: most non-trivial services (web servers, databases, container runtimes) write their own dedicated logs under a subdirectory here (e.g. /var/log/nginx/error.log), which is usually far more specific and actionable than anything in the generic system logs.

Worked example

A web request is returning HTTP 502 errors. The investigation order: (1) /var/log/nginx/error.log (or the equivalent for whatever reverse proxy is in front), since a 502 specifically means the proxy couldn't get a valid response from the upstream, and the proxy's own log usually names which upstream and why (connection refused, timeout); (2) if the upstream application itself crashed, its own application log (or journalctl -u <service> on a systemd host, where systemd is the init system and service manager most modern Linux distributions use to start and supervise processes, and journalctl reads the systemd journal, its structured log store) for a stack trace or crash reason; (3) /var/log/syslog or /var/log/kern.log only if neither of the above shows anything, to check whether the process was killed by something outside its own control, such as the OOM killer.

Trade-offs and pitfalls

• On any modern systemd-based distribution, some of these plain-text files may be thin or entirely absent by default, because journald (systemd's own logging system) is the primary sink and only forwards to /var/log/syslog-style files if rsyslog or a similar bridge is installed and configured; don't assume /var/log/syslog exists or is current on every host without checking.
• Log rotation (via logrotate) means the file you're looking at may only cover a recent window; older entries live in numbered or compressed siblings (syslog.1, syslog.2.gz), and zgrep/zcat are needed to search the compressed ones without decompressing them by hand first.
• Different distributions use different filenames for conceptually the same log (auth.log vs secure, syslog vs messages); scripts and runbooks written against one distribution family will silently find nothing useful (not an error, just an empty or missing file) if pointed at the other family without adjustment.

Direct answer

Layer LUKS (Linux Unified Key Setup, the standard on-disk format for the kernel's dm-crypt block encryption)
directly on the raw partition or disk, and put LVM (Logical Volume Manager, the layer that lets you resize,
snapshot, and combine block devices without repartitioning) on top of the single decrypted mapping, unless
different logical volumes genuinely need different keys or separate access boundaries, in which case put
LVM below and encrypt each logical volume individually and accept multiple unlock prompts. Resizing,
rekeying, and recovery all follow directly from getting that layering right.

How the layers interact

• LUKS-under-LVM (recommended default): one LUKS container wraps the raw partition; LVM's physical
  volume sits on the single decrypted /dev/mapper/<name> device; one passphrase or key unlocks everything
  underneath. Simpler operations, one thing to unlock at boot.
• LVM-under-LUKS: LVM creates logical volumes directly on the raw partition, and each logical volume is
  individually wrapped in its own LUKS container. More keys to manage and back up, but supports genuinely
  separate access boundaries per volume (for example, different tenants or different sensitivity levels on
  the same physical disk).

Resizing an encrypted LVM volume, bottom to top

The sequence matters strictly bottom to top, each layer only sees the space the layer below reports:

1. Grow the underlying block device itself (a cloud provider's disk-resize API, or a physical disk swap).
2. Grow the partition if one is in play (growpart <device> <partition-number>).
3. cryptsetup resize <mapped-name> to grow the LUKS mapping into the newly available space. Both header
   formats support doing this online, while the mapping stays active and mounted, no luksClose/luksOpen
   cycle is required for either one. The real difference is elsewhere: LUKS1's header never stores a size
   at all, so resizing it is a pure device-mapper table operation that needs no passphrase, while LUKS2
   persists the new size into its own authenticated metadata, so cryptsetup resize on a LUKS2 device
   requires a passphrase to write that update even though the mapping itself is never torn down. (Verified
   directly: growing an active, opened LUKS1 mapping with cryptsetup resize succeeded immediately with
   no passphrase prompt and no close/reopen; the equivalent LUKS2 resize on an active mapping needed a
   passphrase before it would proceed, but neither one required closing the mapping.)
4. pvresize on the mapped device, so LVM's physical volume sees the larger space.
5. lvextend -l +100%FREE <volume-group>/<logical-volume> to grow the logical volume into the newly
   available extents.
6. resize2fs (ext4) or xfs_growfs (XFS) to grow the filesystem to fill the logical volume.
   Doing pvresize before cryptsetup resize is a common, cleanly diagnosable mistake: LVM will report zero
   new physical extents available, because the crypt mapping is still reporting its old size to everything
   above it.

Rekeying with minimal downtime

cryptsetup luksAddKey <device> adds a new passphrase into a free keyslot (LUKS2 supports up to 32
keyslots depending on header parameters, LUKS1 exactly 8) alongside the existing one, without touching the
actual data encryption key at all, only the passphrase-wrapped copy of it changes. Verify the new
passphrase unlocks the device, then cryptsetup luksRemoveKey (or luksKillSlot) the old one. Because the
underlying master key never changes, this completes in effectively constant time regardless of volume size,
with zero downtime since the device stays mounted throughout. Contrast this with cryptsetup reencrypt
(LUKS2's online re-encryption, which genuinely rewrites the master key and re-encrypts every block), a much
heavier operation that LUKS2 supports resuming from a checkpoint if interrupted, precisely because a crash
mid-reencrypt without that resume metadata would be dangerous.

Emergency recovery from a corrupted LUKS header

The header holds every keyslot's metadata; if it's destroyed, the master key may be unreachable through any
passphrase, by design, that unrecoverability is the point of full-disk encryption, not a defect. The only
reliable recovery path is a header backup taken proactively: cryptsetup luksHeaderBackup --header-backup-file <file> <device> immediately after luksFormat and after every luksAddKey/
luksKillSlot change. If a backup exists, cryptsetup luksHeaderRestore --header-backup-file <file> <device> restores it and normal unlock resumes. Without a backup, cryptsetup repair can attempt limited
recovery of a LUKS1 header, and cryptsetup luksDump shows whatever keyslot metadata survives; if even one
valid keyslot remains, the data is recoverable, if none do, it genuinely is not.

Worked example

A database host runs LUKS directly on a partition with LVM on top, needing to grow from 500GiB (gibibyte,
2^30 bytes) to 750GiB after the cloud disk itself was resized: growpart /dev/nvme1n1 1, then
cryptsetup resize luks_data, then pvresize /dev/mapper/luks_data, then lvextend -l +100%FREE /dev/vgdata/lvdata, then resize2fs /dev/vgdata/lvdata. Skipping the cryptsetup resize step and running
pvresize directly after growpart is the concrete, checkable symptom of doing this out of order: pvresize
reports the physical volume is unchanged in size, because the crypt mapping in between still reports the old
500GiB.

Trade-offs and pitfalls

LUKS-under-LVM's single unlock is operationally simpler (one boot-time passphrase, or a network-bound
unlock via a TPM, trusted platform module, backed mechanism) but means anyone holding that one key can see
every logical volume underneath; LVM-under-LUKS supports per-volume keys at the cost of managing and backing
up several header backups instead of one. The most common mistake in "rotation" is running luksAddKey and
stopping there: an old, potentially compromised passphrase left in its keyslot after adding a new one has
not actually been rotated, only supplemented, luksRemoveKey on the old slot is the step that completes it.

Direct answer

Partition the new device, format the partition (not the raw disk device) with a filesystem, mount it, set ownership, then make the mount survive a reboot by referencing the partition's UUID (universally unique identifier, a stable label baked into the filesystem itself) in /etc/fstab rather than the device name, because /dev/sdb can become /dev/sdc on the next boot depending on device enumeration order, while the UUID never changes.

Step by step

1. Confirm the device is what you think it is before touching it: lsblk and sudo fdisk -l /dev/sdb, so you do not partition the wrong disk.
2. Create a partition table and a single partition: parted -s /dev/sdb mklabel gpt mkpart primary ext4 1MiB 100%. GPT (GUID Partition Table) is the current default over the legacy MBR (Master Boot Record) scheme; starting at 1MiB rather than sector 0 leaves room for GPT's own metadata and keeps the partition aligned to modern 4K-sector and SSD erase-block boundaries, avoiding a read-modify-write performance penalty from misalignment.
3. Let the kernel see the new partition: partprobe /dev/sdb (re-reads the partition table without a reboot); the new node appears as /dev/sdb1.
4. Format it: mkfs.ext4 /dev/sdb1. Format the partition, never the whole disk device (/dev/sdb), or you lose the partition table structure and get a filesystem with no partition boundary at all, which breaks the moment you want a second partition later.
5. Mount it: mkdir -p /data; mount /dev/sdb1 /data.
6. Set ownership: chown app:app /data (after first confirming both the app user and app group already exist; create them with useradd/groupadd if not).
7. Make it persistent, referencing the UUID rather than the device name: get the UUID with blkid -s UUID -o value /dev/sdb1, then add a line to /etc/fstab: UUID=<uuid> /data ext4 defaults 0 2. The last two fields are the dump flag (0, skip in dump backups) and the fsck pass order (2, checked after the root filesystem which is always 1).
8. Verify before trusting it: mount -a (re-reads fstab and mounts anything not already mounted, the safest way to test a new fstab line without rebooting, since a broken entry here can otherwise leave the system unable to boot cleanly), then mount | grep /data, df -h /data, and ls -ld /data to confirm ownership landed correctly.

Worked example (executed end to end on a loopback block device standing in for /dev/sdb)

LOOPDEV=$(losetup --find --show /tmp/fakedisk.img)   # atomic find-and-attach, avoids a race with any other process also allocating a loop device
parted -s "$LOOPDEV" mklabel gpt mkpart primary ext4 1MiB 100%
kpartx -av "$LOOPDEV"        # creates the partition device node; a real /dev/sdb would get /dev/sdb1 directly via the kernel + udev, this loopback sandbox needs kpartx to do the equivalent
PART="/dev/mapper/$(basename $LOOPDEV)p1"
mkfs.ext4 -q -F "$PART"
mkdir -p /data
groupadd app; useradd -M -N -g app app
mount "$PART" /data
chown app:app /data
UUID=$(blkid -s UUID -o value "$PART")
echo "UUID=$UUID  /data  ext4  defaults  0  2"

Real captured output confirmed the whole chain worked:

fstab entry: UUID=bb30b273-9fc3-41ba-8cbb-55b00328018e  /data  ext4  defaults  0  2
/dev/mapper/loop4p1 on /data type ext4 (rw,relatime)
Filesystem           Size  Used Avail Use% Mounted on
/dev/mapper/loop4p1   55M   24K   51M   1% /data

kpartx (which builds device-mapper nodes for partitions instead of relying on the kernel to auto create /dev/sdb1 style nodes) is a sandbox specific substitute here because this environment has no udev running; on a real VM or bare metal host, partprobe plus the kernel's own partition scanning gives you /dev/sdb1 directly with no extra tool needed, but every other command (parted, mkfs.ext4, mount, chown, blkid) is exactly what you would run against a real disk.

An alternate worked example: numeric UID/GID and extra mount options

The base example above owns the mount with a named user and group and uses defaults for mount options; a common real variant swaps both:

mount -o noatime,nodev "$PART" /data
chown 2000:2000 /data
echo "UUID=$UUID  /data  ext4  noatime,nodev  0  2"

chown 2000:2000 sets ownership by numeric UID/GID directly rather than by name, useful when /data needs to be owned by an identity that does not (yet, or ever) have a corresponding /etc/passwd entry on this particular host, for example a UID that only has meaning inside a container image mounted from this path, or a fleet where the owning account is provisioned slightly out of order relative to the disk. noatime skips updating each file's last-accessed timestamp on every read, a small but real reduction in write traffic for a data volume where nothing actually consults access time; nodev refuses to honor any device special files on this filesystem, a defense in depth measure appropriate for a data volume that should only ever hold regular application files, never a crafted device node smuggled in through some other channel. Both mount options belong in the fstab line itself, not just the interactive mount invocation, or they silently revert to defaults on the next boot.

Trade-offs and pitfalls

The most common real mistake is formatting /dev/sdb instead of /dev/sdb1: it appears to work (the raw disk device can hold a filesystem directly) but throws away the partition table, so there is no room to add a second partition later without destroying the first, and some tools behave inconsistently against a partition-less disk. Second, using the device name (/dev/sdb1) instead of the UUID in fstab works fine until a second disk is added or removed and Linux's device enumeration order shifts, silently mounting the wrong disk at /data on the next boot with no error at all, just wrong data where you expected a working application. Third, always test a new fstab line with mount -a before rebooting: a syntax error in fstab can otherwise leave the system stuck at an emergency shell on the next boot, which is a much worse time to discover the typo.