Linux System Administration Questions

Direct answer

create_secure_tmpdir() {
  local prefix="${1:-app}"
  SECURE_TMPDIR="$(mktemp -d "${TMPDIR:-/tmp}/${prefix}.XXXXXXXX")"
  chmod 700 "$SECURE_TMPDIR"
}

cleanup() {
  local status=$?
  [[ -n "${SECURE_TMPDIR:-}" && -d "$SECURE_TMPDIR" ]] && rm -rf -- "$SECURE_TMPDIR"
  exit "$status"
}

main() {
  create_secure_tmpdir "myapp"
  trap cleanup EXIT

  local target="$SECURE_TMPDIR/a file with spaces.txt"
  printf 'hello world\n' > "$target"
}

main "$@"

mktemp -d creates the directory atomically with a random suffix (no race between "check if a
name is free" and "create it", which is what makes it secure: a predictable name in /tmp
that another user could pre-create or symlink is a classic local privilege-escalation vector).
chmod 700 then restricts it to the owner. The trap cleanup EXIT removes it whenever the
script exits, on success, on error, or on a signal, so a crash mid-script never leaves a
world-creatable temp directory lying around.

Structured elaboration

Why the function sets a global, not a local returned via command substitution

The tempting first draft calls mktemp, chmod, and trap all inside one function and then
captures its output with workdir="$(create_secure_tmpdir)". That is broken, and the breakage
is subtle enough to ship: $(...) runs the function in a subshell. The subshell exits the
instant the function returns (right after it prints the path), which fires the trap ... EXIT
that was set inside that subshell, deleting the directory before the parent script ever
receives the path back. The fix is to never route the path through command substitution:
create_secure_tmpdir sets a global variable directly, and trap is registered afterward in
the same shell that will eventually exit for real (the top-level script), not in a throwaway
subshell.

Handling spaces in filenames correctly

• Quote every expansion of the path: "$SECURE_TMPDIR", "$target". An unquoted $SECURE_TMPDIR
  undergoes word splitting and would be treated as multiple arguments if the prefix or TMPDIR
  contains spaces.
• If you later need to iterate over files inside the directory, never do for f in $(ls "$dir"); use find "$dir" -print0 | while IFS= read -r -d '' f; do ... done, which uses a
  NUL byte as the record separator so spaces (and even newlines) inside filenames cannot be
  mistaken for separators.
• Always terminate option lists with -- before a path argument (rm -rf -- "$dir") so a
  filename that happens to start with - is never parsed as a flag.

Permissions

700 (rwx------) means only the owning user can read, write, or traverse into the directory;
group and others get nothing. mktemp -d on Linux already creates directories 700 by default,
so the explicit chmod 700 here is defense against a non-default umask (the mask that strips
permission bits off every newly created file or directory before any explicit chmod runs;
e.g. a umask of 022 would otherwise leave a fresh directory group- and world-readable) on some other
distribution or in some other user's shell, and it documents the intent instead of relying on an
implicit default.

Worked example

Run against the corrected function on a live container (Ubuntu 24.04):

$ bash myscript.sh
$ ls /tmp
   (directory is already gone: cleanup fired on script exit)

Inside main, before exit, the directory and its file are confirmed present and correctly
permissioned:

$ stat -c '%a' "$SECURE_TMPDIR"
700
$ cat "$SECURE_TMPDIR/a file with spaces.txt"
hello world

By contrast, the broken "function sets its own trap and returns the path via $(...)" version,
run in the same environment, prints the path but the directory is already deleted by the time
the caller's next line runs, because the subshell's EXIT trap already fired.

Trade-offs and pitfalls

• mktemp without -d creates a file; for a directory you must pass -d, and forgetting it
  is a common copy-paste error when adapting a temp-file recipe.
• Do not rely on /tmp being cleaned automatically (some distributions run tmpfiles.d sweeps
  on a schedule, others do not); a script must clean up after itself rather than assuming the OS
  will.
• A trap ... EXIT set inside a function is scoped to whichever shell process actually runs
  that function. In a plain (non-subshell) function call this is the main script's shell, which
  is correct; the moment you wrap the call in $(...), backticks, a pipeline, or &
  (background), you have created a subshell and the trap silently stops protecting the resource
  the caller thinks it owns.
• If the script can be interrupted by a signal other than normal exit (Ctrl-C, kill), trap cleanup EXIT still fires in bash because EXIT is triggered on any script termination path,
  but if you also trap INT/TERM explicitly for other cleanup, make sure that handler does not
  itself exit without eventually reaching the shared cleanup logic, or the temp directory leaks.

What each file is for

• /etc/passwd: the basic user account database. One line per user, defining the mapping from username to numeric UID, primary GID, a description field, home directory, and login shell. Historically it also held the password hash; it no longer does on any modern distribution.
• /etc/shadow: the actual password data, split out of /etc/passwd decades ago specifically so password hashes are not world-readable. Holds the hash plus password-aging fields (last change date, minimum/maximum age, warning period, inactivity period, expiration date).
• /etc/group: the group database, mapping group names to numeric group IDs (GIDs) and listing supplementary (non-primary) members of each group.

Reading a sample line

/etc/passwd:

root:x:0:0:root:/root:/bin/bash

Colon-separated fields, in order: username (root), a placeholder (x, meaning "the real hash lives in /etc/shadow"), UID (0), primary GID (0), the GECOS/comment field (root), home directory (/root), login shell (/bin/bash).

/etc/shadow:

root:*:20664:0:99999:7:::

Fields, in order: username (root), password hash (* here, which means "this account cannot log in via password," as opposed to an empty field, which would mean "no password required"), date of last password change as days since 1 Jan 1970 (20664), minimum days between changes (0), maximum days before a change is required (99999, effectively "never"), warning period in days before expiration (7), then inactivity and expiration fields (empty here). Both examples were read directly from /etc/passwd and /etc/shadow on a running Ubuntu 24.04 container.

Required permissions, and why

-rw-r--r-- 1 root root   888 ... /etc/passwd
-rw-r----- 1 root shadow 502 ... /etc/shadow

(verified on the same container). /etc/passwd is world-readable (mode 644) because many ordinary tools and commands need to resolve a UID to a username (ls -l, ps, id) without needing any privilege, and it contains no secret since the hash was moved out decades ago. /etc/shadow is readable only by root and the shadow group (mode 640, owned by root:shadow) because it holds the actual password hashes: if it were world-readable, any local user could copy it and run an offline brute-force or dictionary attack against every account's hash with no rate limiting. Both files should never be world-writable; writable by anyone but root would let a local user grant themselves UID 0 or blank out another account's password.

Trade-offs and pitfalls

• Never hand-edit /etc/passwd, /etc/shadow, or /etc/group directly with a text editor while other processes might touch them concurrently; use vipw (for passwd) and vipw -s (for shadow), or higher-level tools like usermod/passwd, which take the appropriate lock file so a login attempt mid-edit can't read a half-written file.
• A password hash of ! or !! (as opposed to *) in /etc/shadow conventionally means "account is locked" (often produced by passwd -l), which looks similar to * but is a distinct convention some tooling checks for specifically; don't assume every non-hash value means the same thing.
• UID 0 is what makes an account root, not the username: any account with UID 0 in /etc/passwd, regardless of its name field, has full root privileges. Auditing for a second account with UID 0 (a classic backdoor technique) is a real check, not a theoretical one: awk -F: '$3 == 0 {print $1}' /etc/passwd should return only root on a healthy system.

Direct answer

copytruncate and a postrotate signal solve the same problem (the application must keep writing to a working log file after rotation, not to a now-renamed file descriptor it still has open) in two different ways, and the choice depends on whether the application can be told to reopen its log file on command. postrotate sending a reload signal is the better choice whenever the application supports it, because it has no window where log lines can be lost; copytruncate is the fallback for an application you cannot signal to reopen its files, and it accepts a small, real risk of losing a handful of lines written in the brief gap between the copy and the truncate.

The config

/var/myapp/log/*.log {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
    sharedscripts
    postrotate
        systemctl kill -s HUP myapp.service
    endscript
}

• daily, rotate 7: rotate once a day, keep 7 rotated generations, matching the requirement exactly.
• compress: gzip rotated logs to save space; delaycompress skips compressing the most recently rotated file for one extra cycle, so a log-shipping agent or a human still actively tailing the just-rotated file (app.log.1) gets one uncompressed generation before it disappears into app.log.2.gz.
• missingok: do not error if the log file happens to be absent on a given run (a quiet day, or the app not yet started).
• notifempty: skip rotating an empty file, which avoids burning through the 7 generation history on days with no log output at all.
• sharedscripts: run postrotate once for the whole glob (*.log), not once per matched file, which matters here since sending HUP to the service repeatedly for every log file it owns would be wasteful and could itself cause brief reload churn.
• postrotate ... endscript: after logrotate has renamed the old file and created a fresh, empty one in its place, tell the application to reopen its log file handle by sending it SIGHUP (a signal many daemons treat specifically as "reload config and reopen files," including most that use standard logging libraries), rather than continuing to write into the renamed, no-longer-referenced-by-name file.

copytruncate versus postrotate, and when to use which

• postrotate with a reload signal (shown above) is correct when the application either directly supports SIGHUP for log reopen (nginx, many syslog-style daemons) or exposes an equivalent reload hook (a systemd ExecReload= that does the same thing). It has no data loss window: the application closes its old file handle and opens a fresh one at a point in time it controls, cleanly.
• copytruncate is the right choice when the application cannot be told to reopen its log file at all (no signal handler for it, no reload hook, and you cannot modify it to add one). It works by copying the current log file's contents to the rotated name, then truncating the original file in place to zero length, so the application's already-open file descriptor keeps writing to the same inode (the on-disk record a filesystem uses to track a specific file's data and metadata, independent of the file's name), just now empty. The unavoidable trade-off: anything the application writes between the copy and the truncate is lost, since it was captured in neither the copy (which already happened) nor preserved in the truncated original (which is now empty). For a typical daily rotation this window is milliseconds, and the loss is at most a line or two, which is an acceptable trade for many applications but is a real, honest cost, not a hidden edge case to gloss over.

Centralizing to ELK/Graylog securely from the fleet

Rather than have logrotate or a cron job ship files itself, run a dedicated forwarding agent (Filebeat for an ELK/Elastic stack, or the Graylog sidecar / an rsyslog or Fluent Bit output for Graylog) that tails /var/myapp/log/*.log continuously and follows rotation correctly on its own (both Filebeat and Fluent Bit track files by inode, not by name, so a rotate-and-truncate or a rotate-and-rename is handled without missing or duplicating lines, provided the tool is configured for it, which is a real configuration detail worth testing rather than assuming). Secure the path with TLS between the forwarder and the ingest endpoint (Filebeat's ssl.certificate_authorities, or Graylog's TLS-enabled input) and authenticate the forwarder with a scoped credential or mutual TLS rather than an open, unauthenticated ingest port, since a log ingest pipeline reachable without authentication from inside the fleet is a route for a compromised host to inject fabricated log entries into your centralized logging, undermining its use for incident investigation.

Worked example (executed, config validated with logrotate's own debug mode)

mkdir -p /var/myapp/log
echo "sample log line" > /var/myapp/log/app.log
logrotate -d /etc/logrotate.d/myapp

Real captured output confirming the config parses and logrotate's own rotation-need evaluation runs against it correctly:

reading config file /etc/logrotate.d/myapp
Handling 1 logs
rotating pattern: /var/myapp/log/*.log  after 1 days (7 rotations)
empty log files are not rotated, old logs are removed
considering log /var/myapp/log/app.log
Creating new state
  Now: 2026-09-24 00:23
  Last rotated at 2026-09-24 00:00
  log does not need rotating (log has already been rotated)

-d is logrotate's debug/dry-run mode: it prints exactly what it would do without touching any file, which is the safe way to validate a new config before it runs for real under cron or the systemd timer that drives logrotate on most current distributions.

Trade-offs and pitfalls

The most common mistake is shipping copytruncate by default without checking whether the application actually supports a cleaner reload signal, accepting an avoidable small data loss window for no reason; always check for a reload hook first. A second mistake is forgetting sharedscripts on a glob pattern matching multiple files, which causes postrotate to fire once per matched file instead of once total, potentially reloading a service multiple times in quick succession for what should be a single rotation event. Third, whichever centralization agent is chosen, verify it actually survives rotation correctly in a test before trusting it in production: an agent that tracks files by name rather than inode will either lose the tail of a rotated file or start re-reading the new file from the beginning, both of which are silent failures that only surface later as a gap in the centralized logs during an actual incident.

The states and what each one means

ps and top report a process's state as a single letter (the STAT/S column), sometimes with a modifier appended (like + for foreground, < for high priority):

Transitions and how to observe them

A process moves R <-> S/D constantly as it alternates between running and blocking on something; ps/top only show a snapshot, so a process legitimately flickers through states faster than you can sample. T is entered only by an explicit stop signal (kill -STOP <pid>, or Ctrl-Z from a shell) and left only by SIGCONT (kill -CONT <pid>, or fg/bg). Z is entered automatically the instant a child process calls exit(), and left only when the parent calls wait() on it, which removes the zombie's entry entirely (it isn't "killed," since it's already not running any code: there's nothing left to kill).

ps -o pid,ppid,stat,cmd -p <pid>

Verified transition: with a sleep 100 process running, ps -o pid,ppid,stat,cmd initially reported STAT = S; after kill -STOP <pid>, the same command reported STAT = T; after kill -CONT <pid>, it returned to S. top shows the same single-letter code live in its S column, updating each refresh, which makes it better than repeated ps calls for watching a state change happen in real time rather than reconstructing it from separate snapshots.

When many processes sit in one state for a long time

• Many processes stuck in D for an extended period usually indicates the underlying storage (a disk, a network filesystem like NFS, or a hung block device) is not responding, not that the processes themselves are misbehaving; D-state processes cannot be killed with a normal signal (not even SIGKILL) precisely because they're inside an uninterruptible kernel wait, so the fix is usually at the storage layer, not the process layer.
• A large or growing number of Z (zombie) processes points at a parent process that isn't reaping its children (a bug in supervisor/init logic, commonly seen when a container's PID 1 (the very first process the kernel starts, conventionally responsible for reaping orphaned children) isn't a real init system). Zombies consume a process table entry each but no CPU or memory beyond that, so a handful is harmless, but thousands can exhaust the system's maximum PID count and block new processes from starting at all.
• Many processes stuck in T almost always means something sent them SIGSTOP and never followed up with SIGCONT, commonly a debugging session left attached, or a job control mistake (backgrounding after Ctrl-Z without bg).

Trade-offs and pitfalls

• A single D-state process briefly is completely normal (it's just doing disk I/O); the diagnostic signal is many processes stuck in D simultaneously for an extended period, which points at shared storage, not any one process.
• kill -9 (SIGKILL) has no effect on a D-state process, which is a frequent source of confusion during an incident: the process appears "unkillable" because it genuinely is, until the I/O it's blocked on either completes or the kernel gives up on it.
• Zombie cleanup is not done by killing the zombie (there's no running code to kill); the real fix is making the parent call wait(), either by fixing the parent's own bug or, if the parent itself is gone, by re-parenting to init/PID 1, which does reap it automatically.

How the two package-manager families work at a high level

Debian-family distributions (Debian, Ubuntu) use dpkg as the low-level package tool, which installs, removes, and tracks individual .deb files and their contents on disk, but does not resolve or fetch dependencies on its own. apt (Advanced Package Tool) sits above dpkg: it resolves dependency graphs, downloads packages (and their dependencies) from configured repositories, and calls dpkg to actually perform the install. Red Hat-family distributions (RHEL, Fedora, Rocky Linux, AlmaLinux) mirror the same split: rpm is the low-level tool that installs and tracks individual .rpm files, and dnf (the modern successor to yum) is the high-level tool that resolves dependencies and manages repositories on top of rpm. In both families, the low-level tool is what you'd use to inspect or manually manage an already-downloaded package file, and the high-level tool is what you use for everyday installs, updates, and repository management.

Adding and verifying repository GPG keys

Package repositories sign their package metadata, and the package manager verifies that signature against a trusted GPG (GNU Privacy Guard) public key before installing anything from that repository, which is what stops a compromised or spoofed mirror from serving tampered packages. On Debian/Ubuntu, the modern (post-apt-key-deprecation) pattern is to place the vendor's public key as a dearmored keyring file (dearmoring converts the key from its ASCII-armored, plain-text export format into the raw binary format the keyring file needs; gpg --dearmor is the command that does the conversion) and reference it explicitly from the repository definition:

curl -fsSL https://example.com/repo/gpg-key.asc | gpg --dearmor -o /etc/apt/keyrings/example.gpg
echo "deb [signed-by=/etc/apt/keyrings/example.gpg] https://example.com/repo stable main" \
  | sudo tee /etc/apt/sources.list.d/example.list
sudo apt update

signed-by scopes trust to exactly that keyring for exactly that repository, rather than trusting the key system-wide, which was the security weakness of the older, now-deprecated apt-key add approach. On RHEL-family systems, the equivalent is rpm --import <keyfile> (or a gpgkey= line in the repo's .repo file under /etc/yum.repos.d/), and dnf/yum verify signatures against the imported keyring automatically before installing.

Holding or pinning package versions

Debian/Ubuntu: apt-mark hold <package> prevents that specific package from being upgraded by any future apt upgrade/apt full-upgrade until explicitly unheld (apt-mark unhold <package>); APT pinning (/etc/apt/preferences.d/) offers finer control, letting you pin a specific version or prefer one repository's version over another's by priority. RHEL-family: dnf has a versionlock plugin (dnf versionlock add <package>-<version>) providing the equivalent effect, locking a package to its currently installed (or a specified) version across future dnf update runs.

Safe kernel/package upgrade strategy in production, including rollback

• Never let production hosts run unattended, un-reviewed major-version upgrades; use a staging/canary host (or a small percentage of the fleet) to apply the upgrade first and observe for a soak period before fleet-wide rollout.
• For the kernel specifically, most distributions keep the previous kernel installed alongside the new one by default; the bootloader (GRUB) retains a boot entry for it, so a bad kernel upgrade can often be rolled back by simply selecting the previous kernel at boot, without needing to reinstall or restore from backup, as long as the old kernel package wasn't also removed.
• Immutable or golden-image approaches (baking a new machine image, testing it, then replacing instances rather than upgrading in place) avoid the rollback problem entirely for compute that's disposable (stateless services, autoscaling groups): rollback becomes "redeploy the previous image" rather than "undo an in-place change."
• apt-mark hold/dnf versionlock on specific packages (a database engine at a validated version, for example) prevents a routine apt upgrade/dnf update from silently pulling in an untested major version alongside otherwise-wanted security patches.

Trade-offs and pitfalls

• Holding or pinning a package indefinitely trades short-term stability for long-term risk: a held package stops receiving security patches too, so a hold needs a corresponding follow-up plan (a scheduled review, a ticket) rather than being set once and forgotten.
• Removing the old kernel package too soon after an upgrade (to save disk space, for example) eliminates the GRUB rollback safety net; keeping at least one known-good previous kernel installed is a small disk cost for a meaningful reduction in blast radius from a bad kernel update.
• signed-by scoping in APT sources is a meaningful security improvement over the deprecated system-wide apt-key, but it only protects against a compromised repository; it does nothing against a legitimately-signed but vulnerable package version, which is a separate risk that staged rollout and vulnerability scanning address instead.