Shell Scripting and Automation Questions

Direct answer

The usual culprits are GNU tools on Linux versus BSD tools on macOS: sed -i, date -d, stat -c, head -n -N, base64 -w (older macOS), timeout, and many GNU-only long options such as --reference and --version. Handle them in this order of preference: avoid the feature by using a POSIX equivalent, feature-detect once and call a wrapper, or standardise the userland (install GNU tools on the Mac, or run the script in the same container image everywhere).

Differences that actually bite

I ran each of these on a Mac (BSD userland) and on Ubuntu (GNU coreutils).

"Userland" means the command-line tools an operating system ships (sed, date, stat, cp), as opposed to the kernel. Most Linux distributions ship the GNU versions of these tools (the GNU coreutils package holds ls, cp, date, stat and friends); macOS ships BSD versions, which accept a different set of options. Three rows are the ones scripts reach for first: editing a file in place (sed -i), date arithmetic (date -d) and reading a file's size or modification time (stat -c). The remaining rows, including the long last row, are the same kind of difference for less frequent jobs.

Two terms in the table: "epoch seconds" is time as a count of seconds since 1970-01-01 00:00 UTC (so 86400 is 1970-01-02), which is why date -d @N and date -r N both turn a number into a date. gnubin is a directory that Homebrew creates for GNU tools installed on a Mac; it holds them under their normal names (sed, not gsed), so putting it first on PATH makes sed mean GNU sed.

Newer macOS releases have closed some gaps (for example readlink -f works on the macOS I checked, as do sed -E and sed -i.bak, and so does base64 -w, whose man page says it is accepted for compatibility with GNU base64), so test the macOS versions you must support rather than trust a list.

How to detect

Probe the behaviour, not the operating system name: sed --version prints a first line containing (GNU sed) on GNU sed, while BSD sed rejects the option; test that text rather than the exit status, because BusyBox sed --version exits 0 and prints This is not GNU sed version 4.0. stat -c %s / succeeds only on GNU or BusyBox stat; date -d @0 only on GNU or BusyBox date. Probing the tool is better than uname (which prints the operating system name) because Linux with BusyBox or macOS with Homebrew GNU tools first on PATH break any uname-based assumption. BusyBox is one small program that provides cut-down versions of many standard tools; Alpine Linux and many container images use it, and its sed and date accept fewer options than the GNU ones. command -v name prints where the shell would find name and fails if there is no such command, which is the portable way to ask "is this tool installed?".

A wrapper layer

#!/usr/bin/env bash
# portable.sh : feature-detect the userland once, then call wrappers instead of raw flags
set -euo pipefail

if [[ $(sed --version 2>&1 || true) == *"(GNU sed)"* ]]; then SED_FLAVOR=gnu; else SED_FLAVOR=other; fi
if stat -c %s / >/dev/null 2>&1;    then STAT_FLAVOR=gnu; else STAT_FLAVOR=bsd; fi
if date -d @0 >/dev/null 2>&1;      then DATE_FLAVOR=gnu; else DATE_FLAVOR=bsd; fi

sed_inplace() {                 # sed_inplace SCRIPT FILE  (backup suffix attached, works on both)
  sed -i".bak.$$" "$1" "$2" && rm -f -- "$2.bak.$$"
}
file_size() {                   # bytes, no flag differences at all
  wc -c < "$1" | tr -d ' '
}
mtime_epoch() {
  if [[ $STAT_FLAVOR == gnu ]]; then stat -c %Y "$1"; else stat -f %m "$1"; fi
}
epoch_to_date() {
  if [[ $DATE_FLAVOR == gnu ]]; then date -u -d "@$1" +%F; else date -u -r "$1" +%F; fi
}
sha256() {
  if command -v sha256sum >/dev/null 2>&1; then sha256sum "$1" | cut -d' ' -f1
  else shasum -a 256 "$1" | cut -d' ' -f1; fi
}
drop_last_line() {              # replaces GNU-only `head -n -1`
  sed '$d' "$1"
}

printf 'flavors: sed=%s stat=%s date=%s\n' "$SED_FLAVOR" "$STAT_FLAVOR" "$DATE_FLAVOR"
f=$(mktemp); printf 'one\ntwo\nthree\n' > "$f"; touch -t 202001020304 "$f"
sed_inplace 's/two/2/' "$f"
echo "size:   $(file_size "$f")"
echo "digest: $(sha256 "$f" | cut -c1-16)"
echo "drop:   $(drop_last_line "$f" | tr '\n' ',')"
epoch_to_date 86400
rm -f -- "$f"

Output on Ubuntu 24.04 (GNU branches taken):

flavors: sed=gnu stat=gnu date=gnu
size:   12
digest: bc85caa9b61bcf3a
drop:   one,2,
1970-01-02

Reading the wrappers:

• sed_inplace 's/two/2/' "$f" runs sed -i.bak.PID (PID is the shell's process ID, $$), which edits the file in place and keeps the original as file.bak.PID. Attaching the suffix directly to -i is accepted by both GNU and BSD sed, and then rm -f -- "$2.bak.$$" deletes that backup, so the caller gets the effect of sed -i without a flag that differs. The per-process name matters: a plain .bak suffix would silently overwrite and then delete a file.bak the user already had (checked: sed -i.bak replaced an existing f.bak). The demo file changed from one two three lines to one, 2, three.
• epoch_to_date 86400 converts a count of seconds to a calendar date. GNU date does it with -d "@86400", BSD date with -r 86400; the wrapper picks one using the flavor probed at the top, and both print 1970-01-02 (-u keeps the result in UTC so the time zone cannot shift the day).
• touch -t 202001020304 "$f" sets the file's modification time to 2020-01-02 03:04 so that mtime_epoch "$f" would have a fixed value to read back (on Ubuntu with the container's UTC clock it prints 1577934240); the demo does not print it.

I ran the BSD side of each wrapper by hand on macOS: the sed probe reports other and the stat and date probes report bsd, stat -f %m printed the epoch mtime, date -u -r 86400 +%F gave 1970-01-02, shasum -a 256 gave a digest, and sed '$d' and sed -i.bak behaved as in the table. On Alpine with BusyBox tools (bash installed) the same script prints flavors: sed=other stat=gnu date=gnu: BusyBox stat -c and date -d @N work, but its sed is not GNU sed (it has no -z, for example), which is why the sed probe looks for the text (GNU sed). The script file itself was executed on macOS, Ubuntu and Alpine.

The three strategies, and which to pick

1. Avoid. Prefer POSIX options and tools that exist everywhere (wc -c, sed '$d', tr, awk). Cheapest and the most robust. Choose this for anything small.
2. Detect and wrap. Needed when the feature has no POSIX equivalent (date arithmetic, mtimes). Keep all variants in one small library so every script uses the same wrappers.
3. Standardise. For build and deploy scripts that many people run, pin the environment: on macOS brew install coreutils gnu-sed provides gsed, gdate, gstat (and unprefixed copies under a gnubin directory), or run the script in the CI container image on laptops (a dev container). Then document "requires GNU userland" and fail fast with a clear message if sed --version does not report (GNU sed).

For the colleague's Mac failure specifically: reproduce it on a Mac, read the first error, and fix it with strategy 1 where possible. If Macs are a supported platform, add a macOS job to continuous integration (CI); otherwise add a check at the top of the script that refuses to run on a non-GNU userland rather than failing halfway through a file edit.

Pitfalls

• A script that half-works on BSD is worse than one that stops at once. Fail at the top with a GNU-userland check such as the (GNU sed) probe above.
• Even bash differs: macOS ships bash 3.2, so associative arrays (variables that map a text key to a value), ${v,,} (lower-case the value of v), mapfile (read lines of text into an array) and local -n (a nameref: a local variable that is another name for a variable whose name you pass in) are unavailable there.
• echo -e and echo -n are not portable; use printf.

Direct answer

Read the CSV row by row, validate every field before it reaches a command, skip a user that getent passwd already knows (that is what makes the run idempotent: a second run changes nothing), create the account with useradd, set the home directory to 0700, and never put passwords in the file: create the account with its password locked and install an SSH public key, or force a password change at first login. Running it as root means every unchecked field is a potential root-level injection or a mis-grant of privilege, so the script validates names, shells and groups against allowlists, refuses a CSV that other users can write, and supports --dry-run.

The script

Columns: username,comment,shell,groups,ssh_public_key, with groups separated by ;. useradd creates the group named like the user and (when -m is given) the home directory.

#!/usr/bin/env bash
# provision-users.sh [--dry-run] users.csv      (run as root)
# CSV columns: username,comment,shell,groups(;-separated),ssh_public_key
set -uo pipefail          # no -e: one bad row must not abort the rest
umask 077

allowed_groups=(developers ops)
# key type, base64 blob, then an optional free-text comment such as alice@laptop
key_re='^(ssh-ed25519|ecdsa-sha2-nistp256|ssh-rsa) [A-Za-z0-9+/=]+( [A-Za-z0-9@._ +-]+)?$'
dry=0
[[ ${1:-} == --dry-run ]] && { dry=1; shift; }
csv=${1:?usage: $0 [--dry-run] users.csv}

[[ $EUID -eq 0 ]] || { echo "must run as root" >&2; exit 1; }
read -r mode owner < <(stat -c '%a %u' -- "$csv")
if [[ $owner != 0 ]] || (( 8#$mode & 8#022 )); then
  echo "refusing: $csv must be root-owned and not group/world-writable" >&2; exit 1
fi

run() { if (( dry )); then echo "  DRY: $*"; else "$@"; fi; }
created=0; skipped=0; failed=0
fail() { printf 'FAIL  %s: %s\n' "$user" "$1"; failed=$((failed + 1)); }

while IFS= read -r line || [[ -n $line ]]; do
  line=${line%$'\r'}                              # tolerate CRLF files
  [[ -z $line || $line == \#* || $line == username,* ]] && continue
  IFS=, read -r user comment shell groups key <<< "$line"

  [[ $user =~ ^[a-z_][a-z0-9_-]{0,31}$ ]]        || { fail "bad username"; continue; }
  if getent passwd -- "$user" > /dev/null; then
    printf 'SKIP  %s: exists\n' "$user"; skipped=$((skipped + 1)); continue
  fi
  [[ $comment =~ ^[A-Za-z0-9\ .\'-]*$ ]]         || { fail "bad comment"; continue; }
  grep -qxF -- "$shell" /etc/shells               || { fail "shell not in /etc/shells"; continue; }
  home=/home/$user
  [[ ! -e $home ]]                                || { fail "$home exists but user does not"; continue; }

  gcsv=''; bad=0
  IFS=';' read -ra glist <<< "$groups"
  for g in "${glist[@]}"; do
    [[ " ${allowed_groups[*]} " == *" $g "* ]] || { fail "group '$g' not allowed"; bad=1; break; }
  done
  (( bad )) && continue
  gcsv=$(IFS=,; echo "${glist[*]}")

  if [[ -n $key && ! $key =~ $key_re ]]; then
    fail "malformed ssh key"; continue
  fi

  printf 'CREATE %s\n' "$user"
  run useradd --create-home --home-dir "$home" --shell "$shell" \
      --comment "$comment" ${gcsv:+--groups "$gcsv"} -- "$user" \
    || { fail "useradd failed"; continue; }
  run chmod 0700 -- "$home"
  if [[ -n $key ]]; then
    run install -d -m 0700 -o "$user" -g "$user" -- "$home/.ssh"
    if (( ! dry )); then
      printf '%s\n' "$key" > "$home/.ssh/authorized_keys"
      chown "$user:$user" -- "$home/.ssh/authorized_keys"
    fi
  fi
  created=$((created + 1))
done < "$csv"

printf 'created=%d skipped=%d failed=%d\n' "$created" "$skipped" "$failed"
(( failed == 0 ))

Reading the Bash idioms

• IFS=, read -r user comment shell groups key <<< "$line". read splits one line into variables at the characters in IFS (the internal field separator); setting IFS=, on the same line applies only to this one read. <<< (a here-string) feeds the text of $line to read as its input, and -r stops backslashes being treated as escapes. The last variable receives whatever is left, so the key may contain spaces.
• IFS=';' read -ra glist <<< "$groups". The same idea with ;; -a stores the pieces in an array named glist.
• ${gcsv:+--groups "$gcsv"}. This expands to the text after :+ only when gcsv is non-empty, and to nothing at all otherwise. A user with no groups therefore gets no --groups option instead of an empty, invalid one.
• [[ " ${allowed_groups[*]} " == *" $g "* ]]. ${allowed_groups[*]} joins the array with spaces and the extra spaces on both ends give " developers ops ". The test then asks whether " $g " (the name with a space each side) appears inside it, where * matches anything. The padding makes it a whole-word match: dev is rejected even though it is a prefix of developers.
• read -r mode owner < <(stat -c '%a %u' -- "$csv"). <( ... ) runs the command and lets read take its output like a file. stat prints the mode and the owner's numeric user ID.
• (( 8#$mode & 8#022 )). 8# tells the shell to read the digits as octal. 022 is the group-write bit (020) plus the other-write bit (002). The & is non-zero when either is set, which means somebody other than the owner could edit the CSV.
• --. Marks the end of options, so a value that starts with - is read as a name, not as an option.
• umask 077. A umask is a set of permission bits removed from everything the process creates. With 077, new files get 600 and new directories 700, so nothing the script creates is readable by other users.

Each idiom run on its own (Ubuntu 24.04 container):

allowed_groups=(developers ops)
for g in ops dev developers 'ops developers'; do
  if [[ " ${allowed_groups[*]} " == *" $g "* ]]; then echo "'$g' allowed"; else echo "'$g' rejected"; fi
done
line='alice,Alice Nguyen,/bin/bash,developers;ops,ssh-ed25519 AAAA'
IFS=, read -r user comment shell groups key <<< "$line"
echo "user=$user comment=$comment shell=$shell groups=$groups key=$key"
IFS=';' read -ra glist <<< "$groups"
echo "glist has ${#glist[@]} items: ${glist[0]} and ${glist[1]}"
for gcsv in 'developers,ops' ''; do
  printf 'gcsv=[%s] ->' "$gcsv"; printf ' <%s>' ${gcsv:+--groups "$gcsv"}; echo
done
for mode in 600 644 664 666; do
  if (( 8#$mode & 8#022 )); then echo "$mode: group or others can write"; else echo "$mode: only the owner can write"; fi
done
( umask 077; d=$(mktemp -d); touch $d/f; mkdir $d/dd; stat -c '%a %n' $d/f $d/dd | sed "s#$d/##" )

'ops' allowed
'dev' rejected
'developers' allowed
'ops developers' rejected
user=alice comment=Alice Nguyen shell=/bin/bash groups=developers;ops key=ssh-ed25519 AAAA
glist has 2 items: developers and ops
gcsv=[developers,ops] -> <--groups> <developers,ops>
gcsv=[] -> <>
600: only the owner can write
644: only the owner can write
664: group or others can write
666: group or others can write
600 f
700 dd

Reading it: whole-word matching accepts ops and developers but rejects dev and the two-word string. The read calls split the line and the group list as described. When gcsv is empty the printf receives no arguments from the expansion and prints a single empty <>. Modes 664 and 666 trip the 022 test, while 600 and 644 do not, which is why a group-writable or world-writable CSV is refused.

What each guard does

• Idempotency: getent passwd is the existence test, because it asks the name service switch (the system's configured list of places to look users up: local files, LDAP, SSSD) rather than only grepping /etc/passwd. An existing user is reported as SKIP and not modified, so running the script twice changes nothing the second time.
• Credentials: there is no password column. useradd without a password leaves the account with a locked password (no typed password can ever match, so password login is impossible), which passwd -S reports as L. Access then comes from the SSH key. If a password is unavoidable, generate a random one, feed it to chpasswd on standard input (never as an argument, because arguments are visible to every user in ps) and run chage -d 0 user, which per the chage manual forces a password change at the next login. Deliver the temporary secret out of band. Whether key login works for a locked-password account depends on your sshd and PAM (pluggable authentication modules) settings, so test one canary account (a single throwaway account you create first, to see the whole flow work) before the bulk run.
• Permissions: umask 077 for the script, and an explicit chmod 0700 on the home. The default home mode comes from distribution settings (HOME_MODE in /etc/login.defs) and varies. .ssh is 0700 and authorized_keys is 0600, owned by the user.
• Allowlists: the username must match ^[a-z_][a-z0-9_-]{0,31}$; the shell must be a line of /etc/shells; each group must be in allowed_groups; the key must start with a listed key type and a base64 blob and may end with the usual free-text comment (alice@laptop), so a key as ssh-keygen writes it is accepted, while a line that starts with authorized_keys options such as command= is rejected because those options run commands. The -- before the username stops a name starting with - being read as an option.
• Partial failure: no set -e. One bad row prints FAIL, the loop carries on, and the exit code is non-zero if any row failed, so cron or CI notices.
• CRLF: spreadsheets export Windows line endings, so the trailing carriage return is stripped.

Worked example

Run as root in a Linux container. The CSV has Windows line endings and six data rows: a good user with a key and two groups, a good user with no key, a user that already exists, an uppercase username, a shell that is not in /etc/shells, and a user asking for the sudo group.

set -u
groupadd developers; groupadd ops
K='ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl alice@laptop'
printf '%s\r\n' 'username,comment,shell,groups,ssh_public_key' \
  "alice,Alice Nguyen,/bin/bash,developers;ops,$K" \
  'bob,Bob Okafor,/bin/sh,developers,' \
  'ubuntu,Already Here,/bin/bash,,' \
  'Eve,Bad Name,/bin/bash,,' \
  'mallory,Mal,/usr/bin/evil,,' \
  'carol,Carol Diaz,/bin/bash,sudo,' > /root/users.csv
echo '--- dry run'; bash provision-users.sh --dry-run /root/users.csv; echo "exit=$?"
echo '--- real run'; bash provision-users.sh /root/users.csv; echo "exit=$?"
echo '--- second run'; bash provision-users.sh /root/users.csv; echo "exit=$?"
echo '--- result'
stat -c '%a %U:%G %n' /home/alice /home/alice/.ssh /home/alice/.ssh/authorized_keys /home/bob
id alice; passwd -S alice | awk '{print $1, $2}'
echo '--- unsafe csv'; chmod 666 /root/users.csv; bash provision-users.sh /root/users.csv; echo "exit=$?"

Output:

--- dry run
CREATE alice
  DRY: useradd --create-home --home-dir /home/alice --shell /bin/bash --comment Alice Nguyen --groups developers,ops -- alice
  DRY: chmod 0700 -- /home/alice
  DRY: install -d -m 0700 -o alice -g alice -- /home/alice/.ssh
CREATE bob
  DRY: useradd --create-home --home-dir /home/bob --shell /bin/sh --comment Bob Okafor --groups developers -- bob
  DRY: chmod 0700 -- /home/bob
SKIP  ubuntu: exists
FAIL  Eve: bad username
FAIL  mallory: shell not in /etc/shells
FAIL  carol: group 'sudo' not allowed
created=2 skipped=1 failed=3
exit=1
--- real run
CREATE alice
CREATE bob
SKIP  ubuntu: exists
FAIL  Eve: bad username
FAIL  mallory: shell not in /etc/shells
FAIL  carol: group 'sudo' not allowed
created=2 skipped=1 failed=3
exit=1
--- second run
SKIP  alice: exists
SKIP  bob: exists
SKIP  ubuntu: exists
FAIL  Eve: bad username
FAIL  mallory: shell not in /etc/shells
FAIL  carol: group 'sudo' not allowed
created=0 skipped=3 failed=3
exit=1
--- result
700 alice:alice /home/alice
700 alice:alice /home/alice/.ssh
600 alice:alice /home/alice/.ssh/authorized_keys
700 bob:bob /home/bob
uid=1001(alice) gid=1003(alice) groups=1003(alice),1001(developers),1002(ops)
alice L
--- unsafe csv
refusing: /root/users.csv must be root-owned and not group/world-writable
exit=1

The dry run prints the exact useradd lines it would run (quoting is lost in the echo, not in the real call). The second real run creates nothing: created=0. The three bad rows keep failing, and the final check shows the unsafe CSV (mode 666) is refused outright. alice is in developers and ops, her home and .ssh are 0700, authorized_keys is 0600, and her password status is L.

Risks of running it as root

Trade-offs and pitfalls

• Skip versus reconcile. The brief says skip existing users, and the script does. The cost is drift: if the CSV later changes a shell or group for an existing person, nothing happens. A second mode that compares and usermods is the next step, and it needs its own dry-run output.
• A real fleet usually should not use a script at all for this: a directory service (LDAP, SSSD) or configuration management makes the state declarative. A script is right for a single host, an image build or a bootstrap.
• userdel is the other half of the lifecycle. Offboarding by removing a CSV row does not delete anything here, which is the safer default.

Direct answer

Turn the user's name into a canonical absolute path first (every .. collapsed and every symlink replaced by what it points to), then check that the canonical result sits under the canonical base directory. A check on the raw text, such as "reject names containing ..", is not enough, because a symlink that already lives inside the base can point anywhere. Because the file can change between the check and the open (a time-of-check to time-of-use race, TOCTOU), open the file and then check again what the open descriptor really refers to.

The artifact

This was run as root inside a throwaway ubuntu:24.04 container (GNU coreutils), because it creates /srv directories. resolve_in_base does the check; open_and_recheck opens the checked path and re-checks the open file; read_in_base runs the two in order. Two pieces of Bash syntax appear in the code: set -u makes the script stop with an error if it uses a variable that was never set (so a typo cannot silently become an empty string), and $'\n' is Bash quoting that produces a real newline character.

#!/usr/bin/env bash
set -u

# Prints the canonical path of $2 inside base directory $1, or fails.
resolve_in_base() {
  local base=$1 name=$2 root target
  [[ -n $name ]]            || { echo "empty name" >&2; return 1; }
  [[ $name != /* ]]         || { echo "absolute path refused: $name" >&2; return 1; }
  [[ $name != *$'\n'* ]]    || { echo "newline in name refused" >&2; return 1; }
  root=$(realpath -e -- "$base") || return 1
  [[ $root != / ]]          || { echo "base must not be /" >&2; return 1; }
  target=$(realpath -m -- "$root/$name") || return 1
  [[ $target == "$root"/* ]] || { echo "escapes base: $name -> $target" >&2; return 1; }
  printf '%s\n' "$target"
}

# Opens an already-checked path, then re-checks what the descriptor really points at.
open_and_recheck() {
  local base=$1 target=$2 root real status
  root=$(realpath -e -- "$base")
  exec 3< "$target" || return 1
  real=$(readlink -f /proc/self/fd/3)
  if [[ $real != "$root"/* ]]; then
    exec 3<&-
    echo "descriptor escaped base: $real" >&2
    return 1
  fi
  cat <&3
  status=$?                 # keep the read's own status; closing below would overwrite it
  exec 3<&-
  return "$status"
}

read_in_base() {
  local target
  target=$(resolve_in_base "$1" "$2") || return 1
  open_and_recheck "$1" "$target"
}

base=/srv/uploads
mkdir -p "$base/reports" /srv/uploads-evil /srv/secret
echo "quarterly numbers" > "$base/reports/q1.txt"
echo "evil sibling"  > /srv/uploads-evil/x.txt
echo "root password" > /srv/secret/shadow
echo "attacker copy of q1" > /srv/secret/q1.txt
ln -s /srv/secret/shadow      "$base/reports/link-to-file"
ln -s /srv/secret             "$base/reports/link-to-dir"
ln -s /srv/secret/not-yet     "$base/reports/dangling-out"
ln -s q1.txt                  "$base/reports/alias-inside"

for n in "reports/q1.txt" "reports/alias-inside" "reports/new file.txt" \
         "../uploads-evil/x.txt" "reports/../../secret/shadow" "/srv/secret/shadow" \
         "reports/link-to-file" "reports/link-to-dir/shadow" "reports/dangling-out" \
         "-rf" ""; do
  printf '%-30s => ' "[$n]"
  resolve_in_base "$base" "$n" 2>&1
done
echo "--- read"
read_in_base "$base" "reports/q1.txt"
read_in_base "$base" "reports/link-to-file"
read_in_base "$base" "reports"; echo "directory read status: $?"

echo "--- swap between the check and the open"
target=$(resolve_in_base "$base" "reports/q1.txt")
echo "checked: $target"
mv "$base/reports" "$base/reports.old"          # attacker swaps the directory ...
ln -s /srv/secret "$base/reports"               # ... for a symlink to elsewhere
open_and_recheck "$base" "$target"

Output:

[reports/q1.txt]               => /srv/uploads/reports/q1.txt
[reports/alias-inside]         => /srv/uploads/reports/q1.txt
[reports/new file.txt]         => /srv/uploads/reports/new file.txt
[../uploads-evil/x.txt]        => escapes base: ../uploads-evil/x.txt -> /srv/uploads-evil/x.txt
[reports/../../secret/shadow]  => escapes base: reports/../../secret/shadow -> /srv/secret/shadow
[/srv/secret/shadow]           => absolute path refused: /srv/secret/shadow
[reports/link-to-file]         => escapes base: reports/link-to-file -> /srv/secret/shadow
[reports/link-to-dir/shadow]   => escapes base: reports/link-to-dir/shadow -> /srv/secret/shadow
[reports/dangling-out]         => escapes base: reports/dangling-out -> /srv/secret/not-yet
[-rf]                          => /srv/uploads/-rf
[]                             => empty name
--- read
quarterly numbers
escapes base: reports/link-to-file -> /srv/secret/shadow
cat: -: Is a directory
directory read status: 1
--- swap between the check and the open
checked: /srv/uploads/reports/q1.txt
descriptor escaped base: /srv/secret/q1.txt

Reading the re-check, line by line

A file descriptor is a small number the kernel gives a process for each file it has open; the process uses the number as a handle instead of the name. Descriptors 0, 1 and 2 are already taken (standard input, output and error), so 3 is the first free one, and nothing more special than that.

• exec 3< "$target" opens the file for reading and attaches it to descriptor 3 for the rest of the script. If the open fails, || return 1 stops.
• /proc/self/fd/3 is Linux's window onto the process's own open files: /proc is a virtual directory the kernel fills in, self means the process doing the looking, and fd/3 is a symlink to whatever descriptor 3 is open on. Here the looking is done by readlink, a child process started by $(...); it inherits descriptor 3 from the script (descriptors opened with exec are inherited by child programs), so its /proc/self/fd/3 is the very same open file. readlink -f follows it and prints the real canonical path, so real is the file that was actually opened, not the name you asked for.
• [[ $real != "$root"/* ]] is the same trailing-slash comparison as before, now applied to real. If it fails, exec 3<&- closes descriptor 3 (<&- means "close"), the function reports descriptor escaped base, and returns 1 before any data is read.
• cat <&3 reads from descriptor 3 (the already-checked file) and prints it. Its exit status is saved in status before the final exec 3<&- closes the descriptor and return "$status" hands it back; without that, the exec would overwrite the status with 0 and a failed read would look like success. The demo's last read shows it: reading the directory reports makes cat print cat: -: Is a directory, and the function returns 1 rather than 0.

Why each line is there

• realpath -e -- "$base" canonicalizes the base. GNU realpath -e requires every component to exist, so a mistyped base fails instead of silently matching nothing.
• realpath -m -- "$root/$name" canonicalizes the target. -m allows missing components, so a file you are about to create (reports/new file.txt) can be validated, while symlinks that do exist are still resolved. That is why link-to-file, link-to-dir/shadow and even the dangling dangling-out link (its target does not exist yet, but it would be created outside the base) are all refused, while alias-inside, a link that stays inside the base, is allowed and resolves to q1.txt.
• [[ $target == "$root"/* ]] has a trailing slash on purpose. /srv/uploads-evil/x.txt starts with the characters /srv/uploads, but not with /srv/uploads/, and the sibling-directory row shows it being refused. The quoted "$root" makes any glob characters in the base literal; the unquoted /* is the pattern.
• -- after realpath, and joining the name onto the base, mean a name like -rf is a file called -rf, never an option.
• Absolute names are refused as a policy: joined onto the base they would just be read as relative, so refusing makes an attack attempt visible instead of quietly "working". Names containing a newline are refused because $(...) strips trailing newlines, which would make the checked path differ from the path the user meant.
• Empty names and a base of / are refused, since / would make every path "inside".

The remaining race, and what actually closes it

Between resolve_in_base returning and the open, anyone who can write inside the base could swap a directory for a symlink. read_in_base closes that for reads: after exec 3< "$target" it asks the kernel where descriptor 3 really points (readlink -f /proc/self/fd/3, Linux only) and refuses unless that is under the base. The check is on the file that was opened, so a swap after the check cannot be used.

The last block of the demo plays that attack out with real paths. resolve_in_base approves /srv/uploads/reports/q1.txt (a real file inside the base). Before the open, the attacker renames the reports directory and puts a symlink named reports pointing at /srv/secret in its place. The path string is unchanged, so the open succeeds, but it now lands on /srv/secret/q1.txt. The first check passed on the old layout; the descriptor check asks the kernel what descriptor 3 really points at, gets /srv/secret/q1.txt, and refuses with descriptor escaped base: /srv/secret/q1.txt. This runs as root in a throwaway container, so the attacker's mv and ln are simulated by the script itself.

That does not help a write or create, because by the time you can inspect the result the data has landed. For writes, remove the attacker's ability to plant symlinks (the process that fills the base writes regular files only, and untrusted users cannot create links there), or, outside Bash, do the open in a language that can call openat2 with RESOLVE_BENEATH, which makes the kernel refuse any path resolution that leaves the directory (Linux 5.6 and later; RESOLVE_NO_SYMLINKS refuses symlinks entirely). Bash has no way to pass those flags, so this is a note about other languages, and the realpath plus descriptor re-check above is the approach for a Bash script.

Trade-offs and pitfalls

• realpath -m and readlink -f are GNU/Linux behaviour; macOS and BusyBox differ, so say which platform the script targets.
• Opening a path that an attacker has swapped for a FIFO (a named pipe) blocks until a writer appears: in a test, a read of a FIFO inside the base hung until a 3-second timeout killed it. The descriptor check cannot help because the open itself never returns, so wrap reads of a shared base in timeout or refuse non-regular files before opening.
• A symlink loop does not make realpath -m fail, it just returns a path; the later open of such a path fails, which is the safe outcome.
• Do not "sanitize" by stripping ../ with sed: ....// becomes ../ after one pass, and it ignores symlinks completely.
• A canonical-path check follows symlinks but cannot see hard links. A hard link is a second name for the same file data, so a hard link inside the base to a sensitive file looks like any other file in the base and is allowed. Prevent that by controlling who can create links in the base.

Do not rewrite it. Freeze its current behavior in an automated comparison first (a characterization test, also called a golden-master test: record what the script does today, right or wrong, and fail on any difference), then change it in small steps, each one verified against that recording and shipped on its own. The goal is that after every commit the old and new versions are provably identical on everything you recorded, except for differences you chose on purpose.

1. Learn what it does and how it is run

Before touching code, collect facts: who calls it (cron, systemd timer, CI, other scripts), with what arguments, as which user, with what environment. List every external effect: files written, commands called, network calls, mail sent, exit codes callers depend on. In a 2,000 to 10,000 line script, most of the risk is in those effects, not in the logic.

2. Build the safety net before refactoring

1. Run ShellCheck (a static analyzer for shell) and keep the output as a baseline of known problems. Do not fix them yet.
2. Make the script testable without a production machine: put stubs for external commands (date, mail, curl, systemctl) in a directory placed first on PATH; each stub logs its arguments and stdin to a file. Run against throwaway copies of fixture directories.
3. Record, per case, stdout, stderr, exit status and the stub call log. Cases come from real inputs (sanitized) and from the paths that matter most: the success path, an empty input, a missing file, a failing command, a name with a space.
4. Run the same harness against the old script to prove it is deterministic before you trust it. A harness that flaps cannot guard anything.

3. Refactor in small, reversible steps

• Wrap top-level code in functions and a main, with a guard so the file can be sourced for tests without running: if [[ ${BASH_SOURCE[0]} == "$0" ]]; then main "$@"; fi. Use the if form: [[ ... ]] && main returns status 1 when sourced.
• Extract one block at a time into a function with local variables and explicit arguments. Replace global state (variables shared by accident) with parameters. Move constants and paths to the top.
• Mechanical safety fixes next, one rule per commit: quote expansions, replace `cmd` with $(cmd), replace ls | ... and for f in $(ls) with globs, expr with $(( )).
• Only after the above, consider set -euo pipefail. It changes behavior (a failing command now aborts the script), so each place it matters must be checked separately. Example: grep -c exits with status 1 when the count is zero, so under set -e a plain c=$(grep -c ...) kills the script.
• Split into sourced library files only when a function has proven stable and has its own tests (the file structure is the last change, not the first).
• Where logic is genuinely complex (parsing, math, data munging), that is the moment to ask whether that part belongs in a language with tests and libraries, replacing it behind the same function boundary.

4. Keep regression risk low in production

Ship each step separately and small, so a failure points at one diff. Keep the old script available (legacy.sh plus a switch or a symlink) for a few cycles, run old and new side by side in a read-only or dry-run mode where the effects allow it, and compare outputs from real data before cutting over. Add logging that identifies the version and the path taken, and have a one-command rollback.

Worked example

A small legacy script counts HTTP 500 lines per log file and sends mail when the total exceeds 2. It is deliberately in the old style.

#!/bin/bash
cd $1 || exit 2
total=0
for f in `ls *.log`
do
  c=`grep -c ' 500 ' $f`
  echo "$f: $c"
  total=`expr $total + $c`
done
echo "total: $total"
if [ $total -gt 2 ]; then
  echo "5xx total $total on `date +%F`" | mail -s alert oncall@example.com
fi

The harness runs a script once per case, with stubbed date and mail, and compares stdout, stderr, exit status and recorded calls with a stored snapshot. A stub is a tiny fake program that stands in for a real command. It lives in its own file in a stubs/ directory, and the harness puts that directory first on PATH (the list of directories the shell searches for commands), so the script finds the fake date and mail before the real ones. The directory layout:

golden.sh
legacy.sh
refactored.sh
stubs/date
stubs/mail
cases/busy/input/api.log
cases/busy/input/web.log
cases/quiet/input/app.log
cases/nologs/input/.gitkeep

Each case is a directory holding an input/ folder (the files the script will be pointed at) and, once recorded, an expected file (the snapshot). The fixture log lines are plain access-log style text; the script counts lines containing 500 (a space, 500, a space):

cases/busy/input/api.log:   GET /a 200 512  and  GET /b 500 118
cases/busy/input/web.log:   GET /a 500 97, GET /b 200 40, GET /c 500 130
cases/quiet/input/app.log:  GET /a 200 512  and  GET /b 404 33
cases/nologs/input/:        only an empty .gitkeep placeholder, no .log file

So busy has one 500 in api.log and two in web.log (three in total, which is above the threshold of 2, so a mail is expected), and quiet has none. stubs/date always prints a fixed date, so the output cannot change from day to day:

#!/bin/sh
echo 2026-01-15

stubs/mail appends its own arguments and whatever arrives on its standard input to the file named by $STUB_LOG, so the harness can later see that a mail was "sent" and what it said:

#!/bin/sh
{ echo "mail $*"; cat; } >> "$STUB_LOG"

golden.sh itself:

#!/usr/bin/env bash
# usage: golden.sh record|check SCRIPT
# Runs SCRIPT once per case with stubbed date/mail and compares stdout, stderr,
# exit status and recorded stub calls against the stored snapshot.
set -uo pipefail
mode=$1
script=$(realpath "$2")
root=$(dirname "$(realpath "$0")")
fail=0
for case in "$root"/cases/*/; do
  name=$(basename "$case")
  work=$(mktemp -d)
  cp -r "$case/input/." "$work/"
  : > "$work.calls"
  snap=$(
    cd "$work" || exit
    PATH="$root/stubs:$PATH" STUB_LOG="$work.calls" bash "$script" "$work" >"$work.out" 2>"$work.err"
    rc=$?
    printf '== stdout\n'; cat "$work.out"
    printf '== stderr\n'; cat "$work.err"
    printf '== exit %s\n' "$rc"
    printf '== calls\n';  cat "$work.calls"
  )
  if [[ $mode == record ]]; then
    printf '%s\n' "$snap" > "$case/expected"
    echo "recorded $name"
  elif diff -u --label expected --label actual "$case/expected" <(printf '%s\n' "$snap"); then
    echo "PASS $name"
  else
    echo "FAIL $name"; fail=1
  fi
  rm -rf "$work" "$work.out" "$work.err" "$work.calls"
done
exit "$fail"

Reading golden.sh from the top:

• mode=$1 is record or check; script=$(realpath "$2") turns the script path into an absolute path (realpath resolves it fully) because the harness changes directory later; root is the folder holding golden.sh.
• For each case directory, mktemp -d makes a fresh empty scratch directory in work, and cp -r "$case/input/." "$work/" copies the fixture into it, so the script can never modify the stored fixtures. : > "$work.calls" creates an empty call log next to it.
• snap=$( ... ) is a command substitution: everything printed inside the parentheses is captured into the variable snap. Inside it runs a subshell (a child copy of the shell), so the cd "$work" does not move the harness itself. The script runs with PATH set to stubs first and STUB_LOG pointing at the call log; its stdout and stderr are redirected to two files so they can be labeled separately, and rc=$? keeps its exit status.
• The printf lines then print four labeled sections (== stdout, == stderr, == exit, == calls) into snap. That text is the snapshot.
• In record mode the snapshot is written to expected. In check mode, diff -u expected <(printf '%s\n' "$snap") compares the stored file with the fresh snapshot; <( ... ) is process substitution, which hands diff the output of the command as if it were a file. diff exits 0 when they match (print PASS) and prints the differences otherwise (print FAIL and set fail=1).
• The last lines delete the scratch files and exit non-zero if any case failed, so CI can use the harness as a gate.

The refactored version is the file refactored.sh:

#!/usr/bin/env bash
count_5xx() { grep -c ' 500 ' -- "$1"; }

main() {
  cd -- "$1" || exit 2
  local total=0 f c
  shopt -s nullglob
  for f in *.log; do
    c=$(count_5xx "$f")
    echo "$f: $c"
    total=$((total + c))
  done
  echo "total: $total"
  if ((total > 2)); then
    echo "5xx total $total on $(date +%F)" | mail -s alert oncall@example.com
  fi
}

if [[ ${BASH_SOURCE[0]} == "$0" ]]; then
  main "$@"
fi

Driver (run.sh, run in an Ubuntu 24.04 container with the directory holding the files above mounted at /w):

cd /w
echo "### record from legacy"; bash golden.sh record legacy.sh
cat cases/busy/expected
echo "### check refactored"; bash golden.sh check refactored.sh
echo "### check legacy against itself"; bash golden.sh check legacy.sh

Output:

### record from legacy
recorded busy
recorded nologs
recorded quiet
== stdout
api.log: 1
web.log: 2
total: 3
== stderr
== exit 0
== calls
mail -s alert oncall@example.com
5xx total 3 on 2026-01-15
### check refactored
PASS busy
--- expected
+++ actual
@@ -1,6 +1,5 @@
 == stdout
 total: 0
 == stderr
-ls: cannot access '*.log': No such file or directory
 == exit 0
 == calls
FAIL nologs
PASS quiet
### check legacy against itself
PASS busy
PASS nologs
PASS quiet

The first section records snapshots from the legacy script and prints the busy snapshot: stdout, empty stderr, exit 0 and one captured mail call. Checking the refactor, busy and quiet pass. nologs fails, and the diff is informative: the legacy script wrote ls: cannot access '*.log' to stderr, while the refactor (using nullglob, which makes an unmatched glob expand to nothing) is silent. The harness has found a behavior change that a reading of the code would not have flagged. The next step is a decision, not a fix: if no caller parses that message, record it as an intended change in the commit message and re-record that one snapshot; if some monitoring greps for it, keep it. Last, the harness run against the legacy script itself passes all three, which shows the recording is stable.

ShellCheck is clean on the refactored script and on the harness (run with the koalaman/shellcheck:stable image).

Pitfalls

• A characterization test records bugs as faithfully as features. Decide per difference whether it is a bug worth fixing, but fix it in a separate commit after the refactor so a test failure always means one thing.
• Tests with real date or real network calls flap. Stub time, randomness and the network.
• The refactor "while I am here" feature change is the usual cause of regressions: separate them.
• Cases you did not record are unprotected. After each production incident or surprising input, add a case first, then fix.

Direct answer

The script takes exactly one path, rejects anything else with a usage message and exit code 2, prints whether the file exists, is readable, writable and executable, plus its size in bytes, and exits with a distinct code for each outcome a caller might need to branch on. The permission results are report lines, not failures: exit 0 means "the file exists, is a regular file, and the report was printed". A caller that needs a single yes or no uses test -r file directly.

Exit code contract

Codes 126 and 127 are avoided on purpose because the shell uses them for "found but not executable" and "command not found". Code 1 is avoided so that "the script crashed" is not confused with a deliberate answer.

The script

#!/usr/bin/env bash
# fileinfo.sh [-c] PATH
# Exit codes: 0 PATH is a regular file and the report was printed
#             2 usage error (nothing is checked)
#             3 PATH does not exist (and -c was not given)
#             4 PATH exists but is not a regular file
#             5 -c was given and the file could not be created
set -u

usage() { printf 'usage: %s [-c] PATH\n  -c  create PATH with mode 0640 if it is missing\n' "${0##*/}"; }

create=0
if [[ ${1-} == -c ]]; then create=1; shift; fi
if [[ $# -ne 1 || -z $1 || $1 == -* ]]; then
  usage >&2
  exit 2
fi
path=$1

if [[ ! -e $path && ! -L $path ]]; then
  if (( create )); then
    # umask 027 turns the default 0666 into 0640 at creation: no window with wider permissions
    ( umask 027; : > "$path" ) 2>/dev/null || { echo "cannot create: $path" >&2; exit 5; }
  else
    echo "not found: $path" >&2
    exit 3
  fi
fi

if [[ ! -f $path ]]; then
  echo "not a regular file: $path" >&2
  exit 4
fi

yn() { if "$@"; then echo yes; else echo no; fi; }
printf 'path:       %s\n' "$path"
printf 'readable:   %s\n' "$(yn test -r "$path")"
printf 'writable:   %s\n' "$(yn test -w "$path")"
printf 'executable: %s\n' "$(yn test -x "$path")"
printf 'size:       %s bytes\n' "$(stat -c %s -- "$path")"

Choices:

• All usage and error text goes to standard error, so a caller capturing standard output gets either the report or nothing.
• [[ ! -e $path && ! -L $path ]] distinguishes a missing path from a dangling symlink, which -e alone would call missing. The dangling link then falls through to exit 4.
• -c creates the file with mode 0640 (owner read and write, group read, others nothing). The creation runs in a subshell with umask 027, so the file is born with the right mode and never exists with wider permissions.
• stat -c %s is GNU; on BSD or macOS the equivalent is stat -f %z. wc -c < file is portable but needs read permission.
• test -r and friends answer for the user running the script, not for the file in general.

Reading the script

• set -u makes the script stop with an error if it uses a variable that was never set. That is why the script writes ${1-} (the first argument, or empty if there is none) instead of plain $1.
• ${0##*/} strips everything up to the last / from $0, the name the script was run as: /usr/local/bin/fileinfo.sh becomes fileinfo.sh, so the usage message shows a short name.
• $# is the number of arguments. The check [[ $# -ne 1 || -z $1 || $1 == -* ]] rejects zero or several arguments, an empty argument and one that starts with -.
• (( create )) is arithmetic evaluation: it is true when the number inside is not zero, so it reads as "if create is on".
• yn() { if "$@"; then echo yes; else echo no; fi; } is a helper that runs whatever command it is given and prints yes or no from that command's exit status. yn test -r "$path" runs test -r "$path"; "$@" is "all the arguments, each kept as one word". The $( ... ) around it captures the printed word for printf.
• A dangling symlink is a symbolic link whose target does not exist (ln -s missing dangling). -e follows the link, finds nothing and says false, while -L asks about the link itself and says true; the script uses both so the dangling link is reported as "not a regular file" (exit 4) instead of "not found" (exit 3).
• ( umask 027; : > "$path" ): the parentheses run in a subshell, so the umask change ends with it. : > "$path" is a command that does nothing and, through the redirect, creates an empty file.
• What umask 027 does. A umask lists permission bits to take away from newly created files. A new file starts at 0666 (read and write for everyone; files never start executable). The mask removes bits, it is not a decimal subtraction: 0666 with the bits of 027 removed is 0640. Subtracting the numbers gives the wrong 0637, and a mask of 025 would give 0642, not the 0641 that subtraction suggests (computed in python3: 0o666 & ~0o027 is 0o640, 0o666 - 0o027 is 0o637, 0o666 & ~0o025 is 0o642).
• Reading mode digits. Each digit is a sum for one group, owner then group then others: read is 4, write is 2, execute is 1. So 640 is owner 6 = read and write, group 4 = read, others 0 = nothing, shown by ls -l as -rw-r-----. In the tests, chmod 000 removes every permission (----------), chmod 755 gives the owner everything and the others read and execute (-rwxr-xr-x). Both were checked with stat -c '%a %A'.

Tests with the exit codes asserted

#!/usr/bin/env bash
# Run as a non-root user: root passes every -r and -w test regardless of mode.
S=${1:?path to fileinfo.sh}
cd "$(mktemp -d)" || exit 1
printf 'hello\n' > 'my file.txt'
printf '#!/bin/sh\n' > run.sh && chmod 755 run.sh
: > locked.txt && chmod 000 locked.txt
ln -s missing dangling

check() {                         # check WANT_EXIT ARGS...
  local want=$1 rc; shift
  "$S" "$@" >/dev/null 2>&1; rc=$?
  printf '%s exit=%s want=%s args: %s\n' "$([[ $rc == "$want" ]] && echo PASS || echo FAIL)" "$rc" "$want" "$*"
}
check 0 'my file.txt'
check 0 run.sh
check 0 locked.txt
check 0 -c made.txt
check 2
check 2 a b
check 2 -rf
check 2 ''
check 3 nothing.txt
check 4 .
check 4 dangling
check 5 -c /no/such/dir/x
stat -c 'made.txt mode: %a' made.txt

Output (run as an unprivileged user inside a Linux container):

PASS exit=0 want=0 args: my file.txt
PASS exit=0 want=0 args: run.sh
PASS exit=0 want=0 args: locked.txt
PASS exit=0 want=0 args: -c made.txt
PASS exit=2 want=2 args: 
PASS exit=2 want=2 args: a b
PASS exit=2 want=2 args: -rf
PASS exit=2 want=2 args: 
PASS exit=3 want=3 args: nothing.txt
PASS exit=4 want=4 args: .
PASS exit=4 want=4 args: dangling
PASS exit=5 want=5 args: -c /no/such/dir/x
made.txt mode: 640

Every case passed, including the unreadable file (locked.txt is mode 000, the script still exits 0 and prints readable: no) and the dangling symlink (exit 4). The file created with -c has mode 640.

Pitfalls

• Root is different. For root, test -r and test -w succeed on nearly any file regardless of its mode bits, so test as a normal user. That is why the test harness above is run with a non-root user.
• Time of check versus time of use. The file can change between the test and the caller's next action. For security-sensitive code, open the file and check the descriptor rather than testing the path.
• Arguments starting with - are rejected. If you need to accept them, require -- and pass ./name.
• The same shape for services. To report whether a service is active, run systemctl is-active --quiet NAME: it prints nothing with --quiet and exits 0 only when the unit is active, so the same table of exit codes applies with different case labels.