Shell Scripting and Automation Questions

Brief approach
Use a safe edit: write changes to a secure temp file (same dir), preserve mode/ownership, fsync, then atomically rename into place. Avoid sed -i portability; run sed/awk reading original and writing temp. Use traps to cleanup on interrupt.

Robust pattern (bash)

#!/usr/bin/env bash
set -euo pipefail

file="/etc/myapp/config.conf"
dir=$(dirname "$file")
tmp=$(mktemp -p "$dir" "$(basename "$file").tmp.XXXXXX")
trap 'rm -f "$tmp"' EXIT

# preserve perms/ownership
perm=$(stat -c %a "$file" 2>/dev/null || stat -f %p "$file")
owner=$(stat -c %u:%g "$file" 2>/dev/null || stat -f %u:%g "$file")

# produce edited output safely (use sed/awk; no -i)
sed 's/OLD/NEW/g' "$file" > "$tmp"

# ensure data hits disk
sync
# set metadata
chmod "$perm" "$tmp"
chown "$owner" "$tmp" || true   # may require root; ignore if not permitted

# atomic replace
mv -f "$tmp" "$file"
trap - EXIT

Why this is safe

• Writing to a temp file in same directory allows atomic mv (rename).
• Preserves permissions and ownership so services keep access.
• mktemp avoids predictable names; trap ensures cleanup on interrupt.
• Avoids sed -i portability issues by not using in-place flags.
• mv is atomic on the same filesystem — prevents partial writes and recovers by leaving original until swap.

Edge notes

• If editing symlinks, decide whether to replace symlink or target (use mv --backup or follow symlink explicitly).
• For critical systems, call fsync on temp file descriptor before chmod/mv.
• Run as appropriate user/root to preserve ownership; otherwise chown may fail.

What "trap" is (brief)
A trap registers a handler to run when the shell receives signals or special events (EXIT, ERR, INT, etc.). Useful in DevOps scripts for cleanup, graceful shutdown, and error logging.

Example: EXIT trap to clean temp resources (works with set -e)

#!/usr/bin/env bash
set -euo pipefail
tmpdir="$(mktemp -d)"
cleanup() {
  rm -rf "$tmpdir"
  echo "Cleaned $tmpdir"
}
trap cleanup EXIT

# Simulate early exit due to error
cp non_existent_file "$tmpdir/"   # set -e causes exit, EXIT trap still runs

SIGINT trap for graceful shutdown

#!/usr/bin/env bash
trap 'echo "Interrupted, shutting down..."; stop_services; exit 130' INT
start_services
# long-running work

ERR trap to log failing command

#!/usr/bin/env bash
trap 'rc=$?; echo "ERROR: command \"$BASH_COMMAND\" exited $rc" >&2' ERR
set -e
some_command_that_may_fail

Notes, limitations & portability

• ERR and BASH_COMMAND are Bash-specific; POSIX sh may not support ERR or $BASH_COMMAND. Use portable patterns for wide compatibility.
• Traps run in the current shell; they do NOT fire for signals delivered to commands in a separate process group or for commands running in subshells (e.g., within parentheses). A subshell's EXIT runs its own traps; parent traps won’t catch failures inside () unless you propagate status.
• In pipelines, earlier shells differ: in Bash set -o errtrace and set -o functrace help propagate ERR to functions/traps; use set -o pipefail to catch pipeline failures.
• Test scripts on target shell (dash, bash, ksh) and in container/CI environments to ensure behavior matches production.

Overview — why it matters
Portability prevents CI/CD flakiness and production failures when automation runs on different images (Debian, Alpine, macOS runners). Key problems: differing flags, output formats, and missing utilities.

Concrete differences

• stat
  • GNU: stat -c %Y file returns mtime in seconds.
  • BSD/macOS: stat -f %m file (different format and flags).
• sed -i
  • GNU: sed -i 's/x/y/' file
  • BSD/macOS: sed -i '' 's/x/y/' file (requires an explicit empty backup suffix).
• readlink
  • GNU: readlink -f path resolves symlinks recursively.
  • BSD/macOS: readlink lacks -f; realpath may or may not exist.
• date
  • GNU: date -d '2020-01-01' +%s
  • BSD/macOS: date -j -f '%Y-%m-%d' '2020-01-01' +%s (different flags).

Detection & fallback strategies

• Feature-detect at runtime (prefer) instead of OS sniffing.
• Use command existence and test flags/outputs (e.g., try stat -c on /dev/null).
• Provide small pure-sh shell fallbacks (Perl/python) when coreutils missing.
• Prefer bundling GNU coreutils in CI images or use containerized helpers.

Detect GNU utilities snippet

#!/usr/bin/env bash
# returns "gnu" or "bsd"
is_gnu() {
  cmd=$1
  case "$cmd" in
    stat) stat --version >/dev/null 2>&1 && return 0 || return 1 ;;
    sed) sed --version >/dev/null 2>&1 && return 0 || return 1 ;;
    readlink) readlink -f / >/dev/null 2>&1 && return 0 || return 1 ;;
    date) date --version >/dev/null 2>&1 && return 0 || return 1 ;;
    *) command -v "$cmd" >/dev/null 2>&1 && return 0 || return 1 ;;
  esac
}
# usage examples
if is_gnu stat; then
  mtime=$(stat -c %Y file)
else
  mtime=$(stat -f %m file)
fi

Best practice

• Write small abstraction functions (do_stat, do_sed_i, do_readlink, parse_date) used across scripts, test them in CI on all target runners, and document assumptions.

Solution (streaming, low-memory, robust)

Code:

#!/usr/bin/env bash
# Reads NDJSON from stdin or file arg, writes unique count to stdout,
# warnings about malformed lines to stderr.
file="${1:-/dev/stdin}"

# Parse line-by-line in Python (built-in json), emit user_id on stdout,
# warnings to stderr. Then use GNU sort's external merge to count uniques.
python3 -u -c '
import sys, json
fh = open(sys.argv[1]) if len(sys.argv)>1 and sys.argv[1]!="-" else sys.stdin
for i, line in enumerate(fh,1):
    line=line.rstrip("\n")
    if not line: 
        continue
    try:
        obj = json.loads(line)
        uid = obj.get("user_id")
        if isinstance(uid, str):
            print(uid)
    except Exception as e:
        print(f"Warning: malformed JSON at line {i}: {e}", file=sys.stderr)
' "$file" \
| LC_ALL=C sort -S 25% --temporary-directory=/tmp -T /tmp -u \
| wc -l

Why this approach

• Streaming: Python reads one line at a time — constant memory regardless of file size.
• Robust parsing: Python's json correctly handles escapes and Unicode; exceptions for malformed lines are caught and printed to stderr with line numbers.
• Low-memory unique counting: sort -u uses external merge (disk-backed) when data exceeds RAM; using LC_ALL=C speeds comparisons; -S controls memory footprint.
• Performance tips: run on multiple cores by splitting file (if needed) and merging, or increase -S if RAM available. Avoids per-line process fork overhead by using a single Python process rather than launching jq for every line.

Overview
Using set -euo pipefail is common defensive Bash for DevOps: stop on errors, fail on unset vars, and propagate pipeline failures. Below I explain each, caveats, and safer patterns.

-e (errexit)

• Behavior: exit script immediately if a simple command returns non‑zero.
• Use: prevents silent failures in deployments or provisioning scripts.
• Caveats: does not always trigger in conditional tests, lists (cmd1 || cmd2), pipelines (without pipefail), or commands in while/tests. Also subshell failures may not abort parent.
• Safer pattern: check return codes explicitly or use command || exit for expected non-zero.

-u (nounset)

• Behavior: exit when referencing undefined variables.
• Use: catches typos in env vars, secrets, config.
• Caveats: parameter expansions like ${VAR:-} can avoid exit; arrays and indirections need care.
• Safer pattern: use default expansions ${VAR?err} to give clear messages.

-o pipefail

• Behavior: pipeline exits with the rightmost non‑zero status (first failing command) instead of last command's status.
• Use: catches failures in any stage (e.g., grep | awk in CI checks).
• Caveats: not POSIX; in some shells pipelines in subshells still mask behavior.
• Safer pattern: prefer set -o pipefail and avoid constructs that swallow exits; or capture intermediate statuses via ${PIPESTATUS[@]} (Bash).

Examples

#!/usr/bin/env bash
set -euo pipefail
IFS=$'\n\t'
output=$(grep pattern file | awk '{print $1}')   # fails if grep fails

Best practices

• Set IFS, use trap 'echo "failed at $BASH_COMMAND"; exit 1' ERR for diagnostics.
• Use explicit checks for commands you expect may fail.
• Use ${VAR:-default} or ${VAR?Missing VAR} to control nounset behavior.
• Test scripts in dash/bash versions used in CI to ensure portability.