Bash and Shell Scripting Questions

Covers proficiency in writing reliable Bash and POSIX shell scripts to automate common Linux system administration and operational tasks. Topics include shell syntax, variables, parameter expansion, arrays, control flow such as conditionals and loops, functions and modular script design, input and output redirection and pipes, and use of core Unix utilities for text processing such as grep, sed, and awk. Emphasizes defensive and maintainable scripting practices including error handling, exit codes, trap usage, logging, input validation, command substitution, process and job management, debugging techniques, performance considerations, and secure handling of file and process permissions. Typical use cases include service management, backups, log parsing and rotation, user provisioning, monitoring checks, and small operational tooling.

MediumTechnical

33 practiced

Given a tab-separated metrics file with columns: epoch_seconds\thost\tmetric_name\tvalue, write an awk pipeline to compute the sum of 'requests' per host for the last hour (current time obtained via date +%s). The script should accept the TSV path as input and output 'host\trequests_sum'.

Sample Answer

Approach: compute the cutoff as current_time - 3600, then use awk to filter rows with epoch >= cutoff and metric_name == "requests", aggregate value per host, and print host\trequests_sum. Handle TSV, numeric values, and missing files.

Example script (save as sum_requests_last_hour.sh):

bash

#!/usr/bin/env bash
# Usage: ./sum_requests_last_hour.sh path/to/metrics.tsv

if [ -z "$1" ]; then
  echo "Usage: $0 path/to/metrics.tsv" >&2
  exit 2
fi

file="$1"
cutoff=$(date +%s)
cutoff=$((cutoff - 3600))

awk -F'\t' -v cutoff="$cutoff" '
# Columns: epoch_seconds \t host \t metric_name \t value
$1 ~ /^[0-9]+$/ && $1 >= cutoff && $3 == "requests" {
  # coerce value to number
  v = ($4 == "" ? 0 : $4 + 0)
  sums[$2] += v
}
END {
  for (h in sums) {
    printf("%s\t%g\n", h, sums[h])
  }
}
' "$file" | sort

Key points:- awk does streaming aggregation (low memory for many lines; memory grows with unique hosts).- Uses numeric coercion for value; non-numeric -> 0.- Filters by epoch and metric_name to limit processing.- Output is host\trequests_sum; final sort for deterministic order.

Edge cases:- Empty file -> no output.- Timezones irrelevant because epoch_seconds is UTC-based.- If values are floats, they’re summed correctly; use printf "%.0f" if integers needed.

Alternatives:- Use GNU awk’s mktime for parsing other time formats, or use join/cut + python for more complex validation.

EasyTechnical

44 practiced

Explain how here-documents (heredocs) work in shell scripts. Show an example that expands variables and one that prevents expansion (using <<'EOF'). Discuss use-cases like generating configuration files or embedding SQL, and explain precautions when heredoc content contains sensitive values like passwords.

Sample Answer

A heredoc is a shell I/O redirection that feeds a block of text to a command’s standard input. Syntax: <<WORD starts a literal until WORD appears on its own line. By default the shell performs expansions (variables, command substitution, escapes). Quoting the delimiter (<<'EOF' or <<"EOF") prevents expansion.

Example — variables expanded:

bash

#!/bin/bash
name="alice"
cat <<EOF > /tmp/greeting.txt
Hello, $name
Today is $(date +%Y-%m-%d)
EOF
# /tmp/greeting.txt contains expanded values

Example — prevent expansion (literal):

bash

#!/bin/bash
password='$SECRET_PASS'
cat <<'EOF' > /tmp/template.txt
user: $USER
password: $password
# these lines are written exactly as shown (no expansion)
EOF

Use-cases:- Generate config files (nginx, systemd unit files) dynamically during provisioning.- Embed SQL or multiline queries to pass to psql/mysql.- Supply multiline input to interactive tools in automation scripts.

Precautions with sensitive data:- Avoid placing secrets in world-readable files. Use secure file permissions (chmod 600) immediately after creation.- Prefer passing secrets via secure mechanisms: environment variables with restricted scope, in-memory pipes, secret managers (Vault), or tools’ native secret handling.- Beware of shell history and process listings — heredoc content might be exposed if command captures it; writing to disk should be minimized.- If you must use heredoc for secrets, use <<'EOF' to avoid accidental expansion and ensure the receiving command reads from a pipe or fd with restricted permissions, then immediately delete traces.

MediumTechnical

73 practiced

Given a common log format line such as: 127.0.0.1 - - [2025-01-01T12:00:00Z] "GET /api/v1/users?id=1 HTTP/1.1" 200 1234write an awk script or one-liner that counts unique client IPs per request path and prints the top 10 paths by unique-IP count. Explain memory considerations and approaches for very large logs (streaming vs approximation).

Sample Answer

Approach: extract client IP and request path (drop query string), build a set of IPs per path, then count unique IPs and print top 10. Pure awk keeps per-path associative arrays of IPs.

Awk one-liner (GNU awk for nested arrays):

bash

awk '
{
  ip=$1
  # extract request field like: "GET /api/v1/users?id=1 HTTP/1.1"
  match($0, /"[^"]+"/, req)
  split(req[0], parts, " ")
  path = parts[2]
  sub(/\?.*$/, "", path)         # drop query string
  seen[path,ip]=1               # record ip seen for path
  paths[path]=1
}
END {
  for (p in paths) {
    cnt=0
    for (k in seen) {
      split(k, a, SUBSEP)
      if (a[1]==p) cnt++
    }
    counts[p]=cnt
  }
  # print top 10 by unique IPs
  n=0
  for (p in counts) {
    n++
    arr[n]=counts[p] SUBSEP p
  }
  # simple sort by count descending
  asort(arr)
  for (i=n; i>n-10 && i>0; i--) {
    split(arr[i], t, SUBSEP)
    printf "%d\t%s\n", t[1], t[2]
  }
}
' access.log

Memory considerations:- This awk solution stores one entry per unique (path, ip). If logs are large or cardinality is high (millions of IPs × many paths) memory will explode.- For long-running or high-volume processing prefer streaming/aggregation strategies.

Scalable alternatives:- Stream and shard by path: use awk to emit "path\tip", then external sort/uniq per path: awk ... | sort -T /tmp -S 50% | uniq | cut -f1 | sort | uniq -c | sort -rn | head -10- Approximate counting: use HyperLogLog (e.g., using Redis HLL, bloom/hll libs) to track unique IPs per path with bounded memory and merge across shards.- MapReduce / Spark: map(path, ip) → reduce unique counts per path; good for very large datasets.- Hybrid: maintain exact counts for hot paths, HLL for low-frequency ones.

Trade-offs: exact results vs memory; HLL gives small error but fixed footprint; external sort trades disk I/O for bounded memory. For SREs, pick based on required accuracy, latency, and resource constraints.

HardTechnical

33 practiced

A script uses 'set -euo pipefail' and runs a pipeline 'cmd1 | cmd2 | cmd3'. Sometimes a failure in cmd2 does not cause the script to exit. Explain why that can happen in Bash and demonstrate code changes (or a pattern) so that any command in the pipeline failing will cause the script to exit. Show how to inspect the exit status of each pipeline element.

Sample Answer

Brief explanation:Bash runs each pipeline element in its own process; by default the pipeline's exit status is that of the last command. set -e exits on non-zero in many contexts but won't catch a non-last failing command unless you also enable pipefail. Use set -o pipefail so the pipeline returns a failure if any element fails. To inspect each element's status, use the PIPESTATUS array (Bash) immediately after the pipeline.

Example demonstrating problem and fix:

bash

#!/usr/bin/env bash
set -euo pipefail   # recommended for scripts

# Example pipeline where middle command fails
# cmd1: produces lines, cmd2: returns non-zero, cmd3: consumes
false_cmd() { false; }   # simulates failing command
true_cmd()  { true;  }

# Problem reproduced: without pipefail this may succeed if last command returns 0
# (For demonstration, run with/without 'set -o pipefail')
echo "a" | ( false_cmd ) | true_cmd
# Immediately after the pipeline inspect individual statuses:
echo "PIPESTATUS: ${PIPESTATUS[@]}"   # bash-only: indexes 0,1,2 map to each command
# If pipefail is enabled, the script will have already exited due to the failure of middle command.

Pattern to guarantee exit on any pipeline failure:- Keep set -euo pipefail at top.- If you need to capture outputs, avoid subshells that swallow exit codes.- After a pipeline, check "${PIPESTATUS[@]}" to see each element's exit code and act accordingly.

Robust wrapper example that logs and exits with first non-zero code:

bash

#!/usr/bin/env bash
set -euo pipefail

run_and_check() {
  # run pipeline
  cmd1 | cmd2 | cmd3
  # capture statuses (must check immediately)
  local -a sts=("${PIPESTATUS[@]}")
  for i in "${!sts[@]}"; do
    if [[ "${sts[i]}" -ne 0 ]]; then
      echo "Pipeline element $i failed with status ${sts[i]}" >&2
      exit "${sts[i]}"
    fi
  done
}

# Example commands
cmd1(){ printf '%s\n' a; }
cmd2(){ false; }    # will fail
cmd3(){ cat; }

run_and_check

Notes / pitfalls:- PIPESTATUS is Bash-specific and only valid immediately after the pipeline.- If you use a subshell to run a pipeline (e.g., in parentheses), that subshell's failure might be masked depending on context.- Ensure /bin/sh is actually bash-compatible or invoke bash explicitly (#!/usr/bin/env bash).- For POSIX shells lacking PIPESTATUS, capture exit codes by splitting pipeline and using temporary files or process substitution, e.g., run commands in sequence and collect statuses via wait for backgrounded processes.

HardSystem Design

36 practiced

Design a portable 'shell-lib.sh' helper library for SRE scripts that provides logging, config parsing (KEY=VALUE style), acquire/release lock primitives, retry wrapper, and assertions. Describe the API (function names and expected behavior), how to keep it POSIX-compliant, how to test it, and distribution strategy (vendoring vs packaging).

Sample Answer

Overview: shell-lib.sh is a compact, POSIX-compliant helper library for SRE scripts providing: logging, KEY=VALUE config parsing, advisory lock primitives, retry wrapper, and assertions. Aim: <300 lines, no bash-only features, no arrays, portable /bin/sh.

API (function name → behavior)- log_info "msg" — writes timestamped INFO to stderr, no newline trimming- log_warn "msg"- log_error "msg" — exits? no, just logs; use assert_* to exit- log_debug "msg" — enabled when SHELL_LIB_DEBUG=1- cfg_load /path/to/file — parses KEY=VALUE (ignores blank/comments), exports keys (valid names only), returns 0/1- cfg_get KEY — prints value (or returns non-zero if unset)- lock_acquire NAME timeout_seconds — creates/flock-style lock under ${SHELL_LIB_LOCKDIR:-/var/lock}/NAME.pid using mkdir+pidfile loop (atomic), waits up to timeout; prints owner PID; returns 0 on success- lock_release NAME — releases lock only if owner PID matches; safe no-op otherwise- with_lock NAME timeout -- command... — convenience: acquire, run command, release (trap safe)- retry N DELAY CMD... — retries CMD up to N times with DELAY (secs) backoff; returns last exit code- assert_true CMD || assert_fail "msg" — run CMD, on failure log_error and exit 2- assert_file_exists PATH — exits non-zero with message

POSIX compliance strategies- Use /bin/sh constructs: avoid arrays, process substitution, [[ ]], let, (( )), local in functions is OK in many shells but avoid — instead prefix temp vars with __sl_- Use printf, date +%Y-%m-%dT%H:%M:%S%z for timestamps (fallback if not available)- Use mkdir lock pattern (atomic) instead of flock- Use POSIX test [ ] and read without -r? use IFS= read -r to preserve backslashes- Avoid here-strings; use heredocs or printf

Testing- Unit tests with shunit2 or bats-core (preferred: bats for readability). Tests: - Logging: verify output format via capturing stderr - cfg_load: valid, invalid lines, export behavior, injection attempts - lock: concurrent processes (spawn background subshells) ensure mutual exclusion and timeout - retry: simulate transient failure commands (helper script that fails first N times) - assertions: ensure correct exit codes and messages- Integration tests in CI matrix on ubuntu, alpine (/bin/sh = dash / ash) and macOS- Fuzz tests: malformed configs, long names, permission-denied lockdir

Distribution strategy- Vendoring: include shell-lib.sh in repo for critical operational scripts to guarantee version & availability (recommended for infra repos)- Packaging: provide a small tarball + checksum and optionally a system package (deb/rpm) if central deployment desired- Versioning: semver, install script /usr/local/lib/shell-lib.d/ with symlinked version; support SHELL_LIB_PATH override- CI: Sign artifacts, run lint (shellcheck with POSIX mode), run tests, publish release assets- Backward compatibility: keep API stable, add feature flags (SHELL_LIB_DEBUG) for opt-ins

Security and best practices- Validate config keys before export (allow [A-Z_][A-Z0-9_]*)- Avoid eval; parse lines safely (split on first '=')- Use restrictive umask when creating lockdir/pidfile- Use traps to release locks on exit/ERR signals

This design balances portability, safety, testability, and easy adoption by SRE teams.

Unlock Full Question Bank

Get access to hundreds of Bash and Shell Scripting interview questions and detailed answers.

Join thousands of developers preparing for their dream job.