Shell Scripting and Automation Questions

Direct answer

On Alpine, /bin/sh is the ash shell from BusyBox (one small program that bundles minimal versions of the standard utilities), not Bash. It follows POSIX (the standard that defines portable shell behaviour) plus some extras. A script that begins #!/bin/sh but uses Bash-only syntax (arrays, ${var^^}, here-strings <<<, (( )), brace ranges like {1..3}) fails there, or worse, silently behaves differently. Find them by running the script with the target sh in continuous integration (CI) and by running ShellCheck in sh mode. Then decide: rewrite to POSIX when the script is short glue that has to run in minimal images; require Bash (change the shebang to #!/usr/bin/env bash and install it) when the logic really needs arrays or Bash features.

Why it works on the laptop

The laptop's /bin/sh is usually more permissive. On macOS and on Red Hat style systems /bin/sh is Bash itself (I checked an AlmaLinux 9 image: /bin/sh is bash, and the constructs I probed matched Bash), and Bash in sh mode still accepts most Bash syntax. Debian and Ubuntu use dash (the Debian Almquist shell, a small, fast POSIX shell), which is stricter. Alpine uses BusyBox ash (BusyBox's version of the same Almquist shell), whose feature set depends on how that BusyBox was built.

What crept in

This script has the usual suspects:

#!/bin/sh
set -eu
hosts=(web1 web2 db1)
env_name=${1:-prod}
if [[ $env_name == prod* ]]; then
  echo "production run"
fi
for h in "${hosts[@]}"; do
  echo "checking $h"
done
echo "${env_name^^} done" 
count=$(( ${#hosts[@]} * 2 ))
echo "total checks: $count"
echo -e "tab:\tseparated"

set -eu makes the script stop at the first failing command (-e) and treat any use of an unset variable as an error (-u); both are POSIX.

$ sh deploy-check.sh
deploy-check.sh: line 3: syntax error: unexpected "("
exit=2

BusyBox stops at the first syntax error (the array on line 3), so one message can hide other problems. ShellCheck in sh mode reports the Bash-only constructs it recognises (the SC3xxx codes are its "not defined in POSIX sh" family):

$ shellcheck -s sh -f gcc deploy-check.sh
deploy-check.sh:3:7: warning: In POSIX sh, arrays are undefined. [SC3030]
deploy-check.sh:5:4: warning: In POSIX sh, [[ ]] is undefined. [SC3010]
deploy-check.sh:5:17: warning: In POSIX sh, == in place of = is undefined. [SC3014]
deploy-check.sh:8:11: warning: In POSIX sh, array references are undefined. [SC3054]
deploy-check.sh:11:7: warning: In POSIX sh, case modification is undefined. [SC3059]
deploy-check.sh:14:6: warning: In POSIX sh, echo flags are undefined. [SC3037]
$ shellcheck -s sh deploy-check-posix.sh; echo "exit=$?"
exit=0

Each line reads file:line:column: severity: message [code]. The report has six entries, but line 12, count=$(( ${#hosts[@]} * 2 )), also uses array syntax and ShellCheck does not list it here (checked by running it alone: it reports only an unused variable and an unassigned hosts). A linter is a net with holes, so a clean run does not prove the script is POSIX: run the script itself under the target shell as well.

Measure, do not assume: constructs against three shells

probe.sh runs one snippet per construct under a named shell and compares the output with what Bash prints. The constructs, in plain words:

• a=(x y) is an array, a variable holding a numbered list; [[ ]] is Bash's extended test (pattern matching); (( )) is an arithmetic command; {1..3} is brace expansion (the shell writes out 1 2 3).
• echo -e makes echo interpret backslash escapes such as \t; ${v^^} upper-cases a value; ${v//a/X} replaces every a with X; ${v:1:2} takes two characters starting at offset 1.
• <<< (here-string) feeds a string to a command's standard input; <(cmd) (process substitution) presents a command's output as if it were a file; source file runs a file's commands in the current shell (POSIX spells it .).
• function f { ...; } is the Bash keyword form of defining a function; local makes a variable private to a function; set -o pipefail makes a pipeline fail if any stage fails, not only the last; $'a\tb' is ANSI-C quoting, which turns \t into a real tab.

How the helper works: probe label want snippet runs $sh_bin -c "$snippet" (-c means "take the commands from this string"), merges error messages into the output with 2>&1, keeps only the first line with head -n 1, and captures it with $(...). If that equals want (what Bash prints) the verdict is same, otherwise DIFFERENT; printf '%-18s %-9s %s\n' pads the three columns. tab=$(printf '\t') stores a tab character so tab results can be compared.

The last probe line looks odd because the snippet contains single quotes. A single quote cannot appear inside a single-quoted string, so the line splices four pieces together: 'printf "%s\n" $', then "'" (a quote character in double quotes), then 'a\tb', then "'" again. The snippet that reaches the shell is printf "%s\n" $'a\tb'.

# usage: probe.sh SHELL ; prints what each construct does under SHELL and whether it matches bash
sh_bin=$1
tab=$(printf '\t')
probe() {
  label=$1; want=$2; snippet=$3
  got=$($sh_bin -c "$snippet" 2>&1 | head -n 1)
  if [ "$got" = "$want" ]; then verdict=same; else verdict=DIFFERENT; fi
  printf '%-18s %-9s %s\n' "$label" "$verdict" "$got"
}
probe 'array'           'y'       'a=(x y); echo ${a[1]}'
probe '[[ ]]'           'match'   '[[ abc == a* ]] && echo match'
probe '(( ))'           '2'       'i=1; ((i++)); echo $i'
probe 'brace expansion' '1 2 3'   'echo {1..3}'
probe 'echo -e'         "a${tab}b" 'echo -e "a\tb"'
probe '${v^^}'          'ABC'     'v=abc; echo ${v^^}'
probe '${v//a/X}'       'bXnXnX'  'v=banana; echo ${v//a/X}'
probe '${v:1:2}'        'el'      'v=hello; echo ${v:1:2}'
probe 'here-string'     'hi'      'cat <<< hi'
probe 'process subst'   'hi'      'cat <(echo hi)'
probe 'source'          '1'       'echo x=1 > /tmp/s.sh; source /tmp/s.sh; echo $x'
probe 'function keyword' 'hi'     'function f { echo hi; }; f'
probe 'local'           '1'       'f() { local x=1; echo $x; }; f'
probe 'pipefail'        '1'       'set -o pipefail; false | true; echo $?'
probe 'dollar-quote'    "a${tab}b" 'printf "%s\n" $'"'"'a\tb'"'"

The reference run under Bash matches every expectation. In the two tables after it, DIFFERENT followed by an error message means the construct failed outright. DIFFERENT with no error (brace expansion on both shells, and echo -e and $'...' on dash) means the shell ran the command and printed something else, which is the dangerous case because nothing stops the script. same means the construct works in that shell.

$ sh probe.sh bash   # reference
array              same      y
[[ ]]              same      match
(( ))              same      2
brace expansion    same      1 2 3
echo -e            same      a	b
${v^^}             same      ABC
${v//a/X}          same      bXnXnX
${v:1:2}           same      el
here-string        same      hi
process subst      same      hi
source             same      1
function keyword   same      hi
local              same      1
pipefail           same      1
dollar-quote       same      a	b

Alpine 3 (BusyBox v1.37.0 here):

$ busybox | head -n 1   # the version banner; probe.sh itself does not print it
BusyBox v1.37.0 (2026-01-10 15:38:28 UTC) multi-call binary.
$ sh probe.sh sh   # Alpine 3, BusyBox ash
array              DIFFERENT sh: syntax error: unexpected "("
[[ ]]              same      match
(( ))              DIFFERENT sh: i++: not found
brace expansion    DIFFERENT {1..3}
echo -e            same      a	b
${v^^}             DIFFERENT sh: syntax error: bad substitution
${v//a/X}          same      bXnXnX
${v:1:2}           same      el
here-string        DIFFERENT sh: syntax error: unexpected redirection
process subst      same      hi
source             same      1
function keyword   same      hi
local              same      1
pipefail           same      1
dollar-quote       same      a	b

Ubuntu 24.04 (dash):

$ sh probe.sh sh   # Ubuntu 24.04, dash
array              DIFFERENT sh: 1: Syntax error: "(" unexpected
[[ ]]              DIFFERENT sh: 1: [[: not found
(( ))              DIFFERENT sh: 1: i++: not found
brace expansion    DIFFERENT {1..3}
echo -e            DIFFERENT -e a	b
${v^^}             DIFFERENT sh: 1: Bad substitution
${v//a/X}          DIFFERENT sh: 1: Bad substitution
${v:1:2}           DIFFERENT sh: 1: Bad substitution
here-string        DIFFERENT sh: 1: Syntax error: redirection unexpected
process subst      DIFFERENT sh: 1: Syntax error: "(" unexpected
source             DIFFERENT sh: 1: source: not found
function keyword   DIFFERENT sh: 1: Syntax error: "}" unexpected
local              same      1
pipefail           DIFFERENT sh: 1: set: Illegal option -o pipefail
dollar-quote       DIFFERENT $a\tb

Three lessons from the data. First, BusyBox 1.37 accepted [[ ]], ${v//a/X}, ${v:1:2}, source, pipefail, echo -e, process substitution, the function keyword and local, but rejected arrays, (( )), brace expansion, ${v^^} and here-strings, so "Alpine has no Bash features" is too strong and "it works on Alpine" depends on the BusyBox version. Second, brace expansion and (( )) fail silently or with a confusing message: echo {1..3} prints the literal {1..3}, and ((i++)) prints i++: not found (ash reads ((i++)) as a subshell inside a subshell running a command named i++, so the status is 127). Without set -e the script carries on past it; with set -e it stops there. Both were checked on BusyBox ash. Third, dash rejects most of the list and differs on echo -e (prints -e), so testing on both catches more than either alone.

Finding them systematically

1. Run the script under the real target shell in the real image (docker run --rm -v "$PWD":/w alpine:3 sh /w/script.sh), including every code path.
2. Run shellcheck -s sh script.sh; its SC3xxx warnings are exactly the "undefined in POSIX sh" constructs. Debian's checkbashisms is another tool for this (I did not run it here).
3. Test under dash as well, because it is the strictest common /bin/sh.
4. Note that bash --posix is not a POSIX checker: it still accepts many extensions.

The POSIX rewrite

#!/bin/sh
set -eu
hosts="web1 web2 db1"
env_name=${1:-prod}
case $env_name in
  prod*) echo "production run" ;;
esac
count=0
for h in $hosts; do
  echo "checking $h"
  count=$((count + 2))
done
echo "$(printf '%s' "$env_name" | tr '[:lower:]' '[:upper:]') done"
echo "total checks: $count"
printf 'tab:\tseparated\n'

$ sh deploy-check-posix.sh prod   # Alpine
production run
checking web1
checking web2
checking db1
PROD done
total checks: 6
tab:	separated
exit=0

Arrays became a space-separated list (safe here because the words contain no spaces), [[ == prod* ]] became case, ${v^^} became tr, (( )) became $(( )), and echo -e became printf.

Rewrite or require Bash?

Recommendation: rewrite to POSIX if the script is under roughly a screen, has to run in minimal images (FROM alpine, distroless images that hold only the application and its libraries with no shell at all, and init containers, the short setup containers that run before the main one in Kubernetes) and needs only simple lists; require Bash when it needs arrays of values with spaces, associative arrays, [[ ]] regex matching or mapfile, because emulating those in POSIX is where bugs come from. Requiring Bash costs apk add bash (an extra package and a slightly bigger image) plus a shebang that says what you mean. Whichever you pick, make the shebang match the dialect, put shellcheck -s sh or -s bash in CI, and run the script in the target image in CI so a regression fails the build, not the deploy.

Direct answer

A pipeline of awk, sed and jq is the right tool when the job is one streaming pass over JSON Lines (one JSON document per line) that filters, projects and counts, keeps small state, and is owned by people who live in a shell. Replace it with a small program (Python, Go, Rust) or an embedded query engine (a database library that runs inside your own process, with no server to install) such as DuckDB when state grows with the data, when records must be validated or joined, or when the job has become something that needs unit tests and error handling per record. What you keep, you speed up by moving less data through fewer, longer-lived processes: filter early, run one jq per file instead of one per line, split work by file across cores, and fix the locale.

When the shell pipeline is the right tool

• The work is a single pass: each line is judged on its own, and the answer is a filter, a projection or a counter.
• State is bounded: an awk array keyed by something with a few thousand distinct values is fine.
• The code is short and read by operators; a five-line pipeline that ships today beats a service that ships next quarter.
• Throughput is limited by disk or decompression, which no language choice improves.

When to replace it

DuckDB deserves a specific mention because it removes most of the glue: it reads JSON Lines directly and runs SQL on it, using all cores.

The table's first row is the one a shell cannot work around, so here is what it looks like when measured. The same awk program, mem.awk, counts lines per key over a 2,000,000-line file, once with 1,000 distinct keys and once with 2,000,000 distinct keys (gawk 5.2.1 in an Ubuntu 24.04 container; VmHWM is the process's peak resident memory, read from /proc/self/status):

# mem.awk: count lines per key, then report peak resident memory.
{ n[$1]++ }
END {
    k = 0
    for (key in n) k++
    while ((getline line < "/proc/self/status") > 0)
        if (line ~ /^VmHWM/) hwm = line
    close("/proc/self/status")
    printf "%d keys held,\t%s\n", k, hwm
}

echo "2,000,000 lines, 1000 distinct keys:"
gawk 'BEGIN { for (i = 0; i < 2000000; i++) print "key" (i % 1000) }' | gawk -f mem.awk
echo "2,000,000 lines, 2000000 distinct keys:"
gawk 'BEGIN { for (i = 0; i < 2000000; i++) print "key" i }' | gawk -f mem.awk

2,000,000 lines, 1000 distinct keys:
1000 keys held,	VmHWM:	    3368 kB
2,000,000 lines, 2000000 distinct keys:
2000000 keys held,	VmHWM:	  620880 kB

The peak figures move by a few hundred kB from run to run.

The line count never changed; the distinct-key count did, and memory went from about 3 MB to about 620 MB. At 20 million distinct keys the same program would need roughly ten times that (an extrapolation, not a measurement), which is where the table says to move to something that can spill to disk.

Demonstrations

Three small files a.jsonl, b.jsonl and c.jsonl hold requests; b.jsonl ends with a line torn mid-write and one line mentions /pay only in a free-text note. The script that follows was run in a Debian container with jq 1.7 and GNU findutils. Two jq options appear in it: -R reads each input line as plain text instead of as JSON, and -r prints strings without their quotes. In the filter, fromjson then parses that text as JSON, and the trailing ? in fromjson? suppresses the error, so a line that is not valid JSON produces no output instead of stopping the run.

set -euo pipefail
mkdir -p data && cd data
cat > a.jsonl <<'J'
{"ts":"2024-05-01T00:00:01Z","path":"/pay","status":500,"ms":120}
{"ts":"2024-05-01T00:00:02Z","path":"/pay","status":200,"ms":30}
{"ts":"2024-05-01T00:00:03Z","path":"/cart","status":503,"ms":900}
J
cat > b.jsonl <<'J'
{"ts":"2024-05-01T00:01:01Z","path":"/pay","status":502,"ms":80}
{"ts":"2024-05-01T00:01:02Z","path":"/cart","status":200,"ms":15,"note":"retried via /pay"}
{"ts":"2024-05-01T00:01:03Z","path":"/pay","stat
J
cat > c.jsonl <<'J'
{"ts":"2024-05-01T00:02:01Z","path":"/cart","status":500,"ms":70}
{"ts":"2024-05-01T00:02:02Z","path":"/pay","status":500,"ms":65}
J
cd ..

echo "--- plain jq stops at the torn line in b.jsonl:"
jq -c 'select(.status >= 500) | .path' data/b.jsonl || echo "jq exit status $?"

echo "--- tolerant parse, one jq per file, 2 files at a time, then one merge step:"
export LC_ALL=C
printf '%s\0' data/*.jsonl |      # file names separated by NUL bytes, safe for odd names
  xargs -0 -n1 -P2 jq -Rr 'fromjson? | select(.status >= 500) | .path' |
  sort | uniq -c

echo "--- grep -F prefilter for /pay: cheap, but jq still decides:"
grep -hF '/pay' data/*.jsonl | wc -l
grep -hF '/pay' data/*.jsonl | jq -Rr 'fromjson? | select(.path == "/pay" and .status >= 500) | .path' | wc -l

--- plain jq stops at the torn line in b.jsonl:
"/pay"
jq: parse error: Unfinished string at EOF at line 4, column 0
jq exit status 5
--- tolerant parse, one jq per file, 2 files at a time, then one merge step:
      2 /cart
      3 /pay
--- grep -F prefilter for /pay: cheap, but jq still decides:
6
3

The shell version silently drops the torn line, so the output alone does not show that a record was lost. Count the rejects as a separate step, jq -Rr 'try (fromjson | empty) catch "x"' data/*.jsonl | wc -l, which prints one line per unparsable record: it printed 1 here, against cat data/*.jsonl | wc -l giving 8 lines in all, and the job should fail or alert when that count is not zero (or exceeds a budget you agreed in advance).

The same count in Python's standard library (skips undecodable lines) and in DuckDB (read_json with format='newline_delimited' and ignore_errors=true, DuckDB 1.5.6) gives the identical result:

import json, glob, collections
c = collections.Counter()
for p in sorted(glob.glob("data/*.jsonl")):
    with open(p) as f:
        for line in f:
            try: r = json.loads(line)
            except json.JSONDecodeError: continue
            if r.get("status", 0) >= 500: c[r["path"]] += 1
for k, v in sorted(c.items()): print(v, k)

select path, count(*) n
from read_json('data/*.jsonl', format='newline_delimited', ignore_errors=true)
where status >= 500 group by path order by path;

Both print /cart 2 and /pay 3.

How to speed up what you keep

The first three items change how much work is done (how much data is read, how many processes are started, how many cores are used); items 4 to 8 refine the result around them.

1. Read less. Filter on the cheapest test first. grep -F on a fixed string does much less work per line than a JSON parser, but it can only discard lines, never accept them: in the demo it passes 6 lines and jq still rejects 3 of them. Project only the needed fields (jq -r '[.ts,.path]|@tsv') so later stages handle short lines.
2. One process per file, never per line. A while read loop that starts jq for each line starts one process per line; for a 4 GB file of 200-byte lines (an assumed size, 4,000,000,000 / 200) that is 20 million process launches. The xargs -0 -n1 -P2 jq ... form starts one per file (-n1 hands jq one file name per run, -P2 keeps two runs going at once).
3. Parallelise by file. xargs -P N (-P is the maximum number of simultaneous processes) over a NUL-separated file list scales until disk or decompression throughput is the limit; pick N by measuring on a sample, not by assuming the core count.
4. Reduce in two stages. Let each worker emit partial counts, then merge them with one sort | awk step. Per-file partial results also make a failed file retryable.
5. Set LC_ALL=C for grep, sort and awk when the data is ASCII-safe: byte comparison instead of locale collation. Do not do it if you rely on locale-aware sorting of non-ASCII text.
6. Never jq -s (slurp) a large file. Slurp reads all input into one array first; memory grows with the file. Stream with -c per line.
7. Tune sort with -S (buffer size), -T (temporary directory on a big disk) and --parallel=N when a sort over the whole output is unavoidable.
8. Convert once if the data is queried repeatedly. If the same files are scanned daily with different questions, convert them once to a columnar format and query that. A columnar format such as Parquet stores each field's values together and compressed, so a question that touches 2 of 20 fields reads about a tenth of the bytes (assuming similar field sizes; this is an illustration, not a measurement), whereas JSON Lines must be read whole every time. The shell suits one-off questions.

Deciding in practice

Measure on one representative file first (record bytes read, wall time and peak memory with /usr/bin/time -v), change one thing at a time, and keep the shell version if the next step up only buys a factor you do not need. Promote it when a failure would be silent or when someone asks to change it and nobody dares.

Direct answer

Chain tar | zstd | age | split so the only bytes that ever reach disk are already-encrypted chunks, and upload each chunk as it is produced with retries. A single unbroken stream cannot resume after a network failure, so chunking is what makes the pipeline survive failures: a failed chunk is retried on its own, and the whole job restarts only if a chunk exhausts its retries. You know the backup is complete and restorable from three things: a manifest of chunk checksums that is uploaded last (no manifest means incomplete), a chunk-by-chunk checksum verification, and a periodic real restore of the whole archive compared against hashes of the source files.

Design

age is a modern file-encryption tool: you encrypt to a public key (the recipient) and decrypt with a private key stored in a file (the identity file), with no passphrase prompts in the pipeline. The spool is the temporary local folder that holds one chunk at a time, and a manifest is a list of the chunk file names with their checksums. Compression comes before encryption because encrypted data does not compress.

flowchart LR
  A[source dir] --> B[tar -cf -]
  B --> C[zstd]
  C --> D[age -r recipient]
  D --> E[split --filter]
  E --> F[chunk spool: one ciphertext chunk]
  F -->|retry up to 5x| G[(remote storage)]
  F --> H[chunks.sha256]
  H -->|uploaded last| G

The script

UPLOAD_CMD is the transport and receives LOCAL_FILE DESTINATION. The default cp writes to a mounted path. For S3 it could be aws s3 cp --only-show-errors, because that command takes the same two arguments and documents --only-show-errors. That call, and S3 itself, were not run here: the demo's remote is a directory.

#!/usr/bin/env bash
# stream-backup.sh backup SRC_DIR DEST PREFIX AGE_RECIPIENT
# stream-backup.sh verify DEST PREFIX AGE_IDENTITY_FILE RESTORE_DIR
# DEST is a path or URL prefix; UPLOAD_CMD copies one local file to DEST/PREFIX/name.
set -euo pipefail

UPLOAD_CMD=${UPLOAD_CMD:-cp}          # for S3, for example: "aws s3 cp --only-show-errors"
CHUNK=${CHUNK:-256M}
BACKOFF=${BACKOFF:-2}                 # seconds multiplied by the attempt number

upload_chunk() {                      # run by split once per chunk; $FILE is the chunk name
  local f="$spool/$FILE" try
  cat > "$f"                          # ciphertext only: tar, zstd and age already ran upstream
  sha256sum -- "$f" | { read -r h _; printf '%s  %s\n' "$h" "$FILE" >> "$spool/chunks.sha256"; }
  for try in 1 2 3 4 5; do
    if $UPLOAD_CMD "$f" "$dest/$prefix/$FILE"; then rm -f -- "$f"; return 0; fi
    echo "upload of $FILE failed (attempt $try), retrying" >&2
    sleep $(( try * BACKOFF ))
  done
  echo "giving up on $FILE" >&2; return 1
}
export -f upload_chunk

backup() {
  local src=$1 recipient=$4
  dest=$2; prefix=$3                  # globals on purpose: the chunk filter reads them
  spool=$(mktemp -d); trap 'rm -rf "$spool"' EXIT
  export spool dest prefix UPLOAD_CMD BACKOFF
  : > "$spool/chunks.sha256"

  tar -C "$src" -cf - . | zstd -q -T0 | age -r "$recipient" \
    | SHELL=$BASH split -a 4 -b "$CHUNK" --filter='bash -c upload_chunk' - part.

  # Hashes of the source files, encrypted, so a restore can be compared with the original
  ( cd "$src" && find . -type f -print0 | sort -z | xargs -0 sha256sum ) \
    | age -r "$recipient" > "$spool/files.sha256.age"
  $UPLOAD_CMD "$spool/files.sha256.age" "$dest/$prefix/files.sha256.age"

  # The manifest goes up LAST. No manifest means the backup is not complete.
  sort -k2 "$spool/chunks.sha256" > "$spool/MANIFEST.sha256"
  $UPLOAD_CMD "$spool/MANIFEST.sha256" "$dest/$prefix/MANIFEST.sha256"
  echo "backup complete: $(wc -l < "$spool/MANIFEST.sha256") chunk(s)"
}

verify() {
  local dir=$1/$2 key=$3 out=$4
  [[ -s $dir/MANIFEST.sha256 ]] || { echo "no MANIFEST: backup is incomplete" >&2; return 1; }
  ( cd "$dir" && sha256sum --check --quiet MANIFEST.sha256 )       # every listed chunk present and intact
  awk '{print $2}' "$dir/MANIFEST.sha256" | ( cd "$dir" && xargs cat ) \
    | age -d -i "$key" | zstd -d -q | tar -x -C "$out"             # decrypt, decompress, unpack
  age -d -i "$key" < "$dir/files.sha256.age" | ( cd "$out" && sha256sum --check --quiet )
  echo "verify OK: restored tree matches the source hashes"
}

case ${1:-} in
  backup) shift; backup "$@" ;;
  verify) shift; verify "$@" ;;
  *) echo "usage: see header" >&2; exit 2 ;;
esac

How the pipeline hands each chunk to the upload function

• - as a file name. In tar -cf - . the - means "write the archive to standard output", and in split ... - part. it means "read from standard input". That is what lets the stages join with pipes and keeps the plaintext off disk.
• What split does. It cuts its input into pieces of -b bytes. Normally it writes each piece to a file named by the prefix (part.) plus a suffix of -a 4 letters that it generates itself: aaaa, aaab, aaac and so on. That is where part.aaaa and part.aaab come from.
• --filter='CMD'. Instead of writing each piece to a file, split starts CMD once per piece with the piece on its standard input, and sets the environment variable FILE to the name the piece would have had. So FILE is set by split, not by the script, and cat > "$spool/$FILE" inside the function saves the piece under that name.
• Why SHELL=$BASH, bash -c and export -f. split runs the filter through the shell named in $SHELL, so SHELL=$BASH makes sure that shell is Bash. The filter bash -c upload_chunk starts a fresh Bash that must already know the function. export -f upload_chunk puts the function's definition into the environment, which Bash children read on startup. The variables the function uses (spool, dest, prefix, UPLOAD_CMD, BACKOFF) are exported for the same reason. Without the export the child shell reports command not found.

The mechanism on its own (Ubuntu 24.04 container):

mkdir d; echo hi > d/a
tar -C d -cf - . | tar -tf -
show() { echo "filter got $(wc -c) bytes, FILE=$FILE"; }
export -f show
head -c 25 /dev/zero | SHELL=$BASH split -a 4 -b 10 --filter='bash -c show' - part.
ls part.* 2>/dev/null | wc -l | sed 's/^/files written by split itself: /'
f2() { echo "never runs"; }
head -c 5 /dev/zero | SHELL=$BASH split -b 10 --filter='bash -c f2' - p. ; echo "status=$?"

./
./a
filter got 10 bytes, FILE=part.aaaa
filter got 10 bytes, FILE=part.aaab
filter got 5 bytes, FILE=part.aaac
files written by split itself: 0
bash: line 1: f2: command not found
split: with FILE=p.aa, exit 127 from command: bash -c f2
status=127

Reading it: the first tar writes an archive to standard output and the second lists it from standard input. A 25-byte stream becomes three pieces of 10, 10 and 5 bytes, and the filter sees the names part.aaaa, part.aaab and part.aaac. split itself wrote no files, because the filter received the data instead. A function that was not exported (f2) cannot be found by the child Bash, and split stops with the filter's exit status.

Worked example

Run in an Ubuntu container with age, zstd and gawk installed. The source holds a 150,000-line pseudo-random CSV (seeded with srand(1)) and a file with spaces in its name; chunk size is 1 MiB so there is more than one chunk. A stand-in transport flaky-cp fails once for the second chunk, like a dropped connection. The test then truncates the last chunk, removes the manifest to look like an interrupted run, and runs a backup against a transport that always fails.

set -euo pipefail
apt-get update -qq >/dev/null && apt-get install -y -qq age zstd gawk >/dev/null 2>&1
mkdir -p /data/sub /remote /restore
awk 'BEGIN{srand(1); for(i=0;i<150000;i++) printf "%d,%.6f,%d\n", i, rand(), int(rand()*1e9)}' > /data/big.csv
echo hello > '/data/sub/a file with spaces.txt'
age-keygen -o /root/key.txt 2>/dev/null
R=$(age-keygen -y /root/key.txt)
# transport that fails the first attempt for the second chunk, like a dropped connection
cat > /usr/local/bin/flaky-cp <<'EOT'
#!/bin/bash
if [[ $2 == */part.aaab && ! -e /root/failed-once ]]; then touch /root/failed-once; exit 1; fi
exec cp -- "$1" "$2"
EOT
chmod +x /usr/local/bin/flaky-cp; mkdir -p /remote/nightly
echo '--- backup'
UPLOAD_CMD=flaky-cp CHUNK=1M BACKOFF=1 bash stream-backup.sh backup /data /remote nightly "$R"
ls /remote/nightly
echo '--- verify'
bash stream-backup.sh verify /remote nightly /root/key.txt /restore
echo '--- truncated last chunk'
last=$(tail -1 /remote/nightly/MANIFEST.sha256 | awk '{print $2}')
truncate -s -100 /remote/nightly/$last
bash stream-backup.sh verify /remote nightly /root/key.txt /restore || echo "exit=$?"
echo '--- same truncation, checksum step skipped: age itself refuses'
cat /remote/nightly/part.* | age -d -i /root/key.txt | zstd -d -q | tar -t > /dev/null || echo "exit=$?"
echo '--- no manifest (interrupted run)'
rm /remote/nightly/MANIFEST.sha256
bash stream-backup.sh verify /remote nightly /root/key.txt /restore || echo "exit=$?"
echo '--- permanent outage'
mkdir -p /remote2/n2
UPLOAD_CMD=false CHUNK=1M BACKOFF=0 bash stream-backup.sh backup /data /remote2 n2 "$R" || echo "exit=$?"
echo "files in remote2/n2: $(ls /remote2/n2 | wc -l)"

Output:

--- backup
upload of part.aaab failed (attempt 1), retrying
backup complete: 2 chunk(s)
MANIFEST.sha256
files.sha256.age
part.aaaa
part.aaab
--- verify
verify OK: restored tree matches the source hashes
--- truncated last chunk
part.aaab: FAILED
sha256sum: WARNING: 1 computed checksum did NOT match
exit=1
--- same truncation, checksum step skipped: age itself refuses
age: error: failed to decrypt and authenticate payload chunk
age: report unexpected or unhelpful errors at https://filippo.io/age/report
/*stdin*\ : Read error (39) : premature end 
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now
exit=2
--- no manifest (interrupted run)
no MANIFEST: backup is incomplete
exit=1
--- permanent outage
upload of part.aaaa failed (attempt 1), retrying
upload of part.aaaa failed (attempt 2), retrying
upload of part.aaaa failed (attempt 3), retrying
upload of part.aaaa failed (attempt 4), retrying
upload of part.aaaa failed (attempt 5), retrying
giving up on part.aaaa
split: with FILE=part.aaaa, exit 1 from command: bash -c upload_chunk
exit=1
files in remote2/n2: 0

Reading it: the second chunk failed once, was retried automatically, and the backup finished with two chunks. Verification restored the tree and the hashes matched. After truncating the last chunk the checksum check names the damaged chunk. With the checksum step skipped, age itself refuses ("failed to decrypt and authenticate payload chunk"), so a truncated ciphertext cannot be silently restored as a shorter archive. Without a manifest, verify says the backup is incomplete. In the permanent outage the script stops after five attempts, no manifest is written, and nothing is mistaken for a completed backup (the destination directory is empty because the transport always failed).

Operational notes

• Keys. Keep the age identity (private key) apart from the backups and back it up separately, or no restore is possible. Test that a person other than the author can find it.
• Completeness is not only "upload returned 0". Check the chunk list against the manifest, compare sizes or provider checksums nightly (cheap), and do a full decrypt-and-compare restore on a schedule (for example monthly) into an encrypted or in-memory scratch location, because that restore deliberately writes plaintext.
• Source consistency. tar reading a live directory can archive a file mid-write, and it warns when a file changed while being read. Back up from a snapshot (a frozen, read-only, point-in-time view of the storage, offered by LVM, ZFS and Btrfs) when the data is a database or similar, or use the database's own dump tool.
• Cost. Chunks bound local disk to one chunk, at the cost of more objects and requests. Larger chunks mean fewer requests but more re-upload per failure.
• For a single stream straight to S3, aws s3 cp - s3://bucket/key accepts standard input, and for a stream larger than 50 GB its documentation says to pass --expected-size or the upload may fail for too many parts. It still cannot resume, which is why this design chunks.

Direct answer. set -e (errexit) makes the shell exit when a simple command returns a non-zero status. set -u (nounset) makes expanding a variable that was never set an error instead of silently producing an empty string. set -o pipefail makes a pipeline's status the status of the rightmost command that failed, rather than always the status of the last command. Together they turn silent failures, typos and hidden pipeline failures into loud, early exits. They are a backstop, not a guarantee: -e in particular has exemptions that regularly surprise people, so critical steps still need explicit checks.

A "simple command" is one command with its arguments, such as false or cp a b, as opposed to a pipeline, an if block or a function. A "tested position" is a place where the shell is already using a command's status to make a decision, such as the condition of an if; there a failure is information, so -e does not exit. Of the surprises listed later, three turn up in everyday scripts: tested positions (1), local x=$(cmd) (2) and grep exiting 1 on no match (4). The others matter in particular situations: inherit_errexit (3), ((i++)) (5), SIGPIPE (6), -u with arrays (7) and the ERR trap (8).

What each setting changes

The classic disaster -u prevents is rm -rf "$BUILD_DIR/" when BUILD_DIR is misspelled earlier: without -u it expands to rm -rf /. The classic one pipefail prevents is pg_dump db | gzip > backup.gz: without it a failed dump still leaves a valid, tiny gzip file and the script reports success.

Running every case in its own shell

Each case runs in a separate bash -c so an outer if or || cannot switch -e off and hide the behaviour being tested (that exemption is itself one of the surprises).

The harness is one line: run() { bash -c "$1"; echo " -> exit status $?"; } defines a function whose first argument $1 is a string of script text. bash -c "$1" runs that text in a brand-new shell, and $? (the exit status of the last command, 0 for success) is then printed as the status of that shell. Each echo line before a run is just a label.

#!/usr/bin/env bash
# Each case runs in its own bash so an outer || or if cannot switch -e off.
run() { bash -c "$1"; echo "   -> exit status $?"; }

echo "-e: false stops the script";        run 'set -e; false; echo "not reached"'
echo "-u: unset variable is an error";    run 'set -u; echo "value: $missing"; echo "not reached"'
echo "no pipefail: false | true";         run 'false | true; echo "pipeline status was $?"'
echo "pipefail: false | true";            run 'set -eo pipefail; false | true; echo "not reached"'
echo "-e: ((i++)) when i is 0";           run 'set -e; i=0; ((i++)); echo "not reached"'
echo "-e: i=\$((i+1)) when i is 0";       run 'set -e; i=0; i=$((i+1)); echo "reached, i=$i"'
echo "pipefail: yes | head -n1";          run 'set -eo pipefail; yes | head -n1 >/dev/null; echo "not reached"'
echo "-e: function called in an if";      run 'set -e; f() { false; echo "f kept going"; }; if f; then echo "f ok"; fi'
echo "-e: local x=\$(false)";             run 'set -e; g() { local x=$(false); echo "g kept going"; }; g'
echo "-e: declare then assign";           run 'set -e; g() { local x; x=$(false); echo "not reached"; }; g'
echo "-e: false && true";                 run 'set -e; false && true; echo "reached"'
echo "-e: true && false";                 run 'set -e; true && false; echo "not reached"'
echo "-e: grep -c finds nothing";         run 'set -e; n=$(grep -c zzz /etc/hostname); echo "not reached"'
echo "-e: failure inside \$( ) is not inherited"; run 'set -e; x=$(false; echo hi); echo "x=$x"'
echo "-e: same, with inherit_errexit";           run 'set -e; shopt -s inherit_errexit; x=$(false; echo hi); echo "x=$x"'
echo "-e: export X=\$(false)";                   run 'set -e; export X=$(false); echo "reached"'

Output (GNU bash 5.2 on Ubuntu 24.04):

-e: false stops the script
   -> exit status 1
-u: unset variable is an error
bash: line 1: missing: unbound variable
   -> exit status 127
no pipefail: false | true
pipeline status was 0
   -> exit status 0
pipefail: false | true
   -> exit status 1
-e: ((i++)) when i is 0
   -> exit status 1
-e: i=$((i+1)) when i is 0
reached, i=1
   -> exit status 0
pipefail: yes | head -n1
   -> exit status 141
-e: function called in an if
f kept going
f ok
   -> exit status 0
-e: local x=$(false)
g kept going
   -> exit status 0
-e: declare then assign
   -> exit status 1
-e: false && true
reached
   -> exit status 0
-e: true && false
   -> exit status 1
-e: grep -c finds nothing
   -> exit status 1
-e: failure inside $( ) is not inherited
x=hi
   -> exit status 0
-e: same, with inherit_errexit
   -> exit status 1
-e: export X=$(false)
reached
   -> exit status 0

(The unset-variable case exits with status 127 on this build; the exact number is not something to rely on, only that it is non-zero.)

Reading the output:

• -e: false stops the script printed no "not reached" line and status 1: the shell quit at false.
• no pipefail: false | true reported 0 although false failed, because only the last command in a pipeline counts by default; with pipefail the same pipeline gave 1.
• ((i++)) when i is 0 stopped with status 1, while i=$((i+1)) carried on and printed reached, i=1. Same increment, different status.
• local x=$(false) printed g kept going: the failure was hidden. Splitting it into local x; and x=$(false) stopped the function (status 1).
• false && true printed reached, because the left side of && is a tested position; true && false exited 1, because the failing command is the last one in the list and nothing tests it.
• grep -c finds nothing exited 1 even though nothing is wrong with the script: no match is status 1 for grep.
• x=$(false; echo hi) gave x=hi and carried on; with inherit_errexit the substitution stopped at false and the script exited 1.
• export X=$(false) printed reached: as with local, the status of the line is export's, not the substitution's.

Situations that still surprise you

1. Tested positions turn -e off. A command that is the condition of if, while or until, the left side of && or ||, or negated with ! never triggers an exit. This also applies to everything inside a function you call from such a position: in the demo, f runs false, prints f kept going, and reports success to the if. A script that calls if check_config; then therefore loses -e throughout check_config.
2. local x=$(cmd) and export X=$(cmd) hide failures. The status of the line is the status of local or export, which succeeds. Declare first and assign on a second line (local x; x=$(cmd)) so the assignment's own status counts.
3. $( ) does not inherit -e by default. In x=$(false; echo hi) the false is ignored and x becomes hi. shopt -s inherit_errexit (bash 4.4 and later) makes command substitutions inherit it, as the demo shows.
4. Commands whose non-zero status is not an error. grep exits 1 for "no match", diff exits 1 for "files differ", and test/[ exit 1 for "false". Under -e a bare grep -q pattern file kills the script when nothing matches. Be deliberate: put it in an if, or write grep -q pattern file || true when no match is acceptable. Do not blanket-append || true, which also swallows exit 2 (real error).
5. Arithmetic that evaluates to zero. ((i++)) when i is 0 has status 1 because the expression's value is 0, so -e exits. i=$((i+1)) is an assignment and is always safe.
6. pipefail and SIGPIPE. yes | head -n1 fails under pipefail with status 141 (128 plus 13, the number of SIGPIPE, the signal sent to a writer whose reader has closed the pipe). head finished early and was happy; the producer was killed for writing to a closed pipe. Patterns like cmd | head -1 or cmd | grep -q x can therefore fail a script that worked. Capture to a variable first, or handle 141 explicitly.
7. -u edge cases. $1 with no argument is fatal, so use ${1:-} or ${1:-default} for optional values. In bash older than 4.4, expanding an empty array as "${arr[@]}" under -u raised unbound variable (checked on 4.3.48; 4.4.23 does not). Portable code uses ${arr[@]+"${arr[@]}"}.
8. No message on exit. -e exits silently, so you do not learn which line failed. Add set -E and trap 'echo "line $LINENO: $BASH_COMMAND" >&2' ERR. A trap is a command the shell runs when an event happens, and the ERR trap fires when a command fails. set -E lets functions and substitutions inherit that trap. Run as bash -c 'set -eE; trap ... ERR; f() { false; }; echo start; f; echo not reached', it printed start, then line 1: false on stderr, then exited with status 1 (the line number is 1 because the whole script was one -c string).

Trade-offs and best practice

• Treat strict mode as a safety net and still write explicit handling for steps that matter: cp a b || die "copy failed". Control flow that depends on -e's exemptions is hard to read and breaks silently when someone wraps the function in an if.
• Strict mode is a property of the script, so it must come before the first real command, and any file you source runs under it too.
• Run ShellCheck. It flags the unquoted and masked-status patterns above (for example SC2155 for local x=$(cmd)).
• Interactive shells and .bashrc files should not enable it: one failed command would close your terminal.
• dash, the /bin/sh on Debian and Ubuntu, rejects set -o pipefail (Illegal option -o pipefail), so a #!/bin/sh script cannot assume it; use #!/usr/bin/env bash when you need it.

Direct answer

Keep one counter pair per second of the window instead of keeping the requests. With a 300-second window that is two arrays of 300 slots (requests and errors per second) plus two running totals. Each slot is chosen by the timestamp modulo 300, meaning the remainder after dividing the timestamp's epoch seconds (seconds since 1970-01-01 UTC) by 300, so the slots are numbered 0 to 299 and then reused in a circle (a ring buffer). When a newer timestamp arrives, subtract the slots for the seconds that just slid out of the window. Memory stays constant however fast the stream is, and each input line costs a few operations. Define an error as an HTTP status of 500 or more (server-side failures; 4xx are client mistakes) and the window as the 300 seconds ending at the newest timestamp seen (event time, the time written on each log line, not wall time, the machine's own clock right now). It needs gawk (GNU awk), because it uses mktime (turns year, month, day, hour, minute and second into epoch seconds) and strftime (turns epoch seconds back into text), both with their UTC flag.

Design

1. Parse the time once per line. ISO 8601 (the 2024-05-01T12:00:03Z style) is converted to epoch seconds with mktime(..., 1), where the 1 means "treat the fields as UTC". A numeric offset such as +02:00 is subtracted, and fractional seconds are ignored.
2. Bucket by second. tot[t % 300] and err[t % 300] hold the counts for second t. Because the window is exactly 300 distinct seconds, no two live seconds share a slot. A worked trace with the sample file below (all values measured with gawk): 12:00:00 on 2024-05-01 is epoch second 1714564800, and 1714564800 % 300 is 0, so it lives in slot 0. 12:00:10 is 1714564810, slot 10. 12:04:59 is 1714565099, slot 299. 12:05:00 is 1714565100, slot 0 again. The two requests that share slot 0 are 300 seconds apart, so they are never both inside the window, and the slot must be emptied in between.
3. Slide by clearing. If t is newer than the newest second seen, every second in between (at most 300 of them) leaves the window: subtract its slot from the totals and zero it. In the trace above, when 12:05:00 arrives the newest second was 12:04:59, so one second is retired: slot (12:04:59 + 1) % 300 = 0. That is exactly where the 12:00:00 request was counted, so it is subtracted from the totals just before the 12:05:00 request is added to the now-empty slot. A quiet hour costs at most 300 clears, not an hour of work.
4. Late and bad lines. A line older than the window is counted and dropped. A line inside the window but behind the newest second is added to its own slot, so mild reordering is handled. A malformed timestamp or status is counted and skipped.
5. Emit once per advancing second, with fflush() (push the buffered output out now instead of waiting for the buffer to fill) so a downstream reader on a pipe gets it without extra delay. A row is printed when the next newer second arrives, and it describes the window as it stood just before that line, so each row appears one advance late and the final row appears only when the input ends.

# Usage: gawk -f errrate.awk
# Input: "<ISO 8601 timestamp> <HTTP status> ..." one request per line.
# Window: the last 300 seconds ending at the newest timestamp seen.
# Memory: two fixed arrays of 300 slots, whatever the volume.
function epoch(ts,    d, t, off, sign, e) {
    # 2024-05-01T12:00:03Z, ...03.250Z, or ...03+02:00
    # match(string, regex, arr) is gawk's: arr[1], arr[2], ... receive the text matched by each
    # parenthesised group, so d[1] is the year and d[8] is the Z or +02:00 part.
    if (!match(ts, /^([0-9]{4})-([0-9]{2})-([0-9]{2})[T ]([0-9]{2}):([0-9]{2}):([0-9]{2})(\.[0-9]+)?(Z|[+-][0-9]{2}:[0-9]{2})$/, d))
        return -1
    e = mktime(d[1] " " d[2] " " d[3] " " d[4] " " d[5] " " d[6], 1)   # 1 = interpret as UTC
    if (d[8] != "Z") {
        sign = (substr(d[8], 1, 1) == "-") ? -1 : 1
        off = substr(d[8], 2, 2) * 3600 + substr(d[8], 5, 2) * 60
        e -= sign * off
    }
    return e
}
function emit(    pct) {
    pct = (sumT > 0) ? 100 * sumE / sumT : 0
    printf "%s requests=%d errors=%d error_rate=%.2f%%\n", strftime("%Y-%m-%dT%H:%M:%SZ", maxT, 1), sumT, sumE, pct
    fflush()
}
BEGIN { W = 300; maxT = -1 }
{
    t = epoch($1)
    if (t < 0 || $2 !~ /^[0-9][0-9][0-9]$/) { bad++; next }
    if (maxT < 0) maxT = t
    if (t > maxT) {
        emit()                                   # report the window as it stood
        n = t - maxT; if (n > W) n = W
        for (i = 1; i <= n; i++) {               # retire the seconds that slide out
            s = (maxT + i) % W
            sumT -= tot[s]; sumE -= err[s]; tot[s] = 0; err[s] = 0
        }
        maxT = t
    } else if (t <= maxT - W) { late++; next }   # older than the window: drop
    s = t % W
    tot[s]++; sumT++
    if ($2 >= 500) { err[s]++; sumE++ }
}
END {
    if (maxT >= 0) emit()                    # empty input: nothing to report
    if (bad || late) printf "skipped: %d malformed, %d too late\n", bad, late > "/dev/stderr"
}

Run on a sample

Pipe a live log into it with tail -F access.log | gawk -f errrate.awk. For the file below (UTC, one line has an offset and one has fractional seconds, one is garbage):

2024-05-01T12:00:00Z 200 GET /a
2024-05-01T12:00:10Z 500 GET /b
2024-05-01T12:00:20Z 200 GET /c
2024-05-01T12:04:59Z 503 GET /d
2024-05-01T12:05:00Z 200 GET /e
2024-05-01T12:05:10Z 200 GET /f
2024-05-01T14:05:15+02:00 404 GET /g
2024-05-01T12:05:30.250Z 200 GET /h
not-a-timestamp 200 GET /x
2024-05-01T12:05:31Z 200 GET /i

gawk -f errrate.awk in1.txt prints this on stdout (the skipped: ... summary goes to stderr when the input ends):

2024-05-01T12:00:00Z requests=1 errors=0 error_rate=0.00%
2024-05-01T12:00:10Z requests=2 errors=1 error_rate=50.00%
2024-05-01T12:00:20Z requests=3 errors=1 error_rate=33.33%
2024-05-01T12:04:59Z requests=4 errors=2 error_rate=50.00%
2024-05-01T12:05:00Z requests=4 errors=2 error_rate=50.00%
2024-05-01T12:05:10Z requests=4 errors=1 error_rate=25.00%
2024-05-01T12:05:15Z requests=5 errors=1 error_rate=20.00%
2024-05-01T12:05:30Z requests=5 errors=1 error_rate=20.00%
2024-05-01T12:05:31Z requests=6 errors=1 error_rate=16.67%

Each row describes the window as it stood at that timestamp (and, as step 5 says, it prints once the next newer timestamp has arrived). Check one by hand. In the interval notation (12:00:00, 12:05:00], a round bracket means that end is excluded and a square bracket means that end is included (half-open). At 12:05:00 the window is therefore everything after 12:00:00 up to and including 12:05:00, so the 12:00:00 request has just left. What remains is 12:00:10 (500), 12:00:20 (200), 12:04:59 (503) and 12:05:00 (200): 4 requests, 2 errors, 50.00%. By 12:05:10 the 12:00:10 error has left too, leaving 4 requests and 1 error (25.00%). The +02:00 line (14:05:15+02:00) lands at 12:05:15 UTC, and the 12:05:30.250Z line counts in second 12:05:30.

Checking it

The same rule was implemented a second, slow way in Python (keep every request, filter the list on each step) and both were run on three sets of 30,000 generated lines. In each set a line was 0 to 5 seconds after the previous one (chosen uniformly, so one line in six shares its predecessor's second), about one line in 20 was delivered between 1 and 301 seconds late, and with probability 0.0005 each a line jumped 310 or 700 seconds ahead. In all three runs the two programs printed identical rows (request counts, error counts and rates): 23,870, 24,018 and 23,980 rows, and the awk program reported the same number of too-late lines as the Python version (3, 0 and 4). A brute-force twin like this is the right test for any windowing code, because off-by-one errors at the window edge look plausible by eye.

Trade-offs and pitfalls

• Event time versus wall time. The window advances only when a new line arrives. If traffic stops, no row prints, so "the error rate went to 100% and then the site went silent" is invisible. Alerting on silence needs a separate wall-clock check.
• Resolution. One-second buckets mean the window edge is accurate to one second. Smaller windows (say 10 seconds) would use 10 slots and the same code with W = 10.
• Why not an array of timestamps? Storing every request grows with traffic (10,000 requests per second would hold 3 million entries). The bucket version does not.
• Only works for a count-style metric. Percentiles need a histogram per bucket instead of a single counter; the sliding structure is the same.
• Portability. mktime with a UTC flag and strftime with a UTC flag are gawk extensions. On a minimal image that ships only mawk or BusyBox awk, install gawk or move the parsing upstream.
• Parsing assumptions. The status is the second field and must be exactly three digits. Change the two field references if the log format differs.