Shell Scripting and Automation Questions

Direct answer

Decompress each file with gzip -dc as a stream, parse each line with jq into a CSV row, and remove duplicate ids with awk '!seen[$1]++', which keeps the first row seen for each id. Print the header once, outside the pipeline. All stages read and write one line at a time, so nothing buffers a whole file. The one exception is the de-duplication table, which holds every distinct id; that is the part to swap for sort -u when ids no longer fit in memory.

The pipeline

(A quick decode of the less obvious pieces. awk '!seen[$1]++' uses an array seen keyed by the first field; seen[$1]++ returns the old count and then adds 1, so it is 0 the first time an id appears and 1, 2, ... after; ! turns 0 into true, so awk's default action, print the line, runs only on the first sight of each id. For ids 1, 2, 1, 3, 2 the expression gives 1, 1, 0, 1, 0, and the output is ids 1, 2, 3. In the sed stage, $a\ means "at the last line, append nothing"; GNU sed appends a newline only if the last line lacks one, so every file ends cleanly. In the script it is written "\$a\\" because it sits inside double quotes, where the shell would otherwise read $a as a variable and eat one backslash. xargs -0 -r -n1 reads NUL-separated names, does nothing on empty input and runs the command once per name.)

#!/usr/bin/env bash
# usage: merge_dedupe.sh DIR > merged.csv     (keeps the first record seen for each id)
set -euo pipefail
dir=${1:?usage: merge_dedupe.sh DIR}

printf 'id,timestamp,value\n'
find "$dir" -name '*.jsonl.gz' -print0 | sort -z |
  xargs -0 -r -n1 bash -o pipefail -c 'gzip -dc -- "$1" | sed -e "\$a\\"' _ |
  jq -Rr 'fromjson? | select(.id != null) | [.id, .ts, .value] | @csv' |
  awk -F, '!seen[$1]++'

Test data: four files, one of them without a trailing newline on its last line. Four cases are in there: id 2 appears in two files (the first has value null), id 1 appears twice, one record has no id, and one record has no value.

rm -rf in; mkdir in
printf '%s\n' '{"id":1,"ts":"2026-10-01T00:00:00Z","value":10}' '{"id":2,"ts":"2026-10-01T00:00:05Z","value":null}' | gzip > in/a.jsonl.gz
printf '%s\n' '{"id":2,"ts":"2026-10-01T00:01:00Z","value":7}' '{"id":3,"ts":"2026-10-01T00:01:05Z","value":3.5}' '{"id":1,"ts":"2026-10-01T00:01:09Z","value":11}' | gzip > in/b.jsonl.gz
printf '%s\n' '{"ts":"2026-10-01T00:02:00Z","value":1}' | gzip > in/c.jsonl.gz
printf '%s' '{"id":4,"ts":"2026-10-01T00:02:05Z"}' | gzip > in/d.jsonl.gz   # no trailing newline
bash merge_dedupe.sh in; echo "exit=$?"
echo "--- corrupt file:"
head -c 20 in/b.jsonl.gz > in/e.jsonl.gz
bash merge_dedupe.sh in > out.csv || echo "exit=$?"

Output:

id,timestamp,value
1,"2026-10-01T00:00:00Z",10
2,"2026-10-01T00:00:05Z",
3,"2026-10-01T00:01:05Z",3.5
4,"2026-10-01T00:02:05Z",
exit=0
--- corrupt file:

gzip: in/e.jsonl.gz: unexpected end of file
exit=123

Reading the result:

• One header line, then ids 1, 2, 3, 4 once each. The record with no id is dropped by select(.id != null).
• Id 1 keeps value 10 from a.jsonl.gz and the later 11 is discarded; id 2 keeps the earlier row, whose value is empty (null). "First wins" is a policy, and here it kept a worse row. If the latest ts should win, the dedupe must compare timestamps, which needs a sort by id then ts descending (see trade-offs).
• Id 4 appears even though its file has no final newline: each file goes through sed -e '$a\' (GNU sed), which adds a newline only when one is missing. Without it, gzip -dc a b c glues the last line of one file onto the first line of the next and silently corrupts two records.
• The last block is a deliberately truncated archive: gzip reports unexpected end of file (its message starts with a newline, which is the blank line in the transcript, not output from the script), and xargs exits 123, the value xargs uses for "a command it ran exited with status 1 to 125", so the script fails instead of publishing a short CSV as if it were complete. The inner bash -o pipefail -c (pipefail makes a pipeline fail if any stage fails, not only the last) is what carries gzip's failure through the | sed; without it the pipe would report only sed's success.

Why it stays streaming

find lists names; xargs -n1 runs one decompress per file in sorted order (sort -z keeps NUL separators and a deterministic order, which matters because "first wins" depends on file order); jq -R reads raw lines and @csv emits a row immediately; awk prints a row the moment it is judged new. At any instant memory holds a few pipe buffers plus awk's seen table. fromjson? skips a malformed line; add a counter if silent skipping is not acceptable.

Trade-offs and pitfalls

• Bounded memory dedupe: for hundreds of millions of ids, replace the awk stage with sort -s -t, -k1,1 -u -S 1G -T /big/tmp, an external merge sort (sorts chunks in memory, writes them to temporary files, merges them) that spills to disk. The flags: -t, uses a comma as the field separator, -k1,1 compares only field 1 (the id), -u keeps one row per equal key, -s makes the sort stable so the first of equal ids in input order is the one kept, -S 1G caps the memory buffer at 1 GB, and -T /big/tmp names the directory for temporary files. With -s and -u GNU sort keeps the first of each run of equal ids in input order (checked on a four-line sample), and the output is ordered by id, not arrival.
• Latest wins instead of first wins: emit ts first, sort by id then ts descending and take the first row per id.
• Assumption: ids contain no commas or quotes, so the first CSV field is the whole id. A string id arrives quoted ("abc"); that still works as a dedupe key. If ids can contain commas, dedupe inside jq before formatting.
• One thread: the pipeline is limited by the slowest stage, usually jq. For hundreds of files use xargs -P for the decompress-and-parse step and dedupe once at the end, because dedupe has to see all rows.
• Cost of -n1: one bash start per file, fine for thousands of files.

Direct answer

The failure goes unnoticed for three reasons. First, the exit status of a pipeline is the status of its last command only (here ssh, which reports the remote cat), so a failing tar on the left is invisible. Second, the script has no set -e and does not test $?, so nothing stops echo 'backup complete' from running. Third, a failing tar still closes its output, the remote cat happily writes whatever arrived (possibly nothing useful) and exits 0, and it overwrites last night's good archive in place. The fix is set -euo pipefail, an explicit check of the transfer, writing to a temporary name on the remote and only renaming after validating the archive. It still would not catch an archive that is valid but wrong, such as an empty directory where a file system failed to mount.

What the original does

tar czf - /srv/data | ssh backup@host 'cat > /backups/data.tgz'
echo 'backup complete'

cat > /backups/data.tgz truncates the destination the moment it starts, so the previous good backup is gone before the new one is known to be good. If tar fails (the source is missing, a read error, the disk is full), tar prints to stderr (which cron mails only if mail is configured) and exits non-zero, but the pipeline's status is ssh's: 0.

The fix

• set -o pipefail makes the pipeline's status the rightmost non-zero stage. With set -e the script then stops, or the if ! pipeline form lets you clean up first.
• Write to data.tgz.partial on the remote host, and replace the real file only after the archive validates (gzip -t checks the compressed stream, tar tzf lists entries). The mv within one remote directory is an atomic rename, so the destination is always either the old good archive or the new good one.
• Fail loudly: exit non-zero and send the message to stderr, so the scheduler or monitoring sees it. Print "complete" only at the very end.

Reading the key lines

• tar czf - /srv/data: c creates an archive, z compresses it with gzip, f - says the archive goes to the file named -, which by convention means standard output, so the bytes flow into the pipe instead of landing on disk.
• ssh "$host" "gzip -t $tmp && test \$(tar tzf $tmp | wc -l) -gt 0 && mv -f $tmp $dest" is one command line that the remote host runs, read left to right. gzip -t tests the compressed stream and prints nothing if it is intact. tar tzf $tmp lists the archive's entries, one per line; wc -l counts the lines; test N -gt 0 succeeds only when that count is greater than zero. && runs the next step only if the previous one succeeded, so mv happens only for an archive that passed both checks.
• Why \$(: the whole remote command sits inside double quotes, so your local shell expands $tmp and $dest before ssh runs (the values are known locally), but the backslash stops it from running $( ... ) locally. The text reaches the remote shell intact, and the remote shell runs tar tzf on the remote file. Without the backslash, the local shell would try to list a file that does not exist on your machine.
• || true after the cleanup ssh: the "backup FAILED" message has already been printed by then. If the host is unreachable, the cleanup itself fails, and set -e is still active inside the if body, so the script would end at that line with ssh's own status (255 for a connection failure) instead of reaching exit 1. || true says "ignore this failure", so the script always exits 1 for a failed transfer, which keeps the exit code meaningful to the scheduler.
• "Atomic" rename: a copy writes bytes gradually, so a crash or a reader in the middle sees half a file. A rename on the same filesystem changes which file the name points to in one step, so the name always leads to a complete file.
• The ssh stub in the demo below: shift drops the first argument (the backup@host word), and exec sh -c "$*" runs the remaining words as a command on this machine, which is what the remote side would do.

Worked example

The sandbox has no network, so the demo replaces ssh with a stub that runs the remote command locally. Real ssh joins the arguments the same way and runs them through the login shell on the other side. The connection step itself (authentication, exit status 255 when the connection fails) is not exercised here.

set -u
mkdir -p /backups /srv/data bin
printf 'row1\n' > /srv/data/a.txt
# stand-in for ssh: run the remote command locally (no network in the sandbox)
cat > bin/ssh <<'EOF'
#!/bin/sh
shift
exec sh -c "$*"
EOF
chmod +x bin/ssh; export PATH="$PWD/bin:$PATH"

cat > original.sh <<'EOF'
tar czf - "$1" | ssh backup@host 'cat > /backups/data.tgz'
echo 'backup complete'
EOF

cat > fixed.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
src=$1 host=backup@host dest=/backups/data.tgz
tmp=$dest.partial

# Any failing stage (tar, ssh, remote cat) now fails the pipeline and, with set -e, the script.
if ! tar czf - "$src" | ssh "$host" "cat > $tmp"; then
    echo "backup FAILED: transfer" >&2
    ssh "$host" "rm -f $tmp" || true
    exit 1
fi
# Validate on the remote side, and only then replace the previous good archive.
ssh "$host" "gzip -t $tmp && test \$(tar tzf $tmp | wc -l) -gt 0 && mv -f $tmp $dest"
echo 'backup complete'
EOF

echo '=== original, source directory missing'
rm -f /backups/*; bash original.sh /srv/nope 2>&1; echo "exit=$?"
ls -l /backups/data.tgz | awk '{print $5, $9}'; echo "entries in archive: $(tar tzf /backups/data.tgz | wc -l)"
echo '=== fixed, source directory missing'
rm -f /backups/*; bash fixed.sh /srv/nope 2>&1; echo "exit=$?"; ls /backups
echo '=== fixed, good source'
bash fixed.sh /srv/data 2>&1; echo "exit=$?"; ls /backups; tar tzf /backups/data.tgz
echo '=== fixed, source is an empty mount point'
mkdir -p /srv/empty; rm -f /backups/*; bash fixed.sh /srv/empty 2>&1; echo "exit=$?"; ls /backups

Output:

=== original, source directory missing
tar: Removing leading `/' from member names
tar: /srv/nope: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors
backup complete
exit=0
45 /backups/data.tgz
entries in archive: 0
=== fixed, source directory missing
tar: Removing leading `/' from member names
tar: /srv/nope: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors
backup FAILED: transfer
exit=1
=== fixed, good source
tar: Removing leading `/' from member names
backup complete
exit=0
data.tgz
srv/data/
srv/data/a.txt
=== fixed, source is an empty mount point
tar: Removing leading `/' from member names
backup complete
exit=0
data.tgz

The original prints "backup complete" with exit 0 for a source that does not exist, and leaves a 45 byte data.tgz that lists zero entries (a valid, empty archive). The fixed script reports the failure, exits 1, and leaves no archive in /backups in this first run. The ls output for the good source shows data.tgz is the only file, so the .partial name was renamed.

What the fix still does not catch

• A valid archive of the wrong content. The empty mount point case above passes: tar exits 0 and the archive has one entry (the directory itself). Add a content check that matters to you: a minimum entry count or size compared with the previous night, or a sentinel file that must be present.
• Files changing while tar reads them. GNU tar exits 1 ("file changed as we read it"), which pipefail treats as failure. For a busy database, snapshot first (an LVM, the Logical Volume Manager, or filesystem snapshot) instead of reading live files.
• Corruption after the transfer (bit rot, a later disk failure) and a restore that does not work. Only a periodic test restore proves recoverability.
• The remote host being the only copy. A second off-site copy covers loss of that machine.

Pitfalls

• Quote variables that hold paths when they come from outside. Here the paths are constants.
• tar and ssh both write to stderr, so redirect the script's stderr to a log or to your alerting tool; otherwise failures only appear in local mail.
• Check the exit status of the final mv too. With set -e it is checked.

Direct answer

For top-N by count the idiom is: extract the key, sort, uniq -c, sort -rn, head, or the same in one awk pass with an associative array (awk's lookup table, whose keys can be text such as an IP address: n["10.0.0.1"]++ counts that key). For time-window questions (last 7 days, last 24 hours), compute the cutoff timestamp once with GNU date -d, then compare it with the line's timestamp. That works with a plain string comparison only when the log uses a fixed-width, sortable format such as ISO 8601 in UTC (2025-01-19T12:00:00Z: year, month, day, T, then hour, minute, second, with Z meaning UTC). Web server access logs use [19/Jan/2025:10:00:01 +0000], which does not sort, so convert it to epoch seconds (the number of seconds since 1 January 1970 UTC, so a later time is always a bigger number) with gawk's mktime. Distinct sources come from sort -u, or from an awk array whose keys you count.

Method

1. Top five client IPs: count $1 of the access log in an array, print count ip, sort by count descending, break ties by IP so the output is stable, and keep five lines with head -n 5.
2. Last 7 days of ERROR lines: build the cutoff from a fixed "now" (pinned here so the example is repeatable; in production use date -u +%Y-%m-%dT%H:%M:%SZ), then keep lines whose first field is at least the cutoff and which contain ERROR.
3. Timeout messages in the last 24 hours: same shape with a different cutoff and tolower($0) ~ /timeout/ so "Timeout" and "timeout" both match.
4. Distinct sources: extract the src= value (this could equally be a user_id= field) and apply sort -u, or count keys in an array. To get the top five sources instead of the full list, count each source and keep the first five: grep -oE 'src=[^ ]+' app.log | sort | uniq -c | sort -rn | head -n 5.

Worked example

set -eu
cat > access.log <<'EOF'
10.0.0.1 - - [19/Jan/2025:10:00:01 +0000] "GET / HTTP/1.1" 200 512
10.0.0.1 - - [19/Jan/2025:10:00:02 +0000] "GET /a HTTP/1.1" 200 512
10.0.0.1 - - [19/Jan/2025:10:00:03 +0000] "GET /b HTTP/1.1" 200 512
10.0.0.2 - - [19/Jan/2025:10:00:04 +0000] "GET / HTTP/1.1" 200 512
10.0.0.2 - - [19/Jan/2025:10:00:05 +0000] "GET /a HTTP/1.1" 404 120
10.0.0.3 - - [19/Jan/2025:10:00:06 +0000] "GET / HTTP/1.1" 200 512
10.0.0.4 - - [19/Jan/2025:10:00:07 +0000] "GET / HTTP/1.1" 200 512
10.0.0.5 - - [19/Jan/2025:10:00:08 +0000] "GET / HTTP/1.1" 200 512
10.0.0.6 - - [19/Jan/2025:10:00:09 +0000] "GET / HTTP/1.1" 200 512
10.0.0.6 - - [19/Jan/2025:10:00:10 +0000] "GET /x HTTP/1.1" 200 512
10.0.0.6 - - [19/Jan/2025:10:00:11 +0000] "GET /y HTTP/1.1" 200 512
10.0.0.6 - - [19/Jan/2025:10:00:12 +0000] "GET /z HTTP/1.1" 200 512
EOF
echo '--- top 5 client IPs'
awk '{n[$1]++} END {for (ip in n) print n[ip], ip}' access.log | sort -k1,1nr -k2,2V | head -n 5

cat > app.log <<'EOF'
2025-01-10T09:00:00Z src=10.1.1.1 ERROR db down
2025-01-14T08:00:00Z src=10.1.1.1 ERROR db down
2025-01-14T09:30:00Z src=10.1.1.2 ERROR cache miss storm
2025-01-18T23:00:00Z src=10.1.1.3 ERROR disk full
2025-01-19T11:59:00Z src=10.1.1.2 ERROR db down
2025-01-19T11:59:30Z src=10.1.1.2 WARN slow
2025-01-19T12:00:00Z src=10.1.1.9 timeout calling billing
2025-01-19T12:00:01Z src=10.1.1.8 timeout calling billing
2025-01-20T09:00:00Z src=10.1.1.9 Timeout calling auth
2025-01-20T11:00:00Z src=10.1.1.9 timeout calling billing
2025-01-18T11:00:00Z src=10.1.1.7 timeout calling billing
EOF
now='2025-01-20T12:00:00Z'
cut7=$(date -u -d "$now - 7 days" +%Y-%m-%dT%H:%M:%SZ)
cut1=$(date -u -d "$now - 24 hours" +%Y-%m-%dT%H:%M:%SZ)
echo "cutoffs: $cut7 $cut1"
echo '--- ERROR lines in the last 7 days, count and distinct sources'
awk -v c="$cut7" '$1 >= c && /ERROR/ {n++; match($0, /src=[^ ]+/); s[substr($0, RSTART+4, RLENGTH-4)]=1}
  END {k=0; for (x in s) k++; print n " errors from " k " distinct sources"}' app.log
echo '--- timeout lines in the last 24 hours, distinct sources'
awk -v c="$cut1" '$1 >= c && tolower($0) ~ /timeout/ {match($0, /src=[^ ]+/); print substr($0, RSTART+4, RLENGTH-4)}' app.log | sort -u
echo '--- same count with grep, sort, uniq'
grep -h 'ERROR' app.log | awk -v c="$cut7" '$1 >= c' | grep -oE 'src=[^ ]+' | sort | uniq -c | sort -k1,1nr -k2
echo '--- combined-format timestamps need conversion (gawk mktime)'
gawk -v c="$(date -u -d '2025-01-19T10:00:08Z' +%s)" '
  BEGIN { split("Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec", m, " "); for (i in m) mon[m[i]] = i }
  { t = substr($4, 2); split(t, p, /[\/:]/)
    ts = mktime(sprintf("%d %d %d %d %d %d", p[3], mon[p[2]], p[1], p[4], p[5], p[6]), 1)
    if (ts >= c) n[$1]++ }
  END { for (ip in n) print n[ip], ip }' access.log | sort -k1,1nr -k2,2V

Output:

--- top 5 client IPs
4 10.0.0.6
3 10.0.0.1
2 10.0.0.2
1 10.0.0.3
1 10.0.0.4
cutoffs: 2025-01-13T12:00:00Z 2025-01-19T12:00:00Z
--- ERROR lines in the last 7 days, count and distinct sources
4 errors from 3 distinct sources
--- timeout lines in the last 24 hours, distinct sources
10.1.1.8
10.1.1.9
--- same count with grep, sort, uniq
      2 src=10.1.1.2
      1 src=10.1.1.1
      1 src=10.1.1.3
--- combined-format timestamps need conversion (gawk mktime)
4 10.0.0.6
1 10.0.0.5

Reading the commands

• awk -v c="$cut7" '$1 >= c && /ERROR/ {...}'. An awk program is a series of pattern { action } pairs: for every line, if the pattern is true, the action runs. -v c="$cut7" hands the shell variable cut7 to awk as awk variable c. Here the pattern says "field 1 (the timestamp) is at least the cutoff, and the line contains ERROR".
• match($0, /src=[^ ]+/). It searches the whole line ($0) for src= followed by one or more non-space characters. On success it sets RSTART, the position where the match begins, and RLENGTH, its length. For 2025-01-14T09:30:00Z src=10.1.1.2 ERROR cache miss storm the run printed RSTART=22 RLENGTH=12: the match src=10.1.1.2 starts at character 22 and is 12 characters long.
• substr($0, RSTART+4, RLENGTH-4). substr(text, start, length) cuts out a piece. The +4 and -4 skip the four characters of src=, leaving 10.1.1.2. s[...]=1 then records that address as a key, so each distinct source appears once however often it repeats, and the for (x in s) k++ loop counts the keys.
• sort -k1,1nr -k2,2V. -k1,1nr sorts on field 1 only (1,1), numerically (n), in reverse (r), so the largest count comes first. -k2,2V breaks ties on field 2 with version sort (V), which orders 10.0.0.9 before 10.0.0.10 by comparing the numbers inside it, not the characters. Run on the counts 3 10.0.0.10, 3 10.0.0.9 and 12 10.0.0.1 it printed 12 10.0.0.1, 3 10.0.0.9, 3 10.0.0.10.
• The gawk block, line by line. Line 1 (BEGIN) builds a lookup table mon that maps Jan to 1 through Dec to 12. For each log line, substr($4, 2) takes the fourth field, [19/Jan/2025:10:00:09, and drops the leading [. split(t, p, /[\/:]/) cuts that text at every / or : into p[1]=19, p[2]=Jan, p[3]=2025, p[4]=10, p[5]=00, p[6]=09. sprintf reassembles them in the order mktime wants, year month day hour minute second, with the month turned into a number: 2025 1 19 10 0 9. mktime(..., 1) turns that into epoch seconds (the final 1 means the time is UTC), which was 1737280809 for this line, the same value date -u -d '2025-01-19T10:00:09Z' +%s prints. ts >= c then compares plain numbers, and n[$1]++ counts the line against its client address.

Checking by hand: the cutoff for 7 days before 2025-01-20T12:00:00Z is 2025-01-13T12:00:00Z. ERROR lines at or after it are 14 Jan 08:00 (10.1.1.1), 14 Jan 09:30 (10.1.1.2), 18 Jan 23:00 (10.1.1.3) and 19 Jan 11:59 (10.1.1.2), which is 4 lines from 3 sources. The 10 Jan line is outside the window. For the 24-hour window (cutoff 2025-01-19T12:00:00Z) the timeouts are from 10.1.1.9, 10.1.1.8, then 10.1.1.9 twice more, so 2 distinct sources. The 18 Jan timeout from 10.1.1.7 is older than 24 hours and is excluded.

Trade-offs and pitfalls

• The window test >= includes a line stamped exactly at the cutoff (the 12:00:00 timeout above). Decide whether the boundary is inclusive and keep it consistent.
• Comparing strings only works when every timestamp has the same width, the same time zone, and the same field position. A log with a local-time zone offset needs conversion first.
• Counting with sort | uniq -c is simple but uses a sort over every matching line. For a very large log, an awk array keeps memory proportional to distinct keys instead.
• "Distinct sources" depends on what the source is. Behind a load balancer the client IP in $1 may be the balancer, and the real address sits in a header field such as X-Forwarded-For.
• Rotated and compressed logs need zcat (or zgrep) in front of awk to cover the whole window.

Direct answer

The loop pays for a process, not for the work. For every file it starts a subshell, then grep, then wc: with tens of thousands of files that is tens of thousands of fork and exec calls, each loading a program, setting up its libraries and tearing it down, which costs far more than reading a small file. Restructure so one grep handles many files at once: find ... -print0 | xargs -0 grep -c PATTERN, or grep -rc PATTERN dir. grep -c already counts matching lines, so wc -l disappears.

Terms and lines used in the measurement

• Fork and exec. To start a program the shell first forks (makes a copy of itself) and then execs (replaces that copy with the new program). execve is the system call that does the second step, so counting successful execve calls counts programs started. A subshell is the forked copy of the shell that $(...) creates.
• strace -f -z -e trace=execve. strace is a tool that logs the system calls a program makes. -f also follows child processes, -z shows only the calls that succeeded, and -e trace=execve limits the log to that one call.
• find -print0 with read -r -d ''. A file name may contain spaces or even a newline, but never a NUL byte. -print0 ends each name with NUL, and read -d '' reads up to the next NUL instead of the next newline, so any name survives intact.
• sed 's/:\([0-9]*\)$/\t\1/'. s/FIND/REPLACE/ substitutes text. : is a literal colon, \( ... \) captures what is inside, [0-9]* is any run of digits, and $ anchors the match to the end of the line, so only a colon followed by digits at the very end matches. \t\1 writes a tab followed by the captured digits.
• ARG_MAX. The most bytes of arguments and environment one command may receive. When a list is longer, xargs splits it into several runs of the command, and GNU xargs starts a new run at 128 KiB by default even when ARG_MAX is larger.

The sed rewrite and the batching on their own (Ubuntu 24.04 container):

printf '%s\n' 'data/log 1.txt:1' 'we:ird:name.txt:12' 'x.txt:0' | sed 's/:\([0-9]*\)$/\t\1/' | cat -A | sed 's/\^I/<TAB>/'
seq 1 10 | xargs -n 4 echo batch:
echo "ARG_MAX=$(getconf ARG_MAX)"

data/log 1.txt<TAB>1$
we:ird:name.txt<TAB>12$
x.txt<TAB>0$
batch: 1 2 3 4
batch: 5 6 7 8
batch: 9 10
ARG_MAX=2097152

Reading it: only the final :digits becomes a tab, so a name with colons in it (we:ird:name.txt) keeps them. xargs -n 4 forced a batch limit of four items, giving three commands for ten items; with no -n, xargs fills each command up to its own default limit instead. The xargs manual says that if the system maximum is more than 128 KiB, 128 KiB is used as the default, and xargs --show-limits reports the buffer actually used (131072 bytes in this container). ARG_MAX shown here is this container's system value, in bytes.

Why it is slow, with measurement

Counting the programs started (not wall-clock time, which depends on the machine) for 1,000 files, using strace -f -z -e trace=execve to record every successful execve (the system call that starts a program):

#!/usr/bin/env bash
# per-file counts, one grep and one wc per file
pattern=$1 dir=$2
while IFS= read -r -d '' f; do
  n=$(grep -- "$pattern" "$f" | wc -l)
  printf '%s\t%s\n' "$f" "$n"
done < <(find "$dir" -type f -print0)

#!/usr/bin/env bash
# same output, one grep process per batch of files
pattern=$1 dir=$2
find "$dir" -type f -print0 | xargs -0 grep -c -- "$pattern" | sed 's/:\([0-9]*\)$/\t\1/'

The test builds 1,000 files named log N.txt containing between zero and six ERROR lines, runs both scripts, compares their sorted output, and counts programs started:

set -u
apt-get update -qq >/dev/null && apt-get install -y -qq strace >/dev/null 2>&1
mkdir -p data
for i in $(seq 1 1000); do
  { for j in $(seq 1 $((i % 7))); do echo "ERROR line $j"; done; echo "ok"; } > "data/log $i.txt"
done
bash slow.sh ERROR data | sort > slow.out
bash fast.sh ERROR data | sort > fast.out
cmp slow.out fast.out && echo "identical output, $(wc -l < fast.out) files, $(awk -F'\t' '{s+=$2} END{print s}' fast.out) matching lines"
for s in slow fast; do
  strace -f -qq -z -e trace=execve -o $s.trace bash $s.sh ERROR data > /dev/null
  echo "$s: programs started"
  grep -o 'execve("[^"]*"' $s.trace | sed 's/execve("//; s/"$//' | sort | uniq -c
done

identical output, 1000 files, 3003 matching lines
slow: programs started
      1 /usr/bin/bash
      1 /usr/bin/find
   1000 /usr/bin/grep
   1000 /usr/bin/wc
fast: programs started
      1 /usr/bin/bash
      1 /usr/bin/find
      1 /usr/bin/grep
      1 /usr/bin/sed
      1 /usr/bin/xargs

The outputs are identical (1,000 files, 3,003 matching lines). The loop starts 1,000 grep plus 1,000 wc processes, two per file. The restructured version starts one grep in total (plus find, xargs and sed, one each), regardless of how many files there are, until a batch fills the default 128 KiB argument buffer of GNU xargs, after which it starts another (the 1,000 short names here fit in one). Save the two scripts as slow.sh and fast.sh and the test as demo.sh in one directory, then run docker run --rm --cap-add SYS_PTRACE -v "$PWD":/w ubuntu:24.04 bash -c 'cp /w/slow.sh /w/fast.sh /root/ && cd /root && bash /w/demo.sh'; the strace tool needs that capability, and the scripts must be in the directory the test runs from.

Restructuring options

Other causes to check in the same script

• Unquoted $file or for f in $(find ...) splits names at spaces and expands globs. The shown loop reads NUL-delimited names (-print0, read -r -d ''), which survives any filename.
• Wrong pattern semantics: if the pattern is a literal string, grep -F avoids the regex engine; LC_ALL=C avoids multibyte locale work on ASCII logs.
• Same file read repeatedly: if the loop runs the grep for many patterns per file, use one grep -f patterns.txt or a single awk pass.
• The disk, not the CPU: if the files sit on network storage, per-file open latency also dominates, and batching helps for the same reason.

Trade-offs and edge cases

• grep -c with several files prints name:count, which is why fast.sh rewrites the last colon into a tab. A file name that contains a colon still works because only the final colon is replaced.
• grep -c counts matching lines, the same as grep | wc -l; it does not count occurrences within a line. For that use grep -o ... | wc -l.
• Exit status: grep exits 1 when no line is selected in a whole batch, and xargs reports any child status from 1 to 125 as its own status 123 (xargs manual), so under set -e or pipefail a zero-match batch makes the xargs stage fail with 123 and a || [[ $? -eq 1 ]] after the pipeline would never match. Tolerate status 1 inside each batch instead: xargs -0 sh -c 'grep -c -- "$0" "$@" || [ "$?" -eq 1 ]' "$pattern". Run on two files with no match, this exits 0 under pipefail; a real grep error (a directory argument) still ends as 123.
• xargs splits the list into batches of at most 128 KiB of arguments by default (the system limit from getconf ARG_MAX is higher, about 2 MiB here, but the manual caps the default at 128 KiB unless you raise it with -s), so "one grep" really means one per 128 KiB of file names: roughly nine batches for 30,000 paths averaging 35 bytes (a computed estimate, not a measurement), still nothing next to 30,000 processes.

Direct answer. Standard output is file descriptor 1 and standard error is descriptor 2. To split them into two files use cmd > out.log 2> err.log. To also run the output through a filter, put a pipe after the redirections and use tee so the stream goes to the file and on down the pipe: cmd 2> err.log | tee out.log | grep .... Redirections are processed left to right, so > file 2>&1 sends both streams to the file but 2>&1 > file does not. To capture only the errors in a pipeline, send stdout away and keep stderr on the pipe: cmd 2>&1 > /dev/null | filter.

Concepts

• A redirection changes where a descriptor points. 2>&1 means "make descriptor 2 point at wherever descriptor 1 points right now", which is why order matters.
• A pipe a | b connects a's descriptor 1 to b's standard input. Descriptor 2 is not piped and stays on the terminal unless you redirect it.
• The pipe is set up before the redirections of each command are applied.
• PIPESTATUS is a Bash array holding the exit status of every stage of the most recent pipeline, left to right (0 means success); example 5 prints it.
• In Bash, a |& b is shorthand for a 2>&1 | b (example 7), and tee -a file appends to the file instead of overwriting it (see the pitfalls).

Worked example

noisy writes two lines to stdout and two to stderr.

#!/usr/bin/env bash
cd "$(mktemp -d)" || exit 1

# A command that writes to both streams
noisy() { echo "row 1"; echo "warn: bad checksum" >&2; echo "row 2"; echo "error: disk slow" >&2; }

echo "--- 1. stdout to one file, stderr to another ---"
noisy > out.log 2> err.log
echo "out.log:"; sed 's/^/  /' out.log
echo "err.log:"; sed 's/^/  /' err.log

echo "--- 2. same, and the filter also sees stdout ---"
noisy 2> err.log | tee out.log | grep -c row
echo "err.log has $(wc -l < err.log) lines; out.log has $(wc -l < out.log) lines"

echo "--- 3. order matters ---"
echo "[>file 2>&1] both streams into the file:"
noisy > both.log 2>&1; sed 's/^/  /' both.log
echo "[2>&1 >file] stderr copies stdout's destination first (here the pipe), then stdout moves to the file:"
noisy 2>&1 > only-out.log | sed 's/^/  via pipe: /'
echo "  only-out.log has $(wc -l < only-out.log) lines"

echo "--- 4. only the errors, through a pipe ---"
noisy 2>&1 >/dev/null | tr a-z A-Z

echo "--- 5. errors from every stage of a pipeline ---"
cat missing.csv 2>> errors.log | sort 2>> errors.log | wc -l 2>> errors.log
echo "PIPESTATUS: ${PIPESTATUS[*]}"
cat errors.log

echo "--- 6. swap the streams: filter the errors, keep stdout untouched ---"
{ noisy 2>&1 1>&3 3>&- | tr a-z A-Z > errors.upper; } 3> clean.out
echo "clean.out:";    sed 's/^/  /' clean.out
echo "errors.upper:"; sed 's/^/  /' errors.upper

echo "--- 7. |& is shorthand for 2>&1 | ---"
noisy |& grep -c .

Output (GNU bash 5.2, Ubuntu 24.04):

--- 1. stdout to one file, stderr to another ---
out.log:
  row 1
  row 2
err.log:
  warn: bad checksum
  error: disk slow
--- 2. same, and the filter also sees stdout ---
2
err.log has 2 lines; out.log has 2 lines
--- 3. order matters ---
[>file 2>&1] both streams into the file:
  row 1
  warn: bad checksum
  row 2
  error: disk slow
[2>&1 >file] stderr copies stdout's destination first (here the pipe), then stdout moves to the file:
  via pipe: warn: bad checksum
  via pipe: error: disk slow
  only-out.log has 2 lines
--- 4. only the errors, through a pipe ---
WARN: BAD CHECKSUM
ERROR: DISK SLOW
--- 5. errors from every stage of a pipeline ---
0
PIPESTATUS: 1 0 0
cat: missing.csv: No such file or directory
--- 6. swap the streams: filter the errors, keep stdout untouched ---
clean.out:
  row 1
  row 2
errors.upper:
  WARN: BAD CHECKSUM
  ERROR: DISK SLOW
--- 7. |& is shorthand for 2>&1 | ---
4

How to read it:

1. Two files. > out.log 2> err.log separates the streams cleanly.
2. File plus filter. 2> err.log | tee out.log | grep -c row kept errors in err.log, wrote stdout to out.log and let grep count the 2 row lines. > out.log | filter alone would leave the filter with nothing to read, because stdout is already redirected away from the pipe.
3. Order. noisy > both.log 2>&1 put all four lines into one file, in order. noisy 2>&1 > only-out.log | ... first pointed descriptor 2 at the pipe (descriptor 1's destination at that moment), then moved descriptor 1 to the file, so the filter received only the errors and the file only the stdout lines.
4. Only the errors. 2>&1 >/dev/null | is the same trick on purpose: errors flow into the filter, stdout is discarded. Write it in that order. >/dev/null 2>&1 would discard both.
5. Errors from every stage. Each stage of a pipeline has its own stderr. cat missing.csv 2>> errors.log | sort 2>> errors.log | wc -l 2>> errors.log appends every stage's complaints to one file. wc still printed 0, so a silent success would hide the failure, and ${PIPESTATUS[*]} shows 1 0 0: the first stage failed. Use set -o pipefail to make the whole pipeline's status non-zero.
6. Swapping streams. { cmd 2>&1 1>&3 3>&- | filter; } 3> file is a specialised idiom for the case where one stream must be filtered while the other goes untouched into a file. The filter sees only the errors and the file gets only the normal output. The demo upper-cased the errors into errors.upper while clean.out kept the two clean rows. The next section reads the line one redirection at a time.
7. Shorthand. In bash, |& means 2>&1 |, so both streams go down the pipe: 4 lines counted.

Reading demo 6 one redirection at a time

{ noisy 2>&1 1>&3 3>&- | tr a-z A-Z > errors.upper; } 3> clean.out

The outer 3> clean.out belongs to the whole { ...; } group: it opens clean.out on a new descriptor 3 before anything inside runs. Then noisy's own redirections are applied left to right, exactly like the 2>&1 > file case in example 3:

step                            fd1 (stdout)    fd2 (stderr)    fd3
3> clean.out  (whole group)     terminal        terminal        clean.out
pipe to tr    (noisy's side)    pipe to tr      terminal        clean.out
2>&1                            pipe to tr      pipe to tr      clean.out
1>&3                            clean.out       pipe to tr      clean.out
3>&-                            clean.out       pipe to tr      closed

• 2>&1 points stderr where stdout points right now, the pipe.
• 1>&3 then points stdout at descriptor 3's file. This is the swap: stderr kept the pipe, stdout moved to the file.
• 3>&- closes descriptor 3 for noisy, which no longer needs it (a long-running command would otherwise keep the extra copy open for its whole life).
• tr a-z A-Z > errors.upper reads the pipe, so only the error lines get upper-cased.

A small program that reports where its descriptors point, run through the same line (the pipe number is illustrative and changes per run):

#!/bin/bash
# showfds.sh: report where descriptors 1, 2 and 3 point, writing the report to descriptor 2
for n in 1 2 3; do
  printf 'fd%s -> ' "$n" >&2
  readlink "/proc/$$/fd/$n" >&2 || echo closed >&2
done

{ ./showfds.sh 2>&1 1>&3 3>&- | cat; } 3> clean.out

fd1 -> /tmp/tmp.nAlQQGvVFZ/clean.out
fd2 -> pipe:[1390980]
fd3 -> closed

Descriptor 1 is the file, descriptor 2 is the pipe that cat printed this report from, and descriptor 3 is closed, matching the last row of the table.

Trade-offs and pitfalls

• Interleaving: two files, or a pipe plus a file, lose the relative order of output and errors. If a person needs the story in time order, send both to one file (> log 2>&1) and prefix lines if needed.
• Order mistakes are silent. The wrong order does not error, it just sends text somewhere unexpected.
• |& is a bash extension and is not available in a plain POSIX sh such as dash; 2>&1 | is portable.
• > truncates the file when the command starts; >> appends, which is what you want for a log that several runs share.
• Use tee -a to append, and note that tee discards the command's exit status: the pipeline status is tee's unless pipefail is set.