Command Line Proficiency and Troubleshooting Questions

Comfortable using shell commands for system troubleshooting and administration. Key tools: ps (process information), top and htop (real-time monitoring), grep, sed, awk (text processing), curl (HTTP requests), netstat and ss (network statistics), du and df (disk usage), kill and killall (process termination), tar and gzip (compression). Know how to redirect I/O, pipe commands together, and create simple scripts. Understand how to use man pages.

EasyTechnical

67 practiced

You're the on-call SRE and a service process is unresponsive. Using only local CLI tools and man pages, describe how you'd discover available Unix signals, learn their meanings, and decide whether to send SIGTERM or SIGKILL. Include the specific commands you would run to list signals and to read man-page sections for signals and kill semantics, and explain your decision criteria for which signal to use in production.

HardSystem Design

50 practiced

Design a small, human-readable on-call CLI runbook structure (sections and example commands) for diagnosing a web service outage using only shell tools. The runbook should prioritize quick evidence collection, safe remediation actions, and escalation steps. Provide sample commands and explain how to make the runbook executable and version-controlled.

Sample Answer

Requirements (implicit): fast, human-readable, single-file CLI runbook using only shell tools; prioritizes evidence collection, safe remediation, and clear escalation.

Runbook structure (top-level sections):- Header: service, owners, SLAs, pager- Quick status checks (non-destructive)- Evidence collection (logs, metrics, traces)- Safe remediation (read-only first, then restart/scale)- Post-action verification- Rollback & mitigation steps- Escalation & contact matrix- Notes & links

Example runbook (excerpts — save as runbook.sh and mark executable):

bash

#!/usr/bin/env bash
# runbook.sh - On-call runbook for my-service

SERVICE="my-service"
OWNERS="team-oncall@example.com"
PAGER="pagerduty:PD-1234"

cmd() { echo "+ $*"; "$@"; }

section_status() {
  echo "== Quick status =="
  cmd curl -sS --max-time 5 http://localhost:8080/health || true
  cmd ss -tulpn | grep :8080 || true
  cmd ps aux | grep my-service | grep -v grep || true
}

section_evidence() {
  echo "== Collect evidence =="
  TIMESTAMP=$(date -u +"%Y%m%dT%H%M%SZ")
  DST="/tmp/${SERVICE}-evidence-${TIMESTAMP}"
  mkdir -p "$DST"
  cmd journalctl -u my-service --no-pager -n 500 > "$DST/journal.log" || true
  cmd tail -n 500 /var/log/my-service/*.log > "$DST/app.log" || true
  cmd curl -sS 'http://metrics:9090/api/v1/query?query=up' > "$DST/metrics.json" || true
  cmd tar czf "${DST}.tgz" -C /tmp "$(basename "$DST")"
  echo "Evidence saved to ${DST}.tgz"
}

section_safe_remediate() {
  echo "== Safe remediation =="
  echo "1) Scale up (if k8s):"
  cmd kubectl -n prod scale deployment my-service --replicas=3
  echo "2) Restart one pod at a time (k8s):"
  cmd kubectl -n prod rollout restart deployment/my-service
  echo "3) Local process restart (if systemd):"
  cmd systemctl restart my-service.service
}

section_verify() {
  echo "== Verify =="
  cmd curl -sS http://localhost:8080/health
  cmd curl -sS -I http://localhost:8080/ | head -n 5
  cmd kubectl -n prod get pods -l app=my-service -o wide || true
}

section_escalate() {
  echo "== Escalation =="
  echo "If >15m no recovery: notify on-call, then team lead, then SRE manager."
  echo "Pager: $PAGER"
  echo "Owners: $OWNERS"
}

case "$1" in
  status) section_status ;;
  evidence) section_evidence ;;
  remediate) section_safe_remediate ;;
  verify) section_verify ;;
  escalate) section_escalate ;;
  *) echo "Usage: $0 {status|evidence|remediate|verify|escalate}" ;;
esac

How it prioritizes:- Quick status: non-destructive, fast checks- Evidence first: collect logs/metrics before remediation for RCA- Safe remediation: scale/restart via orchestration; avoid destructive mass restarts- Verify after each action; escalate if no progress

Making the runbook executable and version-controlled:- Store runbook.sh in a git repo (./runbooks/my-service/runbook.sh). Protect main branch, require PRs and review for edits.- Add executable bit in git: chmod +x runbook.sh; commit preserves mode.- CI lint: add shellcheck in CI (e.g., GitHub Actions) to prevent bad changes.- Tag releases (v1.0) and maintain CHANGELOG.md; require PR template asking: "Did you update runbook tests?"- Keep runbook small, update during postmortem; sign commits or protect branches for auditability.- Optionally wrap with Makefile: make evidence -> ./runbook.sh evidence

Best practices:- Keep commands idempotent and safe; avoid destructive defaults.- Short, explicit contact/escalation steps with time thresholds.- Regularly exercise runbook in game days and keep evidence collection commands current.

MediumTechnical

40 practiced

Describe a reliable one-liner using ss (or netstat) that lists listening TCP ports and the associated process names/PIDs, excluding IPv6, and explain how to interpret the output to map a socket to a service unit or binary.

Sample Answer

To list only IPv4 listening TCP sockets with their process name and PID in one line (suitable for diagnostics), use:

sudo ss -4 -tnlp

If you want a compact, parsed one-liner that shows Local:Port, PID and command:

sudo ss -4 -tnlp | awk '/LISTEN/ {
  sub(/.*LISTEN/, "", $0);
  split($1,a,":"); port=a[length(a)];
  pid=""; cmd="";
  if (match($0, /pid=[0-9]+/)) { pid=substr($0, RSTART+4, RLENGTH-4) }
  if (pid != "") { cmd = (getline_cmd = ("ps -p " pid " -o comm=") | getline; close(getline_cmd) ? cmd=$0 : cmd=""); }
  printf "%-22s %-8s %s\n", $4, pid, cmd
}'

How to interpret and map a socket to a service unit or binary:- ss columns: "State", "Recv-Q", "Send-Q", "Local Address:Port", "Peer Address:Port", "Process". The process info contains "pid=NNN,fd=X" — extract the PID.- Given the PID: - Inspect the binary: sudo readlink -f /proc/NNN/exe or ps -p NNN -o pid,cmd to see the exact command and args. - Map to systemd unit: sudo systemctl status $(systemctl list-units --type=service --all | awk -v p="$(ps -p NNN -o comm=)" 'tolower($0) ~ tolower(p) {print $1; exit}') or check cgroup: awk -F: '/^0:/{print;}' /proc/NNN/cgroup then correlate the cgroup to a unit name (systemd places service unit names in the cgroup path).- Example: ss shows pid=2345 for :80 → sudo readlink -f /proc/2345/exe shows /usr/bin/nginx; sudo systemctl status nginx.service confirms the unit and its status.

Notes / alternatives:- netstat equivalent: sudo netstat -tnlp | grep LISTEN- ss is faster and preferred on modern Linux.- Use sudo because kernel exposes process info only to privileged users.

HardTechnical

48 practiced

Security/privacy: When using tcpdump or similar packet-capture tools on production to troubleshoot API failures, what CLI options and practices would you use to minimize exposure of sensitive data (PII, auth tokens) while still collecting useful network metadata? Provide example tcpdump commands and redaction strategies.

Sample Answer

Good practices (high-level)- Minimize scope: capture only needed hosts/ports/protocols and limit duration. - Avoid full payloads: capture headers/first N bytes only. - Use write-to-file and post-process in a controlled environment (not broad/shared storage). - Prefer metadata extraction (timestamps, IPs, ports, TCP flags, RTT, sizes, TLS handshake info) over raw bodies. - Mask/redact sensitive fields ASAP, either by capture-time truncation or immediate post-capture processing. - Audit and approve captures; rotate and delete pcaps; encrypt at rest; notify stakeholders.

Useful tcpdump options and why- -s snaplen: set small snap length to avoid payload (e.g., 96 or 128 bytes to include headers but not body). - -w file: write binary pcap for controlled post-processing. - -c count and timeout via timeout / runit: limit size/time. - -nn: no name resolution (avoids extra lookups/leaks). - -U: packet-buffered output for piping. - filter expressions: host, net, port, and tcp[tcpflags] to reduce noise (e.g., capture SYN/ACK only). - -v/-vv for richer header fields (but still textual).

Example commands- Capture only TLS handshakes and headers between app host and API backend (no payload):

bash

# snaplen 128 keeps headers and TLS ClientHello but not application payload
sudo tcpdump -i eth0 -nn -s 128 -w /secure/path/app_api_hdrs.pcap \
  'host 10.0.1.12 and host 10.0.2.20 and tcp and port 443' -c 10000

- Capture metadata to stdout, pipe to a safe processor (no bodies):

bash

# -s 96 small snaplen, -U for line-buffered, limit 600 seconds
sudo timeout 600 tcpdump -i eth0 -nn -s 96 -U 'tcp port 443 and host 10.0.1.12' \
  -w - | tee /secure/path/stream.pcap > /dev/null

- Capture only TCP handshake events (useful for connectivity issues):

bash

sudo tcpdump -i eth0 -nn 'tcp[tcpflags] & (tcp-syn|tcp-ack) != 0 and host 10.0.1.12' -c 5000 -w syns_only.pcap

Post-capture redaction and metadata extraction- Use tshark to extract fields (no payload) and redact sensitive headers:

bash

# extract timestamp, src/dst IP:port, tcp flags, tls.handshake.version, http.request.method, http.request.uri
tshark -r app_api_hdrs.pcap -T fields -E occurrence=f \
  -e frame.time_epoch -e ip.src -e tcp.srcport -e ip.dst -e tcp.dstport \
  -e tcp.flags -e tls.handshake.version -e http.request.method -e http.request.uri \
  > metadata.tsv

- Remove application payloads from pcap with editcap (truncate each packet to N bytes):

bash

# truncate packets to 128 bytes to remove bodies permanently
editcap -s 128 app_api_hdrs.pcap app_api_truncated.pcap

- Hash or redact headers in textual exports:

bash

# export HTTP headers then replace Authorization values with SHA256
tshark -r app_api_hdrs.pcap -Y http -T json | \
  jq 'walk(if type=="object" and .\"http.authorization\" then .\"http.authorization\"="REDACTED" else . end)' \
  > http_redacted.json

(If jq not available, use small scripts to replace/mask tokens and email addresses.)

Advanced capture-time mitigation- Use very small snaplen to prevent tokens from being captured at all. - Combine BPF to exclude known sensitive ports/services sending tokens in cleartext (avoid capturing unencrypted HTTP). - For immediacy, pipe tcpdump to a filtering program that scrubs lines before write (e.g., tcpdump -U -A -s 0 | python scrubber.py > safe.log), but only use when scrubber is vetted.

Redaction strategies and controls- Principle: avoid ever capturing PII/auth tokens unless necessary. If unavoidable: - Capture to encrypted storage with strict ACLs; limit retention to the minimum; log who accessed. - Redact ASAP: prefer automated scripts that replace full token strings with deterministic hashes (hash allows correlating same token across events without revealing token). Use HMAC with a key stored in a secrets manager. - Keep an audit trail of who requested capture and why; get approvals per policy.

Edge cases and caveats- TLS: capture will not show HTTP bodies for HTTPS, but SNI and ClientHello may leak hostnames—treat SNI as sensitive if needed. - Short snaplen may miss useful debugging info (e.g., URL path if in first bytes), so balance privacy vs. debug needs; document trade-offs and use targeted short captures when debugging specific issues. - Scrubbing must be deterministic and tested; partial redaction can still leak tokens—validate redaction on copies before sharing.

Summary- Use narrow BPF filters, small snaplen (-s), limited duration/count, write to secure storage, post-process with tshark/editcap for metadata extraction and truncation, and apply vetted redaction (HMAC-hash sensitive fields) with strict access/retention policies. These controls let you solve outages while minimizing exposure.

MediumTechnical

35 practiced

The systemd journal shows frequent service restarts. From CLI, how would you collect recent journal entries for a specific service, correlate restarts with timestamps, extract the Restart= or exit code information, and save the artifact for a postmortem? Provide the commands (journalctl or alternatives) and explain how you'd filter by boot id and time-range.

Sample Answer

Situation: You suspect frequent restarts for my-service and need a reproducible artifact for postmortem showing when restarts happened, exit codes/Restart= lines, correlated by boot and time.

Commands and explanation:

1) Quick recent entries for the unit with readable timestamps:journalctl -u my-service -o short-iso --since "2 days ago" > my-service.journal.txt- -u filters by systemd unit; short-iso gives ISO timestamps; redirect saves artifact.

2) Limit to a specific boot (use --boot or boot id):journalctl -u my-service --boot=$(journalctl --grep=_BOOT_ID -n1 -o json | jq -r '._BOOT_ID') --since "2025-11-20 00:00" > my-service.boot.journal.txtOr simpler for last boot:journalctl -u my-service --boot=-1 --since "2025-11-20" -o short-iso > my-service.lastboot.journal.txt

3) Extract structured fields (exit codes, messages) for programmatic analysis:journalctl -u my-service --since "2025-11-20" -o json | jq -r '[.__REALTIME_TIMESTAMP, ._BOOT_ID, .PRIORITY, .MESSAGE] | @tsv' > my-service.jsonlines.tsv- json output makes it possible to parse CODE=... and "Main process exited" messages.

5) Correlate restarts across boots (group by boot id):journalctl -u my-service -o json | jq -r '{"boot":._BOOT_ID, "time":.__REALTIME_TIMESTAMP, "msg":.MESSAGE} | @json' > my-service.events.jsonThen:jq -s 'group_by(.boot)[] | {boot: .[0].boot, events: .}' my-service.events.json > grouped-by-boot.json

6) Save and compress artifact for upload:tar czf my-service-postmortem-$(date +%F).tgz my-service.* && sha256sum my-service-postmortem-$(date +%F).tgz > checksum.txt

Notes / rationale:- Use --since/--until to narrow time windows and --boot to avoid cross-boot noise.- Use -o json + jq to reliably extract fields like MESSAGE and __REALTIME_TIMESTAMP; grep can miss variations.- Look for "Main process exited, code=exited, status=..." and "Unit my-service.service entered failed state" to see exit codes and Restart= triggered restarts.- Always save raw journal output and a parsed summary so postmortem can reproduce timeline.

Unlock Full Question Bank

Get access to hundreds of Command Line Proficiency and Troubleshooting interview questions and detailed answers.

Join thousands of developers preparing for their dream job.