Linux System Administration Questions
Operating and maintaining Linux and Unix systems: filesystems and permissions, process and service management, package management and updates, shell and scripting, storage, and network configuration on the host. Covers day-to-day administration, troubleshooting, monitoring and tuning host resource usage (CPU, memory, swap, and I/O), and hardening of Linux servers that underpin most infrastructure. The Linux operator's core skill set.
Demonstrate how to use journalctl to: 1) show logs for the previous boot, 2) follow logs in real-time for the unit nginx.service, and 3) filter messages by priority and timestamp. Explain the difference between persistent and volatile journals and how to enable persistent journaling.
Sample Answer
Direct answer
journalctl is the query tool for the systemd journal (systemd-journald, the logging service that replaced plain text /var/log files with an indexed, structured binary log on most current distributions). Use -b to select a boot, -f -u <unit> to follow one service's logs live, and -p/--since/--until to filter by priority and time. Whether that data survives a reboot depends entirely on whether persistent journaling is enabled, and on a fresh install of several distributions it is not.
The three asks
1. Logs for the previous boot. journalctl -b -1 shows the boot immediately before the current one (-b -2 the one before that, and so on); journalctl --list-boots shows every boot the journal actually has recorded, each with an index and a time range, which is the safer first step since -1 only works if persistent journaling was already on during that earlier boot.
2. Follow logs live for one unit. journalctl -u nginx.service -f: -u filters to a single systemd unit, -f behaves like tail -f, streaming new entries as they arrive. Add -n 50 to also show the last 50 existing lines before it starts following, since -f alone starts from "now" with no history.
3. Filter by priority and timestamp. journalctl -p err --since "2026-09-01 00:00:00" --until "2026-09-01 06:00:00": -p accepts either a numeric syslog priority (0 emerg through 7 debug) or its name (err, warning, crit, and so on), and by default means "this priority or more severe," so -p err includes crit, alert, and emerg too, not only exact matches. --since/--until accept both absolute timestamps and relative ones like --since "1 hour ago".
Persistent versus volatile journals
By default on many distributions the journal lives only in /run/log/journal, which is a tmpfs (RAM backed filesystem) and is wiped on every reboot: that is the "volatile" journal. A "persistent" journal lives in /var/log/journal on disk and survives reboots, which is what makes journalctl -b -1 actually useful days after the fact instead of only immediately after a crash.
To enable it: mkdir -p /etc/systemd/journald.conf.d, create a drop in with Storage=persistent under [Journal], then restart the journal service (systemctl restart systemd-journald). Journald also switches to persistent mode on its own the moment /var/log/journal exists, even without touching the config: mkdir -p /var/log/journal followed by systemd-tmpfiles --create --prefix /var/log/journal (the real systemd utility that applies tmpfiles.d-style directory and permission rules; there is no separate systemd-tmpfsutils binary) and a restart is enough, because journald detects the directory's existence and switches to persistent mode without a config change at all. The config drop in is for making that explicit and permanent regardless of whether the directory happens to exist. Confirm which mode is currently active with journalctl --disk-usage, which reports 0 or near it when everything is still volatile and about to disappear on reboot.
Worked example
# previous boot's logs
journalctl -b -1
# see what boots the journal actually has on disk
journalctl --list-boots
# follow nginx live, with 50 lines of history first
journalctl -u nginx.service -n 50 -f
# everything at error severity or worse in a specific window
journalctl -p err --since "2026-09-01 00:00:00" --until "2026-09-01 06:00:00"
# enable persistent journaling
mkdir -p /var/log/journal
systemd-tmpfiles --create --prefix /var/log/journal
systemctl restart systemd-journald
journalctl --disk-usage
I ran the journalctl --version and flag inventory (journalctl --help) against a real systemd 255 install to confirm the exact flag names above (-b, -u, -f, -p, -g, --since, --until, -n/--lines) rather than trusting memory, since a wrong flag name here is exactly the kind of small error that looks plausible and is not. This sandbox has no running systemd-journald (containers do not run systemd as PID 1, the very first process a Linux system starts at boot and normally responsible for bringing up and supervising every other process, here), so journalctl -k correctly reports "No journal files were found" rather than showing fabricated log lines: any example log content in an answer like this should be labeled illustrative, because inventing plausible looking denial or error messages as if captured would be worse than saying nothing.
Trade-offs and pitfalls
Forgetting to enable persistent journaling is the single most common mistake here: a host that only ever logs to the volatile, RAM backed journal loses everything the moment it reboots, including the exact evidence you need to diagnose why it rebooted. A second mistake is treating -p err as "only errors" and being confused when crit and emerg lines show up too; the priority filter is a floor of severity, not an exact match, by design. Persistent journaling does have a real cost worth naming: unbounded disk growth if you never set SystemMaxUse= in journald.conf, so enabling persistence should come with an explicit size cap, not just flipping the storage mode on and walking away.
Write a Bash function that creates a secure temporary directory, sets permissions so only the owner can access it, and registers a trap to remove it on EXIT. Provide the function and an example of how to use it in a script safely handling spaces in filenames.
Sample Answer
Direct answer
create_secure_tmpdir() {
local prefix="${1:-app}"
SECURE_TMPDIR="$(mktemp -d "${TMPDIR:-/tmp}/${prefix}.XXXXXXXX")"
chmod 700 "$SECURE_TMPDIR"
}
cleanup() {
local status=$?
[[ -n "${SECURE_TMPDIR:-}" && -d "$SECURE_TMPDIR" ]] && rm -rf -- "$SECURE_TMPDIR"
exit "$status"
}
main() {
create_secure_tmpdir "myapp"
trap cleanup EXIT
local target="$SECURE_TMPDIR/a file with spaces.txt"
printf 'hello world\n' > "$target"
}
main "$@"
mktemp -d creates the directory atomically with a random suffix (no race between "check if a
name is free" and "create it", which is what makes it secure: a predictable name in /tmp
that another user could pre-create or symlink is a classic local privilege-escalation vector).
chmod 700 then restricts it to the owner. The trap cleanup EXIT removes it whenever the
script exits, on success, on error, or on a signal, so a crash mid-script never leaves a
world-creatable temp directory lying around.
Structured elaboration
Why the function sets a global, not a local returned via command substitution
The tempting first draft calls mktemp, chmod, and trap all inside one function and then
captures its output with workdir="$(create_secure_tmpdir)". That is broken, and the breakage
is subtle enough to ship: $(...) runs the function in a subshell. The subshell exits the
instant the function returns (right after it prints the path), which fires the trap ... EXIT
that was set inside that subshell, deleting the directory before the parent script ever
receives the path back. The fix is to never route the path through command substitution:
create_secure_tmpdir sets a global variable directly, and trap is registered afterward in
the same shell that will eventually exit for real (the top-level script), not in a throwaway
subshell.
Handling spaces in filenames correctly
- Quote every expansion of the path:
"$SECURE_TMPDIR","$target". An unquoted$SECURE_TMPDIR
undergoes word splitting and would be treated as multiple arguments if the prefix orTMPDIR
contains spaces. - If you later need to iterate over files inside the directory, never do
for f in $(ls "$dir"); usefind "$dir" -print0 | while IFS= read -r -d '' f; do ... done, which uses a
NUL byte as the record separator so spaces (and even newlines) inside filenames cannot be
mistaken for separators. - Always terminate option lists with
--before a path argument (rm -rf -- "$dir") so a
filename that happens to start with-is never parsed as a flag.
Permissions
700 (rwx------) means only the owning user can read, write, or traverse into the directory;
group and others get nothing. mktemp -d on Linux already creates directories 700 by default,
so the explicit chmod 700 here is defense against a non-default umask (the mask that strips
permission bits off every newly created file or directory before any explicit chmod runs;
e.g. a umask of 022 would otherwise leave a fresh directory group- and world-readable) on some other
distribution or in some other user's shell, and it documents the intent instead of relying on an
implicit default.
Worked example
Run against the corrected function on a live container (Ubuntu 24.04):
$ bash myscript.sh
$ ls /tmp
(directory is already gone: cleanup fired on script exit)
Inside main, before exit, the directory and its file are confirmed present and correctly
permissioned:
$ stat -c '%a' "$SECURE_TMPDIR"
700
$ cat "$SECURE_TMPDIR/a file with spaces.txt"
hello world
By contrast, the broken "function sets its own trap and returns the path via $(...)" version,
run in the same environment, prints the path but the directory is already deleted by the time
the caller's next line runs, because the subshell's EXIT trap already fired.
Trade-offs and pitfalls
mktempwithout-dcreates a file; for a directory you must pass-d, and forgetting it
is a common copy-paste error when adapting a temp-file recipe.- Do not rely on
/tmpbeing cleaned automatically (some distributions runtmpfiles.dsweeps
on a schedule, others do not); a script must clean up after itself rather than assuming the OS
will. - A
trap ... EXITset inside a function is scoped to whichever shell process actually runs
that function. In a plain (non-subshell) function call this is the main script's shell, which
is correct; the moment you wrap the call in$(...), backticks, a pipeline, or&
(background), you have created a subshell and the trap silently stops protecting the resource
the caller thinks it owns. - If the script can be interrupted by a signal other than normal exit (Ctrl-C,
kill),trap cleanup EXITstill fires in bash because EXIT is triggered on any script termination path,
but if you also trapINT/TERMexplicitly for other cleanup, make sure that handler does not
itselfexitwithout eventually reaching the shared cleanup logic, or the temp directory leaks.
You are responsible for hardening SSH across a fleet. Provide an actionable plan covering /etc/ssh/sshd_config changes, centralized key or certificate management, rate limiting/brute force protection, use of bastion hosts, monitoring and alerting for suspicious access, and safe rollout steps to avoid locking out administrators.
Sample Answer
Direct answer
Hardening SSH fleet-wide means treating it as a single, deliberate rollout, not a checklist you paste into every host and hope for the best: lock down sshd_config to remove the weak defaults, move key management off individual hosts and onto something centralized and rotatable, add rate limiting so brute-force attempts can't run unbounded, funnel access through bastion hosts so there's one narrow, monitored front door instead of many, alert on suspicious access patterns, and roll every change out in a way that guarantees you can't lock yourself out of a fleet you can no longer reach.
Structured elaboration
sshd_config changes (at least eight concrete ones worth making):
PermitRootLogin no(no direct root login over SSH at all; usesudoafter logging in as a named user).PasswordAuthentication no(key-based or certificate-based auth only; passwords are brute-forceable, keys practically aren't).PubkeyAuthentication yes(explicit, though it's the default on modern OpenSSH; being explicit avoids surprises across distro defaults).AllowUsersorAllowGroupsrestricting SSH to a named allowlist of accounts or groups, so a stray local account with a weak password (if password auth were ever accidentally re-enabled) still can't SSH in.MaxAuthTries 3(caps how many authentication attempts a single connection gets before the server drops it, slowing down automated guessing).ClientAliveInterval 300paired withClientAliveCountMax 2(server sends a keepalive probe every 5 minutes and drops the session after 2 missed responses, so a stale or hijacked idle session doesn't linger indefinitely).LoginGraceTime 30(a connection that doesn't complete authentication within 30 seconds is dropped, limiting how long a slow-drip attack can hold a connection slot open).X11Forwarding noandAllowTcpForwarding nounless a specific host genuinely needs them, since both expand what a compromised SSH session can be used for.- Restricting
Ciphers,MACs(Message Authentication Codes, which verify that each packet
wasn't tampered with in transit), andKexAlgorithms(Key Exchange algorithms, which negotiate
the shared session key when a connection is first established) to modern, non-deprecated options
only (removes legacy algorithms a downgrade attack could target).
Changing the default port (22) is a genuine trade-off, not a free win: it cuts down the sheer volume of automated scanning noise hitting your logs, which has real operational value (smaller haystack for real signal), but it is security through obscurity, not a security control on its own: a targeted attacker doing a full port scan finds it in seconds. Do it for noise reduction if you want it, never present it as a substitute for the actual auth hardening above.
Centralized key or certificate management: distributing and rotating individual authorized_keys files across a fleet by hand doesn't scale and makes revocation slow (you have to touch every host). An SSH certificate authority (issuing short-lived signed certificates instead of long-lived static keys) lets you revoke access by simply not renewing a certificate rather than hunting down and removing a key from every host it was copied to, and it gives you a single place to audit who was issued access and when.
Rate limiting and brute-force protection: something watching auth logs and temporarily banning source IPs after repeated failures (fail2ban or an equivalent) adds a layer beyond MaxAuthTries, which only limits attempts within a single connection, not across many connection attempts from the same source.
Bastion hosts: route all SSH access through a small number of hardened, heavily monitored jump hosts rather than exposing every fleet member directly to the internet or even to the whole internal network; this shrinks the actual attack surface to those few hosts and gives you one place to centralize logging and session recording.
Monitoring and alerting: ship sshd auth logs (successful and failed logins, key fingerprints used) to centralized logging, and alert on patterns like repeated failures from one source, logins at unusual hours for that account, or logins from a new geography for a user who normally connects from one office.
Safe rollout, so you don't lock out your own administrators:
- Never deploy
PasswordAuthentication noto a fleet until you've confirmed every administrator who needs access already has a working key deployed and tested. - Always test the config file for syntax errors before applying:
sshd -tvalidatessshd_configwithout touching the running daemon. - Reload (
systemctl reload sshd), don't restart, where possible: a reload applies new config to the existing daemon without dropping already-established connections, so if something is subtly wrong you still have your current session to fix it from. - Roll out through configuration management in waves (a canary group of a few non-critical hosts first, verified, then the rest), never as a single fleet-wide push, and keep a documented, tested out-of-band access path (console access, a cloud provider's serial console, or a bastion path outside the change) as a safety net for the rollout itself.
Worked example
For a 200-host fleet, the rollout order matters as much as the content: first, issue everyone SSH certificates from a central certificate authority and confirm every admin can authenticate with the new certificate while password auth is STILL enabled as a fallback. Second, push the full sshd_config hardening set above to a 5-host canary group via configuration management, with PasswordAuthentication left yes for one more cycle even there, and confirm certificate-based login works cleanly and sshd -t passes on every canary host. Third, once the canary group has been stable for a monitoring cycle (confirming no locked-out sessions, no unexpected auth-log errors), flip PasswordAuthentication no on the canary group specifically, verify again, then roll the complete config (hardening plus password auth disabled) to the rest of the fleet in batches, keeping the documented out-of-band console access path live and tested throughout in case any batch goes wrong.
Trade-offs & pitfalls
- Disabling password authentication before every administrator has a verified, working key is the single most common way this kind of hardening locks a team out of their own fleet; sequencing that verification first is not optional.
- A restart (versus a reload) of
sshdmid-change, on a host you're connected to only via SSH, can drop your own session before you've confirmed the new config actually works, leaving you locked out with no way back in except out-of-band access. - Renaming the SSH port reduces log noise but provides no real protection against a targeted attacker; treat it as a nice-to-have, not a control you'd cite in a security review.
Describe the lifecycle commands for services and the difference between stop, restart, reload and daemon-reload in systemd. Provide examples of when you would prefer reload over restart and the trade-offs involved (state preservation vs configuration application).
Sample Answer
Direct answer
stop and start (or restart, which is just the two back to back) tear the process down and bring up a brand new one, losing whatever state only existed in that process's memory. reload asks the running process to re-read its configuration without dying at all, by sending it whatever signal or command the unit defines for that purpose, so in-memory state and existing connections survive. daemon-reload is different from all three: it does not touch any running service; it tells systemd itself (systemd being the init system and service manager most modern Linux distributions use to start and supervise processes) to re-read unit files from disk after you have edited one, so systemd's own idea of the configuration is current.
The four commands and what each actually does
| Command | What happens to the running process | What happens to systemd's unit definitions | Typical use |
|---|---|---|---|
systemctl stop svc | Terminated | Unchanged | Taking a service down deliberately |
systemctl start svc | New process launched | Unchanged | Bringing a stopped service up |
systemctl restart svc | Terminated, then a new process launched | Unchanged | Applying a change that requires a fresh process (new binary, changed environment) |
systemctl reload svc | Same process, told to re-read its config in place (via its ExecReload= command, commonly a SIGHUP) | Unchanged | Applying a config change without dropping connections or in-memory state |
systemctl daemon-reload | Nothing (no service is touched) | systemd re-parses all unit files from disk | After editing a unit file or adding/removing a drop-in |
The critical detail: reload only works if the unit defines an ExecReload= line. If it does not, systemd refuses the request outright rather than silently doing nothing or falling back to a restart:
$ systemctl reload noreload.service
Failed to reload noreload.service: Job type reload is not applicable for unit noreload.service.
That exact error means the unit has no ExecReload=, and a restart is the only option for that particular service, regardless of how the change looks conceptually.
Worked example: reload versus restart on a stateful service
Take a web server holding open, long-lived client connections, with ExecReload=/bin/kill -HUP $MAINPID in its unit (a common pattern: send the running process SIGHUP and let the application's own signal handler re-read its config file in place). Changing a setting like a log level or an upstream backend address and running systemctl reload webserver sends that signal to the same running process (its PID is unchanged before and after), which means every connection the server currently holds open stays open through the change, and only the parts of its configuration the application chooses to re-read on SIGHUP actually take effect. Running systemctl restart webserver instead terminates that process entirely and starts a fresh one: every open connection is dropped, clients get connection-reset errors mid-request, and the new process starts from a completely clean slate, which is exactly what you want if the change is a new binary version rather than a config tweak, since a signal-based reload cannot swap out running code, only re-read configuration.
Trade-offs and pitfalls
reloadis not free of risk just because it is gentler: an application whose reload path has bugs (a known category of bug class: config reload leaking memory or file descriptors over many reload cycles) can degrade slowly in ways a clean restart would not, so a service that gets reloaded dozens of times a day without ever being restarted is worth watching for resource growth over time.- Do not assume every service that accepts
reloadre-reads its entire configuration on that signal; many applications intentionally only re-read a subset (log level, TLS certificate paths) and require a full restart for anything structural (listening port, worker process count). Check the application's own documentation for what itsExecReload=command actually causes it to do, not just whether the unit defines one. daemon-reloadis easy to forget after hand-editing a unit file directly (as opposed to usingsystemctl edit, which does it for you): systemd keeps using the previously loaded definition until told otherwise, so a subsequentrestartcan appear to "not pick up" a change that was in fact saved correctly to disk, purely because systemd itself never re-read it.
Given the following vmstat sample line: 'procs -----------memory---------- ---swap-- -----io---- -system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 1 1 0 2048000 102400 512000 0 0 0 128 256 1024 2 1 79 18 0', explain what the most important fields indicate and what the values suggest about CPU and I/O behavior on the host.
Sample Answer
Direct answer
This line describes a mostly idle host with one meaningful flag: CPU is barely touched (3% combined user
plus system time), but wa (I/O wait) sits at 18%, and there's a process sitting in uninterruptible sleep
(b = 1) at the same instant. Those two facts together describe a host waiting on a slow disk operation,
not one struggling with CPU or memory pressure.
Reading the fields
vmstat's output groups columns by category, all counts are per second unless noted, and by default sizes
are reported in kilobytes:
- procs:
r= 1 process currently runnable (not overloaded, most healthy hosts run well above this
without trouble),b= 1 process in uninterruptible sleep ("D state"): blocked on a request to the kernel
that it cannot be interrupted out of, almost always disk I/O or a network filesystem call. Seeingb= 1
at the same momentwais elevated is not a coincidence, that process is very likely the one behind the
wait. - memory:
swpd= 0 (no swap space currently in use, a healthy sign),free= 2,048,000 KB (about 2GiB
free),buff= 102,400 KB (buffer cache: memory the kernel uses to cache block-device metadata such as
filesystem structures) andcache= 512,000 KB (page cache: memory holding recently read or written file
contents so repeat access skips the disk), both modest and unremarkable given how much is free. - swap:
si/so(kilobytes/sec swapped in and out) both 0, so nothing here is disguised swapping. - io:
bi= 0 blocks (of 1024 bytes each by default) read in during the sample,bo= 128 blocks/s
written out, roughly 0 KB/s of reads and about 128KB/s of writes: a genuinely small I/O volume. - system:
in= 256 interrupts/sec,cs= 1024 context switches/sec, both unremarkable for a lightly
loaded host. - cpu: percentages of total CPU time, and here they add up cleanly to 100 (2+1+79+18+0):
us= 2% user
time,sy= 1% kernel/system time,id= 79% idle,wa= 18% waiting on outstanding I/O,st= 0%
stolen by a hypervisor (only relevant on virtualized hosts; 0 means no visible contention from a
neighboring guest).
What the values suggest
CPU is essentially unused (3% combined user and system time), which rules out a compute-bound host. The
notable figure is wa at 18% against bi/bo showing almost no throughput (0 KB/s reads, ~128KB/s
writes): wa measures CPU time spent idle while at least one I/O is outstanding, not I/O volume, so a high
wa paired with low bi/bo points at a small number of I/O operations each taking unusually long (high
per-operation latency), rather than the host being overwhelmed with I/O volume. The b = 1 confirms there
is in fact a process stuck waiting on the kernel for exactly this kind of operation right now, not a
transient artifact of the sample. That combination is consistent with a slow or contended backing device (a
throttled cloud volume, a network filesystem under load) rather than a workload that is simply doing a lot
of I/O. Still, r = 1 and only one process blocked mean there's no visible CPU contention and just one
thing stuck at this exact snapshot, so this single sample alone doesn't prove a sustained problem: it's the
kind of line that should prompt watching a few more samples and checking iostat -x for the affected
device's average wait time (await) before concluding anything.
Trade-offs and pitfalls
A single vmstat line is a one-second snapshot; the classic mistake is reading meaning into it as if it
were a trend. The very first line vmstat ever prints on a running system is a since-boot average, not a
live sample, always discard it and read the second line onward. And wa alone never tells you WHICH device
or process is responsible, that requires iostat -x (per-device latency) and pidstat -d (per-process I/O)
as the next steps, vmstat only tells you that something worth investigating is happening.
Unlock Full Question Bank
Get access to all Linux System Administration interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.