Linux System Administration Questions
Operating and maintaining Linux and Unix systems: filesystems and permissions, process and service management, package management and updates, shell and scripting, storage, and network configuration on the host. Covers day-to-day administration, troubleshooting, monitoring and tuning host resource usage (CPU, memory, swap, and I/O), and hardening of Linux servers that underpin most infrastructure. The Linux operator's core skill set.
Provide a sed one-liner to replace all occurrences of the string foo with bar in place for all .conf files under /etc, making a backup with extension .bak. Explain the sed flags you used and how to test changes before mass-apply.
Sample Answer
Direct answer
find /etc -type f -name '*.conf' -exec sed -i.bak 's/foo/bar/g' {} +
-i.bak edits each file in place while first saving the original as <file>.bak in the same
directory, so nothing is destroyed if the substitution turns out wrong. s/foo/bar/g is the
standard substitute command: replace every (g = global, not just the first) occurrence of
foo with bar on each line. find ... -exec ... {} + batches as many matched files as fit on
one command line into a single sed invocation (far cheaper than -exec ... {} \;, which forks
a new sed process per file).
Structured elaboration
Flag by flag
| Piece | Meaning |
|---|---|
find /etc -type f -name '*.conf' | Only regular files, recursively, whose name ends in .conf |
sed -i.bak | In-place edit; .bak is the backup suffix, applied to the original filename |
s/foo/bar/g | Substitute foo with bar, g for every match per line, not just the first |
-exec ... {} + | Pass matched files in batches as arguments to one sed call, not one call per file |
Why -i.bak specifically, not bare -i
Bare sed -i 's/foo/bar/g' file on GNU sed edits with no backup at all; on BSD/macOS sed, -i
requires a suffix argument (even -i '' for none) or it errors out or, worse, consumes the
next argument as the suffix by mistake. Writing -i.bak (no space) is the portable, safe form
that always leaves a recovery copy and works the same way with both sed implementations'
sane behaviors.
Testing before mass-apply
Never run the in-place version first against a directory tree you have not previewed:
- Preview matches without editing anything:
find /etc -type f -name '*.conf' -exec grep -l 'foo' {} +lists exactly which files containfooat all. - Preview the resulting text without writing it:
sed -n 's/foo/bar/gp' fileprints only the
lines that would change, post-substitution, without touching the file (-nsuppresses normal
output,pexplicitly prints matched-and-substituted lines). - Run the real in-place command only after step 1 and 2 look right, then diff a sample:
diff file.bak fileon one or two files to confirm the actual change matches what step 2 predicted. - Clean up backups once confirmed:
find /etc -name '*.conf.bak' -delete, or keep them if the
change touches anything you might need to roll back on a live service.
Worked example
Verified in a container against a small /etc-style tree:
$ cat a.conf
listen=foo
name=foobar
other=baz
$ sed -n 's/foo/bar/gp' a.conf # dry run preview, no file touched
listen=bar
name=barbar
$ find . -type f -name '*.conf' -exec sed -i.bak 's/foo/bar/g' {} +
$ cat a.conf
listen=bar
name=barbar
other=baz
$ diff a.conf.bak a.conf
1,2c1,2
< listen=foo
< name=foobar
---
> listen=bar
> name=barbar
A sibling c.txt (not matching *.conf) containing unrelated=foo was confirmed untouched,
proving the -name '*.conf' filter correctly scoped the change to configuration files only.
Trade-offs and pitfalls
s/foo/bar/gmatchesfooanywhere in a line, including inside a longer token
(myfoobar123). If the intent is a whole-word or whole-value replacement, use a
word-boundary anchor (s/\bfoo\b/bar/gwith GNU sed's\b, ors/\<foo\>/bar/gportably) or
anchor to the key (s/^foo=/bar=/) so you do not corrupt unrelated identifiers that merely
contain the substring.-i.bakwrites the backup next to the original; on a read-only or size-constrained/etc
partition (uncommon but real on some embedded or hardened images) doubling every touched file
temporarily can matter. Sweep and remove.bakfiles once verified.- A restart of the affected service is a separate step this command does not do; changing a
.conffile does not make a running daemon re-read it, plan the reload (systemctl reload <service>or equivalent) as part of the change, not as an afterthought. - Running this as a single blanket sweep across every
*.confunder/etcconflates unrelated
services; a safer production pattern scopes thefindto a specific subdirectory
(/etc/myapp/) rather than the entire/etctree, unless the change genuinely is meant to be
system-wide.
Write a Bash one-liner or a short Python script (specify which language you choose) that lists the top 5 processes by memory usage on a Linux server, outputting PID, user, RSS memory in MB, and command. This script will be used as a quick diagnostic during on-call shifts; state any assumptions about the environment.
Sample Answer
Direct answer
Bash, since it needs nothing beyond what every Linux box already has installed:
ps -eo pid,user,rss,comm --sort=-rss | head -n 6 | \
awk 'NR==1{printf "%-8s %-12s %10s %s\n","PID","USER","RSS(MB)","CMD"; next}
{printf "%-8s %-12s %10.1f %s\n",$1,$2,$3/1024,$4}'
ps -eo pid,user,rss,comm lists every process with exactly the four fields asked for; --sort=-rss
orders by resident memory descending; head -n 6 keeps the header plus the top 5; the awk
pass reformats RSS from kilobytes (what ps reports natively) into megabytes and aligns columns
for fast eyeballing during an incident.
Structured elaboration
What RSS actually measures, and why it is the right column here
RSS (resident set size) is how much physical RAM a process is currently occupying, as opposed to
its virtual size (VSZ), which includes memory it has merely reserved or memory-mapped but may
never touch. For "what is eating memory on this box right now," RSS is the relevant number;
VSZ routinely reads far larger and is not comparable across processes the way RSS is.
Assumptions this quick diagnostic makes
procps(which providesps, and the GNU--sortflag) is installed; it is on essentially
every mainstream distribution by default, but a minimal container image can lack it.- The command is run with enough privilege to see other users' processes; an unprivileged user's
ps -eo ...on a hardened system withhidepidset on/procmay only show their own
processes, silently narrowing "top 5 on the server" to "top 5 of mine." - RSS double-counts shared memory: two processes that both map the same shared library each
report that library's pages in their own RSS, so summing RSS across processes overstates true
physical usage. For a single "who is biggest right now" snapshot during an incident this does
not matter; for capacity planning across a whole process tree,PSS(proportional set size,
from/proc/<pid>/smaps_rollup) divides shared pages fairly and is the more honest number. - A single snapshot; if the top consumer is a short-lived burst, this catches it only if it is
still running at the moment you run the command.
Worked example
Run in a live container to prove the exact format and sort order:
$ ps -eo pid,user,rss,comm --sort=-rss | head -n 6 | awk '...'
PID USER RSS(MB) CMD
208 root 3.5 ps
1 root 3.5 bash
210 root 1.5 awk
205 root 1.3 sleep
209 root 1.2 head
On a real application server the same command surfaces the actual memory hog, for example a
leaking worker process climbing steadily across repeated runs, which is the on-call signal this
is built for: run it once to see the current top 5, run it again a minute later, and a process
whose RSS keeps climbing between the two runs is the leak candidate, not just the biggest number
in one snapshot.
Trade-offs and pitfalls
commtruncates to the process's short name (15 characters on Linux) and drops arguments; use
argsorcmdinstead ofcommin the-eofield list if you need to distinguish, say, three
differentpython3workers by their script name or flags.- A one-shot snapshot cannot show a growth rate; if the question during the incident is "is
this leaking or just big," run the same command twice a minute apart, or reach fortop -b -n 1in a loop, rather than trusting a single reading. - Summing the RSS column to estimate "total memory in use" overstates reality because of shared
pages (see PSS above); do not report that sum as the host's actual memory pressure, compare
againstMemAvailablein/proc/meminfoinstead. - If the goal shifts from "quick eyeball during an incident" to "durable metric for
dashboards/alerting," a one-offpscommand is the wrong tool; that belongs in something like
node_exporter scraping/proccontinuously, not a manually re-run diagnostic.
Practical file operations: you're asked to move /var/www/myapp to /mnt/fast-storage while preserving permissions, ACLs, extended attributes, and SELinux contexts with minimal downtime. What commands or sequence would you use on a Linux host? Explain why you chose those commands and mention safety measures (verification, dry-run, backup).
Sample Answer
Direct answer
Do not use mv across filesystems and do not use plain cp -r: neither preserves POSIX ACLs (Access Control Lists, the extended per-user/per-group permission entries beyond owner/group/other) or extended attributes by default, and mv across a filesystem boundary is really a copy-then-delete with no atomicity or verification built in. The right tool is rsync -aAX (archive mode plus ACLs plus extended attributes), run once live, verified, then run again as a short final sync during a brief cutover window, with the original left in place until the copy is verified byte for byte.
Why rsync -aAX and not cp or mv
-a(archive) already implies-rlptgoD: recursive, preserve symlinks, permissions, timestamps, group, owner, and device/special files.-Aadds POSIX ACL preservation, which plain-adoes not cover.-Xadds extended attributes (xattrs, arbitrary name/value metadata attached to a file outside its normal inode fields, where an inode is the on-disk record holding a file's core metadata, such as size, permissions, and timestamps, separate from the file's actual data), which is where SELinux (Security-Enhanced Linux, the mandatory access control system) file contexts actually live on disk, stored as thesecurity.selinuxxattr.cp --preserve=allis a legitimate single-host alternative that also preserves ACLs, xattrs, and (per itscontextsub-option) SELinux context, but it lacks rsync's resumability, delta transfer, and built in dry run mode, which matter once the copy is large enough that "just do it again" is not free.mvacross a filesystem boundary is never atomic: the kernel cannot do an in place rename across two different mounted filesystems, somvsilently falls back to copy-then-unlink, with none ofrsync's verification, resumability, or preservation flags. Never use it for this.
Step by step, with safety measures
- Dry run first:
rsync -aAXn --delete /var/www/myapp/ /mnt/fast-storage/myapp/(the trailingnin-aAXnis--dry-run,--deletepreviews removal of anything at the destination not present at the source). This costs nothing and tells you exactly what would move. - Live copy while the app keeps running, source untouched:
rsync -aAXH --numeric-ids --info=progress2 /var/www/myapp/ /mnt/fast-storage/myapp/.-Hpreserves hard links,--numeric-idsavoids UID/GID (user ID / group ID) translation problems if the two hosts'/etc/passwdever differ. - Verify before touching the original.
diff -rq /var/www/myapp /mnt/fast-storage/myappfor content, and separatelygetfacl -R --absolute-names /var/www/myapppiped to a file, same command against the destination, thendiffthe two ACL dumps, since a directory-content diff alone says nothing about permissions. - Brief cutover. Stop or pause the application (or put it in maintenance mode), run the same
rsync -aAXHcommand one more time so only the handful of files that changed during the copy window are re-sent (rsync only transfers deltas by default), then re-verify with the same diff pair from step 3. - Swap the path, not the data: rather than editing every config that references
/var/www/myapp, either bind mount the new location over the old path (mount --bind /mnt/fast-storage/myapp /var/www/myapp) or replace/var/www/myappwith a symlink to/mnt/fast-storage/myapponce nothing still has the old directory open (lsof +D /var/www/myappto check). A symlink is simpler to reason about; a bind mount is transparent to tools that resolve symlinks unexpectedly (some do not follow them for security checks). - Keep the original in place, read only, for a rollback window (
chmod -R a-w /var/www/myapp_oldafter renaming it out of the way) rather than deleting it immediately. Only remove it once the application has been observed healthy at the new path.
Worked example (executed)
Real run against a test tree with an ACL entry and an xattr set, to confirm both actually survive the copy rather than assuming rsync's flags do what the man page says:
setfacl -m u:www-data:rwx /var/www/myapp
setfattr -n user.myattr -v hello /var/www/myapp/index.php
rsync -aAX /var/www/myapp/ /mnt/fast-storage/myapp/
diff -r /var/www/myapp /mnt/fast-storage/myapp && echo "content identical"
getfattr -n user.myattr /mnt/fast-storage/myapp/index.php
Captured output confirmed the ACL entry (user:www-data:rwx) appeared identically on both sides via getfacl, diff -r reported no content differences, and the extended attribute round tripped: getfattr on the destination returned user.myattr="hello", exactly what was set on the source. Nothing here was assumed; every claim above about what -a, -A, and -X preserve was checked against this actual copy, not just the man page.
Trade-offs and pitfalls
The most common mistake is preserving ACLs and xattrs but forgetting SELinux context can still drift on a system where the two paths are labeled differently by policy (/var/www typically carries httpd_sys_content_t, a directory outside the policy's expected paths may not): after the move, run restorecon -Rv /var/www/myapp (or the new path, whichever the app actually reads from) to reapply the policy-correct label rather than trusting whatever xattr rsync copied, since a copied label can be stale rather than wrong if the destination path itself isn't covered by the same policy rule. Second, -X on a source filesystem that has no xattr support (some very old ext configurations, some network filesystems) will simply have nothing to copy, not error, which can create a false sense that preservation "worked" when there was nothing to preserve; confirm with getfattr -R before trusting it mattered. Third, doing the final cutover sync without stopping writers first can leave you copying a file mid write, producing a byte for byte mismatch that has nothing to do with rsync and everything to do with timing; always quiesce the source for the final pass.
A junior engineer asks you how to check disk usage and find the top 5 directories consuming space on a server. Provide the exact command(s) you would give (mention commonly available tools), and briefly explain how they work.
Sample Answer
The commands
du -h --max-depth=1 / 2>/dev/null | sort -rh | head -5
This is the standard, near-universally-available answer for "top 5 directories by disk usage." du (disk usage) walks the filesystem tree and totals up the size of everything under each path; -h prints sizes in human-readable units (K, M, G) instead of raw byte counts; --max-depth=1 stops it from recursing infinitely and reporting every single subdirectory at every depth, limiting output to just the immediate children of the starting path (adjust the path and depth to zoom into whichever directory turns out to be the big one); 2>/dev/null discards "Permission denied" errors from directories the current user can't read, which would otherwise clutter the output without changing the actual size totals; sort -rh sorts the human-readable sizes in reverse (largest first) numerically rather than alphabetically (plain sort -r would incorrectly rank "9.0M" above "1.2G" because it compares the leading character as text); head -5 keeps just the top five.
Companion command to confirm the overall picture first:
df -h
df (disk free) reports overall filesystem-level usage: total size, used, available, and percentage full, per mounted filesystem. The practical order is df -h first (which filesystem is actually full, and by how much), then du narrowed to that filesystem's mount point (which directories on it are responsible).
How they work
du doesn't read any pre-built index; it does a live, recursive walk of the directory tree, stat-ing every file it encounters and summing block usage (not necessarily the same as reported file size, since a file can be sparse or a filesystem can round up to block boundaries), which is why du on a directory with millions of small files can take a genuinely long time and generate meaningful I/O load, especially against network filesystems. df is much cheaper because it just asks the filesystem driver for its own aggregate counters, which the kernel already tracks continuously, rather than walking anything.
Worked example
$ df -h /
Filesystem Size Used Avail Use% Mounted on
overlay 89G 22G 67G 25% /
$ du -h --max-depth=1 / 2>/dev/null | sort -rh | head -5
506M /
438M /usr
66M /var
2.4M /etc
72K /tmp
(verified against a live container; the specific sizes and which directory comes out largest will differ on any other host, but the command mechanics and output shape are exactly reproducible on any Linux host running the same commands). The top line, 506M /, is du's own running total for everything under / that it just walked, not one of the five directories being ranked; it sorts highest precisely because it is the sum of everything below it. Read the next line down, here /usr, as the actual largest individual directory, with /var, /etc, and /tmp following behind it.
Trade-offs and pitfalls
duanddfcan legitimately disagree, most often because a file has been deleted but a process still holds it open:dfreports the space as still in use (correctly, since the data blocks aren't freed until the last open handle closes), whiledu, which only walks visible directory entries, no longer sees the file at all and doesn't count it. Finding the culprit process usually means checkinglsof +L1(files with a link count of zero, i.e. deleted-but-open) rather than moreduruns.- A very large
durun against a directory with millions of files, especially over NFS (Network File System, a protocol for mounting a remote server's filesystem so it behaves like local disk), can be slow enough to itself become a load or latency concern on a production host; scoping the search narrower first (usingdfto find which filesystem is actually the problem, thenduonly on that mount point) avoids an unnecessarily expensive full-tree walk. du's reported size reflects actual disk block usage, which can differ meaningfully from the "apparent size" reported byls -lfor sparse files (files with intentional holes);du --apparent-sizeswitches to reporting the nominal file size instead, which is a different, occasionally more useful, number depending on what question you're actually trying to answer.
Explain umask and how it affects newly created files and directories. If the umask is 027, what are the default permissions for a newly created file and a newly created directory? How can umask be set for systemd services or persistently for interactive shells?
Sample Answer
What umask does
umask sets a mask of permission bits that are removed from the default permissions the kernel would otherwise grant when a new file or directory is created. It doesn't grant permissions; it only takes them away from a baseline. The kernel's baseline is 666 (read+write for owner, group, and others; never execute) for a newly created plain file, and 777 (read+write+execute for owner, group, and others) for a newly created directory. The umask value is subtracted (bitwise) from that baseline to produce the actual permissions the new object gets.
Working out umask 027
Read the umask octal digit by digit, each digit removing the corresponding read/write/execute bit for owner, group, and others respectively:
0(owner): removes nothing, owner keeps full rights from the baseline.2(group): removes the write bit.7(others): removes read, write, and execute entirely.
Applying that to a new file (baseline 666, execute is never set on files by the creation baseline regardless of umask): owner keeps rw (6), group loses w leaving r (4), others lose everything (0). Result: 640, i.e. -rw-r-----.
Applying it to a new directory (baseline 777): owner keeps rwx (7), group loses w leaving r-x (5), others lose everything (0). Result: 750, i.e. drwxr-x---.
Verified: in an Ubuntu 24.04 container, after umask 027, touch umask_test_file produced mode 640 and mkdir umask_test_dir produced mode 750 (checked with stat -c '%a %n'), matching the derivation above exactly.
Setting umask persistently
For interactive shells, put a umask line in the shell's startup file: ~/.bashrc (per-user, interactive non-login) or /etc/profile / /etc/profile.d/*.sh (system-wide, login shells):
umask 027
This only affects shells (and anything launched from them) that source that file; it has no effect on processes started outside a shell, such as services started directly by an init system.
For a systemd service, the shell's umask is irrelevant, because the service is not spawned by a login shell; set it in the unit file instead:
[Service]
UMask=0027
followed by systemctl daemon-reload and restarting the unit. This is the setting that actually governs what permissions files that service's own process creates at runtime (log files, temp files, sockets), independent of whatever umask happens to be configured for interactive users on the box.
Trade-offs and pitfalls
- A umask can only remove permission bits from the creation baseline; it can never add a bit the baseline doesn't already offer, which is why files never get an execute bit from umask math alone (the baseline is
666, with no execute bit to begin with) even if the umask's math would otherwise leave one. - Setting umask in
~/.bashrcgives every interactive login a consistent default, but it does nothing for cron jobs, systemd services, or anything else that doesn't source that file; a common production mistake is tightening a user's shell umask and assuming it also tightened the permissions a service running as that user creates. - An overly permissive umask (like the traditional default
022, which leaves files world-readable) on a multi-user host or one holding sensitive config can expose data to every local account;027or077are common hardening choices specifically because they remove the "others" bits entirely.
Unlock Full Question Bank
Get access to all 41 Linux System Administration interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.