Linux System Administration Questions
Operating and maintaining Linux and Unix systems: filesystems and permissions, process and service management, package management and updates, shell and scripting, storage, and network configuration on the host. Covers day-to-day administration, troubleshooting, monitoring and tuning host resource usage (CPU, memory, swap, and I/O), and hardening of Linux servers that underpin most infrastructure. The Linux operator's core skill set.
Describe how you would use strace to diagnose a program that fails with 'Permission denied' when opening a config file. Include the strace invocation, how to capture output across forks, how to filter to file-related syscalls, and what specific return values you'll look for in the trace.
Sample Answer
Direct answer
Run the failing program under strace, filter to the file-related syscalls, and read the specific syscall's
return value and its error number (errno), since a "Permission denied" surfacing at the application level is
really the LAST in a sequence of file-related syscalls the kernel may have rejected, and strace shows
exactly which call, on exactly which path, with exactly which errno, the actual ground truth for what the
kernel refused and why.
Structured elaboration
Basic invocation:
strace -f -e trace=open,openat,stat,fstat,access -o trace.out <command>
-f (follow forks) matters for anything that forks or execs a child before the actual open() happens,
shell wrappers and many daemons do exactly this, without -f you'd only see the parent's syscalls and miss
the real one entirely. -e trace=<syscalls> filters the noise so you're reading the relevant handful of
lines instead of thousands of unrelated syscalls. -o writes to a file rather than interleaving with the
program's own output, keeping the trace clean to search afterward.
Capturing output across forks: add -ff alongside -f to write one trace file per process
(trace.out.<pid>) instead of interleaving every forked process into one file, useful when many children
are involved and you want to isolate one specifically. -tt adds microsecond timestamps to every line, so
you can correlate exact syscall timing against application-log timestamps.
What to look for: a line such as openat(AT_FDCWD, "/etc/myapp/config.conf", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied). The arguments show the EXACT path actually resolved (catching, for example, a
symlink resolving somewhere unexpected, or a relative path resolving against a different working directory
than assumed), and the return value plus errno pair is the ground truth for what the kernel rejected. If the
trace instead shows the open succeeding but a LATER read or write failing, that points away from basic
file permissions entirely (toward something like a quota, a stale NFS handle, or a security-module check
that gates specific operations differently from the open itself), so read the whole relevant sequence, not
just the first hit.
Attaching to an already-running process instead of launching fresh: strace -f -p <pid> -e trace=open,openat,access, for a failure buried deep inside an already-running long-lived service rather
than at simple startup.
Worked example
A non-root user's Python process attempts open("/etc/myapp/config.conf") on a file chmod'd to 000 (no
permission bits for anyone). The trace shows:
openat(AT_FDCWD, "/etc/myapp/config.conf", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied)
and Python's own traceback confirms PermissionError: [Errno 13] Permission denied, the same EACCES
(errno 13) the kernel returned, simply re-surfaced through the language runtime. This trace line is what
actually PROVES the cause is the Unix permission bits themselves, rather than, for example, the path not
existing at all (which would show ENOENT instead) or a mandatory-access-control denial such as SELinux
(which produces the identical EACCES at this syscall level, and only the audit log, not strace,
distinguishes the two).
Trade-offs and pitfalls
strace adds real overhead, each traced syscall round-trips through the kernel's ptrace mechanism (the
same facility strace itself is built on: the kernel stops the traced process at every syscall entry and
exit so strace can inspect and log it before letting it continue), so attaching it broadly (especially
-f across many threads or children) to a busy, latency-sensitive production process can itself cause
visible latency or timeouts. Scope the syscall filter as tightly as possible and detach as soon as the one
relevant event has been captured, rather than leaving a broad trace running. A genuinely intermittent
permission failure (works most of the time, fails occasionally) is much better caught by attaching with
-p and a narrow filter left running briefly than by trying to reproduce it fresh each time, since a
"works on retry" bug is often a race (a config file briefly holding the wrong permissions during a deploy,
for example) that a single fresh invocation won't reliably hit.
Give a command-line recipe to trace file descriptor leaks in a long-running server process. Include how you'd detect growth over time, correlate FDs to file paths or sockets, and tools to aggregate results across a fleet.
Sample Answer
Direct answer
Count file descriptors (FDs, the small integers a process uses to refer to open files, sockets, and pipes)
over time via /proc/<pid>/fd, resolve what each one actually points at with readlink, and cross-check
names with lsof; a leak is a specific target that keeps reappearing without ever closing, not just a
rising raw count.
Command-line recipe
Detect growth over time: a simple watch loop is often enough for a single host:
watch -n5 'ls /proc/<pid>/fd | wc -l'
For a trend you can graph rather than eyeball, append timestamped rows instead:
while true; do
echo "$(date +%s),$(ls /proc/<pid>/fd 2>/dev/null | wc -l)"
sleep 5
done >> fd-count.csv
Correlate FDs to file paths or sockets: raw counts alone don't tell you what's leaking, resolve the
targets and rank by how often each one repeats:
for fd in /proc/<pid>/fd/*; do readlink "$fd"; done | sort | uniq -c | sort -rn | head
Many identical entries (the same file path, or many socket:[nnnn] entries with no corresponding closes)
is the actual leak signal. lsof -p <pid> gives the same information in a more readable form, including the
type (regular file, socket, pipe) and, for sockets, the remote peer.
Aggregate results across a fleet: run the same one-liner via your existing fan-out mechanism (an SSH
loop over hosts, or your configuration-management tool's ad-hoc exec) and collect hostname,pid,fd_count
rows into one place, so you can rank hosts by growth RATE rather than manually checking one host at a time.
An early, cheaper warning sign worth watching independently of the count itself is /proc/<pid>/limits's
"Max open files" row: a slow leak typically first shows up in application logs as EMFILE (too many open
files) errors right around when the FD count approaches that limit.
Worked example
A reproducible demonstration: a process opens 30 files over about a second and a half without closing any
of them, standing in for a real leak.
import time
fds = []
for i in range(30):
fds.append(open(f"/tmp/leak_{i}.txt", "w"))
time.sleep(0.05)
time.sleep(5)
Sampling the FD count twice, two seconds apart, while this runs:
--- snapshot 1 ---
22
--- snapshot 2 (after 2s) ---
33
(the exact counts will vary run to run by a file or two depending on scheduling, the upward trend across
samples is the reproducible signal, not the precise numbers). Resolving the targets shows the leak clearly:
/proc/<pid>/fd/5 -> /tmp/leak_2.txt
/proc/<pid>/fd/6 -> /tmp/leak_3.txt
...
and lsof -p <pid> cross-checks the same file handles by name, confirming they are ordinary open files that
were never closed, not, for example, sockets still legitimately in use.
Trade-offs and pitfalls
A raw FD count can't distinguish a genuine leak from a busy process that legitimately holds many
connections open at once, such as a connection-pooling server; the readlink-and-aggregate step is what
turns "the count is high" into "these 400 descriptors are all sockets to the same peer that should plausibly
have been closed by now", the actual actionable signal. Also, /proc/<pid>/fd only shows a complete picture
when you're reading it as root or as the process's own owning user, a monitoring agent running under a
different, unprivileged account will silently undercount, which is easy to miss when you're comparing
numbers across a fleet running under different service accounts.
Describe how to ensure correct system time and timezone on a fleet of Linux servers using timedatectl and chrony. Show commands to set the timezone to UTC, enable and start chronyd/chrony, and verify that the system clock is synchronized. Explain why accurate time is important for distributed systems and log correlation.
Sample Answer
Direct answer
timedatectl sets and reports the system's timezone and overall clock status; chrony (the chronyd daemon, current default on RHEL/Fedora and Ubuntu alike, having replaced the older ntpd) is what actually keeps the clock synchronized against remote time sources over NTP (Network Time Protocol). Accurate, synchronized time matters on a fleet for one concrete reason above all others: correlating log lines and trace spans across multiple hosts only means anything if "5ms apart" reflects real elapsed time rather than clock drift between the two machines being compared.
Setting the timezone
timedatectl set-timezone UTC
timedatectl
Standardizing every server's timezone to UTC specifically (rather than each host's local timezone) is standard practice precisely because it removes a whole category of correlation error: comparing timestamps across hosts in different local timezones (or the same timezone during a daylight saving transition) requires converting before comparing, which is exactly the kind of manual step that gets skipped under incident pressure and produces a wrong conclusion about ordering of events.
Enabling and starting chrony
The service name differs by distribution, confirmed against real package contents rather than assumed:
- Debian/Ubuntu: the package installs
chrony.service.systemctl enable --now chrony. - RHEL/Fedora/Rocky: the package installs
chronyd.service.systemctl enable --now chronyd.
Getting this wrong (using the Debian service name on a RHEL host or vice versa) fails immediately with "unit not found," which is a fast, obvious failure, but worth knowing ahead of time rather than discovering it live during an incident.
Verifying synchronization
timedatectl
chronyc tracking
chronyc sources -v
timedatectl's own output includes an NTP service: active (or System clock synchronized: yes) line as a quick top level check. chronyc tracking gives the detail that actually matters: System time (the current offset from the reference source), Last offset, and RMS offset (a running measure of typical error), which tell you not just "is it synced" but "how good is the sync." chronyc sources -v lists every configured time source with a symbol indicating which one chrony currently trusts as its best reference (*) versus ones it is aware of but not currently using (+, or excluded ones marked differently), useful for confirming the fleet is actually reaching its intended internal or external NTP sources rather than silently falling back to something else or free running with no sync at all.
Why accurate time matters for distributed systems and log correlation specifically
- Ordering events across hosts. A distributed trace spanning three services on three hosts is only interpretable if their clocks agree closely; a host running even a few hundred milliseconds fast can make a downstream span appear to start before the upstream request that triggered it, actively misleading whoever is debugging a slow request path.
- Certificate and token validation. TLS certificate validity windows and time bound authentication tokens (session tokens, Kerberos tickets, many OIDC/JWT based flows (OIDC: OpenID Connect, a login-federation protocol; JWT: JSON Web Token, the signed, time-stamped credential those flows issue)) are checked against the local clock; a host with a clock meaningfully wrong can reject valid credentials or accept ones that should have expired, a security relevant failure mode, not merely a cosmetic logging inconvenience.
- Log aggregation and alerting thresholds. Centralized logging (ELK, Graylog, or similar) generally timestamps by the log line's own embedded time, not arrival time; a drifted host silently skews every event it contributes into the aggregate timeline, and a rate based alert (X errors within Y minutes) can misfire or fail to fire at all if the contributing hosts disagree about what time it currently is.
Worked example (verified service naming; live sync not reproducible in this sandbox)
timedatectl set-timezone UTC
systemctl enable --now chrony # Debian/Ubuntu
systemctl enable --now chronyd # RHEL/Fedora/Rocky
chronyc tracking
chronyc sources -v
The two service names above were confirmed by inspecting the actual installed package contents rather than assumed: the Ubuntu chrony package installs chrony.service (along with chrony-wait.service and chrony-dnssrv@.service), while the Rocky Linux chrony package installs chronyd.service (along with chronyd-restricted.service). Actually reaching and synchronizing against a real time server is not something this sandboxed container environment can demonstrate live (it has no route to real NTP infrastructure and, separately, timedatectl itself requires a running systemd/dbus session (systemd being the init system and service manager most current Linux distributions use to start and supervise services) that a plain container does not provide), so no fabricated chronyc tracking numbers are presented as if captured; the service names and command syntax above are real and verified, and any specific offset or stratum value from a live host would need to come from an actual running fleet, not be invented here to look convincing.
Trade-offs and pitfalls
The most common mistake is assuming synchronization is fine because the service is simply running, without ever checking chronyc tracking's actual offset: a service can be active while still meaningfully out of sync during its first few minutes after boot, or if every configured source is unreachable due to a firewall rule blocking outbound UDP 123. Second, mixing timezones across a fleet (some hosts local time, some UTC) is a self-inflicted correlation problem that costs nothing to avoid up front and a great deal of debugging time to untangle after the fact, once years of logs exist in inconsistent zones. Third, do not assume the RHEL service name works on Debian or vice versa when writing fleet wide automation; branch on distribution family explicitly rather than hardcoding one service name everywhere.
Describe how to set up a persistent ip route and a static IP on a modern Linux distribution (systemd-based). Provide the files or commands you would use, and how to verify the configuration.
Sample Answer
Direct answer
On a modern systemd-based Linux distribution the network stack itself is normally managed by one
of two systemd-adjacent tools: Netplan (Ubuntu, rendering to systemd-networkd or
NetworkManager) or NetworkManager directly via nmcli (RHEL/Fedora/most other current
distributions). Both persist a static IP and a route as a declarative config file that survives
reboot; neither uses the legacy ifconfig/route add commands, which do not persist at all.
Structured elaboration
Ubuntu: Netplan
Write a YAML file under /etc/netplan/ (mode 600, root-only, since it can contain credentials
for other network types):
# /etc/netplan/01-static.yaml
network:
version: 2
renderer: networkd
ethernets:
eth0:
dhcp4: false
addresses:
- 192.168.1.50/24
routes:
- to: default
via: 192.168.1.1
nameservers:
addresses: [1.1.1.1, 8.8.8.8]
Apply with sudo netplan apply (or netplan try, which auto-rolls-back if you do not confirm
within a timeout, the safer choice over a remote SSH session where a bad config could lock you
out). Netplan is a generator: it translates this YAML into the actual backend config (a
systemd-networkd .network file, or NetworkManager keyfiles, depending on renderer:) and
that generated config, not the YAML directly, is what persists the setting across reboots.
RHEL 8+/current: NetworkManager via nmcli
nmcli connection add type ethernet con-name static-eth0 ifname eth0 \
ipv4.method manual ipv4.addresses 192.168.1.50/24 \
ipv4.gateway 192.168.1.1 ipv4.dns "1.1.1.1 8.8.8.8"
nmcli connection up static-eth0
This writes a persistent connection profile (a keyfile under
/etc/NetworkManager/system-connections/) that NetworkManager reactivates automatically on every
boot; there is no separate "make it permanent" step, creating the connection profile is the
persistence mechanism.
Verification, either path
ip addr show <iface>confirms the address is actually bound.ip route showconfirms the route table, in particular the default route via the gateway you
set.resolvectl status(networkd) ornmcli device show <iface>(NetworkManager) confirms DNS
servers took effect.journalctl -u systemd-networkdorjournalctl -u NetworkManagerfor anything that failed to
apply.- Reboot (or at minimum,
systemctl restartthe relevant service) once and re-check the same
three commands, since a config that "looks right" in the file but was never actually re-applied
is a common false confidence trap.
Worked example
Both were verified for real. The Netplan YAML above was fed to netplan generate (the step
netplan apply runs internally before touching the live interface), which produced the exact
systemd-networkd unit it is documented to produce:
$ netplan generate && cat /run/systemd/network/10-netplan-eth0.network
[Match]
Name=eth0
[Network]
LinkLocalAddressing=ipv6
Address=192.168.1.50/24
DNS=1.1.1.1
DNS=8.8.8.8
[Route]
Destination=0.0.0.0/0
Gateway=192.168.1.1
The nmcli command above was run against a live NetworkManager instance and produced the
persisted keyfile exactly as documented:
$ cat /etc/NetworkManager/system-connections/static-eth0.nmconnection
[connection]
id=static-eth0
type=ethernet
interface-name=eth0
[ipv4]
address1=192.168.1.50/24
dns=1.1.1.1;8.8.8.8;
gateway=192.168.1.1
method=manual
One caveat worth calling out: this specific container's interface came up as plain eth0
(confirmed with ip -o link show), because Docker's bridge networking assigns veth interfaces
eth0 by convention and there is no PCI bus path for udev's predictable-naming rules to key off
inside a container network namespace. That does not generalize to real hardware or a VM: there,
with systemd-udev predictable network interface naming active (the default on virtually every
current distribution), the interface is almost never called eth0, it is something like
enp1s0, ens192, or enp0s3 depending on the bus and slot. Never hardcode eth0 in either
config on a real host; always confirm the actual name with ip -o link show first, since a
config written against a name the interface does not have will silently fail to apply to
anything.
Trade-offs and pitfalls
- Applying a bad static config over SSH can cut your own access;
netplan try(auto-rollback on
timeout) or bringing the change up on a console/out-of-band path first are the safe patterns,
applying blind over the connection you are about to change is not. - Mixing tools is a real footgun: if
NetworkManageris active and you also hand-edit
/etc/network/interfaces(the old Debian ifupdown format) or vice versa, the two can fight over
the same interface. Check which manager actually owns the interface (nmcli device status,
ornetworkctl statusfor networkd) before adding a second config source. - A route added with the old
ip route add ...command (no persistence file behind it) vanishes
on reboot; this is the single most common "it worked, then broke after a routine reboot"
incident on freshly built hosts, always encode the route in the same persistent config as the
address, never as an ad hocip route add. - DNS set via
nameservers:/ipv4.dnsonly takes effect if nothing else (like a DHCP client on
a different interface, or a stale/etc/resolv.confnot managed byresolvectl) is overriding
it;resolvectl statusshowing the expected servers is the actual proof, not just the config
file content.
A large-capacity filesystem holding billions of small files (think years of tiny telemetry records) is running out of inodes, and the write rate keeps growing. Propose an immediate mitigation and a longer-term architectural fix, and explain how you'd validate the fix is safe before rolling it out, plus how you'd roll back if it isn't.
Sample Answer
Direct answer
Billions of small files exhausting inodes at a growing write rate is a capacity problem that keeps getting worse, so the immediate mitigation buys time (freeing inodes and slowing the growth) while the real fix changes the STORAGE MODEL itself, since no amount of periodic cleanup keeps pace with a write rate that keeps growing. I'd pack or archive existing small files into fewer, larger units immediately to reclaim inode headroom, then move the architecture toward batching writes into larger objects (or a filesystem or storage system that doesn't tie capacity to a fixed inode count the way ext4 does), validating the new write path on a subset of traffic with an easy rollback before cutting everything over.
Structured elaboration
Immediate mitigation, to reclaim inode headroom and slow the bleeding:
- Identify and archive or delete the oldest, least-accessed files first, the same "delete frees inodes, nothing else does" principle as any inode exhaustion, but at this scale the practical mechanism matters: packing many small files that must be retained into a single archive (a tar file, for instance) collapses millions of inodes into one, satisfying both "keep the data" and "free the inodes" at once, rather than forcing a choice between them.
- If write rate is actively growing and outpacing cleanup, consider a temporary write-side throttle or buffer (queueing new records rather than writing each one as an individual file immediately) purely to prevent the emergency from getting worse while the real fix is built, understanding this is a stopgap, not a solution.
Longer-term architectural fix, addressing why this keeps recurring:
- Batch small records into larger files: instead of one file per telemetry record, buffer records for a time window (per minute or per hour, depending on freshness needs) and write them as one larger file (a columnar format like Parquet, or even a simple concatenated batch), which is often the single highest-leverage fix, since it directly attacks the file-count growth rate rather than just cleaning up after it.
- Move to object storage for this class of data: an S3-compatible object store has no inode concept at all in the traditional filesystem sense and is purpose-built for very large numbers of individually-addressable objects, removing this entire failure class rather than managing around it, at the cost of different latency and consistency characteristics than a local filesystem (network round-trip per access instead of local disk access, eventual consistency semantics depending on the specific store).
- Or, if a local filesystem is a hard requirement, choose one built for dynamic inode allocation rather than a fixed count set at format time (XFS allocates inodes as needed, growing with actual usage rather than being capped by a decision made once at
mkfstime), which removes this specific failure mode without changing the application's write pattern at all, though it doesn't address the operational cost of managing billions of small files (backup time,find/directory traversal cost, and so on) the way batching does.
Validating the fix is safe before rolling it out: run the new write path (batching, or the object-store write path) alongside the existing one on a subset of traffic first (a percentage of incoming telemetry, or a single non-critical source), comparing data completeness and correctness against the old path over a representative time window (spanning normal daily/weekly traffic patterns, not just a quiet hour) before trusting it as the sole path. Confirm downstream consumers of this data (anything reading these files today) can actually read the new format or location before cutting them over, since a storage-format change is also a contract change with every reader.
Rollback plan if it isn't safe: keep the existing write path fully intact and simply route ALL new writes back to it (a feature flag on the write path, not a destructive migration), and don't delete or migrate historical data out of its original format until the new path has been validated in production for a meaningful stretch of time; a migration that deletes the old format as it goes has no rollback at all once it's underway.
Worked example
A telemetry system currently writes one file per record, generating roughly 50 million new files per day and growing 5% month over month as more sources are onboarded, on an ext4 filesystem nearing its fixed inode ceiling. Immediate mitigation: identify the oldest 30 days of records (well past any need for per-record access, since queries against this data are always aggregated by hour or day, never by individual record) and pack each day into a single compressed archive, collapsing roughly 1.5 billion individual small-file inodes into 30 archive files, immediately restoring substantial inode headroom. Longer-term fix: change the write path to buffer records in memory and flush one Parquet file per source per hour instead of one file per record, cutting file creation from 50 million per day to roughly 480 per day (20 sources times 24 hours) for the same data volume, a reduction of about five orders of magnitude in file count for equivalent information content. This new hourly-batch write path runs in parallel with the old per-record path for two sources for one week, with a validation job confirming every record present in the old path's output also appears correctly in the new path's Parquet files for that same window, before the remaining sources are cut over; the old per-record write path is disabled only after every source has been validated and the downstream query layer confirmed working against the new format, with the old path's code left in place, dormant, as the rollback option for that transition period.
Trade-offs & pitfalls
- Packing existing files into archives trades immediate inode relief for a different access cost: reading one record out of an archive now requires extracting or seeking within it rather than opening it directly, which is fine for cold, rarely-accessed historical data and a poor fit if that data still needs frequent per-record access.
- Switching to object storage removes the inode ceiling entirely but changes the access latency and consistency model; validate that downstream consumers can tolerate that before treating it as a drop-in replacement.
- A migration that deletes source-format data as it migrates has no rollback path if a problem is discovered partway through; always validate the new path fully before removing the option to fall back to the old one, not simultaneously with the cutover.
Unlock Full Question Bank
Get access to all Linux System Administration interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.