Linux System Administration Questions
Operating and maintaining Linux and Unix systems: filesystems and permissions, process and service management, package management and updates, shell and scripting, storage, and network configuration on the host. Covers day-to-day administration, troubleshooting, monitoring and tuning host resource usage (CPU, memory, swap, and I/O), and hardening of Linux servers that underpin most infrastructure. The Linux operator's core skill set.
You export /srv/export via NFS and observe that root-owned files created on clients appear as 'nobody' on the server. Explain root_squash, all_squash, anonuid/anongid options and how to configure /etc/exports to allow root write access from a specific CIDR while still preventing other hosts from having that access. Describe security considerations and alternatives such as Kerberos-secured NFS.
Sample Answer
Direct answer
Root's UID (user ID, the numeric identity the kernel actually checks) is 0, and by default NFS (Network File System) maps any client request arriving as UID 0 to an unprivileged "nobody" identity on the server, a protection called root squashing, specifically so that having root on some client laptop does not automatically grant root over every file in the export. root_squash (the default) enables that mapping, no_root_squash disables it for a specific client and grants it real root, all_squash extends the same mapping to every UID from that client, not just root, and anonuid/anongid choose which UID/GID a squashed request is mapped to instead of the default "nobody" (usually 65534).
Configuring /etc/exports for the actual requirement
The requirement is: allow root write access from one specific network (a CIDR, meaning Classless Inter-Domain Routing, the address/prefix-length notation like 10.0.5.0/24 that names a whole subnet in one line), while every other host is denied even that basic access, let alone root's.
/srv/export 10.0.5.0/24(rw,sync,no_root_squash,no_subtree_check)
(sync here means the server only acknowledges a write once it is actually committed to stable storage, the safer default over async; no_subtree_check skips an extra verification step NFS otherwise does when only part of a filesystem is exported, recommended on modern kernels for a whole-filesystem export like this one to avoid a class of spurious stale-file-handle errors; both are covered in more depth under Trade-offs and pitfalls below.)
- Nothing else is listed as a client for
/srv/export, so any host outside10.0.5.0/24gets no access at all, not even squashed access; NFS exports are opt in per host or subnet, there is no implicit "everyone else gets root_squash instead." no_root_squashonly applies to the client(s) named on that line. A separate line for a different subnet or host, withoutno_root_squash, keeps the default protection for everyone else:/srv/export 10.0.9.0/24(rw,sync,no_subtree_check)for a different team that should get normal (root squashed) write access but not root.- If you instead wanted to lock every client, including the trusted subnet, down to a single fixed identity regardless of what UID they connect as:
/srv/export 10.0.5.0/24(rw,sync,all_squash,anonuid=2000,anongid=2000,no_subtree_check), mapping every request from that subnet, root or not, to UID/GID 2000. This is the opposite end of the spectrum fromno_root_squash: it trusts the network boundary itself less, not more.
Security considerations
no_root_squash should be treated as a genuinely dangerous exception, not a convenience default: any client granted it, if compromised, has root level write access to the export, which usually means it can write a setuid binary or overwrite a file another, more trusted host will later execute. Scoping it to the narrowest possible CIDR (or better, a single host IP, not a whole subnet) limits the blast radius of that trust to exactly the machines that need it, typically a small number of build or backup hosts that have a specific, understood reason to need real root over the export. NFS as configured above is also fundamentally trusting the client's own claim about which UID a request comes from: the AUTH_SYS (also called AUTH_UNIX) authentication NFS uses by default takes the client's word for its UID/GID with no cryptographic verification at all, which is why no_root_squash matters as much as it does, and it is also why NFS exports should sit behind a firewall or private network boundary regardless of squash settings, never exposed on a public interface.
The stronger alternative: Kerberos secured NFS
NFSv4 with Kerberos (sec=krb5, or sec=krb5i for per-packet integrity checking, or sec=krb5p for full encryption) replaces the client's self-asserted UID with a cryptographically verified identity from a Kerberos ticket, closing the "any client can just claim to be root" gap that AUTH_SYS has by construction. An export line using it looks like /srv/export 10.0.5.0/24(rw,sync,sec=krb5i,no_subtree_check), and the trade-off is real operational cost: it requires a working Kerberos realm (a KDC, or Key Distribution Center, the trusted server that authenticates identities and issues tickets; keytabs, files holding a service's long-term secret key so it can prove its identity to the KDC without a password prompt each time, distributed to every NFS server and client; and clock synchronization tight enough for Kerberos ticket validity, which is itself a reason accurate time sync across a fleet matters well beyond just log correlation), which is meaningfully more infrastructure than a plain /etc/exports line. For an internal, already network-isolated fileserver serving a small trusted set of build hosts, scoped no_root_squash on a narrow CIDR is a common, defensible trade-off; for anything crossing a security boundary you do not fully control, Kerberos secured NFS is the correct answer, not a nice-to-have.
Trade-offs and pitfalls
The most common mistake is granting no_root_squash to an entire broad subnet "to make things easier," which quietly hands every machine on that subnet real root over the export the moment any one of them is compromised; always scope it to the smallest set of hosts that genuinely need it. A second mistake is forgetting that no_subtree_check (recommended on modern kernels for exports of a whole filesystem, as shown above) trades a small amount of extra checking for avoiding a specific class of stale file handle errors after files are renamed on the server; leaving the default subtree checking on for a whole-filesystem export is a common source of confusing intermittent NFS errors that has nothing to do with permissions at all. Third, remember that sync (the safe default, requiring the server to commit writes to stable storage before acknowledging them) versus async is itself a durability trade-off independent of the squash settings: async is faster but can silently lose the last few writes if the server crashes before flushing, which matters more for some exports than others.
Explain the purpose of SELinux or AppArmor and how mandatory access control differs from traditional Unix discretionary permissions. Show commands to check whether SELinux is enforcing, list file contexts with ls -Z, and find denials logged in audit logs.
Sample Answer
Direct answer
Mandatory access control (MAC) is a system-wide policy set by an administrator that constrains what EVERY
process can do regardless of file ownership, not even the owner (and, depending on policy, not even root)
can grant more access than policy allows; discretionary access control (DAC), the traditional Unix
owner/group/other permission bits, instead lets the resource's OWNER decide who else gets access, with no
layer above that decision. The two stack: DAC is checked first, and if it allows the access, MAC (SELinux
or AppArmor) is checked second and can still say no, MAC is strictly additive, it can only restrict further,
never grant what DAC itself denies.
Structured elaboration
A process running as root normally bypasses almost all DAC checks, exactly the scenario MAC exists to
constrain: even a process running as root, if confined by an SELinux or AppArmor policy, is limited to the
specific files, sockets, and capabilities its policy actually names, so a vulnerability in that one service
does not automatically translate into full root-level filesystem access, a real, meaningful reduction in
blast radius.
SELinux (Security-Enhanced Linux, originally developed by the NSA and merged into the mainline kernel)
is label-based: every process and file carries a security context (a type, for example httpd_t for a
running Apache HTTP Server process, httpd_config_t for its config files), and policy is written as rules
over pairs of these types, "type enforcement". It is the default MAC system on Red Hat Enterprise Linux,
Fedora, and the CentOS/Rocky/AlmaLinux family.
AppArmor is path-based instead: profiles are written per executable path (the binary at a given path may
read these paths, write those paths), generally easier to read and write than SELinux's type system, but
inherently a little less precise, since path-based rules can be fooled by symlinks or bind-mounts more
easily than label-based rules that travel with the file's own inode-associated context. It is the default
MAC system on Ubuntu and the SUSE family.
Checking whether SELinux is enforcing: getenforce returns Enforcing (blocking and logging denials),
Permissive (logging only, useful for safely testing a new policy), or Disabled; sestatus gives fuller
detail including the loaded policy name and version.
Listing file contexts: ls -Z <path> appends the SELinux context (user:role:type:level, for example
system_u:object_r:httpd_config_t:s0) to normal ls output; ps -Z does the same for running processes.
Finding denials in the audit logs: ausearch -m avc -ts recent (AVC, access vector cache, the kernel's
decision cache for these checks, "AVC denial" being the standard term for a logged block) pulls matching
entries from /var/log/audit/audit.log; if auditd isn't running, the same denial may still surface via
journalctl or dmesg through the kernel's audit subsystem falling back to printk.
Worked example
Two identical Apache HTTP Server installs, serving from /var/www/html, one on a distribution using
SELinux, one using AppArmor. On the SELinux host, moving a file into that directory with mv from a user's
home directory carries the SOURCE directory's label along with it (every file has an inode, the kernel's
on-disk record of its metadata, ownership, permissions, and, under SELinux, its security label, kept
separately from the file's name; a mv within the same filesystem just re-points a directory entry at the
same inode rather than creating a new file, so the label, being a property of that inode, is preserved
across a same-filesystem move), so the file often ends up mislabeled and unreadable by httpd until
restorecon fixes it, a very common real-world surprise. On the AppArmor host, the equivalent failure
instead comes from a profile written against one specific path that doesn't cover a symlinked alternate
document root, since AppArmor's path matching doesn't automatically follow a symlink's target the way a
label-based check would.
Trade-offs and pitfalls
The single most common operational mistake with either system is treating a MAC denial as a reason to
disable the whole subsystem (setenforce 0 permanently, or leaving every AppArmor profile in complaint-only
mode) rather than fixing the one specific rule, throwing away the entire security benefit fleet-wide for a
one-file problem. Also, don't assume DAC becomes irrelevant once MAC is added, an attacker or a broken
deploy still has to clear the classic Unix permission check FIRST, a hardening review has to check both
layers, not just the newer one.
Design a permissions model for /srv/projects where multiple teams need write access to their own project directories but must not access other teams' data. Use POSIX ACLs and default ACLs to show how you would set this up and how to ensure new files inherit desired ACLs.
Sample Answer
Direct answer
Give each team's directory to a dedicated group, set the directory itself to deny "other" access entirely, add a default ACL (Access Control List, the extended per-user/per-group permission entries beyond the classic owner/group/other bits) on each team directory so new files and subdirectories automatically inherit the team's group permissions, and never rely on umask alone to keep teams apart, since umask is a per-process default, not an enforced boundary.
Design
- Base directory, traversal only.
/srv/projectsitself getschmod 751: owner (root) full access, group read+execute (list and traverse), other execute only (traverse but not list). This means nobody canls /srv/projectsand enumerate every team's name unless they already know it, without blocking legitimate traversal into a subdirectory they do have rights to. - One group per team, one directory per team, setgid plus a matching group owner. For team A:
groupadd teamA,mkdir /srv/projects/teamA,chown root:teamA /srv/projects/teamA,chmod 2770 /srv/projects/teamA. The2sets the setgid bit on the directory, so every file and subdirectory created inside automatically inherits the groupteamAregardless of the creating user's primary group;770gives the owner and group full access and gives other nothing. - Default ACL for the group, so inheritance is explicit and covers ACL entries, not just the base group bit.
setfacl -d -m g:teamA:rwx /srv/projects/teamAandsetfacl -d -m o::--- /srv/projects/teamA. The setgid bit alone already gets you group inheritance for the classic Unix group bit; the default ACL matters once you need more than one principal to inherit access, for example a cross team auditor group that should get read only access to every team's directory without being the owning group. - Repeat per team, each with its own group, directory, and default ACL. Isolation between teamA and teamB comes entirely from teamB never being named in teamA's ACL or group, combined with
other::---at both the base directory (traversal denied without a matching entry) and inside each team directory (no residual access for anyone not explicitly granted it).
Worked example (executed, including a real cross team access attempt)
Set this up for real, created a user in each team, and actually tried to cross the boundary rather than just inspecting the ACL:
groupadd teamA; groupadd teamB
useradd -m -g teamA alice
useradd -m -g teamB bob
chmod 751 /srv/projects
mkdir /srv/projects/teamA
chown root:teamA /srv/projects/teamA
chmod 2770 /srv/projects/teamA
setfacl -d -m g:teamA:rwx /srv/projects/teamA
setfacl -d -m o::--- /srv/projects/teamA
Resulting ACL (getfacl):
user::rwx
group::rwx
other::---
default:user::rwx
default:group::rwx
default:group:teamA:rwx
default:mask::rwx
default:other::---
Alice (teamA) creates a file, and it correctly inherits the team's group:
-rw-rw----+ 1 alice teamA 0 Sep 24 00:24 /srv/projects/teamA/notes.txt
Bob (teamB, not a member of teamA) attempting to touch teamA's directory at all:
$ ls /srv/projects/teamA
ls: cannot open directory '/srv/projects/teamA': Permission denied
$ cat /srv/projects/teamA/notes.txt
cat: /srv/projects/teamA/notes.txt: Permission denied
Both denied, exactly as designed. Alice, meanwhile, reads and writes her own team's file without issue.
To show the actual value a default ACL adds beyond the setgid bit alone, I added a third, cross-cutting audit group with read only access via ACL, not group ownership:
groupadd audit; useradd -m -g audit carol
setfacl -m g:audit:r-x /srv/projects/teamA
setfacl -d -m g:audit:r-x /srv/projects/teamA
A brand new file created afterward by alice automatically picked up the audit entry (getfacl on the new file showed group:audit:r-x #effective:r-- without anyone touching that file's permissions by hand), and carol, who is in audit but not teamA, could read it (cat succeeded) but not write it (echo >> ... was denied). That is what the default ACL buys you that a plain setgid directory cannot: inherited access for a principal other than the one owning group, applied automatically to every new file without a script running behind every touch.
Trade-offs and pitfalls
The most common failure mode is setting the default ACL but leaving the base directory's own access ACL open to other, so the inheritance rule is correct but the door was never actually closed; always set other::--- on the directory itself, not only in the default entries. Second, ACLs are invisible to tools that only look at the classic nine permission bits (ls -l shows a + suffix as the only hint an ACL exists at all), so document that getfacl is the source of truth here, not ls -l, or a future admin will "simplify" the permissions and quietly remove someone's access. Third, the ACL mask (the entry that caps the maximum effective permission for every named user and group entry) can silently reduce a permission you thought you granted; always check the #effective: annotation getfacl prints when the mask is restricting something, not just the nominal entry.
A service running as 'appuser' gets 'Permission denied' opening a config file, even though ls -l shows appuser owns it with read/write permission. The failure is intermittent: it works most of the time, then doesn't. Walk through how you'd systematically narrow down the cause, layer by layer, between the process and the file on disk.
Sample Answer
Direct answer
"Permission denied" despite ls -l showing the right owner and mode means the mode bits you can see are not the only thing deciding access; something else in the path between the process and the file's bytes on disk is intervening. Given that it's intermittent, the fault is likely something that only sometimes applies, and I'd work outward from the process itself (is it really running as the user you think, right now) through the filesystem layer (access control lists, mandatory access control context) to the mount and namespace layer (is this file's real identity even the same one the process sees).
Structured elaboration
Layer 1: is the process actually running as appuser at the moment of the failure? ls -l tells you who owns the file, not who's asking for it. Check ps -o user,pid,cmd -p <pid> or cat /proc/<pid>/status | grep -E "^(Uid|Gid)" for the real, effective, and saved user IDs. A process that drops privileges after startup, or one managed by a supervisor that occasionally restarts it under a different context (a misconfigured systemd User= directive that only applies on some restart paths, for instance), can intermittently run as someone other than appuser without that ever showing up in a static ps aux snapshot taken after the fact.
Layer 2: access control lists (ACLs) layered on top of the basic mode bits. POSIX ACLs let you attach additional, more granular rules to a file beyond the three rwx triads, and a ls -l listing only hints at their presence with a trailing + on the permission string, easy to miss at a glance. getfacl <file> shows the full picture; a deny-style ACL entry for a specific user or group can override what the basic mode bits appear to promise.
Layer 3: mandatory access control (SELinux or AppArmor). These frameworks enforce policy independently of, and on top of, standard Unix permissions; a process can have full rwx per the mode bits and still be denied if its SELinux type doesn't match the file's context. ls -Z <file> shows the SELinux context; ausearch -m avc -ts recent (or journalctl for AppArmor DENIED lines) shows the actual denial event, including a timestamp, which is exactly what you need when the symptom is intermittent: correlate denial timestamps against the failure reports. A context mismatch that appears only after a file gets rewritten by a tool that doesn't preserve SELinux labels (a common one: a deploy pipeline or editor that recreates the file instead of editing it in place) explains intermittency well: the file's context silently reverts to something the process's policy doesn't cover every time it's touched that way.
Layer 4: mount and namespace boundaries. If appuser is a service running inside a container or under a Linux user namespace, the UID that's appuser inside the container may map to a completely different UID on the host, and vice versa; a bind-mounted file "owned by appuser" from the host's perspective can be owned by an unrelated UID from inside the container's remapped view, or the reverse. This class of bug is often intermittent specifically because it depends on WHICH side (host tooling versus the containerized process) most recently touched or recreated the file, and whether that side's UID mapping happened to line up that time. Network File System (NFS) mounts add a related failure mode: root_squash (or more generally, UID mapping between NFS client and server) can translate appuser's UID differently than expected, and if the NFS mount briefly drops and remounts with different options, or the mapping service (like idmapd) has a caching hiccup, access that worked a moment ago can fail without any local file change at all.
A fifth pattern worth ruling out on intermittency specifically: a deploy or log-rotation tool that replaces the file via write-new-then-rename, briefly creating a window where the temporary file has different (often the shell or process's default umask, the mask that strips permission bits off every newly created file before any explicit chmod runs) permissions before a subsequent chmod/chown step restores the intended ones; if the process happens to open the file during that narrow window, it sees the wrong mode.
Worked example
The service runs inside a container. On the host, /data/config.yml is owned appuser:appuser (UID 2000) with mode rw-------, matching appuser's host UID exactly, so ls -l on the host looks correct. Inside the container, the same process also runs as a user named appuser, but the container's user namespace remaps container UID 2000 to host UID 2000 only when the container is started with a specific --userns mapping; on nodes where the container scheduler occasionally falls back to a default mapping (say, after a node drain and reschedule onto a node with a different container runtime config), container UID 2000 maps to host UID 100000 instead. From inside the container, the process still believes it's appuser opening a file appuser owns, but the kernel is actually checking host UID 100000 against a file owned by host UID 2000, and denies it. The intermittency lines up exactly with which node the container landed on, which is the kind of correlation that layer-by-layer checking (starting from "what UID is this process really running as, right now, on this specific host" in layer 1) surfaces, while a plain ls -l on either side alone never would.
Trade-offs & pitfalls
- Trusting
ls -las the complete picture is the core mistake this question is testing: it shows the mode bits and the names currently mapped to the numeric owner/group IDs, nothing about ACLs, mandatory access control context, or namespace remapping. - Intermittent failures are usually intermittent BECAUSE of a race or a context-dependent condition (which node, which restart path, which brief rewrite window), so the fix is to find what varies between the working and failing cases, not to re-verify the same static permission check repeatedly.
- Fixing the symptom (loosening the mode bits, or disabling SELinux/AppArmor enforcement) instead of the actual mismatch (fixing the UID mapping, fixing the NFS mount options, fixing the deploy tool's write-then-rename race) removes a real security boundary to paper over a config bug.
Audit for setuid and setgid programs in your infrastructure. Provide a command to find them and describe steps to minimize risk (e.g., removal, replacing with capabilities, using sudo wrappers). Also describe how to test that removing a setuid binary does not break critical functionality.
Sample Answer
Direct answer
Find every setuid and setgid binary with a single find command, then for each one ask whether it
genuinely needs to run with another user or group's privileges or whether that's a historical accident, and
prefer removing the privilege-escalation mechanism entirely, via Linux capabilities (a narrower, per-binary
grant of just the one specific kernel privilege actually needed) rather than leaving a broad setuid-root
binary in place because "it's always been there".
Structured elaboration
Finding them: find / -xdev -type f \( -perm -4000 -o -perm -2000 \) -exec ls -la {} \; (permission bit
4000 is the setuid bit, meaning the binary runs with the FILE OWNER's privileges rather than the invoking
user's, most dangerous when the owner is root; 2000 is the setgid equivalent for the group). -xdev stays
within one filesystem so the scan doesn't wander into a mounted network share. Run this against a known-good
baseline image and diff future scans against it, the actionable signal is usually "what setuid binary
appeared that wasn't in the base image", not re-auditing the same handful of standard binaries (passwd,
sudo, mount) every time.
Minimizing risk, in order of preference:
- Remove it entirely if the functionality isn't used at all, the least privileged code running is always
the safest state. - Replace the privilege mechanism with a Linux capability: instead of a binary being setuid-root (full root
for its entire runtime), grant it just the one specific operation it needs, for examplesetcap cap_net_bind_service=+ep /usr/bin/myserverlets a process bind to a privileged port (below 1024) without
being setuid-root at all, dramatically narrowing what a vulnerability in that binary could actually do. - Replace ad-hoc setuid scripts or wrappers with
sudorules scoped to the exact command and arguments
needed, centralizing the grant in one auditable, loggable place (everysudoinvocation is logged by
default) instead of a standing setuid bit that grants privilege silently on every invocation with no
audit trail. - Where you genuinely cannot remove or replace it (some vendor binaries ship setuid-root by design), at
minimum restrict who can even execute it via filesystem permissions or group membership, and track it
explicitly in the baseline diff so any accidental regression (it becoming world-executable) is caught.
Testing that removal doesn't break critical functionality:
- Strip the privilege on a single non-production canary first, not fleet-wide, this is fundamentally a "did
anything actually depend on this" question best answered incrementally. - Before removing, build a list of actual callers: an auditd watch rule on the binary's path
(-a always,exit -F path=<binary> -F perm=x -k setuid_audit),strace/ltraceon a representative
workload, or grepping scripts, cron entries, and application code for direct invocations. - After changing it on the canary, run the full relevant test suite, and leave monitoring in place (the same
auditd watch) for a full realistic usage cycle, covering weekly or monthly jobs, not just daily traffic,
before declaring it safe, an infrequent-but-critical caller (a monthly batch job, an annual certificate
renewal) is exactly what a short canary window misses. - Roll out gradually with an easy revert path (the removed binary or capability set kept ready to restore)
rather than a single fleet-wide change.
Worked example
A security audit finds /usr/bin/mount and /usr/bin/umount are setuid-root, standard and expected, since
mounting filesystems is inherently privileged, but also finds a home-grown /usr/local/bin/backup-agent is
setuid-root, absent from the base image, added ad hoc by a previous engineer so a cron job running as an
unprivileged user could read files owned by other users during backups. Tracing a real backup run confirms
the ONLY privileged operation the binary actually performs is bypassing file read permission checks, so the
fix is setcap cap_dac_read_search=+ep /usr/local/bin/backup-agent, followed by removing the setuid bit
entirely and validating a full backup cycle succeeds identically on a canary host before fleet rollout.
Trade-offs and pitfalls
Capabilities are a real security improvement but not a magic fix, a capability like CAP_SYS_ADMIN (a
notoriously broad, catch-all capability that in practice grants nearly as much power as full root for many
purposes) can be almost as dangerous as setuid-root if chosen too broadly, so the audit must name the
SPECIFIC capability needed, not just swap "setuid" for "some capability" reflexively. sudo wrappers shift
privilege to invocation time and rely on the sudoers rule being scoped tightly, an overly broad NOPASSWD: ALL entry recreates the exact same risk as a setuid-root binary, just relocated into a config file instead
of a permission bit.
Unlock Full Question Bank
Get access to all 17 Linux System Administration interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.