Linux System Administration Questions

Direct answer

Read the actual denial with ausearch/sealert to see the exact type mismatch, then choose between a
relabel (when the file's context has simply drifted, commonly after a package update writes it with the
wrong label) and a genuine policy exception (a real, new interaction the policy has never been told is
legitimate), rather than reflexively disabling enforcement or hand-patching one file's label with chcon.

Structured elaboration

1. Confirm SELinux (Security-Enhanced Linux, the kernel's mandatory access control layer) is even the
   cause and its current mode: getenforce (or sestatus for more detail) returns Enforcing,
   Permissive, or Disabled. Only in Enforcing mode does a denial actually block the action; in
   Permissive mode it's logged but allowed anyway.
2. Find the specific denial: ausearch -m avc -ts recent pulls the raw AVC (access vector cache, the
   kernel's decision cache for these checks, and "AVC denial" is the standard term for a logged block) entry
   from the audit log; sealert -a /var/log/audit/audit.log (part of setroubleshoot) explains the same
   event in plain language with a suggested fix.
3. Read the denial's key fields: the source context (scontext, what the running httpd, Apache HTTP
   Server, process is labeled, typically something like httpd_t) and the target context (tcontext, what
   the file is actually labeled right now), and the permission requested. The mismatch between what
   httpd's policy allows it to read and what the file is currently labeled is the whole story.
4. Decide relabel versus policy change: compare what the file's context SHOULD be per policy
   (matchpathcon <path>) against its CURRENT context (ls -Z <path>, the -Z flag on ls, ps, and
   other core tools shows the SELinux context alongside normal output). If they disagree, the file simply
   has the wrong label, commonly because a package's post-install script wrote it with a plain copy rather
   than a relabel-aware install step, and restorecon -v <path> fixes it by resetting the label to the
   policy-defined default. restorecon is almost always the right tool for "this label is wrong": chcon
   sets an arbitrary context by hand and never updates the policy's file-context database, so the very next
   relabel operation (routine after many package updates, or a full restorecon -R /) silently reverts it,
   chcon should be reserved for genuinely deliberate, temporary testing, not a persistent fix.
5. If the label is already correct and policy still denies it, this is a genuine policy gap: generate a
   targeted module with ausearch -m avc -ts recent | audit2allow -M httpd_local_fix, and always review the
   generated type-enforcement rule file BEFORE installing it, audit2allow's suggestions can be broader than
   the one specific access actually needed. Only after review, semodule -i httpd_local_fix.pp installs it;
   semodule -l lists installed modules so accumulated local exceptions stay auditable over time.
6. A fast, reversible way to confirm SELinux is even the real blocker before committing to either fix:
   temporarily flip to Permissive (setenforce 0), retry the failing action once, and return immediately
   to Enforcing (setenforce 1). If the action now succeeds, SELinux was indeed the cause and you proceed
   to the real fix above; if it still fails, you've avoided mislabeling an unrelated bug as an SELinux
   problem. This same confirm-then-fix ordering applies whether the denied operation is a read, as in this
   scenario, or a write, such as an application unable to write into an upload directory: confirm the cause
   with the temporary flip, then implement the reviewed, permanent fix, never leave the permissive state in
   place as the fix itself.
7. Document the decision: record the denial, the fix chosen, and why (relabel drift versus a genuine new
   interaction), since the next package update rewriting the same file the same way reproduces the identical
   denial, and a documented pattern turns a lengthy investigation into a thirty-second recognition next
   time.

Worked example

After a package update, httpd fails to read /etc/httpd/conf.d/app.conf with a generic permission error in
its log, despite normal Unix permissions (644, owned by root) looking completely fine. getenforce returns
Enforcing. ausearch -m avc -ts recent shows source context httpd_t, target context unlabeled_t (a
context meaning SELinux lost track of the file's intended label, commonly seen right after a package's
install script writes a file with a plain, non-relabel-aware tool). matchpathcon reports the expected
context is httpd_config_t; ls -Z confirms the file is currently unlabeled_t, a clear mismatch.
restorecon -v /etc/httpd/conf.d/app.conf relabels it to httpd_config_t, and the next request succeeds
with no further denials, confirming it was pure label drift, not a real policy gap, so no audit2allow
module was needed at all.

Trade-offs and pitfalls

setenforce 0 left in place (or worse, disabling SELinux permanently in /etc/selinux/config) removes an
entire security control fleet-wide to fix one file, and it also means the specific missing interaction is
never actually identified or recorded, so the same bug reappears on the next similarly-updated host with no
diagnostic trail. Installing an audit2allow module without reading it first is a common wrong turn that
can grant far more than the one narrow access actually needed, quietly widening the attack surface every
time a denial is "fixed" this way. And chcon used as a persistent fix, rather than restorecon, looks
fixed until the next routine relabel silently reverts it with no code change to explain why the ticket
reopened.

Direct answer

Write persistent rules under /etc/audit/rules.d/ (loaded automatically by augenrules, the supported,
reboot-safe path, rather than transient auditctl commands that vanish on reboot), scoped tightly to the
sudo binary and the sudoers files, and rely on SSH's own PAM (pluggable authentication modules)-integrated
logging, correlated through the same audit trail, for failed login attempts, since a failed password
doesn't correspond to one clean syscall event auditd can key on the way a file open does.

Structured elaboration

Rule for sudo executions (syscall-based, since "someone ran sudo" is best captured at execution time):

-a always,exit -F arch=b64 -F path=/usr/bin/sudo -F perm=x -F auid>=1000 -F auid!=4294967295 -k sudo_exec
-a always,exit -F arch=b32 -F path=/usr/bin/sudo -F perm=x -F auid>=1000 -F auid!=4294967295 -k sudo_exec

-a always,exit generates a record at syscall exit, the point where the full outcome, including
success or failure, is known. Both arch=b64 and arch=b32 variants are needed, a 64-bit-only rule
silently misses a 32-bit invocation of the same binary on a system where that's still possible. -F path=/usr/bin/sudo -F perm=x scopes this to executions of sudo specifically, a broad, path-less
execve rule would log every program launch fleet-wide, a serious log-volume and performance cost for
little extra signal beyond watching this one binary. auid>=1000 -F auid!=4294967295 filters to real,
interactively logged-in users: auid (the audit user ID, set once at login and unlike the UID a process
might later switch to, never changes for that login session) is the correct field for "which human actually
triggered this" even after a privilege change; UIDs below 1000 are conventionally system accounts, and
4294967295 is the sentinel value meaning "no login at all".

Watching the sudoers files themselves, since auditing executions is only half the picture, someone
editing WHO CAN sudo matters just as much:

-w /etc/sudoers -p wa -k sudoers_changes
-w /etc/sudoers.d/ -p wa -k sudoers_changes

(-w is a simpler, cheaper path-watch for files that rarely change; -p wa logs writes and
attribute changes.)

Failed SSH login attempts: these aren't captured via a syscall rule at all, the authoritative record is
sshd's own log through PAM, in /var/log/secure or /var/log/auth.log depending on distribution, or
centrally via journalctl -u sshd. Where auditd adds real value here is PAM's own audit integration: pam_unix
and similar modules emit USER_AUTH/USER_LOGIN records when compiled with audit support, and ausearch -m USER_AUTH,USER_LOGIN -su failed surfaces those failures alongside sudo executions and sudoers edits in ONE
unified, searchable trail, the real value here is correlation into the same place, not a new detection
mechanism.

Minimizing performance impact: avoid unscoped -S execve or -S open,openat rules with no path or auid
filter, they fire on nearly everything the kernel does and can measurably slow a busy host, since every
matching syscall round-trips through the audit subsystem before returning. Prefer -w path-watches for
rarely-changing files and narrow, path-and-auid-filtered -a rules for anything execution-based, and size
the kernel's audit backlog (auditctl -b 8192 or higher on a busy host) so a burst of matching events
doesn't overflow it and either drop records or, if configured to panic on overflow, affect the system.

Querying: ausearch -k sudo_exec -ts today pulls today's tagged records; ausearch -k sudoers_changes -i (-i interprets numeric UIDs and syscall numbers back into names) for readability; aureport --summary
or aureport -au (authentication report) for a rolled-up overview instead of reading raw records one at a
time.

Correlating with journalctl: ausearch's output timestamp can be cross-referenced against
journalctl --since "<time>" --until "<time>" for the same narrow window, lining a suspicious sudo
execution up against what else was happening on the host (a login, a cron job firing, an application
restart), since neither log alone usually tells the full story of an incident.

Worked example

A persistent rules file, /etc/audit/rules.d/sudo-and-auth.rules, containing the two sudo-exec rules
(b64 and b32) and the two sudoers watches above, loaded with augenrules --load and confirmed present via
auditctl -l. Querying after a suspicious sudo invocation:

ausearch -k sudo_exec -ts today -i

returns records in the standard auditd format (illustrative, field values will differ per event):

type=SYSCALL msg=audit(1699999999.123:456): arch=x86_64 syscall=execve success=yes exit=0
  auid=jdoe uid=jdoe gid=jdoe euid=root exe="/usr/bin/sudo" key="sudo_exec"

The auid=jdoe field is what answers "who", even though euid=root shows the process's effective
privilege after sudo did its job.

Trade-offs and pitfalls

A rule that filters by -F uid=0 instead of auid misses the entire point: by the time sudo has done its
job the EFFECTIVE uid is already 0 regardless of who invoked it, auid (fixed at login) is what actually
answers "who", not uid/euid at execution time. Rules added transiently with a bare auditctl -a ...
command survive only until reboot, a common on-call mistake where a rule is added live to catch something
in progress, the incident resolves, the host reboots for an unrelated patch, and the rule silently vanishes
with no record it ever existed, /etc/audit/rules.d/ plus augenrules is what makes a rule permanent.

Direct answer

Define a Cmnd_Alias naming exactly the two allowed commands with their full absolute paths and no arguments left open ended, then grant the ops group NOPASSWD access to only that alias, never to ALL. Full paths matter because sudo matches against the literal command path given in the rule; if a user's $PATH resolves systemctl to a different binary than the one named in sudoers, sudo simply will not match, which is a safety feature here, not a bug to work around.

The sudoers entry

Cmnd_Alias OPS_CMDS = /usr/bin/systemctl restart httpd, /usr/sbin/ethtool
%ops ALL=(root) NOPASSWD: OPS_CMDS

• Cmnd_Alias OPS_CMDS = ...: names the exact set of allowed invocations. /usr/bin/systemctl restart httpd only matches that specific subcommand and unit; it does not grant systemctl restart on any other unit, and does not grant systemctl stop httpd or any other verb, because sudoers command matching is against the full command line as written, not just the binary.
• /usr/sbin/ethtool with no arguments listed grants it with any arguments, since an alias entry with no trailing arguments matches the command run with any arguments at all; if the intent were to restrict ethtool to read only queries and forbid it from changing NIC settings, the entry would need to spell out the specific allowed flags explicitly (for example /usr/sbin/ethtool -S * for statistics only) rather than leaving it open.
• %ops: the % prefix means this line applies to a group, not a single user; ops must exist as a real group (groupadd ops) with the intended members already added to it.
• ALL=(root): this rule applies regardless of which host the sudoers file is deployed to (ALL), and the command runs as root specifically, not as any user the invoking user chooses; if this should be able to run as some other target user too, that user would need to be added to the parenthesized run-as list.
• NOPASSWD:: skips the password re-prompt for exactly this alias, which is what makes it usable in scripts and automation, at the cost of removing that speed bump, which is why scoping the alias tightly matters more here than on a rule that still requires a password.

Safe editing and testing

Always edit sudoers with visudo (or visudo -f /etc/sudoers.d/ops-restart for a drop in file under /etc/sudoers.d/, the cleaner approach since it keeps custom rules out of the main file and out of the way of package updates), never with a plain text editor directly on /etc/sudoers: visudo locks the file against concurrent edits and, critically, refuses to save a syntactically broken file, which is the single biggest protection against locking every sudo user out of sudo entirely with one typo. visudo -c (or visudo -c -f <path> for a drop in file) checks syntax without opening an editor at all, which is the right tool for a script or a pre-deploy check rather than an interactive edit. Test with the actual target user, not root: sudo -l -U alice (as root) lists exactly what sudo believes alice is authorized to run, without executing anything, and sudo -u alice -l from alice's own session confirms what she sees; only after both agree should you consider the change live.

Worked example (executed)

groupadd ops
cat > /tmp/ops-sudoers <<'EOF'
Cmnd_Alias OPS_CMDS = /usr/bin/systemctl restart httpd, /usr/sbin/ethtool
%ops ALL=(root) NOPASSWD: OPS_CMDS
EOF
visudo -c -f /tmp/ops-sudoers

Real output confirming the syntax is valid before it is ever placed where sudo would actually read it:

/tmp/ops-sudoers: parsed OK

Only after that check passes would this be moved into place with visudo -f /etc/sudoers.d/ops-restart (which re-runs the same validation on save), never copied in directly with cp or a text editor, since neither of those re-validates syntax before the file takes effect.

Trade-offs and pitfalls

The most common mistake is writing the alias against a bare command name (systemctl instead of /usr/bin/systemctl), which either fails to match at all (if sudoers strictly requires the absolute path, which it does) or, worse on a misconfigured or old sudo build, matches more loosely than intended; always use the full path exactly as which systemctl (or command -v) reports it on the actual target host, since that path can differ between distributions. A second mistake is granting ethtool with no argument restriction when the actual intent was "let ops check NIC stats," since unrestricted ethtool can also change link settings, offload features, and other configuration that a "just checking" ticket did not intend to authorize; if the real requirement is read only, the sudoers entry should say so explicitly rather than relying on ops members' good judgment never to run a mutating flag. Third, never test a new sudoers rule by immediately logging out of your only privileged session: keep a second root session open, exactly as with PAM changes, until sudo -l -U <testuser> and an actual test invocation both confirm the rule behaves as intended.

Direct answer

Give each team's directory to a dedicated group, set the directory itself to deny "other" access entirely, add a default ACL (Access Control List, the extended per-user/per-group permission entries beyond the classic owner/group/other bits) on each team directory so new files and subdirectories automatically inherit the team's group permissions, and never rely on umask alone to keep teams apart, since umask is a per-process default, not an enforced boundary.

Design

1. Base directory, traversal only. /srv/projects itself gets chmod 751: owner (root) full access, group read+execute (list and traverse), other execute only (traverse but not list). This means nobody can ls /srv/projects and enumerate every team's name unless they already know it, without blocking legitimate traversal into a subdirectory they do have rights to.
2. One group per team, one directory per team, setgid plus a matching group owner. For team A: groupadd teamA, mkdir /srv/projects/teamA, chown root:teamA /srv/projects/teamA, chmod 2770 /srv/projects/teamA. The 2 sets the setgid bit on the directory, so every file and subdirectory created inside automatically inherits the group teamA regardless of the creating user's primary group; 770 gives the owner and group full access and gives other nothing.
3. Default ACL for the group, so inheritance is explicit and covers ACL entries, not just the base group bit. setfacl -d -m g:teamA:rwx /srv/projects/teamA and setfacl -d -m o::--- /srv/projects/teamA. The setgid bit alone already gets you group inheritance for the classic Unix group bit; the default ACL matters once you need more than one principal to inherit access, for example a cross team auditor group that should get read only access to every team's directory without being the owning group.
4. Repeat per team, each with its own group, directory, and default ACL. Isolation between teamA and teamB comes entirely from teamB never being named in teamA's ACL or group, combined with other::--- at both the base directory (traversal denied without a matching entry) and inside each team directory (no residual access for anyone not explicitly granted it).

Worked example (executed, including a real cross team access attempt)

Set this up for real, created a user in each team, and actually tried to cross the boundary rather than just inspecting the ACL:

groupadd teamA; groupadd teamB
useradd -m -g teamA alice
useradd -m -g teamB bob
chmod 751 /srv/projects
mkdir /srv/projects/teamA
chown root:teamA /srv/projects/teamA
chmod 2770 /srv/projects/teamA
setfacl -d -m g:teamA:rwx /srv/projects/teamA
setfacl -d -m o::--- /srv/projects/teamA

Resulting ACL (getfacl):

user::rwx
group::rwx
other::---
default:user::rwx
default:group::rwx
default:group:teamA:rwx
default:mask::rwx
default:other::---

Alice (teamA) creates a file, and it correctly inherits the team's group:

-rw-rw----+ 1 alice teamA 0 Sep 24 00:24 /srv/projects/teamA/notes.txt

Bob (teamB, not a member of teamA) attempting to touch teamA's directory at all:

$ ls /srv/projects/teamA
ls: cannot open directory '/srv/projects/teamA': Permission denied
$ cat /srv/projects/teamA/notes.txt
cat: /srv/projects/teamA/notes.txt: Permission denied

Both denied, exactly as designed. Alice, meanwhile, reads and writes her own team's file without issue.

To show the actual value a default ACL adds beyond the setgid bit alone, I added a third, cross-cutting audit group with read only access via ACL, not group ownership:

groupadd audit; useradd -m -g audit carol
setfacl -m g:audit:r-x /srv/projects/teamA
setfacl -d -m g:audit:r-x /srv/projects/teamA

A brand new file created afterward by alice automatically picked up the audit entry (getfacl on the new file showed group:audit:r-x #effective:r-- without anyone touching that file's permissions by hand), and carol, who is in audit but not teamA, could read it (cat succeeded) but not write it (echo >> ... was denied). That is what the default ACL buys you that a plain setgid directory cannot: inherited access for a principal other than the one owning group, applied automatically to every new file without a script running behind every touch.

Trade-offs and pitfalls

The most common failure mode is setting the default ACL but leaving the base directory's own access ACL open to other, so the inheritance rule is correct but the door was never actually closed; always set other::--- on the directory itself, not only in the default entries. Second, ACLs are invisible to tools that only look at the classic nine permission bits (ls -l shows a + suffix as the only hint an ACL exists at all), so document that getfacl is the source of truth here, not ls -l, or a future admin will "simplify" the permissions and quietly remove someone's access. Third, the ACL mask (the entry that caps the maximum effective permission for every named user and group entry) can silently reduce a permission you thought you granted; always check the #effective: annotation getfacl prints when the mask is restricting something, not just the nominal entry.

Direct answer

Find every setuid and setgid binary with a single find command, then for each one ask whether it
genuinely needs to run with another user or group's privileges or whether that's a historical accident, and
prefer removing the privilege-escalation mechanism entirely, via Linux capabilities (a narrower, per-binary
grant of just the one specific kernel privilege actually needed) rather than leaving a broad setuid-root
binary in place because "it's always been there".

Structured elaboration

Finding them: find / -xdev -type f \( -perm -4000 -o -perm -2000 \) -exec ls -la {} \; (permission bit
4000 is the setuid bit, meaning the binary runs with the FILE OWNER's privileges rather than the invoking
user's, most dangerous when the owner is root; 2000 is the setgid equivalent for the group). -xdev stays
within one filesystem so the scan doesn't wander into a mounted network share. Run this against a known-good
baseline image and diff future scans against it, the actionable signal is usually "what setuid binary
appeared that wasn't in the base image", not re-auditing the same handful of standard binaries (passwd,
sudo, mount) every time.

Minimizing risk, in order of preference:

1. Remove it entirely if the functionality isn't used at all, the least privileged code running is always
   the safest state.
2. Replace the privilege mechanism with a Linux capability: instead of a binary being setuid-root (full root
   for its entire runtime), grant it just the one specific operation it needs, for example setcap cap_net_bind_service=+ep /usr/bin/myserver lets a process bind to a privileged port (below 1024) without
   being setuid-root at all, dramatically narrowing what a vulnerability in that binary could actually do.
3. Replace ad-hoc setuid scripts or wrappers with sudo rules scoped to the exact command and arguments
   needed, centralizing the grant in one auditable, loggable place (every sudo invocation is logged by
   default) instead of a standing setuid bit that grants privilege silently on every invocation with no
   audit trail.
4. Where you genuinely cannot remove or replace it (some vendor binaries ship setuid-root by design), at
   minimum restrict who can even execute it via filesystem permissions or group membership, and track it
   explicitly in the baseline diff so any accidental regression (it becoming world-executable) is caught.

Testing that removal doesn't break critical functionality:

• Strip the privilege on a single non-production canary first, not fleet-wide, this is fundamentally a "did
  anything actually depend on this" question best answered incrementally.
• Before removing, build a list of actual callers: an auditd watch rule on the binary's path
  (-a always,exit -F path=<binary> -F perm=x -k setuid_audit), strace/ltrace on a representative
  workload, or grepping scripts, cron entries, and application code for direct invocations.
• After changing it on the canary, run the full relevant test suite, and leave monitoring in place (the same
  auditd watch) for a full realistic usage cycle, covering weekly or monthly jobs, not just daily traffic,
  before declaring it safe, an infrequent-but-critical caller (a monthly batch job, an annual certificate
  renewal) is exactly what a short canary window misses.
• Roll out gradually with an easy revert path (the removed binary or capability set kept ready to restore)
  rather than a single fleet-wide change.

Worked example

A security audit finds /usr/bin/mount and /usr/bin/umount are setuid-root, standard and expected, since
mounting filesystems is inherently privileged, but also finds a home-grown /usr/local/bin/backup-agent is
setuid-root, absent from the base image, added ad hoc by a previous engineer so a cron job running as an
unprivileged user could read files owned by other users during backups. Tracing a real backup run confirms
the ONLY privileged operation the binary actually performs is bypassing file read permission checks, so the
fix is setcap cap_dac_read_search=+ep /usr/local/bin/backup-agent, followed by removing the setuid bit
entirely and validating a full backup cycle succeeds identically on a canary host before fleet rollout.

Trade-offs and pitfalls

Capabilities are a real security improvement but not a magic fix, a capability like CAP_SYS_ADMIN (a
notoriously broad, catch-all capability that in practice grants nearly as much power as full root for many
purposes) can be almost as dangerous as setuid-root if chosen too broadly, so the audit must name the
SPECIFIC capability needed, not just swap "setuid" for "some capability" reflexively. sudo wrappers shift
privilege to invocation time and rely on the sudoers rule being scoped tightly, an overly broad NOPASSWD: ALL entry recreates the exact same risk as a setuid-root binary, just relocated into a config file instead
of a permission bit.