Linux System Administration Questions

Direct answer

The fastest way to isolate whether a "can't resolve external domains" symptom is genuinely DNS, versus something else entirely (routing, a firewall, an application-level issue), is to bypass DNS altogether first: try reaching something by raw IP address. If that works, the problem is specifically in name resolution, not connectivity; if it also fails, DNS was never the actual problem, and the investigation moves to routing, firewall rules, or the application layer instead. From there, work inward through the resolver configuration, then the resolution mechanism itself.

Structured elaboration

Step 1: rule DNS in or out entirely, by removing it from the equation. A bare TCP-level check to a known-good public IP, bypassing name lookup completely, tells you immediately whether basic network connectivity out of this host works at all: nc -zv 1.1.1.1 443 (or curl -v telnet://1.1.1.1:443) opens a raw TCP connection with no DNS lookup and no TLS negotiation involved. Reaching for curl -v https://<ip> directly is a trap worth knowing about: many production services today sit behind a shared, SNI-routed IP (a CDN or reverse proxy fronting many hostnames on one address), so connecting straight to that IP over HTTPS without also presenting the right hostname produces a TLS handshake failure that looks exactly like a connectivity problem but is actually just the server rejecting an unrecognized SNI value, nothing to do with DNS, routing, or a firewall at all. If you specifically need to test HTTPS to a named service while forcing a particular IP, curl -v --resolve <domain>:443:<ip> https://<domain> is the correct form, since it forces resolution to that IP while still sending the real domain in the TLS handshake and the Host header, so a failure there is actually meaningful. If even the bare TCP check fails, DNS isn't the culprit and the investigation redirects toward routing (ip route, traceroute toward that IP), a firewall blocking outbound traffic generally, or a network path issue upstream of this host, not toward anything DNS-specific.

Step 2: if direct-IP connectivity works, confirm what resolver configuration this host is actually using. resolvectl status (on hosts running systemd-resolved, the resolver management service that ships with most modern distributions) shows the effective nameservers currently in use, which is not always the same as what's naively expected from reading /etc/resolv.conf directly, since systemd-resolved often manages that file itself and the visible contents can be a stub pointing back at the local resolver service rather than the real upstream nameserver. Older or minimal systems without systemd-resolved read /etc/resolv.conf directly as the source of truth for nameservers.

Step 3: test raw DNS resolution directly, bypassing any local caching layer. dig @<nameserver-ip> example.com queries a specific nameserver directly, letting you determine whether the configured nameserver itself is reachable and answering correctly, independent of anything cached or misconfigured locally on this host. Comparing that against dig example.com (using the host's normal configured resolution path) tells you whether the problem is upstream (the configured nameserver itself failing to answer) or local (something wrong in how this host is routing its own queries to that nameserver, such as a local caching resolver stuck or misconfigured).

Step 4: confirm the resolution order and any local overrides. /etc/nsswitch.conf's hosts: line determines the order name resolution actually checks (commonly files dns, meaning /etc/hosts is consulted before DNS at all); a stale or incorrect entry in /etc/hosts for the domain in question, or for the nameserver's own hostname if it's referenced by name rather than IP, can shadow DNS entirely and produce a misleading symptom that looks like a DNS failure but is actually a local override taking precedence. getent ahosts <domain> resolves through the full configured chain (respecting nsswitch.conf order, unlike dig, which talks to a nameserver directly and skips local overrides entirely), which is useful specifically for confirming whether the application-visible resolution path matches what dig alone would suggest.

Step 5: if a nameserver in the chain appears reachable by IP but queries to it hang or get no response, check whether traffic is actually reaching it. A firewall or network policy blocking outbound traffic on port 53 (the standard DNS port, using UDP for lookups and sometimes TCP for larger or specific queries) produces exactly this symptom: the nameserver's IP is reachable for other things, but DNS queries specifically never get an answer. tcpdump -i any port 53 while retrying a lookup shows whether the query is even leaving the host, and whether any response comes back at all, which tells you definitively whether this is a local firewall problem, a silent network drop somewhere upstream, or a nameserver that received the query and simply isn't answering.

Interpreting each step's result as you go, rather than running the whole checklist blindly: a failure at step 1 means stop, this was never a DNS problem. A pass at step 1 with a failure resolving via the configured nameserver in step 3 means the nameserver itself, or the path to it, is the problem, not this host's configuration. A mismatch between dig's direct answer and getent's full-chain answer in step 4 means a local override (most often /etc/hosts or resolver misconfiguration) is intercepting resolution before DNS is even consulted.

Worked example

A host can't reach api.example.com. nc -zv 1.1.1.1 443 (a stable, dedicated public IP, bypassing DNS entirely) succeeds immediately, confirming basic connectivity out of the host is fine and this is specifically a name-resolution problem, not a routing or firewall issue. resolvectl status shows the effective nameserver configured is 10.0.0.2, an internal resolver. dig @10.0.0.2 api.example.com times out with no response at all, while dig @8.8.8.8 api.example.com (a public nameserver, reached directly) succeeds, isolating the problem specifically to reaching or getting an answer from 10.0.0.2. tcpdump -i any port 53 and host 10.0.0.2 while retrying the failing lookup shows the query leaving the host but no response ever arriving back, which, combined with ping 10.0.0.2 succeeding fine (so the host itself is reachable for other traffic), points at either the internal resolver service being down or overloaded rather than the network path, or a firewall change that specifically blocks port 53 to that resolver while leaving general connectivity to its IP untouched. That combination of evidence (reachable by ping, unreachable specifically on port 53, works fine against an entirely different nameserver) narrows the fix to that specific resolver's own health or its port-53 firewall rule, not this host's own configuration.

Trade-offs & pitfalls

• Jumping straight to dig or resolver-config inspection without first ruling DNS in or out via a direct-IP test risks spending time deep in resolver configuration on a problem that was actually a routing or firewall issue with nothing to do with DNS at all.
• dig talking directly to a nameserver and getent/application-level lookups going through the full nsswitch.conf chain can legitimately disagree; treating them as interchangeable checks misses local override problems like a stale /etc/hosts entry.
• Testing raw connectivity with curl -v https://<ip> directly (instead of a bare TCP check or --resolve) can produce a TLS handshake failure purely because of SNI-based routing on a shared IP, which is easy to misread as "so the network path is broken" when the path is actually fine and only the TLS handshake's hostname failed to match; a bare TCP check (nc -zv) avoids that false signal entirely.
• systemd-resolved environments can make /etc/resolv.conf misleading if read naively, since it may point at a local stub resolver rather than showing the real upstream nameserver directly; resolvectl status is the more reliable source of truth on those hosts.

Direct answer

Count file descriptors (FDs, the small integers a process uses to refer to open files, sockets, and pipes)
over time via /proc/<pid>/fd, resolve what each one actually points at with readlink, and cross-check
names with lsof; a leak is a specific target that keeps reappearing without ever closing, not just a
rising raw count.

Command-line recipe

Detect growth over time: a simple watch loop is often enough for a single host:

watch -n5 'ls /proc/<pid>/fd | wc -l'

For a trend you can graph rather than eyeball, append timestamped rows instead:

while true; do
  echo "$(date +%s),$(ls /proc/<pid>/fd 2>/dev/null | wc -l)"
  sleep 5
done >> fd-count.csv

Correlate FDs to file paths or sockets: raw counts alone don't tell you what's leaking, resolve the
targets and rank by how often each one repeats:

for fd in /proc/<pid>/fd/*; do readlink "$fd"; done | sort | uniq -c | sort -rn | head

Many identical entries (the same file path, or many socket:[nnnn] entries with no corresponding closes)
is the actual leak signal. lsof -p <pid> gives the same information in a more readable form, including the
type (regular file, socket, pipe) and, for sockets, the remote peer.
Aggregate results across a fleet: run the same one-liner via your existing fan-out mechanism (an SSH
loop over hosts, or your configuration-management tool's ad-hoc exec) and collect hostname,pid,fd_count
rows into one place, so you can rank hosts by growth RATE rather than manually checking one host at a time.
An early, cheaper warning sign worth watching independently of the count itself is /proc/<pid>/limits's
"Max open files" row: a slow leak typically first shows up in application logs as EMFILE (too many open
files) errors right around when the FD count approaches that limit.

Worked example

A reproducible demonstration: a process opens 30 files over about a second and a half without closing any
of them, standing in for a real leak.

import time
fds = []
for i in range(30):
    fds.append(open(f"/tmp/leak_{i}.txt", "w"))
    time.sleep(0.05)
time.sleep(5)

Sampling the FD count twice, two seconds apart, while this runs:

--- snapshot 1 ---
22
--- snapshot 2 (after 2s) ---
33

(the exact counts will vary run to run by a file or two depending on scheduling, the upward trend across
samples is the reproducible signal, not the precise numbers). Resolving the targets shows the leak clearly:

/proc/<pid>/fd/5 -> /tmp/leak_2.txt
/proc/<pid>/fd/6 -> /tmp/leak_3.txt
...

and lsof -p <pid> cross-checks the same file handles by name, confirming they are ordinary open files that
were never closed, not, for example, sockets still legitimately in use.

Trade-offs and pitfalls

A raw FD count can't distinguish a genuine leak from a busy process that legitimately holds many
connections open at once, such as a connection-pooling server; the readlink-and-aggregate step is what
turns "the count is high" into "these 400 descriptors are all sockets to the same peer that should plausibly
have been closed by now", the actual actionable signal. Also, /proc/<pid>/fd only shows a complete picture
when you're reading it as root or as the process's own owning user, a monitoring agent running under a
different, unprivileged account will silently undercount, which is easy to miss when you're comparing
numbers across a fleet running under different service accounts.

Direct answer

create_secure_tmpdir() {
  local prefix="${1:-app}"
  SECURE_TMPDIR="$(mktemp -d "${TMPDIR:-/tmp}/${prefix}.XXXXXXXX")"
  chmod 700 "$SECURE_TMPDIR"
}

cleanup() {
  local status=$?
  [[ -n "${SECURE_TMPDIR:-}" && -d "$SECURE_TMPDIR" ]] && rm -rf -- "$SECURE_TMPDIR"
  exit "$status"
}

main() {
  create_secure_tmpdir "myapp"
  trap cleanup EXIT

  local target="$SECURE_TMPDIR/a file with spaces.txt"
  printf 'hello world\n' > "$target"
}

main "$@"

mktemp -d creates the directory atomically with a random suffix (no race between "check if a
name is free" and "create it", which is what makes it secure: a predictable name in /tmp
that another user could pre-create or symlink is a classic local privilege-escalation vector).
chmod 700 then restricts it to the owner. The trap cleanup EXIT removes it whenever the
script exits, on success, on error, or on a signal, so a crash mid-script never leaves a
world-creatable temp directory lying around.

Structured elaboration

Why the function sets a global, not a local returned via command substitution

The tempting first draft calls mktemp, chmod, and trap all inside one function and then
captures its output with workdir="$(create_secure_tmpdir)". That is broken, and the breakage
is subtle enough to ship: $(...) runs the function in a subshell. The subshell exits the
instant the function returns (right after it prints the path), which fires the trap ... EXIT
that was set inside that subshell, deleting the directory before the parent script ever
receives the path back. The fix is to never route the path through command substitution:
create_secure_tmpdir sets a global variable directly, and trap is registered afterward in
the same shell that will eventually exit for real (the top-level script), not in a throwaway
subshell.

Handling spaces in filenames correctly

• Quote every expansion of the path: "$SECURE_TMPDIR", "$target". An unquoted $SECURE_TMPDIR
  undergoes word splitting and would be treated as multiple arguments if the prefix or TMPDIR
  contains spaces.
• If you later need to iterate over files inside the directory, never do for f in $(ls "$dir"); use find "$dir" -print0 | while IFS= read -r -d '' f; do ... done, which uses a
  NUL byte as the record separator so spaces (and even newlines) inside filenames cannot be
  mistaken for separators.
• Always terminate option lists with -- before a path argument (rm -rf -- "$dir") so a
  filename that happens to start with - is never parsed as a flag.

Permissions

700 (rwx------) means only the owning user can read, write, or traverse into the directory;
group and others get nothing. mktemp -d on Linux already creates directories 700 by default,
so the explicit chmod 700 here is defense against a non-default umask (the mask that strips
permission bits off every newly created file or directory before any explicit chmod runs;
e.g. a umask of 022 would otherwise leave a fresh directory group- and world-readable) on some other
distribution or in some other user's shell, and it documents the intent instead of relying on an
implicit default.

Worked example

Run against the corrected function on a live container (Ubuntu 24.04):

$ bash myscript.sh
$ ls /tmp
   (directory is already gone: cleanup fired on script exit)

Inside main, before exit, the directory and its file are confirmed present and correctly
permissioned:

$ stat -c '%a' "$SECURE_TMPDIR"
700
$ cat "$SECURE_TMPDIR/a file with spaces.txt"
hello world

By contrast, the broken "function sets its own trap and returns the path via $(...)" version,
run in the same environment, prints the path but the directory is already deleted by the time
the caller's next line runs, because the subshell's EXIT trap already fired.

Trade-offs and pitfalls

• mktemp without -d creates a file; for a directory you must pass -d, and forgetting it
  is a common copy-paste error when adapting a temp-file recipe.
• Do not rely on /tmp being cleaned automatically (some distributions run tmpfiles.d sweeps
  on a schedule, others do not); a script must clean up after itself rather than assuming the OS
  will.
• A trap ... EXIT set inside a function is scoped to whichever shell process actually runs
  that function. In a plain (non-subshell) function call this is the main script's shell, which
  is correct; the moment you wrap the call in $(...), backticks, a pipeline, or &
  (background), you have created a subshell and the trap silently stops protecting the resource
  the caller thinks it owns.
• If the script can be interrupted by a signal other than normal exit (Ctrl-C, kill), trap cleanup EXIT still fires in bash because EXIT is triggered on any script termination path,
  but if you also trap INT/TERM explicitly for other cleanup, make sure that handler does not
  itself exit without eventually reaching the shared cleanup logic, or the temp directory leaks.

Direct answer

"Permission denied" despite ls -l showing the right owner and mode means the mode bits you can see are not the only thing deciding access; something else in the path between the process and the file's bytes on disk is intervening. Given that it's intermittent, the fault is likely something that only sometimes applies, and I'd work outward from the process itself (is it really running as the user you think, right now) through the filesystem layer (access control lists, mandatory access control context) to the mount and namespace layer (is this file's real identity even the same one the process sees).

Structured elaboration

Layer 1: is the process actually running as appuser at the moment of the failure? ls -l tells you who owns the file, not who's asking for it. Check ps -o user,pid,cmd -p <pid> or cat /proc/<pid>/status | grep -E "^(Uid|Gid)" for the real, effective, and saved user IDs. A process that drops privileges after startup, or one managed by a supervisor that occasionally restarts it under a different context (a misconfigured systemd User= directive that only applies on some restart paths, for instance), can intermittently run as someone other than appuser without that ever showing up in a static ps aux snapshot taken after the fact.

Layer 2: access control lists (ACLs) layered on top of the basic mode bits. POSIX ACLs let you attach additional, more granular rules to a file beyond the three rwx triads, and a ls -l listing only hints at their presence with a trailing + on the permission string, easy to miss at a glance. getfacl <file> shows the full picture; a deny-style ACL entry for a specific user or group can override what the basic mode bits appear to promise.

Layer 3: mandatory access control (SELinux or AppArmor). These frameworks enforce policy independently of, and on top of, standard Unix permissions; a process can have full rwx per the mode bits and still be denied if its SELinux type doesn't match the file's context. ls -Z <file> shows the SELinux context; ausearch -m avc -ts recent (or journalctl for AppArmor DENIED lines) shows the actual denial event, including a timestamp, which is exactly what you need when the symptom is intermittent: correlate denial timestamps against the failure reports. A context mismatch that appears only after a file gets rewritten by a tool that doesn't preserve SELinux labels (a common one: a deploy pipeline or editor that recreates the file instead of editing it in place) explains intermittency well: the file's context silently reverts to something the process's policy doesn't cover every time it's touched that way.

Layer 4: mount and namespace boundaries. If appuser is a service running inside a container or under a Linux user namespace, the UID that's appuser inside the container may map to a completely different UID on the host, and vice versa; a bind-mounted file "owned by appuser" from the host's perspective can be owned by an unrelated UID from inside the container's remapped view, or the reverse. This class of bug is often intermittent specifically because it depends on WHICH side (host tooling versus the containerized process) most recently touched or recreated the file, and whether that side's UID mapping happened to line up that time. Network File System (NFS) mounts add a related failure mode: root_squash (or more generally, UID mapping between NFS client and server) can translate appuser's UID differently than expected, and if the NFS mount briefly drops and remounts with different options, or the mapping service (like idmapd) has a caching hiccup, access that worked a moment ago can fail without any local file change at all.

A fifth pattern worth ruling out on intermittency specifically: a deploy or log-rotation tool that replaces the file via write-new-then-rename, briefly creating a window where the temporary file has different (often the shell or process's default umask, the mask that strips permission bits off every newly created file before any explicit chmod runs) permissions before a subsequent chmod/chown step restores the intended ones; if the process happens to open the file during that narrow window, it sees the wrong mode.

Worked example

The service runs inside a container. On the host, /data/config.yml is owned appuser:appuser (UID 2000) with mode rw-------, matching appuser's host UID exactly, so ls -l on the host looks correct. Inside the container, the same process also runs as a user named appuser, but the container's user namespace remaps container UID 2000 to host UID 2000 only when the container is started with a specific --userns mapping; on nodes where the container scheduler occasionally falls back to a default mapping (say, after a node drain and reschedule onto a node with a different container runtime config), container UID 2000 maps to host UID 100000 instead. From inside the container, the process still believes it's appuser opening a file appuser owns, but the kernel is actually checking host UID 100000 against a file owned by host UID 2000, and denies it. The intermittency lines up exactly with which node the container landed on, which is the kind of correlation that layer-by-layer checking (starting from "what UID is this process really running as, right now, on this specific host" in layer 1) surfaces, while a plain ls -l on either side alone never would.

Trade-offs & pitfalls

• Trusting ls -l as the complete picture is the core mistake this question is testing: it shows the mode bits and the names currently mapped to the numeric owner/group IDs, nothing about ACLs, mandatory access control context, or namespace remapping.
• Intermittent failures are usually intermittent BECAUSE of a race or a context-dependent condition (which node, which restart path, which brief rewrite window), so the fix is to find what varies between the working and failing cases, not to re-verify the same static permission check repeatedly.
• Fixing the symptom (loosening the mode bits, or disabling SELinux/AppArmor enforcement) instead of the actual mismatch (fixing the UID mapping, fixing the NFS mount options, fixing the deploy tool's write-then-rename race) removes a real security boundary to paper over a config bug.

Direct answer

find /var/logs/app -maxdepth 1 -type f -mtime -1 -print0 | xargs -0 -I{} mv {} /tmp/recent/ does this safely in one pipeline: find ... -print0 emits each matching filename terminated by a NUL byte instead of a newline, and xargs -0 reads that same NUL-delimited stream, so a filename containing a space or even an embedded newline is never misread as two separate arguments, which is exactly the failure mode a plain find ... | xargs (newline delimited) has.

Why each part is there

• -maxdepth 1: restricts the search to files directly inside /var/logs/app, not subdirectories, matching "all regular files in /var/logs/app."
• -type f: only regular files, not directories or symlinks that happen to also match the time window.
• -mtime -1: modified within the last 24 hours. find's -mtime counts in whole 24 hour periods from now; -mtime -1 means "modification time is less than 1 day ago," which is the standard, correct way to express "within the last 24 hours," as opposed to -mtime 1 (exactly between 24 and 48 hours ago) or -mtime +1 (more than 48 hours ago), which are easy to confuse with the intended meaning if you have not internalized the off by one convention here.
• -print0: the safety critical flag. Ordinary find output (or -print, its default) separates results with a newline, which is indistinguishable from a filename that legitimately contains a newline, and a plain space in a filename already breaks a naive for f in $(find ...) loop by splitting one filename into two words. -print0 uses the NUL byte instead, which cannot appear inside a POSIX filename, making it an unambiguous delimiter no matter what characters the filename itself contains.
• xargs -0: the matching consumer for NUL delimited input; without it, xargs would still assume newline/whitespace splitting and the safety -print0 provides on the producing side would be thrown away on the consuming side.
• -I{}: substitutes each individual filename into the {} placeholder in the mv {} /tmp/recent/ command, run once per file, which is what you want here since you are moving files individually into a common destination directory rather than passing them all as one argument list to a single invocation.

Worked example (executed, including the actual hostile filenames this pattern is meant to survive)

Created files with a space and with an embedded literal newline in the name specifically to prove the safety claim rather than assert it:

cd /var/logs/app
touch "app one.log" "app two.log" $'weird\nname.log'
touch -d "3 days ago" old.log
find /var/logs/app -maxdepth 1 -type f -mtime -1 -print0 | xargs -0 -I{} mv {} /tmp/recent/

Real captured result, listing what actually ended up in the destination:

-rw-r--r-- 1 root root 0 ... app one.log
-rw-r--r-- 1 root root 0 ... app two.log
-rw-r--r-- 1 root root 0 ... weird
name.log

All three files, including the one whose name contains a literal newline, moved correctly and intact as single files, and the fourth file (old.log, backdated three days) was correctly left behind since it fell outside the -mtime -1 window. The command handled exactly the two hostile cases (embedded space, embedded newline) that a naive unquoted find | xargs pipeline would have split incorrectly.

Trade-offs and pitfalls

The single most common mistake is writing find /var/logs/app -mtime -1 | xargs mv -t /tmp/recent/ without -print0/-0: it looks identical for ordinary filenames and silently breaks the moment a filename contains a space, which is common enough in real log directories (application names, dates, or hostnames embedded in filenames) that this is not a hypothetical edge case. A second, subtler issue: -I{} runs mv once per file, which is correct here but slower than batching for a very large number of files; xargs -0 -t mv -- {} /tmp/recent/ versus a batched form (xargs -0 mv -t /tmp/recent/ without -I, letting xargs pack multiple filenames into fewer mv invocations) is worth knowing as an alternative when moving thousands of files, though the batched form only works because GNU mv -t accepts multiple sources, which not every command does. Finally, always sanity check with a dry run before deleting or moving anything destructively: swap mv for echo first (... | xargs -0 -I{} echo mv {} /tmp/recent/) to see exactly what would run.