Linux and Unix Fundamentals Questions

Practical command line and system administration knowledge for Linux and Unix style operating systems. Candidates should understand the file system hierarchy and common directories, file and directory permissions and ownership, user and group management, package management and software installation, service and startup management, system logging and basic log inspection, process management and signals including job control and background processes, and secure remote access using secure shell. Candidates should be proficient with common command line utilities for searching and text processing, input and output redirection and piping, and basic shell scripting constructs such as variables, conditionals, loops, and simple functions to automate routine tasks. Interview assessments may include hands on tasks and explanations such as navigating the file system, creating and managing files and directories, changing permissions and ownership, monitoring and terminating processes, inspecting logs to diagnose issues, managing services and packages, writing short scripts to automate workflows, and describing troubleshooting approaches for common system problems.

HardTechnical

80 practiced

An XFS filesystem shows metadata corruption. Describe the safe steps to unmount, run xfs_repair, take a block-level backup before repair, and how to restore from backups or LVM snapshots if repair fails. Mention online vs offline considerations and hardware checks you would run first.

Sample Answer

Situation: You detect XFS metadata corruption (kernel logs show XFS errors, mount failures, or xfs_repair suggestions). As an SRE you must preserve data, minimize downtime, and follow safe repair/restore steps.

1) Immediate hardware checks (before touching filesystem)- Check SMART for the underlying disk(s): smartctl -a /dev/sdX; run short/long self-test if suspicious.- Inspect dmesg/journalctl for I/O errors, controller/driver errors, timeouts.- Verify cabling/hot-swap status, RAID controller logs, and battery-backed cache state.- If any hardware fault is suspected, isolate the device (fail the RAID member, replace) before any write operations.

2) Online vs offline considerations- XFS metadata repair cannot be done safely online. xfs_repair must operate on an unmounted filesystem (or on a device mapper device).- You may mount filesystem read-only to extract files while planning repairs: mount -o ro,nouuid /dev/mapper/vg-lv /mnt/recover- Do not attempt xfs_repair on a mounted filesystem (except using -n on a device to simulate — still must be unmounted for real repair).

3) Safe unmount procedure- Notify stakeholders, pause writers, enter maintenance mode.- Stop services using the FS (systemctl stop <svc>, stop containers).- Find and kill remaining processes: fuser -vm /mountpoint or lsof +f -- /mountpoint; gracefully terminate; avoid lazy (-l) unmount for repair — filesystem must be cleanly unmounted.- Umount: umount /mountpoint (or umount -f only for emergency, but avoid unless necessary).- Confirm: mount | grep /mountpoint or lsblk to ensure device is not mounted.

4) Block-level backup BEFORE ANY REPAIR (critical)- Create a full block-level image of the device and store off-host or on separate storage. Prefer ddrescue for resilience to read errors:

bash

# example using ddrescue (recommended)
ddrescue --no-split --logfile=/tmp/ddrescue.log /dev/mapper/vg-lv /backup/images/vg-lv.img /backup/images/vg-lv.map

- Or use dd with pv and compression if device healthy:

bash

dd if=/dev/mapper/vg-lv | pv | gzip -c > /backup/images/vg-lv.img.gz

- Verify image integrity (checksums): sha256sum image > image.sha256- If using LVM, create an LVM snapshot first to avoid long downtime:

bash

lvcreate --size 10G --snapshot --name lv_snap /dev/vg/lv
# mount snapshot read-only and dd from /dev/vg/lv_snap

- For large volumes, stream backup to a remote host (netcat/ssh) to avoid filling local disk.

5) Run xfs_repair safely- First run non-destructive check:

bash

xfs_repair -n /dev/mapper/vg-lv

- Review output carefully. If it reports only metadata inconsistencies that xfs_repair can fix, run repair. Use parallel option for speed:

bash

xfs_repair -P /dev/mapper/vg-lv

- If the log is corrupted and xfs_repair refuses, you can zero the log as last resort (data-risking):

bash

xfs_repair -L /dev/mapper/vg-lv

- Note: -L can lose recent metadata (inode/link) but may be necessary to recover mountability.

6) Post-repair validation- Run xfs_repair -n again to confirm no remaining issues.- Mount read-only, run filesystem checks, and run application-level validations.- Remount read-write in maintenance and gradually bring services back.

7) Restore path if repair fails- If xfs_repair cannot restore or results are unacceptable: - Restore from block-level image: dd or use ddrescue to write back to the block device (ensure target is correct):

bash

dd if=/backup/images/vg-lv.img of=/dev/mapper/vg-lv bs=4M status=progress

- If you backed up via an LVM snapshot or have file-level backups (rsync, Borg, backups), restore files rather than raw blocks: - Mount the snapshot or image loopback: losetup -f --show /backup/images/vg-lv.img; kpartx -av /dev/loopX; mount -o ro /dev/mapper/... - Copy data to new clean LV or newly provisioned storage with rsync preserving attributes. - If only partial recovery possible, use file-level salvage from readonly mounts and rebuild services.

8) Preventive follow-up- Replace failing hardware, run full filesystem scrubs on schedule.- Implement regular block-level snapshots and file-level backups, verify restores.- Document incident, root cause, and update runbooks for future incidents.

Key principles: never run destructive fixes before taking a complete block-level backup; prefer non-destructive checks first; validate hardware health; use LVM snapshots to minimize downtime and enable safe backups; -L is last-resort.

MediumSystem Design

83 practiced

Design a strategy to manage users and groups across hundreds of Linux servers. Compare local /etc-based management to centralized solutions such as LDAP, FreeIPA, or AD via sssd. Consider authentication, sudoers distribution, home directory provisioning, and offline access.

Sample Answer

Requirements:- Hundreds of Linux servers, mixed availability (data center, cloud, remote), need consistent uid/gid, central auth, sudo policy distribution, home dir provisioning, and reasonable offline access.- Non-functional: high availability, auditability, low-latency auth, secure transport, manageable by SREs.

High-level options1) Pure /etc management (local passwd/group, sudoers)- Pros: simple, works offline, no external dependency.- Cons: scale: error-prone, UID drift, hard to audit, labor-intensive; sudoers duplication leads to drift.

2) Centralized identity (LDAP/FreeIPA/Active Directory) + sssd on clients- Pros: single source of truth for accounts, centralized auth, group-based sudo rules, auditing, kerberos support (FreeIPA/AD) for stronger auth and single sign-on, SSSD provides offline caching and NSS/PAM integration.- Cons: more infrastructure, needs HA and monitoring, network dependency for logins (mitigated by SSSD cache).

Design recommendation (SRE-focused):- Use FreeIPA or AD (if enterprise already uses AD); expose via SSSD on all clients for NSS/PAM. - FreeIPA if you want integrated host/service management (certs, host entries, HBAC). - AD if organization policy requires it; use realmd/sssd or winbind.- Authentication: - Kerberos for strong auth and single sign-on (FreeIPA/AD). SSSD handles ticket renewal. - Enforce MFA for admin accounts at IdP level.- Account & sudoers distribution: - Manage users/groups centrally in IdP. - Distribute sudo rules via: - SSSD sudo provider (store sudoers in LDAP/IPA) so clients pull policies. - Or manage /etc/sudoers.d via configuration management (Ansible) with templating based on LDAP group membership. - Audit sudo via centralized logs (syslog to ELK/EFK) and leverage sudo log_options.- Home directory provisioning: - Use network home directories (NFSv4 with Kerberos/FSAL, or SMB for AD) mounted via autofs; or - On first login provision home with a provisioning script (e.g., PAM module that triggers rsync from template) if network homes are undesirable. - For cloud/ephemeral hosts, prefer ephemeral homes with centralized storage or object-backed profiles.- Offline access: - Enable SSSD offline caching (id, passwd, group, and kerberos ticket caching). Configure long cache_ttl and offline_credentials_expiration according to security policy. - Maintain a small set of local admin service accounts (managed via CM) for recovery when IdP is unreachable; protect keys tightly.- Availability & scaling: - Deploy IdP as HA cluster (replicated LDAP/IPA masters, load balancer). Use monitoring (Prometheus, alerts). - Use configuration management (Ansible) to ensure consistent sssd.conf, realmd, autofs, and sudoers files.- Security & compliance: - Enforce TLS for LDAP, restrict IPs, rotate service keys, enforce password/credential policies in IdP. - Role-based groups for sudo and host access; HBAC in FreeIPA or group filters in SSSD.- Trade-offs: - /etc-only: best for tiny isolated fleets or extreme air-gapped systems. - Centralized: operational overhead but massive gains in auditability, onboarding/offboarding speed, and consistency. SSSD mitigates network dependency through caching.

Operational playbook (SRE):- Automate client bootstrap (install sssd, join domain, configure automount).- Run chaos tests: simulate IdP outage to verify offline login and recovery.- Regularly reconcile UID/GID and run audit reports.- Backup IdP, test restores, and document emergency local-admin procedures.

Conclusion:For hundreds of servers, centralized identity with SSSD + network homes (or provisioning) is the scalable, auditable solution; complement it with robust caching, CM-driven local fail-safes, and HA IdP deployment.

HardTechnical

52 practiced

A Linux host experienced a kernel oops and then intermittent panics. Describe steps to collect crash data and configure kdump to capture vmcore, how to analyze vmcore with the crash utility or gdb to extract backtraces and modules, and criteria for filing a kernel bug report with a minimal reproducible test case.

Sample Answer

Situation & goal: When a host has an oops followed by intermittent panics you must (1) collect volatile and persistent crash data immediately, (2) configure kdump so future panics produce a vmcore, (3) analyze the vmcore to get backtraces and module state, and (4) prepare a minimal reproducible bug report if needed.

1) Immediate data collection- Capture live logs: dmesg -T > /tmp/dmesg-before.txt; journalctl -k --since "YYYY-MM-DD HH:MM" > /tmp/journal-k.txt- Save /var/log/messages, /var/log/kern.log and any application logs.- Record uptime, kernel version, uname -a, lsmod, /proc/modules, /proc/cmdline, and sysctl -a.- If system still unstable, reboot into rescue after collecting above.

2) Configure kdump to capture vmcore- Install kexec-tools and kdump: e.g., yum/apt install kexec-tools kdump-tools- Edit /etc/kdump.conf: set path (e.g., /var/crash), core_collector (makedumpfile -c --message-level 1), and reserve memory (crashkernel=auto or crashkernel=256M@16M in kernel cmdline).- Add crashkernel to GRUB: update-grub or grub2-mkconfig; reboot to activate.- Start and enable service: systemctl enable --now kdump.service- Test kdump: echo c > /proc/sysrq-trigger on a test VM (only on non-production) and verify /var/crash/<timestamp>/vmcore exists.

3) Analyze vmcore with crash (preferred) or gdb- Ensure matching vmlinux (unstripped) and System.map for exact kernel build.- Basic crash command: crash /usr/lib/debug/lib/modules/$(uname -r)/vmlinux /var/crash/.../vmcore- Useful commands inside crash: - bt (or bt -a) — show kernel stack traces for all tasks - ps — list tasks and current task - log — show die text and oops message - mod — list loaded modules and addresses - km — examine kernel memory - sym -l <symbol> — check symbol addresses - struct <type> <addr> — inspect structures (e.g., task_struct) - dis -m <addr> — disassemble around address - vm — physical/virtual memory mappings- With gdb (if vmlinux available): gdb vmlinux -c vmcore - bt, info threads, x/20i <addr> etc.- Extract module list: crash> mod; save output. Get per-module backtraces: crash> bt -m <module_name>

4) What to include for a kernel bug report (criteria & minimal reproducible case)- Include: kernel version, full uname -a, exact kernel config (.config), vmlinux (or System.map + build-id), vmcore, dmesg and journal around the oops, module list and versions, lsmod, steps that reliably reproduce the panic/oops, hardware details (model, firmware/BIOS), and whether this happens on multiple machines or kernels.- Minimal reproducible test case: smallest set of actions (or workload) to trigger the oops, VM/container reproducer if possible, and test script that others can run. If timing/scale dependent, provide a load script and expected timing.- Reproduce on latest stable kernel if feasible; report whether issue persists.- If confidential, provide a sanitized reproducer and offer to share artifacts via secure channel to maintain privacy.

Key rules of thumb:- Always capture vmcore + vmlinux/System.map — without matching symbols analysis is limited.- Prefer crash utility for convenience; gdb is fallback.- Make the repro deterministic and minimal; logs and artifacts are essential for triage.

EasyTechnical

50 practiced

Explain job control in a POSIX shell. Show the commands to run a process in the background, list background jobs, stop and resume jobs, bring a job to the foreground, and detach a job so it continues after you log out. Give a short example workflow.

Sample Answer

Job control in a POSIX shell lets you run, stop, resume, and manage processes within your interactive session (foreground vs background), so you can multitask or recover processes during logout.

Common commands:

bash

# start in background
sleep 300 &

# list jobs in this shell
jobs -l

# stop (suspend) current foreground job with keyboard
# Press Ctrl-Z

# resume a stopped job in background
bg %1

# bring a job to foreground
fg %1

# run a command immune to SIGHUP, keep it after logout
nohup long_running_cmd &

# detach an existing job so it won't be killed at logout
disown -h %1

Notes and behavior:- jobs shows job IDs like [1] and PIDs. Use %N to reference job N.- Ctrl-Z sends SIGTSTP (suspend). bg sends SIGCONT to continue in background; fg resumes in foreground.- nohup reroutes SIGHUP and stdout/stderr (creates nohup.out by default).- disown removes job from shell’s job table (with -h it keeps process but prevents SIGHUP).

Short workflow example:1) Start: `./deploy.sh &`2) Check: `jobs -l` (note job %1 PID)3) If suspended: `bg %1`4) Make persistent before logout: `disown -h %1` (or start with `nohup ./deploy.sh &`)

This is essential for SREs to manage long-running ops from interactive sessions without losing processes on disconnect.

EasyTechnical

45 practiced

Explain Unix file permissions for files and directories. Describe the meaning of rwx for user, group, and others, explain numeric octal notation such as 0755 and 0640, and show how chmod u+x differs from chmod 755 with an example scenario for each.

Unlock Full Question Bank

Get access to hundreds of Linux and Unix Fundamentals interview questions and detailed answers.

Join thousands of developers preparing for their dream job.