Comprehensive knowledge of operating system concepts and practical administration across Linux, Unix, and Windows platforms. Core theoretical topics include processes and threads, process creation and termination, scheduling and context switching, synchronization and deadlock conditions, system calls, kernel versus user space, interrupt handling, memory management including virtual memory, paging and swapping, and input and output semantics including file descriptors. Practical administration and tooling expectations include file systems and permission models, user and group account management, common system utilities and commands such as grep, find, ps, and top, package management, service and process management, startup and boot processes, environment variables, shell and scripting basics, system monitoring, and performance tuning. Platform specific knowledge should cover Unix and Linux topics such as signals and signal handling, kernel modules, initialization and service management systems, and command line administration, as well as Windows topics such as the registry, service management, event logs, user account control, and graphical and command line administration tools. Security and infrastructure topics include basic system hardening, common misconfigurations, and an understanding of containerization and virtualization at the operating system level. Interview questions may probe conceptual explanations, platform comparisons, troubleshooting scenarios, or hands on problem solving.
MediumTechnical
57 practiced
On a Windows server, a critical service crashed. Explain how you would use Event Viewer, the System and Application logs, and diagnostic tools (Process Explorer, ProcDump, Performance Monitor) to determine the cause. Highlight the differences compared to Linux diagnostics.
Sample Answer
First I’d triage with Event Viewer to get timeline and correlated symptoms.- Event Viewer (System/Application): open Windows Logs → Application and System. Filter by Time and Level (Error/Critical). Look for: - Application Error / .NET Runtime / Faulting application entries (gives faulting module, exception code, offset). - Service Control Manager events (service stopped/crashed, exit code). - Paired System events (disk, driver, kernel-power) indicating resource or OS-level causes. Use Event IDs and the timestamp to map to alerts and deploys.Next I’d capture process-level evidence.- Process Explorer: inspect the crashed service’s process tree, handles, loaded DLLs, and CPU/memory at rest. Use its properties to identify suspicious DLLs, high handle counts, or threads stuck in kernel. Useful for live inspection and to find parent/child relationships.- ProcDump: if the service might crash again or hang, set up ProcDump to capture a full or mini-dump on exception/CPU/memory threshold: procdump -ma -e -t -s 5 -n 3 ServiceName.exe C:\dumps Analyze dump with WinDbg or dotnet-dump to get stack traces and exception context.- Performance Monitor (PerfMon): collect counters (Process\% Processor Time, Process\Private Bytes, .NET CLR Exceptions, Thread Count), and System counters (Disk Queue Length, Available MBytes). Use a short-range Data Collector Set to correlate metric spikes to the crash time.Workflow & reasoning:1. Use Event Viewer to form hypothesis (exception vs resource starvation vs service termination).2. If transient, use ProcDump to capture reproducible evidence.3. Use Process Explorer and PerfMon to validate resource/handle/thread leaks and DLL issues.4. Analyze dumps for root cause; correlate with recent deployments/config changes.Differences vs Linux:- Windows relies on Event Viewer/ETW and structured Event IDs; Linux uses syslog/journald, kernel messages and core dumps. ProcDump/Process Explorer are Windows equivalents of gdb/strace/top/ps/ss; PerfMon maps to sar/top/iostat/collectd/Prometheus. Dump analysis tools differ (WinDbg vs gdb/objdump); on Windows symbol resolution and PDBs are often required. Windows provides richer built-in GUI tooling; Linux favors CLI toolchains and plain-text logs.
HardSystem Design
71 practiced
Design a rolling kernel update strategy for a multi-region cluster that minimizes downtime and supports automatic rollback if the new kernel causes regressions. Include bootloader (GRUB) management, image promotion, canarying, health checks, and stateful workload considerations.
Sample Answer
Requirements:- Zero/low downtime multi-region rolling kernel updates- Automated canary + progressive promotion- Safe GRUB boot management and atomic image promotion- Automatic rollback on regression with minimal impact to stateful workloads- Observability, safety gates, and auditabilityHigh-level architecture:- Build pipeline (CI) -> Immutable boot images (kernel + initramfs + OS image) stored in signed image registry (artifact store) per region -> Orchestrator (cluster-updater) + Region controllers -> Node agents (updater + GRUB manager) -> Observability & health evaluator (metrics, logs, e2e smoke)Flow:1. Build & sign kernel image; tag as candidate:vX.2. Canary phase: promote candidate to a small fraction of nodes in a single region (configurable: capacity, AZ spread). Orchestrator instructs node agents to fetch image to local disk (do not change boot order yet) and register readiness.3. GRUB management: node agent updates GRUB.cfg atomically by creating a new menuentry pointing to candidate kernel, using a priority scheme (next-boot marker). Keep previous entry intact. Use grub-install+grub-mkconfig with transactional write (write to tmpfile, fsync, rename).4. Controlled reboot: canary nodes are cordoned/drained safely (stateful workloads: prefer replicas on other AZs/regions; use PVC-aware eviction or storage-level handoff). Reboot into candidate kernel.5. Health checks: node-level (systemd unit statuses, dmesg errors), platform-level (kubelet readiness, pod health probes), and service-level synthetic tests (latency, key transaction success). Central evaluator aggregates and scores via SLIs.6. Observe for a defined window. If metrics pass threshold, promote candidate to next batch. Promotion is automated with exponential ramp-up and manual hold points.7. Automatic rollback: if health evaluator detects regressions beyond thresholds, orchestrator issues rollback: node agents mark next-boot to previous kernel (using saved previous GRUB entry), or if not possible, restore from snapshot or boot from rescue image. For nodes that already failed to boot, out-of-band recovery uses IPMI/console automation to select previous GRUB entry.8. Image promotion: only after N regions pass canary do CI mark candidate stable and update global tag (vX -> stable). Image registry records provenance and signed manifests.Stateful workload considerations:- Stateful sets: prefer updating nodes hosting non-migratable PVs last per AZ, and ensure replication factor keeps quorum. Use scheduling policies to avoid simultaneous reboots of majority holders.- Use storage-level fencing and application-aware pre-stop hooks to quiesce I/O.- For databases: use rolling leader demotion/promote strategy; update followers first; run consistency checks post-update.Safety & observability:- Circuit breakers: rate limits, max concurrent reboots, region isolation.- Playbooks & runbooks auto-triggered on rollback.- Audit logs for GRUB changes and image promotions; signed artifacts.- Canary dashboards, automated alerting scaling to SLOs.Trade-offs:- Immutable images + GRUB atomic swaps increase storage/complexity but reduce drift.- Conservative rollback logic may delay rollout; tune thresholds per service.- Full automation reduces human error but requires robust tests and out-of-band rescue for boot failures.This strategy balances automation, GRUB safety, canary validation, and careful handling of stateful services to minimize downtime and enable fast, reliable rollback.
MediumTechnical
56 practiced
You want to harden SSH access across your fleet. Describe SRE-focused best practices (key-only login, disable password auth/root login, use SSH bastions, agent forwarding rules, rate limiting) and how you'd roll these changes safely without locking administrators out.
Sample Answer
Approach: treat SSH hardening as a reversible, automated change with safety gates, observability, and staged rollout so availability and admin access are never lost.Best practices (what & why)- Key-only login: disable PasswordAuthentication to eliminate brute-force/credential-stuffing risk. Use strong ED25519/ECDSA keys and enforced key length/algorithms.- Disable root login: set PermitRootLogin no and require sudo from individual accounts for auditing and least privilege.- SSH bastions: force all inbound SSH through hardened bastion hosts with MFA and strict logging to central SIEM; bastions reduce attack surface and centralize access controls.- Agent forwarding rules: avoid unrestricted ForwardAgent; only allow AgentForwarding on trusted bastions and restrict with Match blocks/CAC to reduce credential theft risk.- Rate limiting: use firewall (iptables/nft) or sshd's MaxStartups and fail2ban/rate-limiter to mitigate brute-force; combine with network ACLs.- Key lifecycle & rotation: store public keys in an authoritative source (LDAP/SSO/CAC/Git-backed), enforce TTLs, automated rotation and revocation, and use ephemeral short-lived certs (ssh-certificates from CA) when possible.- Auditing & monitoring: centralize logs (syslog → ELK) and alert on anomalous auth attempts, new key uploads, or bastion hops.Safe rollout plan (how)1. Prepare automation: write IaC/Ansible/Terraform modules to apply sshd_config, firewall rules, and bastion routing; include an automated rollback path.2. Staging test: apply to non-prod/test instances; verify connectivity, sudo flows, logging, and key acceptance.3. Admin readiness: collect and verify admin public keys, ensure at least two independent admins have validated access, provision break-glass credentials (time-limited) stored in a vault with audited access.4. Gradual production rollout: - Phase A: enforce key-only on a small canary cohort (non-critical services). Monitor authentication metrics and incident queues for 24–72 hours. - Phase B: expand to larger pools, enable rate limiting in parallel. - Phase C: disable root login after confirming sudo works and audit trails are complete.5. Bastion migration: deploy bastion(s) behind MFA+IAM, shift SSH ACLs to allow SSH only from bastion IPs, then remove direct public SSH.6. Verify agent forwarding rules and deploy SSH CA/short-lived certs to replace static keys over weeks.Safety measures (prevent lockout)- Always maintain at least two independent, tested admin keys in authoritative store before disabling passwords.- Implement and test automated rollback that restores previous sshd_config if health checks fail.- Use orchestration to run a connectivity test post-change (attempt SSH and run a noop command); if failures exceed threshold, auto-roll back.- Keep break-glass in secure vault with multi-party approval and expire it automatically.- Use audit logs and realtime alerts for failed access spikes so you can quickly intervene.Metrics & validation- Track auth success/failure rates, number of password attempts blocked, bastion session counts, time-to-recover for access incidents.- Post-change runbook: verify access, check sudo/audit logs, confirm no service breakages.Trade-offs- Short-lived certs and bastions add operational overhead but greatly reduce risk; automation and good key management minimize that burden.This approach balances security hardening with SRE priorities: automated, observable, reversible changes that preserve availability and admin access.
MediumTechnical
68 practiced
Write a command-line one-liner or short shell script (bash) to find the top 5 processes holding the most open file descriptors on a Linux host and print PID, command, and open file count. Describe assumptions and how you would adapt it for use in a monitoring script.
Sample Answer
One-liner (uses /proc, no lsof required):
bash
#!/usr/bin/env bash
# prints PID, command, and number of open fds for top 5 processes
for pid in $(ls -1 /proc | grep -E '^[0-9]+$'); do
if [ -d "/proc/$pid/fd" ] 2>/dev/null; then
echo "$(ls -1 /proc/$pid/fd 2>/dev/null | wc -l) $pid"
fi
done | sort -rn | head -n 5 | while read -r cnt pid; do
cmd=$(ps -p "$pid" -o comm= 2>/dev/null || echo "[exited]")
printf "%6s %6s %s\n" "$pid" "$cnt" "$cmd"
done
Explanation and assumptions:- Reads /proc/<pid>/fd to count file descriptors (fast, avoids invoking lsof for each pid).- Assumes the script runs as root or with sufficient permissions to read /proc/<pid>/fd for other users. As non-root, you'll only see FDs for processes you own.- Accounts for short-lived PIDs by handling missing dirs and suppressed errors.- ps -p ... -o comm= returns the command name; adjust to -o cmd= for full command line.Adapting for monitoring/automation:- Wrap in a small script that emits structured output (JSON or Prometheus metrics). Example metric: process_fd_count{pid="1234",cmd="nginx"} 512- Run periodically (cron/systemd timer or as a collector) and push to monitoring (Prometheus pushgateway or exporter).- Add thresholds and alerting: track top-N trend, alert if any process's FD count > threshold or rate-of-change high.- For containers/namespaces, run inside container or inspect /proc of container PID namespaces (nsenter).- Optimize: avoid shell loops for high PID counts by using a single awk/perl program to reduce overhead if running frequently.
EasyTechnical
73 practiced
Describe virtual memory on Linux: the role of page tables, the TLB, what a page fault is, and the difference between major and minor page faults. Explain why overcommit/commit settings matter for SREs and list commands you'd use to inspect virtual memory usage in production.
Sample Answer
Virtual memory maps each process’s virtual addresses to physical RAM (or swap) so processes get isolated contiguous address spaces. The kernel uses per-process page tables (multi-level on x86_64) to store mappings: virtual page → physical frame + permissions. The TLB (translation lookaside buffer) is a CPU cache of recent page-table translations — it makes address translation fast; TLB misses require walking page tables (hardware or kernel-managed), which is slower.A page fault happens when the CPU cannot translate a virtual address via the page tables/ TLB: either the mapping is not present or access permissions don’t allow the requested operation. The kernel then handles it. Minor (soft) page faults occur when the page is present in RAM but not mapped into the process’s page tables (e.g., copy-on-write or after mmap with MAP_PRIVATE); the kernel fixes metadata and maps it without I/O. Major (hard) page faults require loading the page from disk (swap or file) — this incurs I/O and latency.Overcommit/commit settings matter because Linux can allow allocation beyond physical RAM (overcommit), causing processes to succeed in malloc() but later incur OOM when memory is actually used. For SREs, this affects reliability: unexpected OOM kills, swap storms, or degraded latency. Configure vm.overcommit_memory and vm.overcommit_ratio according to workload (e.g., conservative mode 2 for databases), monitor commit-related stats, and use cgroups to limit memory per service.Useful commands to inspect VM in production:- free -h (high-level memory + swap)- cat /proc/meminfo (detailed stats: MemAvailable, CommitLimit, Committed_AS)- vmstat 1 (system memory, paging, IO)- top / htop (per-process RES/VIRT/SWAP)- ps aux --sort=-rss (top memory consumers)- smem (accurate proportional resident calculations)- pidstat -r or perf stat -e page-faults (per-process page-fault counts)- pmap -x <pid> (process memory map)- sar -B / iostat (swap/page I/O monitoring)- sysctl vm.overcommit_memory vm.overcommit_ratio (view/set commit policy)As an SRE, use these tools to detect swap I/O, high major-fault rates (latency), and tune overcommit, cgroups, and OOM policies to maintain service SLAs.
Unlock Full Question Bank
Get access to hundreds of Operating System Fundamentals interview questions and detailed answers.
