Linux Process and Service Management Questions
Technical topic covering operating system process and service management on Linux. Includes process states and hierarchies, signals and graceful termination, investigating processes with ps and top, interpreting the proc filesystem, managing daemons and services with systemctl and service commands, startup behavior, resource troubleshooting with top and free, log inspection in var log, and basic service restart and health check strategies. Candidates should be able to diagnose runaway processes and explain safe remediation steps.
HardTechnical
16 practiced
Design a systemd-based approach to rotate and retain high-volume logs safely for a service that produces many GBs per hour. Compare using journald retention controls (SystemMaxUse, SystemKeepFree) vs classic logrotate, and describe how you would avoid losing recent logs during rotation and minimize service disruption.
Sample Answer
Situation / requirements (clarify): we have a service that writes many GBs/hour and must retain recent logs reliably, avoid losing the tail during rotation, and minimize service disruption. Goal: safe local retention for short-term troubleshooting + reliable off-host shipping for long-term.Design summary (recommended): use journald for structured capture + a streaming agent (Vector/Fluentd/Fluent-bit) to ship logs off-host, and enforce conservative journald retention settings. For any service that writes to plain files, prefer a signal-based rotate (not copytruncate) using logrotate with a postrotate that tells the process to reopen log fds. Monitor drop/restart metrics and disk usage; alert before retention thresholds.Compare approaches1) journald retention controls (SystemMaxUse, SystemKeepFree, RuntimeMaxUse, VacuumSize):- Pros: atomic journal file handling, per-namespace access control, integrated with systemd, writes are append-only and fsync’d (lower risk of corrupting on rotate). journald deletes oldest archived journals when constraints exceeded—no need to signal the service.- Cons: deletion is based on size/time and can remove older archival files; if you rely solely on local journald for long-term retention you risk losing logs when high-volume bursts exceed thresholds. journald has its own on-disk format; shipping requires a reader/agent. Aggressive limits can delete recent records if misconfigured (e.g., RuntimeMaxUse too small for burst spikes).Example journald.conf settings:Storage=persistentSystemMaxUse=200GSystemKeepFree=10GRuntimeMaxUse=50GVacuumSize=180GRateLimitInterval=30sRateLimitBurst=1000Why safe: journald is transactional when writing; vacuuming removes oldest full journal files, not the active file, so the current tail is preserved. Still, ensure SystemMaxUse + VacuumSize provide headroom for peak bursts.2) classic logrotate (for plain-file logs):- Pros: familiar tool, flexible compression/retention, integrates well if the daemon supports reopen via signal.- Cons: copytruncate (which truncates the original file to avoid signaling) is lossy — it can drop log lines during the small window between copy and truncate and is inefficient for GBs/hour. Restarting the service to reopen files is disruptive.Best practice: Use logrotate without copytruncate. Configure postrotate to signal the process to reopen fds (SIGUSR1, SIGHUP, or use systemctl kill --signal=SIGUSR1 <svc>) or leverage systemd --user-file descriptors via systemd-cat. Example:/var/log/myservice/*.log { daily rotate 7 compress delaycompress missingok notifempty create 0640 myuser mygroup sharedscripts postrotate # instruct service to reopen logs (service-specific) /bin/systemctl kill -s SIGUSR1 myservice.service || true endscript}If the service cannot be signaled, use an external logging agent to tail the file (avoid copytruncate).Avoid losing recent logs during rotation- Prefer signal-based reopen over copytruncate. Signaling is atomic: the process closes old fd and opens a new file immediately; there is no window where logs are dropped.- For journald: ensure SystemMaxUse + VacuumSize are set to accommodate peak hourly throughput; set RateLimit* to avoid accidental suppression; use Storage=persistent and separate journal partition (XFS ext4 with reserved space) so journald deletions don't impact system root.- For file-based logs: rotate frequently (hourly) but keep rotation fast; use delaycompress so the rotated file is available until compression finishes; use O_APPEND in writer code (or stdio with flock) and ensure proper O_SYNC/fdatasync if durability is critical.- Use atomic rename for rotated files (logrotate does this when not using copytruncate).Minimize service disruption- Do not restart service to rotate logs. Signal to reopen or use systemd socket activation / stdout->journal to remove file handling from the app.- Move logging responsibility away from app: write to stdout/stderr and let systemd-journald capture it, or use a sidecar logging agent that tails and ships logs.- If using logrotate and the daemon supports log reopening on SIGHUP/SIGUSR1, use systemctl kill to send signal without altering cgroup or restarting service.Operational controls and safety nets- Ship logs off-host immediately (Vector/Fluent-bit). Treat local logs as ephemeral short-term cache.- Provision a dedicated partition for journals or log files; set SystemKeepFree and filesystem reserved blocks to avoid OOM or system instability.- Monitor: journald/rsyslog drop counters, journal size, free space, and dropped log metrics from the shipper. Alert well before retention thresholds.- Test rotation under load (chaos test): simulate peak generation and validate no lines lost and that rotation/shipper keep up.Trade-offs- journald-centric: simpler, safer write path, but requires shipping/translation for analysis and potentially more disk usage for the binary journal format.- logrotate-centric: more control over final files and retention/compression, but requires careful coordination with the process for safe reopen and can be riskier with copytruncate.Bottom line (SRE play): make journald your write-ahead capture (Storage=persistent + conservative SystemMaxUse) and run a resilient shipper to move logs off-host in near real-time. Use logrotate only for legacy file-output services that support reopen signals; never rely on copytruncate for multi-GB/hour workloads. Add monitoring/alerts and test rotation under realistic traffic.
EasyTechnical
22 practiced
Explain the difference between a zombie process and an orphan process. How do you detect zombies on a system and what are safe remediation steps an SRE can take to remove or prevent zombie accumulation without risking data corruption?
Sample Answer
A zombie process is a child that has exited but still has an entry in the process table because its parent has not called wait()/waitpid() to collect its exit status (its state is "Z" / defunct). An orphan process is a running process whose parent has exited; on UNIX/Linux it is reparented to init/systemd (PID 1) or to a designated subreaper and continues running normally.Detecting zombies:- Quick view: ps aux | awk '$8=="Z" {print $0}'- More explicit: ps -eo pid,ppid,stat,cmd | grep " Z"- Count per parent: ps -eo pid,ppid,stat,cmd | awk '$3 ~ /Z/ {print $2}' | sort | uniq -c- Top/htop show <defunct> entries; ss/strace on parent can help triage.Safe remediation steps (SRE-focused, minimize data-risk):1. Ask parent to reap: - If parent is responsive, request it to reap children (fix in code: call wait/waitpid or install SIGCHLD handler). - You can send SIGCHLD to parent to nudge it: kill -SIGCHLD <parent-pid>. (This only helps if parent has a handler that wakes on the signal.)2. Graceful restart of parent: - Restart the parent process/service cleanly (service restart or systemctl restart) so it reaps children on shutdown or after restart. Prefer rolling/redeploy windows to avoid data loss.3. If parent is hung/unresponsive and cannot be safely restarted: - As a last resort, kill the parent (kill <parent-pid>) — when the parent dies its children are reparented to init/systemd which will reap exited children, clearing zombies. Avoid killing PID 1 or critical system services without careful planning.4. For containers: - Ensure PID 1 in the container reaps children (use tini or proper init). Recreate container if necessary.Prevention and long-term fixes:- Fix application logic: ensure children are wait()ed or use double-fork/daemonize patterns when appropriate.- Use proper signal handling (SIGCHLD) and non-blocking wait loops in long-running parents.- Use init systems or tini for containers so PID 1 reaps children.- Monitor: alert on growth of processes in state Z per host/service; set thresholds.Caveats:- Killing a parent may cause in-flight work or state loss; prefer graceful restarts and code fixes.- Do not kill PID 1; coordinate maintenance for critical services.
HardTechnical
24 practiced
Implement (or outline in pseudocode) a Python script that watches processes and moves any process exceeding a configurable swap usage threshold into a low-priority cgroup to limit its resource impact. Address concurrency, PID reuse, and how you would verify operations did not destabilize critical services. Describe how you would deploy this safely.
Sample Answer
Approach: run a daemon that periodically samples per-process swap usage (from /proc/[pid]/status or smaps_rollup), and atomically moves offending PIDs into a pre-created low-priority cgroup (v1 cpuset/cpu? or unified cgroup v2 with cpu.weight & memory.swap.max controls). Handle concurrency via worker threads + a single cgroup-move queue and optimistic checks to avoid races; guard against PID reuse by verifying process start_time (from /proc/[pid]/stat) before moving.Python pseudocode + example:Key concepts:- Use smaps_rollup for efficient swap accounting.- Verify start_time to prevent acting on a new process that reused the PID.- Serialize writes to cgroup.procs if high concurrency; current code appends but production should use a thread-safe queue and single writer.Safety & verification:- Exclude critical system PIDs (1, kernel threads) and services (via whitelist by cgroup, unit file tags).- Start in "dry-run" logging mode to record intended moves before actuating.- Add health checks: compare service unit health (systemd) and synthetic transactions before/after moves.- Monitor metrics: OOM events, IO wait, load average, latency of critical services; set alerts.Deployment plan:- Ship as a systemd service with capability bounding (CAP_SYS_RESOURCE), run in a dedicated namespace on canary hosts first.- Start in read-only/dry-run, run for 24-48h, review logs/alerts, then enable enforce mode on canaries.- Gradually roll to production with automated rollback if alerts trigger (increase thresholds or disable service).- Documentation, runbook, and a kill-switch (systemd mask) included.Edge cases & improvements:- Use eBPF for low-overhead sampling at scale.- Consider rate-limiting moves to avoid thrashing.- For containers, operate via container cgroup paths and annotate pods/services to exclude.
python
import os, time, threading
CGROUP_PATH = "/sys/fs/cgroup/lowprio" # cgroup v2
SWAP_THRESHOLD_BYTES = 500*1024*1024
SAMPLE_INTERVAL = 5
def read_swap_and_starttime(pid):
try:
with open(f"/proc/{pid}/smaps_rollup") as f:
data = f.read()
swap = int(next(line for line in data.splitlines() if line.startswith("Swap:")).split()[1]) * 1024
with open(f"/proc/{pid}/stat") as f:
parts = f.read().split()
start_time = parts[21] # clock ticks
return swap, start_time
except Exception:
return None, None
def move_to_cgroup(pid, start_time):
# verify still same process
cur = read_swap_and_starttime(pid)
if cur[1] != start_time:
return False
try:
with open(os.path.join(CGROUP_PATH, "cgroup.procs"), "a") as cg:
cg.write(str(pid)+"\n")
return True
except Exception:
return False
def monitor():
while True:
pids = [d for d in os.listdir("/proc") if d.isdigit()]
for pid in pids:
swap, start = read_swap_and_starttime(pid)
if swap and swap > SWAP_THRESHOLD_BYTES:
threading.Thread(target=move_to_cgroup, args=(pid,start)).start()
time.sleep(SAMPLE_INTERVAL)MediumTechnical
17 practiced
Explain how to restrict a service's CPU and memory using cgroups (preferably cgroup v2 or systemd unit settings). Provide example systemd unit directives or commands showing how to limit memory to 2G and CPU to 20% of a single CPU, and explain how to inspect the applied cgroup settings on disk.
Sample Answer
Situation: As SRE you'll often enforce resource limits at the service/unit level using systemd which speaks cgroup v2. The simplest, idiomatic way is to set MemoryMax and CPUQuota in the unit file (systemd converts these into cgroup v2 settings like memory.max and cpu.max).Example systemd unit snippet (drop into /etc/systemd/system/my.service):[Unit]Description=My limited service[Service]ExecStart=/usr/bin/my-binaryMemoryMax=2G # limits memory to 2 GiB (cgroup v2 -> memory.max)CPUQuota=20% # limits CPU to 20% of one CPU (systemd computes cpu.max quota/period)[Install]WantedBy=multi-user.targetApply without editing unit:sudo systemctl set-property my.service MemoryMax=2G CPUQuota=20%Quick alternative with transient scope:sudo systemd-run --scope --unit=temp --property=MemoryMax=2G --property=CPUQuota=20% /usr/bin/my-binaryHow CPUQuota maps to cgroup v2: systemd writes cpu.max as "<quota> <period>". For 20% with default period 100000µs, quota=20000 -> cpu.max "20000 100000".Inspect applied settings on disk (cgroup v2 unified):1) Find the unit cgroup path:systemctl show -p ControlGroup --value my.service2) Read cgroup files:cat /sys/fs/cgroup/<unit-path>/memory.maxcat /sys/fs/cgroup/<unit-path>/cpu.maxOr use systemd queries:systemctl show my.service -p MemoryCurrent -p MemoryMax -p CPUQuotaNotes and tips:- MemoryMax enforces OOM accounting; use MemoryHigh for soft limits.- CPUQuota is relative to a single CPU; for multi-CPU systems use CPUQuotaPerSecUSec or CPUWeight for proportional shares.- Ensure your system is running cgroup v2 (mount at /sys/fs/cgroup and /proc/filesystems contains "cgroup2").
MediumTechnical
19 practiced
Write a short incident runbook for handling a long-running process that is consuming all memory and threatening to trigger the OOM killer on a production host. Include pre-incident checks, safe remediation steps (including graceful termination attempts), communication steps, and how to update SLOs or error budgets after the incident.
Sample Answer
Pre-incident checks (run continuously / before touching host)- Validate alert: confirm host alerts/metrics (memory usage > 85%, RSS near total) and check OOM score. Pull timeline from monitoring (time-series) and confirm it's not a transient spike.- Gather context: uname -a, uptime, kernel logs, recent deploys, pod/container IDs.- Quick commands (SSH to host or use orchestration tooling):Safe remediation steps (order matters; prefer least disruptive → more forceful)1. Rate-limit impact - If possible, disable incoming traffic to the instance (LB remove, cordon node, scale down traffic via feature flag). - For k8s: cordon node + evict safe workloads: `kubectl cordon NODE; kubectl drain NODE --ignore-daemonsets --delete-local-data --force` (use with caution).2. Attempt graceful recovery - Connect to process and request graceful shutdown/reload (HTTP /health endpoint, admin endpoint, SIGHUP if supports reload). - Example: send SIGTERM and wait: `kill -TERM PID` - Wait a configured grace period (e.g., 30s-120s) while monitoring memory decline and request in-flight drains.3. If unresponsive, escalate signals - Check process's OOM score: `cat /proc/PID/oom_score` and adjust `oom_score_adj` if needed to protect/target: `echo 1000 > /proc/PID/oom_score_adj` (requires root). - Use SIGINT then SIGQUIT if supported: `kill -INT PID` then `kill -QUIT PID` with waits between.4. Controlled termination - If still not reclaimed and OOM is imminent, do a forceful kill: `kill -9 PID` and confirm memory reclaimed. - For containers: `docker stop CONTAINER` (wait), then `docker rm` / `docker restart`. - For k8s: `kubectl delete pod POD -n NS --grace-period=0 --force`5. Temporary mitigations - Add swap (short-term) or kill other low-priority processes to buy time. - If recurring, increase replica count or scale vertically temporarily.6. Verification - Confirm memory freed, application healthy, traffic restored, monitoring alerts cleared. Collect logs and core dump if available for triage.Communication steps- Incident channel: open dedicated channel (Slack/Teams), post initial severity, affected service, immediate action plan and ETA.- Template update: "Incident: High memory usage on HOST/POD. Impact: potential OOM and degraded/failed requests. Action: draining node, attempting graceful shutdown of PROCESS. Next update: in 10 minutes."- Notify stakeholders and on-call rotation; escalate to dev owning service if process is application-level.- Provide running updates at regular intervals and on state changes (mitigation, containment, resolution).Post-incident: root cause & SLO/error budget handling- Immediately record incident start/end timestamps; tag alerts.- Short-term: deduct downtime/error budget proportional to user impact window. Example: if 5% of requests failed for 12 minutes and page-level SLO is 99.9%, compute error budget burn and notify product/ops.- Add follow-up actions to ticket: reproduce locally, memory profiling, heap dump analysis, fix (GC tuning, memory leak patch, backpressure, req limits, cgroups, liveness/readiness improvements).- Update SLOs only if systemic capacity issues show SLOs unrealistic; otherwise keep SLO and treat incident as budget burn.- Postmortem: blameless write-up with timeline, root cause, remediation, and action owners with deadlines. Track completed mitigations and re-evaluate monitoring thresholds/alerting to catch growth earlier.Key safeguards and best practices- Implement OOM protections: cgroups or container memory limits, readiness/liveness probes, per-process oom_score_adj policies.- Automated runbook actions where safe (e.g., auto-drain and scale) to reduce manual error.- Practice drills and runbook reviews every 6 months.
bash
# host-level
free -m; vmstat 1 5; dmesg --ctime | tail -n 200
ps aux --sort=-rss | head -n 10
# container/k8s
docker ps --no-trunc; docker stats --no-stream CONTAINER_ID
kubectl get pods -o wide
kubectl top pod POD -n NS
kubectl describe pod POD -n NSUnlock Full Question Bank
Get access to hundreds of Linux Process and Service Management interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.