Linux and Windows System Administration Questions

Requirements clarified:

• Low CPU on hosts, TLS in transit, at-least-once delivery, local buffering during outages, efficient indexing.

Agent choice:

• Fluent Bit as host agent (C, <10MB, low CPU/memory) or Filebeat if deeper Elastic integrations required. I recommend Fluent Bit for lightweight footprint and built-in filesystem buffering.

Pipeline overview:

• Fluent Bit (tail input) → local buffer (filesystem) → batch/compress → TLS to a gateway/ingest tier → EFK (Elasticsearch via Fluentd/Logstash or Elasticsearch HTTP/gRPC).

Buffering & batching strategy:

• Use tail with multiline support, set small in-memory buffer for low-latency. Enable filesystem-based persistent buffer (e.g., fluent-bit storage.path) to ensure durability across restarts and network outages.
• Configure batch_size (e.g., 5–10k events or 5–10 MB) and batch_timeout (1–5s) to balance latency and throughput.
• Implement backpressure: gateway returns 429/5xx; agent retries with exponential backoff and keeps events on disk until ACKed. This yields at-least-once semantics.

Compression & transport:

• Send batches over HTTPS/TLS (mutual TLS optionally) to protect in-transit data.
• Enable gzip compression on HTTP requests to reduce bandwidth; decompress at the ingest tier.
• Use keep-alive connections and connection pooling to minimize CPU overhead.

Gateway/ingest tier:

• Run an autoscaled pool of lightweight gateways (Fluentd/Logstash or nginx + buffering) that accept compressed TLS batches, validate, and forward to Elasticsearch Bulk API. Gateways can handle auth, rate-limit, and parse enrichment.

Indexing & efficient storage:

• Use Elasticsearch index templates and pipeline processors to parse fields, drop noisy fields early, and use appropriate datatypes to reduce index size.
• Route by application/environment to separate indices; use daily or size-based rollovers (ILM) with optimized shard sizing (target ~30-50GB per shard).
• Use bulk sizes tuned for ES (e.g., 5–15MB per bulk) and concurrent bulk threads on gateways; agents should aim for similar batch sizes.

Retention & rotation:

• Use Elasticsearch Index Lifecycle Management (hot-warm-cold-delete):
  • Hot: frequently queried, fast nodes, short period (days)
  • Warm: readonly replica, longer retention for occasional queries
  • Cold/frozen: infrequent access, searchable but slower
  • Delete after retention window (30/90/365 days depending on compliance)
• Compress older indices (force merge, shrink) and use rollover policy based on age/size/doc count.

Operational considerations:

• Monitor agent CPU/memory and disk buffer usage; set local disk quota for buffering and alert when >75%.
• Secure credentials with vault and rotate TLS certs.
• Test failure modes: full disk, prolonged network outage, ES backpressure.
• Provide tooling to rehydrate buffered logs if needed.

Trade-offs:

• Fluent Bit minimizes host load; using gateways adds complexity but centralizes parsing and rate control, reducing host overhead.
• At-least-once may cause duplicates—handle dedupe in consumers or by including unique event IDs/timestamps.

This design meets low host overhead, durable local buffering, TLS in-flight, at-least-once delivery, and efficient indexing/retention via ILM.

Ensure consistent timezone and accurate clock across a fleet by setting timezone to UTC, enabling system NTP sync, installing and running chrony, and verifying sync. Example commands and verification steps:

Set timezone to UTC (idempotent):

# set timezone to UTC
sudo timedatectl set-timezone UTC

# verify
timedatectl status
# or
timedatectl show --property=Timezone --value

Enable system NTP (timedatectl tells systemd/OS that NTP should be used):

sudo timedatectl set-ntp true

Install and enable chrony (package name may be chrony or chronyd depending on distro), start service and enable at boot:

# on RHEL/CentOS/Fedora
sudo yum install -y chrony
sudo systemctl enable --now chronyd

# on Debian/Ubuntu
sudo apt update
sudo apt install -y chrony
sudo systemctl enable --now chrony

Verify chrony is running and synchronized:

# service status
sudo systemctl status chronyd    # or 'chrony' on Debian

# check peers and sync state
chronyc tracking
chronyc sources -v
# or
sudo chronyc sourcestats

Confirm system clock sync and offset:

timedatectl status   # shows "System clock synchronized: yes" and NTP service
# or check kernel clock vs RTC
timedatectl show --property=NTPSynchronized --value

Operational notes and why this matters:

• Use UTC across servers to avoid DST/offset drift and simplify log timestamps and scheduled jobs.
• Chrony is recommended for VMs and unstable networks: faster convergence and better handling of clock drift than traditional ntpd.
• Ensure outbound UDP/123 (NTP) is allowed to reach chosen stratum servers (or internal NTP pool).
• Accurate time is critical for distributed systems: it enables correct ordering of events, causality (e.g., tracing, distributed transactions), certificate validation (TLS/OCSP), cron job correctness, and consistent TTL/expires behavior. For incident response and observability, synchronized timestamps across services make log correlation, tracing, and forensic analysis reliable; even seconds of skew can break join queries, alerting windows, and SLO calculations.

Automation tip: bake these steps into your image/CM tooling (Ansible, Terraform provisioners, cloud-init) and monitor NTP synchronization metrics (chrony metrics) with alerts when offset exceeds acceptable thresholds (e.g., >500 ms).

Situation: You need tamper-evident auditing for /etc/sudoers, be able to query who/what changed it, and forward events to a central SIEM.

1. Configure auditd to watch /etc/sudoers

• Add a rule (persistent):
  • Edit /etc/audit/rules.d/99-sudoers.rules and add:
    -w /etc/sudoers -p wa -k sudoers_mod
  • Explanation: -w watches the path, -p wa watches writes and attribute changes, -k tags events with the key sudoers_mod.
• Load rules:
  sudo augenrules --load
  or
  sudo systemctl restart auditd

2. Query recent changes & extract user/process

• Find events by key (human-readable):
  sudo ausearch -k sudoers_mod -i --start recent
• Show file modification syscall events and include user/process fields:
  sudo ausearch -k sudoers_mod -i --raw | ausearch -i -k sudoers_mod
  or directly filter SYSCALL records for today:
  sudo ausearch -k sudoers_mod -i -ts today -m SYSCALL | aureport -f
• Extract who (auid/uid) and process (pid/exe):
  sudo ausearch -k sudoers_mod -i -ts today -m SYSCALL -sv no
  | awk -F' ' '/node=|type=SYSCALL/ {print $0}'
  | sed -n 's/.auid=([0-9])./auid:\1/p; s/.uid=([0-9])./uid:\1/p; s/.pid=([0-9])./pid:\1/p; s/.exe="(.)"./exe:\1/p'
• Simpler example to see important fields:
  sudo ausearch -k sudoers_mod -i -ts today | aureport --summary -i
• Note: map auid -> username via getent passwd (auid=4294967295 means anonymous/kernel). For automation, parse audit JSON (see next).

3. Forward audit logs to a central SIEM
   Option A — Filebeat/Auditbeat (recommended):

• Use Elastic's auditbeat or Filebeat audit module to tail /var/log/audit/audit.log and forward to Kafka/Elasticsearch/Logstash.
  • Install auditbeat on host, enable auditd module, configure output:
    output.elasticsearch or output.logstash or output.kafka
  • Advantages: structured events, TLS, batching, backpressure.
    Option B — audispd-forward or audisp-remote:
• Configure /etc/audisp/plugins.d/au-remote.conf to forward to a remote auditd receiver (audisp-remote/audit-remote daemon). Use TLS and firewall rules. This sends native audit protocol.
  Option C — rsyslog/imfile:
• Use rsyslog to read /var/log/audit/audit.log (imfile) and forward via RELP/TCP to SIEM: less preferred (less structured).

Best practices for SIEM:

• Ship structured JSON when possible (Filebeat/Auditbeat).
• Use TLS, authentication, and mutual certs.
• Preserve seq numbers/timestamps and hostname; include the audit key as a field (sudoers_mod).
• Rate-limit/alert on rapid changes and correlate with sudo, ssh logins (auid/uid), and process exe.

Example Filebeat snippet:
filebeat.inputs:

• type: log
  enabled: true
  paths: /var/log/audit/audit.log
  processors:
• decode_json_fields: { fields: ["message"], target: "audit" }
  output.logstash:
  hosts: ["siem.example.com:5044"]

Outcome: persistent audit rule with key sudoers_mod, use ausearch/aureport to identify actor (auid/uid, pid, exe), and forward logs securely to SIEM with Filebeat/Auditbeat or audisp-remote for centralized alerting and compliance reporting.

Situation: The new release begins crash-looping in production and we need to rollback immediately with minimal downtime while preserving logs/metrics for post-incident analysis.

Immediate manual rollback procedure (safe, fast)

1. Preserve runtime logs/metrics first

• Collect journal and stdout for the failing unit:
  journalctl -u myservice --since "5 minutes ago" > /tmp/myservice_journal_pre_rollback.log
• If metrics/telemetry agent is running (Prometheus node exporter, pushgateway), ensure scrape targets are healthy and snapshot any recent metrics if possible (e.g., use Prometheus API to export last 15m).
• If core dumps enabled, copy /var/lib/systemd/coredump or /var/crash to /tmp for analysis.

2. Stop the bad process gracefully

• systemctl stop myservice.service
• Confirm it's stopped: systemctl status myservice.service && ps aux | grep myservice

3. Make the old binary available (recommended deployment layout)

• Use release directories: /opt/myservice/releases/v1.2.3/ and /opt/myservice/releases/v1.2.4/
• Keep a current symlink: /opt/myservice/current -> /opt/myservice/releases/v1.2.4
• To rollback atomically:
  • ln -sfn /opt/myservice/releases/v1.2.3 /opt/myservice/current
  • systemctl daemon-reload (only if unit references changed)
  • systemctl start myservice.service
• If unit ExecStart uses absolute path to binary, start explicitly:
  • systemctl stop myservice.service
  • /opt/myservice/releases/v1.2.3/bin/myservice & (for quick emergency)
  • systemctl start myservice.service

4. Verify service health and restore traffic

• Check health endpoints: curl -f http://localhost:8080/health
• Confirm logs: journalctl -u myservice -n 200 --no-pager
• Confirm metrics resume and alerting clears
• Re-enable load balancer / service discovery once healthy

Preserving evidence for post-incident

• Archive journal and stdout: gzip /tmp/myservice_journal_pre_rollback.log && upload to artifact store (S3)
• Save binary artifacts: copy failing binary and working binary to artifact repo: /artifacts/myservice/fail-vX && /artifacts/myservice/good-vY
• Tag release in VCS/registry with incident notes and captured logs/metrics links
• Ensure core dumps, heap profiles, and flamegraphs are collected if applicable

Systemd-specific tips

• Use systemctl --no-block for non-blocking operations if needed
• Use systemctl kill --signal=SIGTERM myservice.service to force-stop remaining processes
• Use systemctl edit myservice.service to temporarily override ExecStart for testing a rollback binary (creates drop-in): edit to point to /opt/.../releases/v1.2.3/bin/myservice then systemctl daemon-reload && systemctl restart myservice.service
• Use PersistentJournal: ensure /etc/systemd/journald.conf has Storage=persistent so logs survive reboots

Automation ideas to make future rollbacks safer

• Blue/green or canary deployments (traffic-shift) so bad releases affect few hosts; automated rollback if health checks fail
• Atomic releases with symlinked /opt/myservice/current and immutable release directories; rollback = symlink swap + service restart
• Automated runbook scripts/ansible playbooks that:
  • fetch last known-good release, stop service, swap symlink, start service, collect logs, and alert owner
  • example: ansible-playbook rollback.yml --extra-vars "target_release=v1.2.3"
• Health-check-driven orchestration: Kubernetes liveness/readiness or a deployment controller that auto-reverts on mass failures
• Automated preservation: on failure, a prehook archives journalctl, binaries, and metrics to central storage
• Add feature flags and gradual rollout with automated abort thresholds
• CI/CD: store every build artifact in an immutable artifact registry with signed tags and quick download URLs for rollbacks
• Test rollback procedure regularly in staging and runbooks in game days

Why this works

• Atomic symlink swap + systemd restart minimizes downtime and is reversible.
• Collecting journals/metrics before changing state preserves forensic data.
• Automation and canary patterns reduce blast radius and allow safe, repeatable rollbacks.

Debian/Ubuntu (apt front-end, dpkg low-level) vs RHEL/CentOS/Fedora (dnf/yum front-end, rpm low-level)

• apt/dpkg: apt handles repos, dependency resolution, higher-level features (apt-get, apt). dpkg installs individual .deb files, no dependency resolution.
• dnf/yum + rpm: dnf (modern) and yum (legacy) handle repos and deps; rpm manages local .rpm files and the RPM DB.
• dnf has better dependency solving, parallel downloads; apt has mature package pinning and apt-key/apt-secure mechanisms.

Common commands (use sudo):

# Install package
apt update && apt install -y package-name
# or
dnf install -y package-name      # or yum install -y package-name

# List upgradable packages
apt list --upgradable
dnf check-update                  # or yum check-update

# Find which package owns a file
dpkg -S /usr/sbin/sshd
rpm -qf /usr/sbin/sshd

Recover from locked DBs:

# dpkg locked (remove stale lock then reconfigure)
sudo rm -f /var/lib/dpkg/lock-frontend /var/lib/dpkg/lock
sudo dpkg --configure -a
sudo apt-get -f install

# rpm DB corruption/locked
sudo rm -f /var/lib/rpm/__db*
sudo rpm --rebuilddb
sudo dnf makecache

Production unattended security updates considerations:

• Prefer staged testing: apply security updates to canaries before fleetwide rollouts.
• Use unattended-upgrades (Debian) or dnf-automatic (RHEL) but limit to security-only, schedule off-peak, and enable automatic reboots only when safe.
• Integrate with orchestration (Ansible, Salt) to coordinate rolling updates, health checks, and automated rollback.
• Monitor after updates (alerts, synthetic tests), maintain changelogs, and pin critical packages where required.