System and Endpoint Hardening Questions

Direct answer

Install only the security origins with unattended-upgrade (Ubuntu's tool that installs only updates from the origins listed in its configuration), compare the installed kernel image packages before and after, set a fact kernel_upgraded from that comparison, and call ansible.builtin.reboot only when the fact is true. Make it safe for production with serial (a few hosts per batch) and max_fail_percentage: 0 (stop the rollout on the first failure), make check mode a read-only preview, and make a second run change nothing (that property is called idempotence: running the play again on an already-patched host reports no change). The playbook below was run in an Ubuntu 24.04 container to verify each of those properties, with one stand-in explained afterwards.

The playbook

---
- name: Apply Ubuntu security updates and reboot only after a kernel upgrade
  hosts: "{{ target_hosts | default('ubuntu_prod') }}"
  become: true
  serial: "{{ batch_size | default(1) }}"
  max_fail_percentage: 0
  vars:
    # Versioned kernel image packages (linux-image-6.8.0-45-generic), not the
    # linux-image-generic meta package, which keeps one name across upgrades.
    kernel_package_regex: '^linux-image-[0-9]'
  tasks:
    - name: Snapshot installed packages before the upgrade
      ansible.builtin.package_facts:
        manager: apt

    - name: Remember the kernel images (name=version) installed before the upgrade
      ansible.builtin.set_fact:
        kernels_before: "{{ ansible_facts.packages | dict2items
          | selectattr('key', 'match', kernel_package_regex)
          | map(attribute='key') | sort
          | map('extract', ansible_facts.packages) | map('first') | map(attribute='version')
          | zip(ansible_facts.packages | dict2items
              | selectattr('key', 'match', kernel_package_regex)
              | map(attribute='key') | sort) | map('reverse') | map('join', '=') | list }}"

    - name: Install the tool that restricts upgrades to the security origins
      ansible.builtin.apt:
        name: unattended-upgrades
        state: present
        update_cache: true
        cache_valid_time: 3600

    - name: Preview what would be upgraded (check mode only, changes nothing)
      ansible.builtin.command: unattended-upgrade --dry-run --verbose
      register: security_preview
      changed_when: false
      check_mode: false
      when:
        - ansible_check_mode
        - "'unattended-upgrades' in ansible_facts.packages"

    - name: Apply security updates (origins come from Allowed-Origins in 50unattended-upgrades)
      ansible.builtin.command: unattended-upgrade --verbose
      register: security_run
      changed_when: "'Packages that will be upgraded' in security_run.stdout"
      when: not ansible_check_mode

    - name: Snapshot installed packages after the upgrade
      ansible.builtin.package_facts:
        manager: apt

    - name: Record whether a new kernel image arrived
      ansible.builtin.set_fact:
        kernels_after: "{{ ansible_facts.packages | dict2items
          | selectattr('key', 'match', kernel_package_regex)
          | map(attribute='key') | sort
          | map('extract', ansible_facts.packages) | map('first') | map(attribute='version')
          | zip(ansible_facts.packages | dict2items
              | selectattr('key', 'match', kernel_package_regex)
              | map(attribute='key') | sort) | map('reverse') | map('join', '=') | list }}"

    - name: Decide
      ansible.builtin.set_fact:
        kernel_upgraded: "{{ (kernels_after | difference(kernels_before)) | length > 0 }}"

    - name: Report the decision per host
      ansible.builtin.debug:
        msg: "{{ inventory_hostname }} kernel_upgraded={{ kernel_upgraded }} before={{ kernels_before }} after={{ kernels_after }}"

    - name: Reboot only when a kernel was upgraded in this run
      ansible.builtin.reboot:
        reboot_timeout: 900
        msg: "Rebooting after kernel upgrade (Ansible)"
      when:
        - kernel_upgraded | bool
        - not ansible_check_mode
      tags: reboot

Inventory used for the test (inventory.ini):

[ubuntu_prod]
localhost ansible_connection=local

Run it as ansible-playbook -i inventory.ini security-updates.yml -e batch_size=2 against real hosts: two hosts per batch, and a failed host stops later batches.

Reading the kernel-detection chain

package_facts fills ansible_facts.packages with a dictionary: each key is a package name and each value is a list of installed versions. The long expression in kernels_before (and the identical one in kernels_after, which runs after the upgrade) turns that dictionary into a list of name=version strings for kernel images only. Read it one filter at a time, using illustrative package data (a kernel image, the linux-image-generic meta package that keeps one name across upgrades, and an unrelated package; the versions are made up). Each stage below was run with ansible-core 2.16.3 in an Ubuntu 24.04 container:

The names are listed twice in the expression because zip needs a second copy of the sorted name list to pair against. The final value on this data was ['linux-image-6.8.0-45-generic=6.8.0-45.45'].

Why name=version pairs and not just names: a new kernel normally arrives as a new package name (...-52-generic), but a rebuilt package can keep its name with a new version, and the pair catches both. difference returns the items of the first list that are not in the second. After an upgrade that installs a second kernel, with illustrative lists, ['linux-image-6.8.0-52-generic=6.8.0-52.53', 'linux-image-6.8.0-45-generic=6.8.0-45.45'] | difference(['linux-image-6.8.0-45-generic=6.8.0-45.45']) printed ['linux-image-6.8.0-52-generic=6.8.0-52.53']: one new pair, so its length is above zero and kernel_upgraded is true. If nothing new arrived the difference is empty and the length is 0.

In the playbook, changed_when: "'Packages that will be upgraded' in security_run.stdout" means Ansible reports the task as changed only when that phrase appears in the tool's output, which unattended-upgrade prints when it is about to upgrade packages.

Why each part is built this way

Security updates only. Ubuntu's unattended-upgrade reads Allowed-Origins (a setting in /etc/apt/apt.conf.d/50unattended-upgrades that lists which package sources it may install from). On the Ubuntu 24.04 image it printed Allowed origins are: o=Ubuntu,a=noble, o=Ubuntu,a=noble-security, o=UbuntuESMApps,a=noble-apps-security, o=UbuntuESM,a=noble-infra-security, and Initial blacklist: was empty, so kernels are not excluded by default. On the same fresh image, apt-get -s dist-upgrade (a simulation) listed libaudit1 coming from noble-updates, while unattended-upgrade reported No packages found that can be upgraded unattended: updates that exist only in the general updates pocket were left alone (a pocket is one of an Ubuntu release's update channels: noble is the release as shipped, noble-security the security fixes, noble-updates the other bug fixes). The alternative of apt with upgrade: dist and default_release: noble-security is weaker, because apt's target-release option raises the priority of that suite for packages that have a candidate in it; it does not exclude the other suites.

Detecting a kernel upgrade. A new kernel arrives as a new package whose name carries the version (for example linux-image-6.8.0-45-generic), while the linux-image-generic meta package keeps its name. The play records name=version pairs for packages matching ^linux-image-[0-9] before and after, and kernel_upgraded is true when the after-list contains a pair the before-list lacks. It uses package_facts, which reads the package database, instead of parsing text output.

Reboot only in that case. The reboot task has two conditions, kernel_upgraded | bool and not ansible_check_mode. ansible.builtin.reboot waits until the host answers again; it compares /proc/sys/kernel/random/boot_id (a random ID the kernel generates at each boot, so a new value proves the machine really restarted) before and after to prove that a real reboot happened, and reboot_timeout: 900 gives the host 900 seconds to come back (the default is 600).

Safe against production.

• serial limits the batch, so capacity is never fully removed. max_fail_percentage: 0 aborts the play when any host in a batch fails, so one bad host stops the rollout. When this was tried with three hosts, serial: 1 and a failure on the second host, the third host was never run.
• Check mode (--check) must not change anything. Here the apply task is skipped under check mode and a separate task runs unattended-upgrade --dry-run (marked check_mode: false so that it really runs, and changed_when: false) to preview what would happen. That preview only runs when unattended-upgrades was already installed when the play started (the package facts from before the install decide this): on a bare host a check-mode run reports the install as the pending change and skips the preview instead of failing, because the command does not exist until the real run installs it.
• The kernel decision is taken after the upgrade from facts, so it does not depend on the text output of the upgrade tool.

Verification that was actually run

The container has no kernel packages, so the printed output below shows a different package where a real run shows a kernel image. The substitution, in full:

In detail: libssl3t64 was downgraded to its release version (3.0.13-0ubuntu3), and the play was run with kernel_package_regex overridden to ^libssl3t64$, so the security update of that package plays the role of a kernel upgrade. The reboot task itself was not executed (a container cannot reboot); --skip-tags reboot was used in the run where the decision was true, and the when condition produced the skip in the other runs.

apt-get install -y ansible-core sudo python3-apt unattended-upgrades
apt-get install -y --allow-downgrades libssl3t64=3.0.13-0ubuntu3 openssl=3.0.13-0ubuntu3
ansible-playbook -i inventory.ini security-updates.yml --syntax-check
ansible-playbook -i inventory.ini security-updates.yml --check -e "kernel_package_regex='^libssl3t64\$'" \
  | grep -E "TASK \[(Preview|Apply|Report|Reboot)|Packages that|kernel_upgraded|skipping|changed=|fatal"
dpkg-query -W libssl3t64

TASK [Preview what would be upgraded (check mode only, changes nothing)] *******
TASK [Apply security updates (origins come from Allowed-Origins in 50unattended-upgrades)] ***
skipping: [localhost]
TASK [Report the decision per host] ********************************************
    "msg": "localhost kernel_upgraded=False before=['libssl3t64=3.0.13-0ubuntu3'] after=['libssl3t64=3.0.13-0ubuntu3']"
TASK [Reboot only when a kernel was upgraded in this run] **********************
skipping: [localhost]
localhost                  : ok=9    changed=0    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0   
libssl3t64:arm64	3.0.13-0ubuntu3

Check mode changed nothing: the package is still at the old version. Now the real run, then a second run:

ansible-playbook -i inventory.ini security-updates.yml -e "kernel_package_regex='^libssl3t64\$'" --skip-tags reboot \
  | grep -E "TASK \[(Preview|Apply|Report|Reboot)|kernel_upgraded|skipping|changed="
ansible-playbook -i inventory.ini security-updates.yml -e "kernel_package_regex='^libssl3t64\$'" \
  | grep -E "TASK \[(Apply|Report|Reboot)|kernel_upgraded|skipping|changed="

TASK [Preview what would be upgraded (check mode only, changes nothing)] *******
skipping: [localhost]
TASK [Apply security updates (origins come from Allowed-Origins in 50unattended-upgrades)] ***
TASK [Report the decision per host] ********************************************
    "msg": "localhost kernel_upgraded=True before=['libssl3t64=3.0.13-0ubuntu3'] after=['libssl3t64=3.0.13-0ubuntu3.16']"
localhost                  : ok=9    changed=1    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0   
skipping: [localhost]
TASK [Apply security updates (origins come from Allowed-Origins in 50unattended-upgrades)] ***
TASK [Report the decision per host] ********************************************
    "msg": "localhost kernel_upgraded=False before=['libssl3t64=3.0.13-0ubuntu3.16'] after=['libssl3t64=3.0.13-0ubuntu3.16']"
TASK [Reboot only when a kernel was upgraded in this run] **********************
skipping: [localhost]
localhost                  : ok=9    changed=0    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0   

The first run reports changed=1 and kernel_upgraded=True, so on a real host the reboot would run. The second run reports changed=0, kernel_upgraded=False and skips the reboot, which is what idempotent means here: running it again on a patched host does nothing.

Edge cases and limits

• A run that dies between the upgrade and the reboot leaves the new kernel installed but not running. The next run sees no new kernel package and does not reboot. Cover this with a separate check that compares the running kernel (ansible_facts.kernel) with the newest installed image, and alert on hosts that differ.
• The same happens if the machine's own automatic updates installed the kernel earlier in the day, before this play took its first snapshot.
• A stateless web tier can be rebooted in batches; a database primary should not. Use serial: 1, patch standbys first, and fail over deliberately before touching a primary. Drain hosts from the load balancer before the reboot, as for any rolling change.
• unattended-upgrade takes the package manager lock and can fail if another apt process is running; the task fails loudly and, with max_fail_percentage: 0, stops the rollout, which is the safe outcome.
• Complexity: one pass over the installed-package list per snapshot, so the work grows with the number of installed packages per host and linearly with the host count, divided by the batch size.

Direct answer

Treat it as an emergency change with two parallel tracks that start at once: close the gap (reduce exposure now with mitigations while the patch is prepared) and close the hole (patch every vulnerable host, internet-facing first, through compressed rings). A third track runs alongside because the flaw is being exploited: assume some hosts may already be compromised and look for evidence going back to when exploitation started, not when we heard about it. Verification is an independent check that no host still runs the vulnerable version, and I close the incident only after mitigations have been unwound deliberately and the hunt has finished.

Advisory to every host patched, step by step

1. Confirm it applies to us. Read the vendor advisory: which product, versions and configurations are affected, and whether exploitation requires a specific feature. Query the inventory and scanner for hosts that run it. This is the first place a poor inventory shows: if you cannot list affected hosts within the hour, that is the first lesson.
2. Declare an emergency change. Name an incident lead and a change approver who can approve out of normal cycle, open a channel, and agree a status cadence. Pre-agreed emergency change rules exist so this is minutes, not days.
3. Rank the exposure. Internet-facing first, then hosts reachable from partner networks or user networks, then isolated hosts. The same ranking decides patch order.
4. Apply mitigations immediately where you can, per exposure (see the next section).
5. Patch a canary in hours, not weeks. A canary is a small first group of hosts, like the canary in the mine, that gets the change before everyone else so a problem shows up on a few machines. Apply the vendor patch to a handful of hosts that represent each affected role, check health, and use that evidence to authorise the wider push. An emergency changes the soak time, not the existence of a test.
6. Roll out in compressed rings (groups of hosts patched one group after another), internet-facing hosts first even if they are in the later ring. Keep the ability to halt.
7. Hunt for compromise. Search logs and endpoint telemetry for the exploit's known indicators from the date exploitation began, using the vendor and public reporting, up to now. Patching closes the door but does not evict someone already inside. Indicators of compromise (IOCs) are observable traces of an attack, such as file hashes, IP addresses, domain names or process names; the vendor and public reporting publish them. Beyond the published list, look for typical evidence, and know where each kind lives:

8. Verify. Re-scan with an authenticated scan (the scanner logs in to each host and reads installed versions, instead of guessing from outside) and query installed versions directly; reconcile against the host list from step 1; chase every host that did not report. Confirm required restarts happened.
9. Unwind and learn. Remove temporary mitigations on purpose (a forgotten block rule is its own outage), record the timeline, and fix whatever slowed you (inventory, approval, rings).

What protects you in the gap

Choose by what the flaw needs: a remotely reachable network flaw gets the network restriction first; a flaw in an optional feature gets the feature disabled. I never rely on a single row: restrict reach and add detection at minimum.

Worked example

Suppose the advisory covers a web application server product, exploitation is unauthenticated over HTTPS, and the inventory shows 40 affected hosts: 6 internet-facing, 14 reachable from the corporate network, 20 internal job-runner hosts. Hour 0 to 2: confirm the list, declare the emergency change, put the 6 internet-facing hosts behind an allow-list or WAF rule. Hours 2 to 6: patch 2 canaries drawn from the 40 (one internet-facing host and one internal job-runner host) and run the application's health checks. Then in stages: the other 5 internet-facing hosts, then the 14 reachable from the corporate network, then the remaining 19 internal job-runner hosts. The count reconciles: 2 + 5 + 14 + 19 = 40. In parallel, start the hunt on the 6 exposed hosts from the first public exploitation date. If an authenticated scan afterwards finds 39 hosts at the fixed version, the one missing host is found by reconciling the scan against the step-1 list, which is why that list is kept.

Trade-offs and pitfalls

• Speed versus safety: an emergency patch that breaks the service turns a security incident into an availability incident. Canary evidence is how you justify going fast.
• Mitigations create false confidence. A WAF rule is not a patch and should carry an expiry.
• Declaring victory at "patch deployed". The hosts that did not report, and the hosts already compromised, are the risk that remains.
• Assuming the clock starts at the advisory. Exploitation may have started earlier, so the hunt window starts at the first reported exploitation.

Direct answer

After patching I check three different things, in this order: (1) the fix is installed (the right package version is on disk and matches the vendor's advisory), (2) the machine is running the fixed code (long-lived processes started before the update still use the old files, and a kernel update does nothing until a reboot), and (3) the workload is healthy (units, ports, logs and an end-to-end test of the application). Installed is not the same as running, and running is not the same as working.

Before the patch: capture something to compare against

Verification is a diff, so save the "before" state while you still can:

dpkg-query -W > /var/tmp/pre-patch-packages.txt        # Debian/Ubuntu (rpm -qa on RHEL family)
ss -tlnp        > /var/tmp/pre-patch-listeners.txt     # listening TCP sockets and their processes
systemctl --failed --no-legend > /var/tmp/pre-patch-failed.txt

If a unit was already failed before the patch, you must not blame the patch for it afterwards.

Step 1: did the package land, and is it the fixed version?

Two traps. First, compare against the fixed version named in the vendor advisory (an Ubuntu Security Notice, a Red Hat Security Advisory, a Debian Security Advisory), never against the upstream version number: distributions usually backport a fix into the version they already ship, so "OpenSSL 3.0.13" can be fixed or unfixed depending on the distribution's build suffix. Second, a security update is confirmed by the CVE (Common Vulnerabilities and Exposures identifier) appearing in the package changelog or advisory, so quote the CVE when you record the result.

How to read a version string such as 3.0.13-0ubuntu3.16. Everything before the hyphen (3.0.13) is the upstream OpenSSL release number. Everything after it (0ubuntu3.16) is Ubuntu's own revision of that package, and it is the part that changes when Ubuntu ships a fix. A backport is a fix taken from a newer upstream version and applied to the older version a distribution already ships, so 3.0.13 can be unfixed in revision .15 and fixed in .16. In the package name, t64 marks the libraries that Ubuntu 24.04 rebuilt for the 64-bit time_t change; it is part of the name to query and not a version. In the changelog header, urgency=medium is the packager's own urgency field, not a severity score from the CVE.

Example from a test container (Ubuntu 24.04), before and after apt-get install --only-upgrade libssl3t64:

$ dpkg-query -W -f='${Package} ${Version}\n' libssl3t64
libssl3t64 3.0.13-0ubuntu3.15          <- before
libssl3t64 3.0.13-0ubuntu3.16          <- after
$ zcat /usr/share/doc/libssl3t64/changelog.Debian.gz | head -5
openssl (3.0.13-0ubuntu3.16) noble-security; urgency=medium

  * SECURITY UPDATE: Excessive Memory Allocation in Relative CRLDP Processing
    - debian/patches/CVE-2026-35189.patch: Defer computation of relative CRLDP
      names in crypto/x509/v3_crld.c, crypto/x509/v3_purp.c,

Step 2: is the host running the fixed code?

A process loads a library or binary when it starts (the file is mapped, meaning attached into the process's memory). When the package manager replaces the file on disk, running processes keep the old copy open. /proc/<pid>/maps is a file the Linux kernel generates that lists every file mapped into the process with that ID. This is easy to demonstrate (here /tmp/sleep.old stands in for a daemon whose binary is replaced by an update):

cp /bin/sleep /tmp/sleep.old
/tmp/sleep.old 300 &
pid=$!
sleep 1
cp /bin/true /tmp/sleep.new && mv /tmp/sleep.new /tmp/sleep.old   # the "update" replaces the file
grep deleted /proc/$pid/maps | head -2

aaaab4eb0000-aaaab4eb5000 r-xp 00000000 00:39 5857720                    /tmp/sleep.old (deleted)
aaaab4ecf000-aaaab4ed0000 r--p 0000f000 00:39 5857720                    /tmp/sleep.old (deleted)

Reading one line, left to right: the memory address range, the permissions (r-xp is read, execute, private, the code segment), the offset into the file, the device (00:39), the inode (5857720, the file's identity on that device) and the path. The cp ... && mv pair is the "update": mv gives the name /tmp/sleep.old to a new file with a different inode, and the running process still holds the old inode, which no longer has a name, so the kernel appends (deleted). A (deleted) mapping therefore means the process is still executing the pre-update file. (The addresses and inode numbers differ on every run.) Package tooling does this scan for you. If you want one order to follow, run the per-process check first, then the kernel comparison:

• Ubuntu / Debian: needrestart -r l lists services (and the kernel check, needrestart -k) that need a restart, without restarting anything. Ubuntu also creates /var/run/reboot-required when a package that needs a reboot (a kernel, for example) was installed; on the test container above the file did not exist because nothing installed required it.
• RHEL family: dnf needs-restarting (from the dnf-plugins-core package, installed as dnf-utils) lists processes using files from packages updated after the process started. dnf needs-restarting -r prints only a reboot verdict and exits 1 if a reboot is needed, 0 if not, which makes it usable in a script. On the freshly started test container it answered "Reboot should not be necessary."
• Kernel (last, and only if a kernel package was updated): compare uname -r (the running kernel) with the newest installed one (rpm -q kernel --last, or dpkg -l 'linux-image-*'). A mismatch means the kernel fix is not active yet.

Decide per case: a changed application library means restart that service; a changed C library, systemd or kernel generally means a reboot, scheduled in the same window and not "later".

Step 3: did anything break?

Work outward from the host to the user:

1. systemctl --failed and compare to the pre-patch file. A new failed unit is the fastest signal.
2. systemctl is-active <your-service> for the services this host exists to run, then ss -tlnp against the saved listener list (a missing listener means a service did not come back).
3. journalctl -p err --since "30 min ago" for new errors after the patch time.
4. An end-to-end test from outside: the health endpoint, a login, a synthetic transaction. Local "service is active" can still hide a broken dependency such as a TLS library mismatch.
5. Dashboards against the prior window: error rate, latency, restarts.

On a pool of servers, patch one node, run steps 1 to 5, watch it through a representative stretch of traffic, then proceed in batches. The first node is the experiment.

Rollback or roll forward

If step 3 fails, first decide whether the patch caused it (did the pre-patch snapshot show the same fault?). Then pick the smallest reversal: restart the dependent service, dnf history undo <id> (RHEL family) or reinstall the previous package version from cache or repository (Debian family), or restore the VM snapshot taken before the window. For a security fix that is being actively exploited, rolling back reopens the hole, so a compensating control (a firewall rule, a disabled feature) is often the better temporary move.

Pitfalls

• Declaring success from "the package manager exited 0" (installed, not running).
• Restarting only the service you expected to be affected while an updated shared library is still mapped by a dozen other processes.
• Comparing to upstream version numbers instead of the advisory's fixed build.
• Rebooting the whole pool at once because the last step was "reboot".

Direct answer

Make the baseline a single versioned source of truth, check it at three points (image build, provisioning, and continuously at runtime), and decide per setting between report-only and auto-fix with a rule: auto-fix only when the correction is deterministic (the same input always gives the same result, with no per-host judgement), idempotent, reversible and cannot take a service or an administrator out; report and ticket everything else. Keep the scan results as dated, tamper-evident evidence (any later edit would be visible) so an auditor can see the history, not only today's state.

Design: three checkpoints and evidence

1. Image build: scan every image before it is published and fail the build on violations. On Linux, OpenSCAP (an open-source compliance scanner) can evaluate a profile and write results and an HTML report with oscap xccdf eval --profile <profile> --results-arf results.xml --report report.html <datastream>. A profile is a named subset of rules, such as a CIS level; a datastream is the single file that bundles the rules and the checks that test them; XCCDF is the XML format those rules are written in and results are reported in; on Windows, build from a hardened image and apply the settings through Group Policy.
2. Provisioning: enforce at first boot with configuration management (Ansible, Group Policy or an endpoint management tool) so no host starts life drifted.
3. Runtime: re-check continuously (for example daily) in detect mode. Ansible can run read-only with --check (and --diff), where modules that support check mode report the changes they would make. Note the documented limits: modules without check-mode support report nothing, and tasks whose conditions depend on registered results from earlier tasks produce no output in check mode.
4. Evidence: store each scan result (the XML results file or the exported report) with the host, time, baseline version and operator, in write-once storage (storage that accepts new files but refuses edits and deletions, for example object storage with a retention lock). This is what an auditor asks for.

Deciding per setting: report only or fix automatically

Rules behind the table:

• Fix automatically when the setting is a pure tightening that no workload legitimately needs and the fix is idempotent (running it twice changes nothing).
• Report only when it can interrupt service or lock people out, needs a reboot, varies by host role, or when the same setting is repeatedly reverted by a person or program. Repeated reverts mean something is fighting your baseline, so raise an alert instead of fixing again.
• Every report-only drift gets an owner and a due date, or a time-limited exception with an expiry.
• Examples of fixes that are not deterministic, so they stay report-only: "set rp_filter to whatever suits this host's routing" (the right value differs per host), "remove local administrators who look unfamiliar" (who is legitimate is a human judgement), and any change that only takes effect after a reboot.

How the pieces fit: the single versioned baseline is what everything is compared against; the three checkpoints decide when you look; the table and rules decide what happens on a mismatch. OpenSCAP and the registry script below are the detectors for Linux and Windows hosts, and Ansible check mode and Group Policy are how you detect or enforce through the tool a fleet is already managed with.

Worked example: Linux check that fixes some settings and reports others

This script reads a table of settings. Rows marked fix are rewritten when they drift. Rows marked report are only flagged. It takes a root directory so it can be tested safely.

#!/usr/bin/env bash
# Report-or-fix drift check for key/value settings in config files.
# Usage: drift-check.sh ROOT        (ROOT is / on a real host)
set -euo pipefail
root=${1:-/}
# file | key | expected value | mode (report|fix)
settings=(
  "etc/ssh/sshd_config|PermitRootLogin|no|fix"
  "etc/ssh/sshd_config|X11Forwarding|no|fix"
  "etc/ssh/sshd_config|MaxAuthTries|4|report"
  "etc/sysctl.d/99-hardening.conf|net.ipv4.conf.all.rp_filter|1|report"
)
drift=0
for row in "${settings[@]}"; do
  IFS='|' read -r file key want mode <<<"$row"
  path="$root/$file"
  have=$({ sed -E 's/[[:space:]]*=[[:space:]]*/ /' "$path" 2>/dev/null || true; } |
    awk -v k="$key" '$1 == k { v = $2 } END { print v }')
  if [[ "$have" == "$want" ]]; then
    printf 'OK     %-40s %s\n' "$key" "$have"
    continue
  fi
  drift=$((drift + 1))
  printf 'DRIFT  %-40s have=%s want=%s mode=%s\n' "$key" "${have:-<unset>}" "$want" "$mode"
  if [[ "$mode" == fix ]]; then
    if [[ ! -f "$path" ]]; then
      printf 'SKIP   %-40s not fixed, %s does not exist\n' "$key" "$file"
    else
      if grep -qE "^[[:space:]]*${key}[[:space:]]" "$path"; then
        sed -i -E "s|^[[:space:]]*${key}[[:space:]].*|${key} ${want}|" "$path"
      else
        printf '%s %s\n' "$key" "$want" >>"$path"
      fi
      printf 'FIXED  %-40s now=%s\n' "$key" "$want"
    fi
  fi
done
echo "drift_count=$drift"
exit $(( drift > 0 ))

Driver and real output (run inside a Debian container with GNU sed):

rm -rf /tmp/root && mkdir -p /tmp/root/etc/ssh /tmp/root/etc/sysctl.d
printf 'PermitRootLogin yes\nMaxAuthTries 6\n' > /tmp/root/etc/ssh/sshd_config
printf 'net.ipv4.conf.all.rp_filter = 0\n' > /tmp/root/etc/sysctl.d/99-hardening.conf
bash drift-check.sh /tmp/root; echo "exit=$?"
bash drift-check.sh /tmp/root; echo "exit=$?"

DRIFT  PermitRootLogin                          have=yes want=no mode=fix
FIXED  PermitRootLogin                          now=no
DRIFT  X11Forwarding                            have=<unset> want=no mode=fix
FIXED  X11Forwarding                            now=no
DRIFT  MaxAuthTries                             have=6 want=4 mode=report
DRIFT  net.ipv4.conf.all.rp_filter              have=0 want=1 mode=report
drift_count=4
exit=1
OK     PermitRootLogin                          no
OK     X11Forwarding                            no
DRIFT  MaxAuthTries                             have=6 want=4 mode=report
DRIFT  net.ipv4.conf.all.rp_filter              have=0 want=1 mode=report
drift_count=2
exit=1

Reading the script, line by line:

• settings=( ... ) is a table with one row per setting; the four fields in each row are separated by |. IFS='|' read -r file key want mode <<<"$row" splits one row on | into four variables (IFS is the field separator read uses, -r stops backslashes being treated as escapes, and <<<"$row" feeds the row to read as its input). For the first row that gives file=etc/ssh/sshd_config key=PermitRootLogin want=no mode=fix.
• The have=$(...) line finds the current value. The { sed ... || true; } group matters: with set -euo pipefail, a missing config file would make sed fail and silently end the whole script with status 2 and no output. With the guard, a missing file simply reads as <unset> and is reported as drift (run on a root whose sshd_config already meets the baseline and which has no sysctl file, it prints DRIFT net.ipv4.conf.all.rp_filter have=<unset> want=1 mode=report and drift_count=1). The same guard does not help a fix row, because the rewrite needs a file to edit: without the [[ ! -f "$path" ]] test in the fix branch, a root with no sshd_config made grep and then the >> append fail and the whole run stop after the first DRIFT line, with no drift_count and exit status 1, which looks the same as ordinary drift. With the test, that row prints SKIP PermitRootLogin not fixed, etc/ssh/sshd_config does not exist, stays counted as drift, and the run finishes (on an empty root all four rows are drift, two of them SKIP, drift_count=4, exit 1; run in the same container). sed -E 's/[[:space:]]*=[[:space:]]*/ /' turns a key = value line into key value by replacing the first = and the spaces around it with one space (lines already written key value pass through unchanged). awk -v k="$key" '$1 == k { v = $2 } END { print v }' then looks for lines whose first word equals the key, remembers the second word, and prints the last one remembered when the file ends. Run on a small file in a Debian container:

--- after sed:
PermitRootLogin yes
MaxAuthTries 6
MaxAuthTries 5
# X11Forwarding yes
--- awk for MaxAuthTries:
5

The commented-out X11Forwarding line is ignored because its first word is #, and when a key appears twice the last line wins. That is how sysctl files behave; sshd keeps the first value instead, one more reason the sketch's reading of a file can differ from the effective configuration.

• [[ "$have" == "$want" ]] prints OK and moves to the next row; anything else counts as drift, increments drift and prints DRIFT. Only rows with mode fix reach the rewrite, and only when the file exists: sed -i replaces the existing line, or printf ... >> appends one if the key was missing.
• exit $(( drift > 0 )) ends the script with status 1 if any drift remains and 0 otherwise, because the arithmetic comparison evaluates to 1 or 0. Run in the same container, drift=0 exits 0 and drift=3 exits 1.

The first run fixes two settings and reports two. The second run finds the fixed ones clean (idempotent) while the report-only ones stay open until a human acts. The non-zero exit code lets a scheduler or pipeline alarm on remaining drift.

Limits of this sketch, so you do not trust it blindly: it reads files, not effective configuration. The sshd manual says that for each keyword the first obtained value is used, and that keywords after a Match line apply only to matching connections, so an appended line can be shadowed or captured by a Match block (a section of the configuration whose settings apply only to connections matching a condition, such as a particular user or source address). In production, read the effective SSH configuration with sshd -T, which prints it, rather than grepping the file.

Worked example: Windows registry-backed setting

On Windows the same decision logic applies; prefer Group Policy or endpoint management to enforce, and use a script only for what they do not cover. This reports drift for SMB signing and fixes it only when run with -Fix:

param([switch]$Fix)
$path = 'HKLM:\SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters'
$name = 'RequireSecuritySignature'
$want = 1
$have = Get-ItemPropertyValue -Path $path -Name $name -ErrorAction SilentlyContinue
if ($have -eq $want) { "OK     $name = $have"; return }
"DRIFT  $name have=$have want=$want"
if ($Fix) {
    Set-ItemProperty -Path $path -Name $name -Value $want
    "FIXED  $name now=$want"
}

Run it on a Windows host, since the registry provider is Windows-only. RequireSecuritySignature is the registry setting Microsoft names for the "always sign" SMB policy. It is deliberately report-only by default.

Trade-offs and pitfalls

• Auto-fix everything and you will eventually take down an application at 3 a.m.; report everything and drift grows faster than people close tickets. The per-setting rule is the compromise.
• Detecting drift is half the job: find out why it happened (manual change, a package script, a stopped agent) or the same drift returns.
• Measure it: drift count per host per week, mean time to close report-only items, and number of exceptions past expiry.
• A scanner finding is only evidence if the scan could authenticate and covered the host.

Direct answer

A ring is a group of servers that receives a patch at the same time, ordered so that each ring is bigger and more important than the one before, with a waiting period (soak) between rings to let problems show up. The blast radius of a bad patch (how many servers it can hurt at once) is therefore limited to the ring it is in. For 5,000 servers I would use four rings: a canary ring of 25 hosts (0.5 percent), a first ring of 250 (5 percent), a second ring of 1,250 (25 percent) and a final ring of 3,475 (the rest). I pick canaries to be representative rather than convenient, set soak times that cover the workloads' real cycles, and promote only when written health criteria pass. A patch is halted automatically when a halt criterion trips, and the halt is owned by a named person who can also decide to resume.

The rings

The sizes sum to 5,000 (25 + 250 + 1,250 + 3,475) and were computed rather than estimated. Microsoft's Windows Update client policies (page "Configure Windows Update client policies" on learn.microsoft.com, formerly called Windows Update for Business) support the same idea for Windows clients: group devices with similar deferral periods into deployment or validation groups, defer quality updates by a number of days after release so a small pool is updated first, and pause quality updates for up to 35 days from a start date if a problem appears. Windows Server only gets the quality update policies from that service. For servers managed by other tooling, the same ring structure can be built from groups in your patch tool or from per-ring package repository snapshots.

Choosing canary hosts

A canary only helps if a bad patch would hurt it the way it would hurt production. Pick canaries by coverage, not by convenience:

• At least one host per operating system and major version, per hardware or hypervisor type, per major application role, and per network segment or region that has unique configuration (a different storage driver or agent version is a reason to add a canary).
• Include hosts that run the heaviest-used software on the box (a Java stack, a database, a web tier), so a library change shows up.
• Mostly non-production, plus a few real production hosts with low blast radius, because staging often lacks production's traffic and data.
• Never both members of a redundant pair, and never the only host of a role.
• Owners must have agreed in advance and have monitoring on.

Why 25 is not the whole answer. The chance a canary ring catches a defect depends on how many hosts the defect affects. If a patch breaks 1 percent of hosts (a specific hardware or driver combination), the chance that a random 25-host ring contains at least one broken host is 1 - 0.99^25 = 22.2 percent. The reasoning: "at least one broken" is the opposite of "none broken", so its probability is 1 minus the probability that none are. Each host is fine with probability 0.99, and with independent hosts all 25 are fine with probability 0.99^25 = 0.778 (0.7778 before rounding), so at least one is broken with probability 1 - 0.778 = 0.222. A random 250-host ring catches it with probability 1 - 0.99^250 = 91.9 percent. The same calculation for a 2 percent and a 5 percent defect, printed by this script (it assumes hosts are picked at random and independently):

for p in (0.01, 0.02, 0.05):
    for k in (25, 250):
        print(f"defect on {p:.0%} of hosts, ring of {k}: P(at least one broken host) = {1 - (1 - p) ** k:.1%}")

Reading the script: the outer loop takes each defect rate p (1, 2 and 5 percent of hosts broken); the inner loop takes each ring size k (25 and 250); (1 - p) ** k is the chance that all k hosts are fine; 1 - (1 - p) ** k is the chance that at least one is broken; the f"..." string formats p as a whole percent (:.0%) and the result to one decimal place (:.1%). Six combinations give six lines:

defect on 1% of hosts, ring of 25: P(at least one broken host) = 22.2%
defect on 1% of hosts, ring of 250: P(at least one broken host) = 91.9%
defect on 2% of hosts, ring of 25: P(at least one broken host) = 39.7%
defect on 2% of hosts, ring of 250: P(at least one broken host) = 99.4%
defect on 5% of hosts, ring of 25: P(at least one broken host) = 72.3%
defect on 5% of hosts, ring of 250: P(at least one broken host) = 100.0%

That is the reason ring 1 exists and the reason canaries must be chosen to cover the rare configurations on purpose instead of hoping random selection finds them.

Soak times

Soak must be at least as long as the shortest cycle that would reveal the problem. These are example values that I would set and justify per estate:

Add time for monthly or quarterly workloads: if a billing or month-end job is the thing a patch could break, either include that system in an earlier ring and let one real run complete, or hold ring 3 for it deliberately. Patches that need a reboot are not "soaked" until the reboot has happened. Under these example values the happy-path campaign takes 24 + 48 + 48 = 120 hours of soak, which is 5 days, plus deployment time inside each ring; that is deliberately slower than an emergency track, and an emergency compresses soak (see the note at the end).

Promote or halt

Decide the criteria before the ring starts, write them down, and make them measurable. Example criteria (thresholds are a team choice and should come from your own baselines):

Halting means the orchestration stops sending the patch to later rings, and the decision is reversible. Pre-agree who may halt (the on-call engineer, without waiting for approval) and who may resume (the change owner). Prepare the rollback before the ring: a snapshot or image for virtual machines, the previous package version available in the repository, or a documented uninstall; and note which patches cannot be rolled back, because those need a longer soak and a rehearsal. After a halt, find the cause, exclude the affected configuration from the ring until fixed, and resume with the same criteria.

Worked example

A monthly operating-system patch. Day 0 evening: ring 0 (25 hosts) is patched, health stays inside baseline through the next 24 hours. Day 1 evening: ring 1 (250 hosts) is patched; on day 2, 7 of the 250 hosts show high disk latency, all of them using one storage driver version. The campaign halts. The halt fires under the service-health row of the criteria table (a sustained rise over baseline on patched hosts, all with the same storage driver), not under the install-failure row, because nothing failed to install. Seven hosts out of 250 is 2.8 percent, which is context for how large the affected group is, not the trigger: the 2 percent example threshold applies to install failures, and the service-health row halts on any patched host. Ring 2 and ring 3 never receive the patch: 1,250 + 3,475 = 4,725 hosts were protected from a problem that the 25 canaries alone did not reveal, since none of them used that driver (a case where coverage-based selection should be revised to add a host with that driver). Fix: update the driver, add a canary with that driver, re-run ring 1's affected hosts, then resume.

Failure modes and pitfalls

• Rings by convenience: all the canaries are the same VM template, so they cannot catch driver-specific or application-specific failures.
• Redundant pairs in one ring: both nodes down at once. Rule: at most one member of any redundant pair per ring.
• No halt authority: the person who sees the problem has to ask permission and the patch keeps rolling.
• Promotion by calendar instead of evidence: the soak ended, so everything proceeds, even though nobody looked.
• Skipping the ring structure for "urgent" patches. A genuine emergency compresses soak and shrinks ring 0 to a few hosts, but still keeps a canary and still keeps the halt.