System and Endpoint Hardening Questions

Direct answer

I would run this as a measured programme, not a one-off hardening project. Pick one published baseline per operating system (CIS Benchmarks, the Center for Internet Security's consensus configuration guides, at Level 1 as the starting profile), measure first without changing anything, enforce a short first tier of low-breakage, high-impact settings through the same tooling that builds and manages the servers, prove compliance with automated scans whose results feed a dashboard, and handle every deliberate omission through an exception record with an owner, a compensating control and an expiry date. The baseline is a living definition in version control, so "keeping them there" is a scheduled enforcement and scan loop, not a recurring audit.

For illustration assume 600 Linux and 400 Windows servers (1,000 in total) in a mix of on-premises and cloud. The split is an assumption; the structure does not depend on it.

What the baseline is, and why Level 1 first

CIS defines a Level 1 profile as a base recommendation that can be implemented fairly promptly without extensive performance impact, and Level 2 as defense in depth for environments where security is paramount, which can have an adverse effect if implemented without due care. CIS also advises testing either level in a test environment first. For Windows Server, Microsoft publishes security baselines as Group Policy Object (GPO) backups in its Security Compliance Toolkit, together with Policy Analyzer (compares sets of GPOs against each other or against local policy) and LGPO.exe (applies local policy to machines that are not domain-joined). So the plan is: CIS Level 1 server profile per Linux distribution, and either the Microsoft baseline or the CIS Windows Server benchmark per Windows version, choosing one per OS and not mixing, because overlapping baselines conflict.

Phase 0: discovery and assessment (measure before enforcing)

You cannot enforce what you have not inventoried. Build the server list from more than one source (the configuration management database, the hypervisor and cloud account inventories, the directory) and reconcile it, because the servers nobody remembers are the ones with no baseline. Record for each server: OS and version, owner, environment, internet exposure, whether it can be rebooted, and whether a vendor supports changes to it.

Then run the benchmark in assessment-only mode everywhere. For Linux, OpenSCAP (oscap, an open-source implementation of SCAP, the Security Content Automation Protocol) with the SCAP Security Guide content (the rule definitions) evaluates a profile and writes results without changing the host. A real run in a test AlmaLinux 9 container against the CIS server Level 1 profile:

dnf -y install openscap-scanner scap-security-guide
oscap xccdf eval \
  --profile xccdf_org.ssgproject.content_profile_cis_server_l1 \
  --results /tmp/results.xml --report /tmp/report.html \
  /usr/share/xml/scap/ssg/content/ssg-almalinux9-ds.xml
echo "exit status: $?"

The run printed one block per rule: a Title, the Rule id and a Result. Three of them, copied from the run:

Title	Ensure AlmaLinux GPG Key Installed
Rule	xccdf_org.ssgproject.content_rule_ensure_almalinux_gpgkey_installed
Result	pass

Title	Implement Custom Crypto Policy Modules for CIS Benchmark
Rule	xccdf_org.ssgproject.content_rule_configure_custom_crypto_policy_cis
Result	fail

Title	Install AIDE
Rule	xccdf_org.ssgproject.content_rule_package_aide_installed
Result	notapplicable

pass means the host matches the rule, fail means it does not and needs a fix or an exception, and notapplicable means the rule does not apply to this host (the scanner decides this from conditions in the content). The --profile value is the XCCDF (the XML format that SCAP uses for benchmarks) profile id, and here it selects the CIS server Level 1 rule set out of the content file. The tally was 70 pass, 5 fail, 220 notapplicable, 0 error, with exit status 2: according to the oscap manual, 0 means every rule passed, 1 means an error occurred during evaluation, and 2 means at least one rule failed or was unknown, so a pipeline can gate on the exit status. The honest compliance figure excludes the not-applicable rules from the denominator: 70 / (70 + 5) = 93.3% of the 75 applicable rules, not 70 of 295. Report both numbers, and say which one is the headline, or the dashboard will quietly flatter itself. (CIS-CAT Pro Assessor is CIS's own tool for assessing conformance, and is the alternative where a vendor-certified assessment is wanted.)

The assessment also tells you how bad it is by item: the fleet-wide failure count per rule is your prioritisation list.

What to enforce first (tier 1)

Rank candidate settings by three questions: does it close a path attackers actually use, how likely is it to break something, and can I verify it automatically? The first tier takes settings that score well on all three:

The Windows LAPS row of tier 1 has a concrete shape. Windows LAPS is configured through a Group Policy setting under Computer Configuration, Policies, Administrative Templates, System, LAPS. Illustrative policy for the fleet, using the setting names and defaults from Microsoft's Windows LAPS documentation: backup directory Active Directory, password age 30 days, password length 14, complexity 4 (upper case, lower case, numbers and special characters). An administrator with the right to decrypt can then read a server's current password:

Get-LapsADPassword -Identity SRV-APP-01 -AsPlainText

Output shaped like Microsoft's documentation example (host name illustrative, password omitted):

ComputerName        : SRV-APP-01
Account             : Administrator
Password            : <the current 14-character password>
PasswordUpdateTime  : 4/9/2023 9:39:38 AM
ExpirationTimestamp : 4/14/2023 9:39:38 AM
Source              : EncryptedPassword
DecryptionStatus    : Success

Read it as: the managed account is Administrator, the password was last rotated at PasswordUpdateTime and will be rotated again after ExpirationTimestamp, it is stored encrypted in Active Directory (Source), and the caller was allowed to decrypt it. A different password per server is the point: a stolen password for one machine opens nothing else.

Everything else in Level 1 follows in later tiers, and Level 2 items are opt-in per workload.

How to enforce: one definition, two delivery paths

The plan rests on three pieces: the published baseline as the definition, one enforcement tool per OS family (Group Policy for domain-joined Windows, Ansible for Linux), and one scanner per OS family (OpenSCAP for Linux, Policy Analyzer and CIS-CAT for Windows). Packer, LGPO.exe, Puppet, Chef and PowerShell DSC appear in the tables as the alternatives that fit particular situations (image builds, machines outside the domain, estates that already run an agent).

Roll out in rings, so a bad setting hurts few servers: ring 0 is 20 servers (2%), ring 1 is 100 (10%), ring 2 is 300 (30%), ring 3 is the remaining 580 (58%): 20 + 100 + 300 + 580 = 1,000. Each ring waits for a clean soak (no baseline-caused incidents) before the next starts. Pick ring 0 deliberately: non-production, then low-risk production, with owners who agreed in advance.

How to prove it, and keep proving it

• Scans on a schedule, results to one store: OpenSCAP or CIS-CAT results for Linux, Policy Analyzer comparisons and CIS-CAT for Windows. Keep the raw results file as evidence, because "dashboard says green" is not evidence.
• Metrics that cannot be gamed: percentage of servers passing tier 1 (applicable rules only), count of open exceptions and how many are expired, and median days from "new failing rule detected" to "fixed or excepted".
• Continuous assessment trade-offs: a daily scan catches drift quickly but costs CPU on the servers and generates volume; weekly scans are cheap and slower to notice. Stagger scan start times so 1,000 servers are not all scanning at once, run daily for tier 1 only and weekly for the full profile.
• Remediation automation: OpenSCAP can emit remediation content from a result (oscap xccdf generate fix, with fix types including bash and ansible) and can apply it with --remediate. Treat generated fixes as a starting point to review and put in version control. Do not run --remediate across production unreviewed.
• Independent check: have someone outside the team sample 25 servers a quarter and compare the scan result with what is actually configured.

How to decide what to leave out

Leave a setting out (or defer it) only when one of these is true, and write it down:

Each exception carries an owner, the exact hosts, the reason, the compensating control and an expiry date (I would default to six months) so it forces a decision again. An exception without an expiry is a permanent hole.

Patch windows and emergency changes

Align the monthly patch window with the rings, so a baseline change and a patch do not land on the same host in the same night, which makes failures impossible to attribute. For an actively exploited vulnerability, keep a pre-approved emergency path outside the monthly cycle; an out-of-band change that touches a hardened setting creates a time-limited exception (72 hours, for example) rather than a silent edit, and the scheduled enforcement run is paused for that host until it expires.

The first 90 days

The 95% figure is a target to agree, not a measurement.

Boundary

Container images and the cluster that schedules them are hardened separately. This programme covers the host operating systems that run them, since a container runtime inherits the host's weaknesses.

Pitfalls

• Enforcing everything in Level 1 on day one because "it is only Level 1".
• Counting not-applicable rules as passes, or excluding failing servers from the denominator.
• Two baselines applied to one host (a CIS GPO plus a Microsoft one) that silently overwrite each other.
• Exceptions with no expiry, which turn the baseline into a suggestion.

Direct answer

Use Azure Update Manager (the Azure service that assesses and installs updates on Windows and Linux machines) with one maintenance configuration per patch ring, attach machines to rings by tag through dynamic scoping, and install only the Critical and Security classifications on a monthly cadence for Windows (anchored to "Patch Tuesday", the second Tuesday of the month) and a weekly security-only cadence for Linux. Use "Reboot if required" for stateless tiers and a controlled reboot for databases. Use pre and post events (small automations triggered before and after each run) to power on machines that are normally off, to snapshot or stop services beforehand, and to run health checks afterwards. For machines that are offline when the window opens, rely on three things: a power-on pre-event for machines you control, a catch-up schedule a few days later, and the 24-hour periodic assessment to show you who was missed.

Azure terms in plain words. A maintenance configuration is a saved schedule object (when, for how long, which update classifications, which reboot option) that you attach to machines. Dynamic scoping is a saved filter (subscription, resource group, location, operating system, tags) that decides which machines a maintenance configuration covers, re-evaluated each time it runs. Customer Managed Schedules is a patch-orchestration setting meaning only your schedule patches the machine. An availability set is an Azure grouping that spreads virtual machines across separate hardware, and an update domain is a slice of it that the platform restarts at one time. Azure Resource Graph is Azure's query service over resource metadata and results.

The estate is hybrid: Azure virtual machines are patched directly, and on-premises servers are patched after they are connected through Azure Arc (the service that represents non-Azure servers as Azure resources). Update Manager honors the update source already configured on each machine, so Windows machines keep using Windows Update, Microsoft Update or WSUS (Windows Server Update Services, where you approve updates yourself; Microsoft documents WSUS as deprecated, meaning no new features but still supported with security and quality updates), and Linux machines keep using their configured repositories. It installs what that source offers; it does not publish updates.

1. Baseline: what gets installed

What one ring looks like as a configuration (illustrative values)

A pre-event handler is a small piece of code (for example an Azure Function, code Azure runs on demand) subscribed through Event Grid to the pre-maintenance event. In this design it does three things: start machines tagged as normally off, take the data-tier snapshot, and call the cancellation API if either step fails.

Which numbers to design around: the 3-hour window with its 25-minute cut-off (decides when the last install can start), the "machine on at least 15 minutes before the start" rule together with the pre-event period (decide when power-on must begin), and the 24-hour assessment (drives the catch-up schedule). The 40-minute edit rule, the 20-machine one-time update limit and the 7 and 30 day retention periods are operating details to look up when you meet them.

2. Rings, schedules and maintenance windows

A maintenance configuration starts updates on all of its attached machines at the same time, so rings must be separate configurations. Dynamic scoping (selecting machines by subscription, resource group, location, operating system and tags) or the built-in Azure Policy named "Schedule recurring updates using Azure Update Manager" attaches new machines automatically, so a server built next month is patched without anyone remembering it.

The offsets use the product's own recurrence rule ("nth weekday of the month, with an offset of up to six days either way"); Tuesday plus 1 is Wednesday, plus 4 is Saturday and plus 6 is Monday.

Window length: the portal accepts 1 hour 30 minutes to 3 hours 55 minutes; the schedule needs to repeat no more often than every 6 hours. Choose 3 hours (180 minutes) for Windows. Update Manager checks the remaining time before each step: per the troubleshooting guidance, with fewer than 25 minutes left (15 for the update plus 10 reserved for the reboot) it does not scan or install, and a service pack needs 30. So 180 - 25 = 155 minutes is the latest point at which a new ordinary update can start. Updates already running are never killed, and anything not attempted is reported as "Not attempted", so a window that is too short shows up as a recurring pattern in the history instead of a silent failure. On Linux a reboot needs 15 minutes left in the window on Azure VMs.

Reboot options are Reboot if required, Never reboot and Always reboot. Two details matter. "Never reboot" leaves a machine whose updates need a restart waiting for one that nothing in the run will perform, so the planned reboot for Production B must be a deliberate step (a post-event or an operations task) and not an assumption. And some Windows registry settings (the Windows Update and restart policies) can cause a reboot even when you chose Never reboot, so set those to match.

Machines in a common availability set are not all updated at once: they are updated within update domain boundaries, and machines across multiple update domains are not updated concurrently. If you patch members of one availability set under different schedules and one window overruns, a member can fail or be skipped. Split such machines across schedules at different times and widen the window.

3. Pre and post events

Update Manager uses Azure Event Grid (a service that routes events to handlers such as a webhook or an Azure Function) for this. The documented order is: pre-event, optional cancellation, update installation, post-event.

Timing facts to design around (from the Microsoft documentation, using its own example of a 3:00 PM start): pre-events run outside the maintenance window, between about 2:30 PM and 2:50 PM, and the cancellation call must be made by 2:50 PM; post-events start as soon as installation ends and can run outside the window; and the status of the run covers update installation only, not the pre and post events, so monitor the event handler separately. If you create or edit a schedule that has a pre-event, do it at least 40 minutes before the start or that run is cancelled automatically.

4. Machines that are offline when the window opens

1. Shut-down machines cannot be patched. The machine must be on at least 15 minutes before the scheduled start. Machines may also appear disassociated from their configuration while off; that is a display issue, and the association is still there.
2. For machines you control that are off by design (test and development virtual machines), power them on from a pre-event, and start the power-on at the beginning of the pre-event period. For a 3:00 PM start the machine must be running by 2:45 PM (3:00 minus 15 minutes), while the documented pre-event period runs from about 2:30 to 2:50, so a power-on that begins at the end of the period is already too late. Power them off from the post-event.
3. For servers that are simply unreachable (a branch server with a network outage), the run misses them. The periodic assessment (every 24 hours when enabled) keeps reporting what is missing, so build two things on it: a dashboard or report of machines with pending Critical and Security updates, and a catch-up maintenance configuration a few days after the main window, scoped to the same tag. The catch-up is safe to run broadly, because every run starts with a fresh assessment, so machines already patched have nothing left to install. For individual stragglers, a one-time update can be started from the portal (up to 20 machines at a time).
4. Keep evidence outside the service. Pending-update data is retained for 7 days and installation results for 30 days in Azure Resource Graph (the tables patchassessmentresources and patchinstallationresources), so export compliance results monthly if audit evidence is needed.
5. Agent health is the other cause of silent misses: Update Manager installs its own extension on each machine the first time it runs, and Azure Arc machines must be connected. Alert on machines that have not been assessed.

Mapping the design to other tools

The same design uses vendor-neutral parts: a baseline (which classifications and exclusions are approved), rings selected by tag, a schedule per ring, hooks before and after, and an assessment that reports who was missed.

Trade-offs and what would change the choices

• Monthly Windows plus weekly Linux security splits the estate by how fast each ecosystem ships fixes. If the audit requires one cadence for both, use weekly Critical and Security for both and accept more reboots.
• Pre-events add moving parts that can fail. Keep them to jobs that really need to happen before patching, and make the failure path explicit (cancel the run, notify).
• If a patch must be validated before it reaches production, hold it at the source (WSUS approval, repository snapshot) and not by delaying a schedule, so the pilot ring still installs what production will later install.