Windows Server Administration Questions

Direct answer

Run every one of these servers on the Long-Term Servicing Channel (LTSC), the traditional channel with a fixed lifecycle, and plan retirements from the lifecycle dates (Windows Server 2016 first). WSUS (Windows Server Update Services) and Configuration Manager fit as the delivery layer: WSUS is the server that downloads, stores and approves updates for machines to fetch, and Configuration Manager's software update point uses a WSUS server underneath. WSUS is deprecated, which means it gets no new features but is still supported and still patched, so keep it healthy, put its database on SQL Server, and treat the long-term replacement as a planning item, not an emergency.

Servicing channel choice

Windows Server has two release channels. The LTSC gives a traditional lifecycle of security and quality updates. The Annual Channel (AC) gives more frequent releases aimed at containers and microservices; Microsoft's release page lists Windows Server, version 23H2 as the latest Annual Channel release and Windows Server 2025 as the current LTSC release. Ordinary infrastructure servers (file, print, IIS, domain members) belong on the LTSC, because they cannot absorb a new feature release every year.

LTSC releases follow the Fixed Lifecycle Policy: 5 years of mainstream support plus 5 years of extended support, 10 years in total. From Microsoft's release information:

Mainstream support is the first phase, in which Microsoft ships security and non-security fixes. Extended support is the second phase, which keeps delivering security updates until the end date and then the regular update stream ends, so extended support is the real deadline for patching. The channel decision for this estate is therefore about order of retirement. Windows Server 2016 reaches the end of extended support on 2027-01-12, so it is the first group to rebuild or upgrade; 2019 follows in January 2029. Windows Server 2022 moves from mainstream to extended support on 2026-10-13 and keeps receiving updates through 2031-10-14, so it is not urgent. New builds go to Windows Server 2025, the current LTSC release.

Where WSUS and Configuration Manager fit

• WSUS gets update metadata and files from Microsoft Update, lets administrators approve them, and serves them to clients. It can be chained: an upstream server talks to Microsoft Update and downstream servers take their updates from it. A downstream server is either a replica (it copies the upstream server's approvals and declines) or autonomous (it keeps its own approvals and declines); the maintenance steps below differ between the two.
• Configuration Manager (the Microsoft endpoint management product, also called ConfigMgr) needs a WSUS server as a prerequisite for its software update point (SUP), the site role that publishes updates to clients. ConfigMgr 1906 and later can automate most WSUS cleanup after each sync (the "WSUS Maintenance" options on the SUP), but you still schedule database backup and reindexing yourself.
• Status of WSUS. Microsoft's documentation says WSUS is deprecated and no longer adding new features, but still supported for production deployments and still receiving security and quality updates according to the product lifecycle. The Windows Server 2025 "no longer in development" list says existing capabilities and content remain available.
• Windows Internal Database (WID), the default database for WSUS, is also on that list: it "will be removed from Windows in a future release" and Microsoft suggests SQL Server instead. So a new or rebuilt WSUS server should use SQL Server for its database (SUSDB, the WSUS database), which also lets you schedule reindexing with SQL tools. WID is a limited, built-in edition of SQL Server that WSUS installs by default; it has no management console of its own.

What to keep an eye on

Four terms carry this section. A superseded update is one that a newer update replaces (each monthly cumulative update supersedes the last), and WSUS keeps listing the old ones, so clients that scan against the catalogue get slower as they pile up. A declined update is one an administrator has marked as not wanted: WSUS stops offering it and clients stop evaluating it. SUSDB is the SQL database in which WSUS stores update metadata, approvals and client status. Reindexing rebuilds the database's indexes (lookup structures that speed up queries), which become fragmented as rows are added and deleted, so a neglected database makes cleanup and client scans crawl.

Count superseded updates with the query from Microsoft's maintenance guide, run against SUSDB:

Select COUNT(UpdateID) from vwMinimalUpdate where IsSuperseded=1 and Declined=0

The query returns one number: how many updates are superseded but not yet declined. An illustrative result of 1,823 is above Microsoft's 1,500 threshold, so the declining step in the routine below is due; a result of 240 needs no action yet.

To find out where SUSDB lives, read the SQLServerName value under HKEY_LOCAL_MACHINE\Software\Microsoft\Update Services\Server\Setup: a value containing ##WID or ##SSEE means Windows Internal Database, while a plain name such as SQL01 or SQL01\WSUS (illustrative) means a full SQL Server.

Maintenance routine (monthly)

Microsoft's guidance is monthly maintenance. Its five steps are back up, create custom indexes, reindex, decline superseded updates and run the Server Cleanup Wizard, and it adds that SUSDB should be reindexed again after declining, which makes six steps here. Each step has a reason:

1. Back up SUSDB, so any later step can be undone.
2. Create the recommended custom indexes (one time only), because they speed up the cleanup queries.
3. Reindex, so the cleanup starts on a healthy database and does not time out.
4. Decline superseded updates, so clients stop scanning for updates nobody needs.
5. Run the Server Cleanup Wizard, which deletes what is now unused.
6. Reindex again after declining, because the deletions fragment the indexes.

The cleanup wizard (a WSUS console tool under Options) removes unused updates and revisions, clients not seen for 30 days, unneeded files and expired updates, and declines superseded updates only when they meet all of its conditions (not approved for 30 days, not needed by any client, and superseded by an approved update). The same operations are available from PowerShell:

Get-WsusServer | Invoke-WsusServerCleanup -DeclineExpiredUpdates -CleanupObsoleteUpdates -CleanupUnneededContentFiles -CompressUpdates

Hierarchy rules from Microsoft's maintenance guide: updates are added from the top down but removed from the bottom up, so cleanup starts at the bottom tier and each tier finishes before the next one starts (servers in the same tier can be done at the same time); the cleanup and reindex steps run on every WSUS server, replica or not; the decline script is needed only on autonomous servers; and sync must not run while maintenance runs, because a sync can undo work already done (the guide's own example is downstream servers re-syncing updates that were just cleaned out). Declines run in the opposite direction from cleanup, and the same guide says so in its closing section: when the decline script is used, run it from the top down, because declining is really a type of addition (a kind of approval) rather than a removal. An upstream decline reaches replica downstream servers at their next sync, while autonomous servers need their own decline run, so the worked example below declines at the upstream server first and then at each autonomous downstream server. For cleanup, which removes items, Microsoft's instruction is to start at the bottom of the hierarchy; the likely reason is that no lower tier is then left holding updates that the tier above has already removed (the guide states the order, not the reason).

Worked example, one upstream server (wsus-up) with two autonomous downstream servers (wsus-dn1, wsus-dn2), all three sync schedules paused during the window. The table covers the overnight window: the one-time custom indexes (step 2) were created earlier, and the pre-decline reindex (step 3) ran earlier that day. Because syncing is paused, declines cannot flow down by synchronization, so the decline step is run on each autonomous server, top-down:

Declines are top-down (upstream at 00:00, downstream at 00:15), cleanup is bottom-up (downstream at 01:00, upstream at 02:00), and each reindex starts after its cleanup should have finished. If the downstream servers were replicas instead, the 00:15 row would not exist: a replica takes its declines from the upstream server at the first sync after the window (06:00), and the following month's cleanup then removes them.

If the server has never been cleaned, the first run can time out. Microsoft's remedy: back up, reindex first, then run the cleanup with only "Unused updates and update revisions" selected and repeat until it completes (it may take several passes, in some cases hours or days), run each other option separately, and finish with a pass of all options. Where SUSDB is on WID you cannot schedule the reindex from SQL tools, so use Task Scheduler with sqlcmd (SQL Server's command-line query tool) against the WID pipe; on SQL Server use a maintenance plan. The scheduled action from Microsoft's guide has this form (the 110 folder depends on the installed tools version): "C:\Program Files\Microsoft SQL Server\110\Tools\Binn\SQLCMD.exe" -S \\.\pipe\Microsoft##WID\tsql\query -i C:\WSUS\SUSDBMaint.sql -o c:\WSUS\reindexout.txt. -S names the server (here the WID connection), -i is the reindex script to run and -o is the file that receives the output.

Trade-offs and pitfalls

• Do not build new tooling that assumes WSUS gains features. Keep the dependency narrow (approval, cleanup, sync health) so a later replacement is a migration, not a rewrite.
• Moving the database to SQL Server is cheap now and removes a second deprecation risk, since WID is also on the removal path.
• A cleanup that is "working" but never finishes is a sign that maintenance was skipped for months, not that the wizard is broken.
• Declining an update is reversible: right-click the declined update, choose Approve, change the approval to Not Approved, and resync the software update point to bring it back, or import it from the Microsoft Update Catalog if it is no longer in WSUS.

Direct answer

For application and file servers I would layer three protections: application-consistent image backups using the Volume Shadow Copy Service (VSS), a bare metal recovery image plus system state for a server that will not boot, and an offsite copy that the production domain cannot modify or delete. The ransomware answer is that the backups sit behind separate credentials, in a location the server's own accounts cannot overwrite. Snapshots and shadow copies on the same storage are convenience, not backup.

Two terms carry the middle layer. A bare metal recovery image is a backup of every critical volume (the operating system, boot files and system settings), so the whole server can be rebuilt onto empty or replacement disks. System state is a smaller bundle of the machine's own configuration: boot files, the registry (Windows' settings database), role and service settings, and the certificate store if the certificate services role is installed. It does not include your data files. The three layers work in sequence: the image backups give you copies of the data, bare metal recovery and system state rebuild the machine around that data, and the separate credentials and immutable copy decide whether those backups can still be trusted after an attack.

Consistent backups of running data

VSS coordinates four parts: the requester (the backup software), the writer (application-aware code that quiesces its data, such as flushing caches and completing transactions), the provider (which makes the shadow copy) and the VSS service itself. A group photo is a fair picture: the requester is the person who asks for the photo, each writer is an application tidying itself up and standing still (to quiesce is to pause and settle data into a consistent state), the provider is the camera, and the VSS service is the coordinator who counts down. Learn gives the timing limits: the application write freeze may not exceed 60 seconds, and creating the shadow copy may not exceed 10 seconds during which writes to the file system stay frozen. If either limit is exceeded the shadow copy can be aborted and the backup retried.

• Check writers before trusting a backup. vssadmin list writers shows each writer and vssadmin list providers shows providers. A writer in a failed state means that application's data will not be consistent.
• Copy versus full. wbadmin start backup by default makes a VSS copy backup, which does not update file history or truncate application logs. -vssFull does update the history and may truncate logs, so do not use it when another product also backs up those applications, because it can break that product's incremental chain. Two terms here: an incremental chain is one full backup plus a series of small backups that each hold only what changed since the previous one, and the backup software decides what changed from per-file history markers. A full VSS backup from a second tool updates those markers, so Learn warns it can break the other product's incremental or differential chain and push it into an unnecessary full backup. To truncate logs means to discard database transaction-log entries that a backup has safely captured, so the log file stops growing.
• Crash-consistent is not application-consistent. A snapshot without a writer is like pulling the plug and may need database recovery on restore. For example, a database restored from a crash-consistent copy starts up, then has to replay or undo whatever its transaction log shows was half finished, which usually works but takes time and can fail if the log and data files were captured at slightly different moments. The same database restored from an application-consistent copy opens straight away because the writer had already settled it.

Restoring a server that will not boot

On sysrecovery, -machine is required whenever -backupTarget is given, and the version identifier is in the MM/DD/YYYY-HH:MM format that wbadmin get versions prints. A backup that supports bare metal recovery is made with -allCritical, and the target cannot be one of the volumes being backed up. On a member server system state includes the registry, boot files and COM settings (the Component Object Model registrations that tell programs how to find each other's components), plus the Internet Information Services (IIS) metadirectory (IIS's configuration store) if the web server role is installed and cluster service information if the server is clustered. It does not include the per-user registry hives. -recreateDisks on a recovery deletes data on volumes that host operating system components and possibly on data volumes, so use -excludeDisks to protect data disks.

wbadmin start backup -backupTarget:\\backup-01\wsb\file-01 -allCritical -include:D: -systemState -noInheritAcl -user:svc-backup -password:<read from the secret vault> -quiet
wbadmin get versions -backupTarget:\\backup-01\wsb\file-01 -machine:file-01

The syntax follows Microsoft's wbadmin reference pages (the tool runs only on Windows). Reading the first command switch by switch:

• -backupTarget:\\backup-01\wsb\file-01 is where the backup is written, here a folder on a share.
• -allCritical adds every critical volume (those holding the operating system and its boot files), which is what makes the backup usable for bare metal recovery.
• -include:D: adds the data volume D:. The data on D: is saved by this switch, not by the others.
• -systemState adds the system state described above (registry, boot files, role and service settings).
• -noInheritAcl limits access to the credentials given plus Administrators and Backup Operators, instead of letting everyone with access to the share read the backup.
• -user and -password are the account that has write permission on the share.
• -quiet runs without prompts so a scheduler can run it.

The second command, wbadmin get versions, lists the backups on that target so you can confirm the new one exists and note its version identifier for a later restore. A backup to a share overwrites the previous backup from the same computer, and if the new backup fails you can be left without a usable one, so Learn recommends organizing backups into subfolders of the share, which needs about twice the space.

Protecting the backups

1. Separate identity. The account that writes to the repository (the -user above) can write only to its own folder and is not a domain admin, so ransomware running as an administrator on the server cannot read, change or delete older backups elsewhere in the repository.
2. Immutability. At least one copy that cannot be altered or deleted before its retention expires (object lock or write-once storage, or a vendor immutable repository).
3. Offline or isolated copy. Disconnected tape or a network-isolated target, so one compromise cannot reach every copy.
4. Restore tests. A backup nobody has restored is a hope. Test bare metal recovery on a spare machine or VM each quarter, and record the time.
5. Shadow copies are not backups. They live in a storage area on the same server, and when that area fills, the oldest shadow copies are deleted. Anyone with administrator rights, including ransomware running as an administrator, can delete them (vssadmin delete shadows exists for that purpose).

Virtual machine snapshot caveats

• A standard checkpoint in Hyper-V includes memory state, is not a full backup, and can cause consistency problems for systems that replicate data between nodes.
• A production checkpoint uses VSS in the guest to produce a data-consistent copy and takes no memory state, so applying it leaves the VM powered off.
• Checkpoint files (.avhdx) sit next to the VM's own .vhdx files, so they share its storage fate: a lost array loses both. Only an exported checkpoint, or a backup taken through the hypervisor and stored elsewhere, is a separate copy.

Worked example: sizing the restore

Assume a 2 TiB file server restored at an assumed sustained 400 MB/s (substitute your measured rate). Keep the units consistent: 2 TiB is 2 x 1,024^4 = 2,199,023,255,552 bytes, and at a decimal 400 MB/s (400,000,000 bytes per second) that is 2,199,023,255,552 / 400,000,000 = 5,497.6 seconds, about 92 minutes. If your tool reports 400 MiB/s (binary) instead, the same 2 TiB is 2,097,152 MiB, and 2,097,152 / 400 = 5,242.9 seconds, about 87 minutes. Either way this is the copy time only, before OS recovery and validation. If the recovery time objective (RTO, how long the business tolerates an outage) is 4 hours, this fits with margin. If it were 1 hour, you would need faster restore storage or a replica.

Pitfalls

• Backing up only files and finding the registry, roles and certificates gone.
• Keeping the backup repository domain-joined and writable by the accounts it protects.
• Treating a hypervisor snapshot as a backup.
• Never testing a bare metal restore until the day it matters.

Direct answer

Do not guess among the four causes: make the logs sort the 500s for you. IIS records a status, a substatus and a Windows error code for every request. Group the 500s by substatus, Windows error code, URL and hour, then line them up against recycle and crash events. Each cause leaves a different fingerprint: permissions give a steady Windows error 5 (access denied) on the same URLs; recycling gives bursts at recycle times with substatus 500.11 or 500.12; resource exhaustion gives 500.13 and rising response times before the errors; an application bug gives substatus 500.0 with Windows error 0 and an exception in the Application log or in a failed-request trace. I fix first whichever fingerprint accounts for the largest share of the 500s. If two tie, I take them in this order: permissions (smallest change, no restart), recycling settings, resource limits, application code (needs a developer).

Key terms

• Application pool and worker process: an application pool is a group of settings, and the worker process (w3wp.exe) it starts, that serve the requests for one or more IIS applications. Learn describes the pool as the isolation boundary: a problem in one pool does not affect applications in another.
• Recycling: IIS shutting a pool's worker process down and starting a fresh one, on a timer or when a limit such as memory or request count is hit, or after a configuration change.
• Substatus and Win32 status: the HTTP status is the three-digit code such as 500. The substatus is a second number that narrows it down (the 13 in 500.13). The Win32 status is the Windows error code behind the request: 0 means no Windows error, 5 means access denied.
• W3C log: IIS's text request log. A #Fields: header line names the columns, then there is one line per request.
• Rapid-fail protection: an IIS safety switch that removes every application in a pool from service when the worker process crashes too many times in a short window (5 crashes in 5 minutes by default).
• ApplicationPoolIdentity is the default virtual account a pool runs as, and loadUserProfile is the pool setting that makes IIS load that account's Windows user profile (its personal folders and registry hive) for the worker process.

Order of work

Start with Step 1, because it only reads logs and costs nothing. Go to Step 2 when Step 1 points at recycling or exhaustion, or when the burst times need to be matched to server events. Use Failed Request Tracing (Step 3) for the application-bug and unclear cases, and the live appcmd and netsh commands in Step 2 only while an incident is happening. The custom-account checks come next, and are the first thing to test when Step 1 shows Windows error 5.

Step 1: group the 500s

IIS writes W3C log files under inetpub\logs\LogFiles\W3SVC<site id>. The #Fields: header names each column, so this function reads the header instead of assuming column positions and reports, per hour, how many requests ended in 500 and which substatus and Windows error code they carried.

function Get-Iis500Summary {
    [CmdletBinding()]
    param([Parameter(Mandatory)][string]$LogDirectory)

    $total  = @{}
    $errors = [System.Collections.Generic.List[object]]::new()
    foreach ($file in Get-ChildItem -Path $LogDirectory -Filter '*.log') {
        $fields = $null
        foreach ($line in [System.IO.File]::ReadLines($file.FullName)) {
            if ($line.StartsWith('#Fields:')) { $fields = $line.Substring(8).Trim().Split(' '); continue }
            if ($line.StartsWith('#') -or -not $fields) { continue }
            $v = $line.Split(' ')
            if ($v.Count -ne $fields.Count) { continue }
            $f = @{}
            for ($i = 0; $i -lt $fields.Count; $i++) { $f[$fields[$i]] = $v[$i] }
            $hour = '{0} {1}:00' -f $f['date'], $f['time'].Substring(0, 2)
            $total[$hour] = 1 + [int]$total[$hour]
            if ($f['sc-status'] -eq '500') {
                $errors.Add([pscustomobject]@{
                    Hour = $hour; SubStatus = $f['sc-substatus']; Win32 = $f['sc-win32-status']
                    Uri = $f['cs-uri-stem']; Ms = [int]$f['time-taken'] })
            }
        }
    }
    $errors | Group-Object -Property Hour, SubStatus, Win32 | ForEach-Object {
        $first = $_.Group[0]
        [pscustomobject]@{
            Hour      = $first.Hour
            SubStatus = $first.SubStatus
            Win32     = $first.Win32
            Count     = $_.Count
            PctOfHour = [math]::Round(100 * $_.Count / $total[$first.Hour], 1)
            TopUri    = ($_.Group | Group-Object Uri | Sort-Object Count -Descending | Select-Object -First 1).Name
        }
    } | Sort-Object -Property Hour, SubStatus
}

Get-Iis500Summary -LogDirectory 'C:\inetpub\logs\LogFiles\W3SVC2' | Format-Table -AutoSize

A request line in the log, with the default columns, looks like this (illustrative values). The last four fields are sc-status, sc-substatus, sc-win32-status and time-taken in milliseconds:

#Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Referer) sc-status sc-substatus sc-win32-status time-taken
2026-10-04 02:45:10 10.0.0.5 GET /reports/export - 443 - 10.0.1.20 Mozilla/5.0 - 500 0 5 12
2026-10-04 03:41:10 10.0.0.5 GET /api/price - 443 - 10.0.1.20 Mozilla/5.0 - 500 0 0 310

Running the function over a made-up log with 40 ordinary requests in each of two hours, six requests for /reports/export that ended 500 with Win32 status 5 in the first hour and three requests for /api/price that ended 500 with Win32 status 0 in the second, printed (real output of that run):

Hour             SubStatus Win32 Count PctOfHour TopUri
----             --------- ----- ----- --------- ------
2026-10-04 02:00 0         5         6    13.000 /reports/export
2026-10-04 03:00 0         0         3     7.000 /api/price

Reading it: Count is how many requests fell in that hour with that substatus and Win32 pair, and PctOfHour is that count as a percentage of all requests logged in the same hour (6 of 46 is 13 percent, 3 of 43 is 7 percent). The first row is the permissions fingerprint (SubStatus 0 with Win32 5, one URL). The second is the application-bug fingerprint (SubStatus 0, Win32 0, one URL). Two different causes can coexist in one log, which is why the rows are grouped before anything is fixed.

How to read the rows. The substatus meanings below come from the IIS status-code reference.

Step 2: line the errors up with what the server did

Recycles are logged by default on current IIS versions (the default logEventOnRecycle flags on IIS 10 and later cover Time, Requests, Schedule, Memory, IsapiUnhealthy, OnDemand, ConfigChange and PrivateMemory). Pull the System log entries for the pool around the burst hours:

$since = (Get-Date).AddDays(-2)
Get-WinEvent -FilterHashtable @{ LogName = 'System'; StartTime = $since } |
    Where-Object { $_.Message -match 'application pool' -and $_.Message -match 'ContosoPool' } |
    Sort-Object -Property TimeCreated |
    Format-Table -Property TimeCreated, Id, Message -AutoSize -Wrap

Then look at the live process and what it is doing during an incident, from an elevated prompt:

%windir%\system32\inetsrv\appcmd.exe list wps /apppool.name:ContosoPool
%windir%\system32\inetsrv\appcmd.exe list requests /apppool.name:ContosoPool
%windir%\system32\inetsrv\appcmd.exe list apppool "ContosoPool" /text:*
netsh http show servicestate view=requestq

Output looks like this, in the formats Learn's AppCmd guide shows (illustrative values):

WP "3577" (apppool:ContosoPool)
REQUEST "fb0000008000000e" (url:GET /api/price,time:4276 msec,client:10.0.1.20)

• A worker-process ID that changes between two samples means the pool recycled or crashed between them.
• list requests shows currently executing requests with their elapsed time; several requests stuck for a long time point at exhaustion or a hung dependency.
• list apppool /text:* prints every setting, including the identity.
• The request-queue view shows whether requests are waiting in HTTP.sys because the pool cannot take them.

Recycling checks against the defaults: the periodic restart interval is 29 hours, the request-count, virtual-memory and private-memory triggers are 0 (disabled), and rapid-fail protection stops a pool after 5 crashes within 5 minutes. A burst that repeats every 29 hours matches the default timer. Anything else (a memory trigger someone set, a configuration change, an on-demand recycle) shows in the event text.

Step 3: capture the failing request itself

For the application-bug and unclear cases, enable Failed Request Tracing: add the Tracing role service under Health and Diagnostics, enable it for the site, and add a rule for status code 500 (and a second rule with a timeTaken threshold to catch slow requests before they fail). IIS writes one XML file per failed request, with a freb.xsl style sheet, to %SystemDrive%\inetpub\logs\FailedReqLogFiles\W3SVC<site id>. The trace names the module that failed and the error details, which separates a permissions failure (access denied against a path) from an exception in application code.

What is special about a custom service account

By default a pool runs as ApplicationPoolIdentity, a virtual account written IIS AppPool\<pool name>, and content is secured to that name. A custom account changes three things you must check:

1. NTFS rights. The account needs Read on the site folder and Modify only where the application writes. Grant exactly that, for example icacls "D:\Sites\Contoso" /grant "CONTOSO\svc-web:(OI)(CI)RX" and icacls "D:\Sites\Contoso\App_Data" /grant "CONTOSO\svc-web:(OI)(CI)M". In that notation (OI) and (CI) mean the entry is inherited by files and by subfolders.
2. Network access. Outbound calls to a database or share run as that account (as the computer account for the built-in identities), so a file share or SQL login that was granted to the machine name will fail for the custom account.
3. User profile. IIS does not load a Windows user profile by default, so code that writes to a profile or temp path can fail unless loadUserProfile is set to true.

A password that expires or is changed, or a locked-out account, stops the worker processes from starting, which shows up as 503 rather than intermittent 500. Microsoft lists application pools among the supported uses of a group Managed Service Account (gMSA), whose password Active Directory rotates automatically; moving the pool to a gMSA removes that failure class.

Verify the fix without fooling yourself

An intermittent error can hide for hours by chance. Whether a clean hour means anything depends on the error rate. The chance that n requests contain no error at all is (1 - p) to the power n, where p is the error rate. It is close to e to the power minus (n x p), the expected error count n x p fed into an exponential decay, which is why the expected count tells you most of what you need:

The three rows, worked out: 0.975^200 = 0.0063, so 0.6 percent. 0.999^200 = 0.819, so 82 percent. 0.999^2000 = 0.135, which the table rounds to 14 percent. The 15-hour figure: 15 x 200 = 3,000 requests, 3,000 x 0.001 = 3 expected errors, and 0.999^3000 = 0.0497, about 5 percent.

At 2.5 percent one clean hour is convincing; at 0.1 percent and 200 requests an hour a clean hour proves nothing, and you need about 15 hours (3 expected errors, about a 5 percent chance of none) before the quiet means anything. Re-run the Step 1 summary over a window sized that way, and compare the count of 500s per hour, not just whether any appeared.

Trade-offs and pitfalls

• Raising memory or request recycling limits to "hide" 500.12 bursts trades visible errors for a longer-lived leak. Fix the leak or schedule the recycle at a quiet hour.
• Adding the service account to local Administrators makes access-denied errors vanish and widens what a compromised application can do. Grant the narrow right instead.
• Do not read a stopped pool as an intermittent 500: rapid-fail protection removes every application in the pool from service, and the client sees 503.

Direct answer

Do not look at the certificate store first. Connect to the site the way a user does, read the certificate it actually presents (thumbprint, expiry, names, chain), and compare that with what the server is configured to present. If the old thumbprint is served, the binding still points at the old certificate (a renewal creates a new certificate with a new thumbprint, and nothing repoints the binding). If the new thumbprint is served but clients still warn, the problem is in the chain, the names, the trust, or a device in front of the server.

Background that explains the symptom: IIS keeps a site's binding (IP, port, host name) in its own configuration, but the HTTPS certificate for that binding is stored separately in HTTP.sys, the Windows kernel HTTP driver that does the TLS handshake. netsh http show sslcert shows that second list, each entry with a Certificate Hash and a store name. A new certificate sitting in the machine's Personal store does nothing until an entry points to its hash.

Key terms

• Thumbprint: a short hash that uniquely identifies one certificate, like a fingerprint. A renewed certificate is a new certificate, so it has a new thumbprint, even if the name is the same. (netsh calls the same value the Certificate Hash.)
• Leaf, intermediate and root certificates; chain: the leaf is the site's own certificate. A certificate authority (CA, the organisation that signs certificates) usually signs it through one or more intermediate CA certificates, and those lead up to a root CA certificate that clients already trust. Leaf, intermediates and root together are the certificate chain. The server has to send the leaf and the intermediates; the client supplies the root from its own trusted store.
• Private key and PFX: the private key is the secret half of the certificate's key pair, which the server needs in order to complete a TLS handshake. A PFX (.pfx) file is a password-protected bundle that carries the certificate and its private key, so it can be moved between machines.
• SAN (Subject Alternative Name): the list of host names a certificate is valid for. Browsers compare the name in the address bar with this list.
• Binding: the link that says "requests for this IP, port and host name use this certificate".

Step 1: see what the site is serving (from a client, not the server)

This function opens a TLS connection, sends the host name the way a browser does (SNI, Server Name Indication), and reports what came back. Test from the network path users take, because a load balancer or reverse proxy may present its own certificate.

function Get-ServedCertificate {
    [CmdletBinding()]
    param(
        [Parameter(Mandatory)][string]$ComputerName,
        [int]$Port = 443,
        [string]$ServerName = $ComputerName
    )
    $tcp = [System.Net.Sockets.TcpClient]::new()
    try {
        $tcp.Connect($ComputerName, $Port)
        # Accept any certificate so the handshake completes and we can inspect what was sent.
        $accept = [System.Net.Security.RemoteCertificateValidationCallback]{ $true }
        $ssl = [System.Net.Security.SslStream]::new($tcp.GetStream(), $false, $accept)
        $ssl.AuthenticateAsClient($ServerName)
        $cert = [System.Security.Cryptography.X509Certificates.X509Certificate2]::new($ssl.RemoteCertificate)
        $chain = [System.Security.Cryptography.X509Certificates.X509Chain]::new()
        $chainOk = $chain.Build($cert)
        $san = $cert.Extensions | Where-Object { $_.Oid.Value -eq '2.5.29.17' } |
            ForEach-Object { $_.Format($false) }
        [pscustomobject]@{
            SniName    = $ServerName
            Subject    = $cert.Subject
            Issuer     = $cert.Issuer
            NotAfter   = $cert.NotAfter.ToUniversalTime().ToString('yyyy-MM-dd HH:mm')
            Thumbprint = $cert.Thumbprint
            Names      = $san
            Protocol   = $ssl.SslProtocol
            ChainValid = $chainOk
            ChainIssue = (($chain.ChainStatus | ForEach-Object { $_.Status }) -join ',')
            ChainDepth = $chain.ChainElements.Count
        }
    }
    finally {
        if ($ssl) { $ssl.Dispose() }
        $tcp.Dispose()
    }
}

Get-ServedCertificate -ComputerName 'intranet.contoso.com' | Format-List

What the function does, block by block. The first block opens a plain TCP connection to the server. The second wraps it in a TLS stream (SslStream) and starts the handshake, sending $ServerName as the SNI value so the server picks the right certificate; the callback that returns $true accepts any certificate so the handshake completes and the function can inspect what came back instead of failing on it. The third block copies the certificate the server presented and asks Windows to build its chain (X509Chain.Build), which is where ChainValid, ChainIssue and ChainDepth come from. The fourth block pulls the SAN list out of the certificate's extension (object identifier 2.5.29.17 is the SAN extension) and puts everything into one object. The finally block closes the connection even if something above failed.

Example output for a site whose binding was never repointed (illustrative values; the field names are what the function returns):

SniName    : intranet.contoso.com
Subject    : CN=intranet.contoso.com
Issuer     : CN=Contoso Issuing CA 01, DC=contoso, DC=com
NotAfter   : 2026-09-30 23:59
Thumbprint : 3F1C9A5B7D2E4F60A1B2C3D4E5F60718293A4B5C
Names      : DNS Name=intranet.contoso.com
Protocol   : Tls12
ChainValid : False
ChainIssue : NotTimeValid
ChainDepth : 3

Reading it: the site is serving the thumbprint ending 4B5C, and NotAfter is already in the past (NotTimeValid is the chain status for an expired certificate). That matches the first row of the table below: the renewal never reached the binding. ChainDepth 3 means leaf, one intermediate and root were all found by the client machine.

If the host name resolves to several servers (a farm), run it against each node's IP address with -ServerName intranet.contoso.com, because one node that missed the renewal produces exactly "it works for some people".

How to read it:

The chain is built by the client machine running the function, so ChainIssue values such as RevocationStatusUnknown reflect what that machine could reach, not necessarily what users see.

Step 2: see what the server is configured to serve

Run on the web server, elevated:

# HTTP.sys list: every IP:port, hostname:port or central-store entry, with its Certificate Hash
netsh http show sslcert

# IIS list: the site's bindings; sslFlags 0 = IP:port certificate, 1 = SNI, 2 = central certificate store
Get-IISSiteBinding -Name 'Contoso Intranet' -Protocol https

# What is in the machine store, newest last, and whether the private key is present
Get-ChildItem -Path Cert:\LocalMachine\My |
    Sort-Object -Property NotAfter |
    Format-Table -Property Thumbprint, Subject, NotAfter, HasPrivateKey -AutoSize

Match the thumbprint from Step 1 against the Certificate Hash values in the first command and the store listing in the third. Illustrative output for the same incident (made-up values):

SSL Certificate bindings:
-------------------------

    IP:port                      : 0.0.0.0:443
    Certificate Hash             : 3f1c9a5b7d2e4f60a1b2c3d4e5f60718293a4b5c
    Application ID               : {11111111-2222-3333-4444-555555555555}
    Certificate Store Name       : My
    ...

Thumbprint                               Subject                 NotAfter              HasPrivateKey
----------                               -------                 --------              -------------
3F1C9A5B7D2E4F60A1B2C3D4E5F60718293A4B5C CN=intranet.contoso.com 9/30/2026 11:59:00 PM          True
8D2E41B7C09A63F5E1D2C3B4A5968778695A4B3C CN=intranet.contoso.com 9/30/2027 11:59:00 PM          True

Reading it: the HTTP.sys entry for 0.0.0.0:443 still carries the hash 3f1c...4b5c, the old certificate. The store already holds the renewed certificate (8D2E...4B3C, valid a year longer, with its private key present), but nothing points at it. That is cause 1 below. The Get-IISSiteBinding line is the IIS side of the same fact: its binding information has the form IP:port:hostname, and its SslFlags value says whether the binding is a plain IP and port entry, an SNI entry or a centralized-store entry.

The Step 1 table tells you which causes to test first. An old thumbprint points at cause 1, 2 or 8. A new thumbprint with PartialChain points at cause 4. A new thumbprint whose Names list lacks the host name points at cause 5.

Why a renewed certificate does not take effect, or the handshake fails

1. The binding still points at the old hash. The most common cause. netsh http show sslcert shows the old Certificate Hash on the entry that matches the site.
2. A different entry wins. There can be an IP:port entry and a host name:port entry, or another site on the same IP and port. With sslFlags 0 only one certificate can be bound to an IP and port combination; with sslFlags 1 (SNI) each host name has its own entry. Fix the entry that matches the connection, and look for the old hash in all of them.
3. The new certificate has no private key on this server (HasPrivateKey is False). This happens when the request was created on another machine or the certificate was imported without its key. The server cannot complete a handshake with it. Complete the request on the same machine, or import the PFX file that includes the key.
4. The intermediate CA certificate is missing. The server must send the chain below the root: the leaf and every intermediate. Without the intermediate, clients that do not already have it cached cannot link the leaf to a root they trust. Install the issuing CA's intermediate certificate in the Intermediate Certification Authorities store (Cert:\LocalMachine\CA).
5. The names do not match. The host name users type is not in the certificate's name list (the Names field above). Browsers check three things: the date range, the name, and that the issuer is trusted.
6. The site uses the centralized certificate store (sslFlags includes 2). There the certificate is a file named after the host name in the store's folder, so renewal means replacing that file, not the machine store.
7. Protocol mismatch. Microsoft's Windows documentation says TLS 1.0 and 1.1 are deprecated and are disabled by default starting with the 2024 Windows Server Insider Preview releases, without changing versions already in the market. So a server built on a newer release can refuse an old client with a protocol error rather than a name warning, while an older server may still accept it. Check the Schannel protocol settings on the server before assuming either way. The Protocol field shows what a modern client negotiated.
8. Something in front terminates TLS (a load balancer, reverse proxy or web application firewall) and still holds the old certificate: the server's own binding is correct but irrelevant. Compare the thumbprint served by the proxy with the one served by the server's own IP address.

Fix and verify

For cause 1, either edit the HTTPS binding in IIS Manager and choose the new certificate, or update the HTTP.sys entry. The appid is a GUID recorded with the entry to say which application registered it; copy the Application ID that netsh http show sslcert printed for the existing entry so the updated entry keeps the same owner:

netsh http update sslcert ipport=0.0.0.0:443 certhash=<NewThumbprint> appid={<ApplicationIdFromShowSslcert>} certstorename=MY

0.0.0.0 is the entry for all IPv4 addresses. Use hostnameport=intranet.contoso.com:443 instead for an SNI entry.

Verify by running Get-ServedCertificate again from a client: the thumbprint must equal the new certificate's, NotAfter must be the new date, and ChainIssue must be empty. Then repeat for every node.

Trade-offs and pitfalls

• Do not restart IIS as a first reaction. It hides the cause and does not change which certificate hash HTTP.sys holds.
• Keep the old certificate in the store until the verification passes on every node, so the rollback is a single netsh http update sslcert pointing back at the old hash.
• To stop this recurring, alert on the expiry of the thumbprint each binding actually serves (the Step 1 function from a monitoring host), not on the store contents.

Direct answer

For about 100 remote users I would build a Remote Desktop Services (RDS) deployment with five parts: session hosts (the servers where users' desktops and apps actually run), a Remote Desktop Connection Broker (tracks who is on which host and reconnects them), Remote Desktop Gateway (tunnels RDP over HTTPS port 443, so RDP itself is never exposed), Remote Desktop Web Access (the web page that lists apps and desktops) and a Remote Desktop Licensing server (issues the client access licenses, or CALs). Every infrastructure role is duplicated on a second machine, the broker's state lives in a shared SQL database (SQL Server or Azure SQL Database, a database service the two brokers both read), and the two gateway/web servers sit behind a load balancer (a device or service that spreads incoming connections across several servers and stops sending them to one that stops responding). Licensing is per-user RDS CALs, one per person, installed on a license server. Internet exposure is limited to the gateway and Web Access servers behind one load balancer on TCP 443 (plus UDP 3391 for the gateway's UDP transport), with a trusted certificate and multifactor authentication on the gateway sign-in.

Components and how each stays up

Microsoft's high-availability guidance is to duplicate each infrastructure role service on a second machine. That is eight servers plus the database (4 session hosts + 2 broker and licensing + 2 gateway and web access), which is the footprint to defend: a smaller one has a single point of failure at some layer. The session hosts, the broker, the gateway and the license server are what a working remote session needs; Web Access is the page where users pick their apps and desktops.

Sizing the session hosts (worked arithmetic)

Capacity is a measured number, not a formula. I run a pilot with the real applications and find how many concurrent users one session host carries at an acceptable response time; assume the pilot shows 35. Worst case all 100 users are connected at once:

• Hosts to carry the load: ceil(100 / 35) = 3.
• One extra host so one can fail or be patched: 3 + 1 = 4 hosts.
• With one host down, 3 hosts carry 100 users: 100 / 3 = 33.3 users each, which is under 35, so the design holds during a failure.

If the pilot measures 50 instead, the answer changes to ceil(100 / 50) = 2, plus 1 = 3 hosts. The measured per-host number is what flips the host count.

Licensing

• Each user and each device that connects to a session host running Windows Server needs an RDS CAL, issued by a Remote Desktop Licensing server.
• How many to buy: 100 users means 100 per-user CALs, not 200 because there are two license servers. Microsoft's roles guidance has both license servers associated with every RD server so issuing continues if one is down; how the purchased license codes are installed across the pair is a licensing-agreement question to settle with your Microsoft reseller before the build.
• Per-user vs per-device: remote workers use their own devices from home, so I would buy per-user CALs (100). Per-user CALs are tied to a user in Active Directory, cannot be tracked in a workgroup, and can be over-allocated, which would breach the licensing terms, so I track usage in Remote Desktop Licensing Manager. Per-device CALs fit shared devices used by several shifts.
• Version rule: the RDS CAL must be the same version as, or newer than, the session host's Windows Server version (a Windows Server 2022 CAL cannot be used on a 2025 session host), and the license server must run the same or a newer Windows Server version than the CALs it holds.
• Grace period: there is a 120 day grace period during which no license server is needed. After it ends, clients must hold a valid CAL to sign in, so the license server is built and activated first, not last.

Protecting access from the internet

1. Expose only the gateway tier, never RDP (3389) or the broker. RD Gateway and RD Web Access run on the same two servers behind the same load balancer, so TCP 443 (and UDP 3391 for the gateway) reaches both, while the brokers, session hosts and SQL database stay unreachable from the internet. Place the gateway servers in a perimeter network (a separate network zone between the internet and the internal network, sometimes called a DMZ), with TCP 443 inbound to the load balancer and rules allowing the gateway to reach the internal brokers and hosts.
2. A trusted TLS certificate on the gateway and web access servers, and a public DNS name that matches it.
3. Authorization policies: RD Gateway requires a Connection Authorization Policy (RD CAP, who may connect) and a Resource Authorization Policy (RD RAP, which internal machines they may reach). By default both are rules stored on the gateway; the CAPs can instead be held on a central Network Policy Server, while RAPs are always evaluated on the gateway itself. Scope users to a named security group and the RAP to the session hosts only.
4. Multifactor authentication: with the RD CAPs held on a central Network Policy Server (NPS, the Windows Server role that answers RADIUS authentication requests), the Microsoft Entra multifactor authentication NPS extension, installed on that NPS server and not on the gateway, adds a second factor to the gateway sign-in. The gateway sign-in supports phone call and Approve/Deny push in the Authenticator app, not SMS or typed codes, so plan enrollment accordingly. This is a layer added on top of a working gateway.
5. Operations: patch the gateway first, alert on repeated sign-in failures, and give both Web Access servers identical validation and decryption machine keys, as Microsoft's farm procedure requires. Those keys are the secrets IIS uses to sign and decrypt the tokens Web Access issues to a browser; with identical keys, a sign-in started on one Web Access server is accepted by the other when the load balancer switches the user over.

Trade-offs and pitfalls

• Recommendation: this on-premises RDS design if the applications must stay on your servers. Choose Azure Virtual Desktop instead if there is no appetite for running broker, gateway and database infrastructure, or if the team is not already staffed to patch it.
• Windows Internal Database (WID, a lightweight database engine built into Windows) is used by roles that include the Connection Broker, but a highly available broker needs a shared database server (SQL Server or Azure SQL Database). Microsoft also lists WID as being removed from Windows in a future release and advises SQL Server for these roles, so use SQL from the start.
• Co-locating roles saves servers but ties failure domains together. I accept brokers plus licensing on one pair at 100 users; I would separate them at larger scale.
• User data on the session hosts disappears with a failed host. Keep profiles and files on a highly available file share, not on the host's local disk.