Components inactive after an update checklist

Symptoms to confirm

The Exchange Management Shell confirms it:

• Mail stops flowing through the server after the update, and queues on other servers that point to it start growing.
• Load balancer health probes fail, so the server drops out of the pool; clients that connect to it directly can't reach Outlook on the web, EWS or ActiveSync.
• The POP3 and IMAP4 services stop on their own. That's by design while their proxy components are Inactive.
• Health Checker shows the yellow Exchange Server Maintenance warning instead of "Server is not in Maintenance Mode".

Get-ServerComponentState -Identity EX01 | Where-Object State -ne 'Active' | Format-Table Component, State

Component          State
---------          -----
ServerWideOffline  Inactive
HubTransport       Inactive
FrontendTransport  Inactive
...

Likely causes

Every Exchange server has a set of components, such as HubTransport, FrontendTransport, OwaProxy, EwsProxy and ActiveSyncProxy, and each one is Active, Inactive or Draining.

• Maintenance mode was never ended. The patching procedure set ServerWideOffline Inactive as Maintenance, and the closing command didn't run. Setup doesn't end your maintenance mode for you.
• The update was interrupted. When setup fails part-way, services can be left Disabled, and admins report components left Inactive as well. If the hold isn't from Maintenance, it didn't come from your maintenance procedure.
• Managed Availability stepped in. HealthApi means Exchange's built-in health engine took a protocol offline because its probes kept failing, for example after the update broke a certificate binding or a config file. Here the component state is the symptom, not the problem.

Checklist

1. 1

   Find out who holds the component

   powershell

   $c = Get-ServerComponentState -Identity EX01 -Component ServerWideOffline
   $c.LocalStates  | Format-List
   $c.RemoteStates | Format-List
   Search-AdminAuditLog -Cmdlets Set-ServerComponentState -StartDate (Get-Date).AddDays(-7) |
     Format-List RunDate, Caller, ObjectModified, CmdletParameters

   Each state entry shows a requester, its state and a timestamp. The admin audit log shows whether a person or a patching script ran the cmdlet.

2. 2

   Make sure the update really finished

   Don't bring a half-upgraded server back into service. Check the build with Get-Command ExSetup.exe | ForEach-Object { $_.FileVersionInfo }, check services with Test-ServiceHealth, and look for leftover setup Watermark or Action registry values.

3. 3

   Release the hold with the same requester

   For maintenance mode, end it the way it was started:

   powershell

   Set-ServerComponentState -Identity EX01 -Component ServerWideOffline -State Active -Requester Maintenance
   Set-ServerComponentState -Identity EX01 -Component HubTransport -State Active -Requester Maintenance

   If step 1 showed a different requester and the update is complete, release the hold with that requester, for example:

   powershell

   Set-ServerComponentState -Identity EX01 -Component ServerWideOffline -State Active -Requester Functional

   Watch out: If the requester is HealthApi, fix the cause before touching the state. Run Get-HealthReport -Identity EX01 | Where-Object AlertValue -ne 'Healthy', repair whatever the unhealthy health set points to, then check the component again. Forcing it Active while its probes still fail only hides the problem.

4. 4

   Restart what depends on the state

   Start POP3 and IMAP4 again if you use them, because they may not come back by themselves once PopProxy and ImapProxy are Active. If mail doesn't resume within a few minutes, restart MSExchangeTransport and MSExchangeFrontEndTransport, and check that the queue database drive has enough free space.

Verify

• Get-ServerComponentState -Identity EX01 | Where-Object State -ne 'Active' returns only ForwardSyncDaemon and ProvisioningRps.
• The Health Checker reports "Server is not in Maintenance Mode" and no difference between LocalStates and RemoteStates.
• Invoke-WebRequest https://ex01.contoso.com/owa/healthcheck.htm -UseBasicParsing returns status code 200, and the load balancer marks the server healthy.
• Queues on the other servers drain (Get-Queue), and a test message shows RECEIVE and DELIVER or SEND events in Get-MessageTrackingLog.

Prevent it next time

• Script maintenance mode as a pair: whatever turns it on also turns it off and then runs the not-Active check above.
• Run Health Checker after every CU and SU; its maintenance check compares LocalStates and RemoteStates for you.
• Treat a HealthApi hold as an alert to investigate, not a setting to override.
• Finish failed updates with setup rather than by flipping component states on a half-upgraded server.