Event ID 4999 on Exchange Server: reading MSExchange Common Watson crash reports

Event ID 4999 from MSExchange Common is Exchange saying that one of its processes hit an unhandled exception or an unexpected condition, and that a Watson (Windows Error Reporting) report is being prepared. It looks cryptic, but it names the process, the component, the method and the exception, which is enough to decide whether you're looking at noise, a known bug fixed in a later update, or third-party code crashing Exchange. In this post I'll show how to read it and what to do with each answer.

Symptoms#

Event ID 4999 (MSExchange Common) errors like this one, taken from a Microsoft support article about Exchange 2019 CU8, appear in the Application log:

Log Name:      Application
Source:        MSExchange Common
Event ID:      4999
Task Category: General
Level:         Error
Description:   Watson report about to be sent for process id: 3292, with parameters: E12IIS, c-RTL-AMD64, 15.02.0659.004, w3wp#MSExchangePowerShellFrontEndAppPool, M.Exchange.Security, M.E.S.A.F.CacheReader.AddEntry, M.E.C.TimeoutCache.DuplicateKeyException, 28e2-dumptidset, 15.02.0659.004. ErrorReportingEnabled: True

On recent builds the description continues with fields such as exStacktrace= that carry the stack trace. Around the same time you may also see:

• ASP.NET event 1325: An unhandled exception occurred and the process was terminated. with an Application ID such as /LM/W3SVC/2/ROOT/EWS.
• .NET Runtime event 1026: The process was terminated due to an unhandled exception., or Application Error event 1000 for the same process.
• For transport crashes: The following messages were loaded at startup before Transport crashed.
• Users seeing brief OWA, ActiveSync or EWS failures and Outlook reconnecting, mail pausing while transport restarts, or no visible impact at all.

Why it happens#

When Exchange code throws an exception nothing catches, Exchange writes event 4999 with the details Watson uses to group identical failures. If it was fatal, the process ends, but IIS starts a new worker process for the next request and the transport service starts a new EdgeTransport.exe, so the event may be the only trace. Some 4999s describe non-fatal conditions, and Microsoft has documented at least one that is safe to ignore. The usual root causes:

• A product bug in your build, fixed in a later cumulative or security update.
• An incomplete or non-elevated update that left binaries and configuration out of step.
• Third-party code inside an Exchange process, such as transport agents, or file-level antivirus scanning Exchange processes and folders.
• The environment: missing permissions (System.UnauthorizedAccessException), memory pressure (System.OutOfMemoryException), or one message or item that triggers the same bug every time.

How to fix it#

1. Group the events#

$since = (Get-Date).AddDays(-7)
Get-WinEvent -FilterHashtable @{LogName='Application'; ProviderName='MSExchange Common'; Id=4999; StartTime=$since} -ErrorAction SilentlyContinue |
  ForEach-Object {
    $p = ($_.Message -split 'with parameters: ')[1] -split ', '
    [pscustomobject]@{ Process = $p[3]; Method = $p[5]; Exception = $p[6]; Hash = $p[7] }
  } |
  Group-Object Process, Exception, Method, Hash | Sort-Object Count -Descending |
  Select-Object -First 15 Count, Name | Format-Table -AutoSize -Wrap

Work on the top line first. A handful of identical reports a week is a different problem from hundreds an hour.

2. Decode the parameters#

ErrorReportingEnabled only tells you whether the server is set to send reports to Microsoft; it doesn't change the diagnosis.

3. Check whether the process actually crashed#

$e = Get-WinEvent -FilterHashtable @{LogName='Application'; ProviderName='MSExchange Common'; Id=4999} -MaxEvents 1
Get-WinEvent -FilterHashtable @{LogName='Application'; StartTime=$e.TimeCreated.AddMinutes(-2); EndTime=$e.TimeCreated.AddMinutes(2)} |
  Where-Object { $_.Id -in 1000, 1026, 1325 } | Format-Table TimeCreated, ProviderName, Id -AutoSize
Get-CimInstance Win32_Process -Filter "Name='w3wp.exe'" |
  Select-Object ProcessId, CreationDate, @{n='AppPool';e={ if ($_.CommandLine -match '-ap "([^"]+)"') { $Matches[1] } }}
Get-Process EdgeTransport | Select-Object Id, StartTime

An exit event, or a worker or EdgeTransport process that started at the time of the 4999, confirms a crash and restart. No exit, no restart and no user impact means a non-fatal report: track it, but it isn't an outage.

4. Identify the exact build#

Get-ExchangeServer | Format-Table Name, Edition, AdminDisplayVersion
Get-Command Exsetup.exe | ForEach-Object { $_.FileVersionInfo }

AdminDisplayVersion shows the cumulative update but not the security or hotfix update on top; the Exsetup.exe file version does. Compare it with Microsoft's build numbers page, or let Health Checker do it.

5. Match it to a known issue#

Search Microsoft Learn and Microsoft Support for "event ID 4999" together with the process and exception, and read the known issues in the Exchange Team Blog post for your update. Two documented examples show the pattern:

If a later update fixes your fingerprint, plan it with the CU pre-flight checklist.

6. Rule out third-party code#

Get-TransportAgent | Format-Table Identity, Enabled, Priority
Get-TransportAgent -TransportService FrontEnd | Format-Table Identity, Enabled, Priority
Disable-TransportAgent -Identity "Contoso Filter Agent"   # test in a maintenance window
Restart-Service MSExchangeTransport

For a front-end agent, add -TransportService FrontEnd and restart MSExchangeFrontEndTransport instead. If the crashes stop with the agent disabled, take the fingerprint to the vendor. Also compare your antivirus configuration with Microsoft's exclusion list. If transport keeps crashing on the same message, see poison message queue.

7. Repair a bad update#

If the reports started right after an update, check that it completed and rerun it from an elevated prompt; see installing Exchange security updates. If nothing matches, open a Microsoft support case with the grouped list, a full sample event and your exact build.

Verify the fix#

• Run step 1 again with $since set to the time of your change: the fingerprint is gone over a full business cycle.
• No new 1000, 1026 or 1325 events, and the affected worker or EdgeTransport keeps its original start time.
• Test-ServiceHealth shows RequiredServicesRunning : True and users confirm the affected client works.

Key takeaways#

• Read 4999 as process, method and exception; the hash is your fingerprint.
• Many crash bugs are fixed in later updates, and Exchange Server SE is only supported while you keep it current.
• Keep third-party code out of Exchange processes where you can, and install updates elevated.
• Keep a weekly grouped report so new fingerprints stand out from known, documented ones.

References#

• Exchange Server build numbers and release dates (Microsoft Learn)
• Windows antivirus software on Exchange servers (Microsoft Learn)