Third-party transport agent isolation checklist

Symptoms to confirm

• Mail through one server slows down or stops, and Get-Queue shows its Submission queue growing.
• Application log warnings from MSExchange Extensibility, event ID 1050, naming an agent and an event:

The execution time of agent '<agent name>' exceeded <n> milliseconds while handling event '<event>'
... However, Transport will continue processing this message.

ScanMail example (Trend Micro):
The execution time of agent 'ScanMail Routing Agent' exceeded X milliseconds while handling event 'OnResolvedMessage'
for message with InternetMessageId: 'Not Available'.

• Crashes: event 1000 shows Faulting application name: EdgeTransport.exe ... Faulting module name: <third-party agent DLL>, often with event 4999 and, in the System log, The Microsoft Exchange Transport service terminated unexpectedly.
• Back pressure (452 4.3.1 Insufficient system resources) as EdgeTransport.exe memory grows.
• The Transport service won't start after a product was removed, with Event ID 1052 (MSExchange Extensibility) or Event ID 16023 (MSExchange Extensibility) in the Application log; ESET documents both for agents left registered after its product was uninstalled.
• It began right after an Exchange cumulative update (CU), security update (SU) or vendor update.

Likely causes

A transport agent is a .NET assembly built with the Exchange Transport Agents SDK and registered in agents.config (%ExchangeInstallPath%TransportRoles\Shared) on each server.

• A slow agent holds every message at its event. Event 1050 means an agent spent unusually long on one event; transport carries on, but a backlog builds. Trend Micro documents one cause for its ScanMail Routing Agent: Detect new spam sources listing domains that aren't yours as organizational MX records.
• An unhandled exception in agent code can bring down EdgeTransport.exe; a message that keeps crashing it lands in the poison queue.
• Memory-hungry scanning inflates the process until back pressure refuses mail.
• Version drift: vendors support specific Exchange builds. A CU or SU replaces Exchange's files underneath the agent, and a version the vendor hasn't released for that build can fail straight after patching.

Checklist

1. 1

   List the agents on every transport service

   powershell

   # Transport service (the default, -TransportService Hub)
   Get-TransportAgent | Format-Table Identity, Enabled, Priority -AutoSize
   
   # Front End Transport service
   Get-TransportAgent -TransportService FrontEnd | Format-Table Identity, Enabled, Priority -AutoSize
   
   # Agents that handled mail since the last service start
   Get-TransportPipeline | Format-List

   Run it on each server. The Microsoft agents you'll see are the Transport Rule, DLP Policy, Retention Policy, Supervisory Review, Malware, Text Messaging and System Probe Drop agents, plus the anti-spam agents if you ran Install-AntiSpamAgents.ps1.

2. 2

   Tie the errors to an agent

   Event 1050 names the agent. For crashes, match the faulting module in event 1000 to an agent's DLL:

   powershell

   Get-WinEvent -FilterHashtable @{LogName='Application'; ProviderName='MSExchange Extensibility'; Id=1050} -MaxEvents 20 |
       Format-List TimeCreated, Message
   Get-WinEvent -FilterHashtable @{LogName='Application'; Id=1000} -MaxEvents 50 |
       Where-Object Message -like '*EdgeTransport.exe*' | Format-List TimeCreated, Message
   
   # The DLL behind each registered agent
   Select-String -Path (Join-Path $env:ExchangeInstallPath 'TransportRoles\Shared\agents.config') -Pattern '\.dll' |
       ForEach-Object { $_.Line.Trim() }
   
   # Agent version and exact Exchange build
   (Get-Item 'C:\Program Files\Vendor\Agent\VendorAgent.dll').VersionInfo.FileVersion
   Get-Command ExSetup.exe | ForEach-Object { $_.FileVersionInfo.ProductVersion }

3. 3

   Disable the suspect agent to prove it

   Watch out: disabling an antivirus or anti-spam agent stops that scanning. Use a maintenance window, make sure your gateway or Exchange's Malware agent covers you, and re-enable it afterwards.

   powershell

   Disable-TransportAgent -Identity "Vendor Routing Agent"
   Restart-Service MSExchangeTransport
   
   # An agent in the Front End Transport service
   Disable-TransportAgent -Identity "Vendor Receive Agent" -TransportService FrontEnd
   Restart-Service MSExchangeFrontEndTransport

   If a product registers several agents, disable them all, then re-enable one at a time. Follow the vendor's procedure where one exists.

4. 4

   Fix the cause

   • Update the product to a version the vendor supports on your exact Exchange build, then Enable-TransportAgent -Identity "Vendor Routing Agent" and restart the service.
   • Correct the slow setting. For ScanMail's event 1050, Trend Micro's fix is at Spam Prevention › Content Scanning › New Spam Sources › Detect new spam sources in the ScanMail console: list only your own MX FQDNs or gateway IP addresses.
   • Change the order only if the vendor says so. Agents run in event order; on the same event, priority 1 runs first: Set-TransportAgent -Identity "Vendor Signature Agent" -Priority 12, then restart.
   • Remove leftovers of uninstalled products with the vendor's uninstaller, or Uninstall-TransportAgent -Identity "Old Vendor Agent" and a restart.

Verify

# Submission drains
Get-Queue -Identity "$env:COMPUTERNAME\Submission" | Format-List MessageCount, Status, Velocity

# No new 1050 warnings in the last 4 hours (no output = good)
Get-WinEvent -FilterHashtable @{LogName='Application'; ProviderName='MSExchange Extensibility'; Id=1050; StartTime=(Get-Date).AddHours(-4)} -ErrorAction SilentlyContinue

# The worker process stays up
(Get-Process EdgeTransport).StartTime

# A test message flows in seconds
Get-MessageTrackingLog -Sender test@contoso.com -Start (Get-Date).AddMinutes(-15) |
    Sort-Object Timestamp | Format-Table Timestamp, EventId, Source, ServerHostname

• Good looks like: each agent in the state you intended, a re-enabled agent back in Get-TransportPipeline once mail has passed, Submission near zero and the same EdgeTransport.exe start time for days.
• For agents that change messages (signatures, disclaimers), pipeline tracing snapshots a message before and after each agent: Set-TransportService EX01 -PipelineTracingEnabled $true -PipelineTracingSenderAddress test@contoso.com, send a test, read the folder in PipelineTracingPath, then switch it off and delete the snapshots, which are full message copies.

Prevent it next time

• Keep an inventory of agents per server (vendor, DLL version, priority) and refresh it after every change.
• Before each CU or SU, check the vendor's compatibility statement for that build and its advice on suspending agents during updates; afterwards, confirm the agents are enabled and send a test (see the CU pre-flight checklist).
• Run as few agents as possible. If a gateway in front of Exchange already filters mail, decide whether you need on-box scanning too; for signatures, compare cloud signature services.
• File-level antivirus needs Microsoft's exclusions, and AMSI integration scans HTTP requests, not SMTP mail; Get-TransportAgent controls neither.