Symantec DLP Network Prevent for Email with Exchange Server: modes, loops and queue errors

Symantec Data Loss Prevention's Network Prevent for Email server inspects outbound mail inline and blocks or tags messages that break policy. Behind Exchange Server, most failures come from one choice, reflecting versus forwarding mode, plus ports, relay and outages. Here's how to spot each one from the Exchange side and fix it.

Symptoms#

Typical errors after you point a send connector at the detection server:

554 5.4.6 Hop count exceeded - possible mail loop

441 4.4.1 Error encountered while communicating with primary target IP address: "Failed to connect. Winsock error code: 10060, Win32 error code: 10060."
Attempted failover to alternate host, but that did not succeed. Either there are no alternate hosts,
or delivery failed to all alternate hosts. The last endpoint attempted was 10.0.5.21:10025

550 4.4.7 QUEUE.Expired; message expired

• The hop count NDR means messages are circling between Exchange and the detection server.
• 441 4.4.1 in a Retry queue means Exchange can't open a session to any detection server on the connector: Winsock 10060 is a timeout, 10061 a refused connection (nothing listening on that port).
• After the expiration timeout (two days by default), held messages bounce with 550 4.4.7.
• An NDR carrying a 550 response in your own wording isn't a fault: that's a Block SMTP Message response rule at work.

Why it happens#

It's a proxy, not a mail server#

Broadcom's MTA integration guide says Network Prevent for Email doesn't store messages locally, so it isn't an MTA: it holds the inbound SMTP transaction open until the outbound one closes and, in forwarding mode, relays the next hop's responses back. So Exchange owns each message until the next hop accepts it. Outages leave mail in Exchange's queue; a policy block returns a 550 response, and Exchange writes the NDR. Symantec also says to place it only in your outbound SMTP stream.

Reflecting mode needs routing Exchange doesn't have#

In reflecting mode the detection server sends each message back to the MTA it came from. Symantec requires that MTA to route each message for inspection only once and to send returned messages on to their recipients, deciding by the port the message returns on (10026 by default), the client IP address, the HELO string or an added header. Symantec rates the header method least secure, because users can add headers themselves.

Exchange Server can't do that. It selects a send connector by size limit, enabled state, scope, the most specific address space and then aggregate cost, never by how a message arrived, and its mail flow rules have no action that picks a connector. A reflected message matches the same * connector again and loops until the hop count limit produces 554 5.4.6.

Ports and relay settings that don't match#

• RequestProcessor.ServerSocketPort (default 10025) is where the detection server listens; Exchange must send to that port.
• RequestProcessor.MTAResubmitPort (default 10026) is the port it sends to. In forwarding mode that's the next hop's listener; left at 10026 when the next hop listens only on 25, every forward fails. In reflecting mode, Broadcom warns that using one port for both settings can cause a loop.
• RequestProcessor.AllowHosts (default any) limits which hosts may connect. If you set it, list every sending Exchange server by IP address or FQDN, separated by commas without spaces, and restart the detection server for the change to apply.

How to fix it#

1. Use forwarding mode#

In the Enforce Server administration console, go to System › Servers and Detectors › Overview, open the Network Prevent for Email server and click Configure. Set Next Hop Configuration to Forward, then choose Disable MX Lookup with next-hop host names (tried in order) or Enable MX Lookup with a domain. The next hop must deliver to the internet itself, such as your outbound email gateway or a hosted email service. Leave Trial Mode selected until testing is done; deselecting it enables blocking.

2. Align the ports#

Under Server Settings, set RequestProcessor.ServerSocketPort to the port Exchange will use and RequestProcessor.MTAResubmitPort to the next hop's SMTP port. One value each: Broadcom notes that a comma-separated list stops the RequestProcessor from starting. If Exchange must deliver on port 25, remember that many Linux systems restrict ports below 1024; Broadcom's answer is to route traffic from port 25 to the unrestricted listener port (10025 by default).

3. Point Exchange at the detection servers#

New-SendConnector -Name "Outbound via Symantec DLP" -Usage Custom -AddressSpaces "SMTP:*;1" `
  -SmartHosts dlp01.contoso.com,dlp02.contoso.com -Port 10025 -DNSRoutingEnabled $false `
  -SmartHostAuthMechanism None -SourceTransportServers EX01,EX02
Get-SendConnector | Format-Table Name,AddressSpaces,SmartHosts,Port,MaxMessageSize,Enabled -AutoSize

Disable the old internet connector so it can't win the selection. More specific address spaces, such as the hybrid connector to contoso.mail.onmicrosoft.com, still bypass the DLP connector by design. And no other enabled * connector should accept larger messages: Exchange drops a connector from selection when a message exceeds its limit, so big messages would leave uninspected.

4. Let the next hop accept relay#

Allow the detection servers' IP addresses to relay through the next hop. If scanned mail must come back into Exchange, forward it to relay servers in a separate Active Directory site, each site with its own scoped connector; the layout is in routing Exchange mail through DLP gateways.

5. Plan for outages#

An outage leaves mail in Exchange's queues until it expires: fail-closed. List at least two detection servers as smart hosts so Exchange can fail over. Symantec documents letting the upstream MTA bypass an unavailable detection server, but Exchange won't switch connectors by itself; for a manual fail-open, keep a disabled bypass connector and swap:

Set-SendConnector "Outbound via Symantec DLP" -Enabled $false
Set-SendConnector "Internet direct (DLP bypass)" -Enabled $true

Mail sent during a bypass isn't inspected; log its use.

6. Add TLS if you need it#

For TLS in forwarding mode, Broadcom requires every hop in the proxy chain to support TLS and authenticate itself to the previous hop, so each mail server needs the next hop's public key certificate in its keystore. Follow Broadcom's TLS procedure for the detection server's keystore and certificate import. If the Exchange connector uses -TlsAuthLevel CertificateValidation, each source server must trust the detection server's certificate.

Verify the fix#

Get-TransportService | Get-MessageTrackingLog -Recipients test@fabrikam.com -Start (Get-Date).AddHours(-1) -ResultSize Unlimited |
  Sort-Object Timestamp | Format-Table Timestamp,ServerHostname,EventId,Source,ConnectorId -AutoSize
Get-TransportService | Get-Queue | Where-Object { $_.DeliveryType -eq 'SmartHostConnectorDelivery' } |
  Format-Table Identity,Status,MessageCount,NextHopDomain,LastError -AutoSize

• Good looks like one SEND through Outbound via Symantec DLP per message, no repeating RECEIVE and SEND pairs, and no LastError on the queue.
• The delivered message carries the header set in RequestProcessor.DefaultPassHeader, unless RequestProcessor.AddDefaultHeader is false.
• Content that matches a blocking policy produces an NDR with your 550 text and an incident in the Enforce console.
• With one detection server stopped, mail keeps flowing.

Key takeaways#

• With Exchange as the upstream MTA, use forwarding mode; reflecting needs routing by arrival port, IP or header, which Exchange lacks.
• Match both RequestProcessor ports to the real listeners.
• An outage means queued mail, not lost mail: monitor the queue and decide in advance whether you'll ever bypass.
• Related guides: hop count NDRs, 441 4.4.1 queues and SMTP protocol logging.

References#

• Connector selection in external message routing (Microsoft Learn)
• Configuring Network Prevent for Email Server for Reflecting or Forwarding Mode (Broadcom Tech Docs)
• MTA Integration Guide for Network Prevent for Email, version 15.7 (Broadcom Tech Docs, PDF)