Email gateway in front of Exchange Server: connectors, InternalSMTPServers and loops

FortiMail, Barracuda, Mimecast, Proofpoint, Cisco, Sophos or Trend Micro: whichever gateway filters your mail, the Exchange side comes down to the same few settings. Get them wrong and you get spoofed "internal" mail, a hidden open relay, misleading Sender ID results or mail loops. Here's the vendor-neutral checklist for Exchange Server SE and 2019 (same cmdlets on 2016).

How it works#

• Inbound: MX points at the gateway, which delivers to the Front End Transport service on your Mailbox servers on port 25. Exchange picks the receive connector by the connecting IP; when remote IP ranges overlap, the most specific match wins.
• Outbound: a send connector uses the gateway as its smart host, and the gateway relays based on your egress IP or SMTP authentication.
• Filtering: the anti-spam agents on Mailbox servers aren't installed by default, and Microsoft notes you typically don't need them when another product filters inbound mail.

Collect first: the addresses the gateway delivers from (vendor documentation for cloud services; the delivering interface for appliances), its outbound relay hostnames, the public IP your Exchange servers appear from, and a certificate matching your connector FQDN.

Step-by-step#

1. Create a dedicated receive connector#

New-ReceiveConnector -Name "Inbound from Gateway" -Server EX01 -TransportRole FrontendTransport -Custom `
    -Bindings 0.0.0.0:25 -RemoteIPRanges 203.0.113.0/28,198.51.100.20 `
    -PermissionGroups AnonymousUsers -AuthMechanism Tls -Fqdn mail.contoso.com -ProtocolLoggingLevel Verbose
Set-ReceiveConnector "EX01\Inbound from Gateway" -MaxInboundConnectionPerSource 100 -MaxMessageSize 50MB

Repeat on every Mailbox server the gateway delivers to. AnonymousUsers is deliberate: it's internet mail, so Exchange should treat it that way. Because a gateway funnels all inbound mail through a few IPs, per-source limits trip first, and Exchange answers 421 4.3.2 The maximum number of concurrent server connections has exceeded a per-source limit, closing transmission channel in the gateway's delivery log. Raise MaxInboundConnectionPerSource (also check MaxInboundConnectionPercentagePerSource) here only, or lower the gateway's concurrency towards Exchange, and align MaxMessageSize with the gateway's limit.

2. Don't make internet mail "Externally secured"#

Externally secured (ExternalAuthoritative with the ExchangeServers permission group) is for links protected outside Exchange, such as IPsec or a physically controlled network. Per Microsoft's permission table it grants relay to any recipient, bypass of anti-spam and size limits, acceptance of any sender address, and the flag that marks messages authenticated, which also resolves senders against your directory. For internet mail that means a forged message "from" your CEO arrives as trusted internal mail, internal-only groups accept it, and Exchange relays whatever the gateway sends. Reserve it for traffic that's really yours, such as a DLP or encryption appliance returning your outbound mail.

Get-ReceiveConnector | Where-Object { $_.AuthMechanism -match 'ExternalAuthoritative' } |
    Format-Table Identity, RemoteIPRanges, PermissionGroups -AutoSize

3. Tell Exchange which hops are yours#

The Sender ID agent uses the Received headers to find the IP that handed the message to your organization and checks it against the sender's SPF record. Behind a gateway that's the gateway's IP, so legitimate mail can fail. InternalSMTPServers tells the agents which addresses to skip; Microsoft includes it as a step when enabling anti-spam on Mailbox servers.

Set-TransportConfig -InternalSMTPServers @{Add="203.0.113.0/28","198.51.100.20","10.0.1.11","10.0.1.12"}
Get-TransportConfig | Format-List InternalSMTPServers

It's organization-wide, and it relies on the gateway adding a Received header with the sender's real IP, so leave that option on. Validate recipients on the gateway (directory or LDAP lookup), not with the Recipient Filter agent: Microsoft advises against configuring it on Mailbox servers, because one invalid recipient gets the whole message rejected.

4. Work with the header firewall#

Without the ms-Exch-Accept-Headers-Organization and -Forest permissions, which anonymous users don't have, Exchange removes every X-MS-Exchange-Organization-* and X-MS-Exchange-Forest-* header. So the gateway can't set the SCL by writing X-MS-Exchange-Organization-SCL, but its own verdict headers survive. Map one to an SCL (use your gateway's real header and value):

New-TransportRule -Name "Gateway spam verdict to Junk" -HeaderContainsMessageHeader "X-Gateway-Spam" `
    -HeaderContainsWords "yes" -SetSCL 6
Get-OrganizationConfig | Format-List SCLJunkThreshold

Messages with an SCL above SCLJunkThreshold go to Junk Email. Don't grant the header permissions to anonymous senders instead; internet senders could then plant trusted Exchange headers.

5. TLS in both directions#

Inbound, Exchange offers STARTTLS only with a valid certificate for the connector's FQDN (event ID 12014 says it found none). Pin the certificate, and require TLS only once the gateway uses it; otherwise it gets 451 5.7.3 Must issue a STARTTLS command first.

$cert = Get-ExchangeCertificate -Thumbprint "<thumbprint>"
Set-ReceiveConnector "EX01\Inbound from Gateway" -TlsCertificateName "<I>$($cert.Issuer)<S>$($cert.Subject)" -RequireTLS $true

Outbound TLS is opportunistic by default; enforce it on the send connector (next step). Update TlsCertificateName after each renewal and check protocol versions against your TLS hardening.

6. Route outbound mail through the gateway#

New-SendConnector -Name "Outbound via Gateway" -AddressSpaces "*" -DNSRoutingEnabled $false `
    -SmartHosts "relay1.gateway.example","relay2.gateway.example" -SourceTransportServers EX01,EX02 `
    -RequireTLS $true -TlsAuthLevel CertificateValidation
Test-NetConnection relay1.gateway.example -Port 25

Register your public egress IP (after NAT) with the gateway, or authenticate with -SmartHostAuthMechanism BasicAuthRequireTLS and -AuthenticationCredential; cloud services usually also want every sending domain registered. When the gateway refuses mail, the queue's LastError or the NDR quotes its response, so read that before changing Exchange.

7. Give every domain one final owner#

Loops start when each side thinks the other owns an address. Typical case: a domain is InternalRelay in Exchange, unknown recipients leave via the * connector, and the gateway routes them straight back, until the sender gets 554 5.4.6 Hop count exceeded - possible mail loop. Make fully hosted domains Authoritative (more on loops). Conversely, every domain the gateway delivers must be an accepted domain, or Exchange refuses it with 550 5.7.54 SMTP; Unable to relay recipient in non-accepted domain.

Get-AcceptedDomain | Format-Table DomainName, DomainType, Default -AutoSize

Verify#

$log = Join-Path $env:ExchangeInstallPath "TransportRoles\Logs\FrontEnd\ProtocolLog\SmtpReceive"
Get-ChildItem $log -Filter "RECV*.LOG" | Sort-Object LastWriteTime | Select-Object -Last 1 |
    Select-String -Pattern "Inbound from Gateway" | Select-Object -Last 10
Get-TransportService | Get-Queue | Where-Object { $_.DeliveryType -eq 'SmartHostConnectorDelivery' } |
    Format-Table Identity, Status, MessageCount, LastError -AutoSize

• Gateway sessions show your connector name, a gateway IP and STARTTLS. Search for Default Frontend too: public IPs that aren't the gateway's mean senders can still bypass it.
• A test message from outside shows X-MS-Exchange-Organization-AuthAs: Anonymous (not Internal) and the gateway's Received header with the original sender's IP.
• Outbound, message tracking shows a SEND through Outbound via Gateway, and the smart host queue stays empty.

Tips & gotchas#

• Keep one list of gateway addresses and update the connector, InternalSMTPServers and the firewall together; cloud services add ranges over time.
• Without the Recipient Filter agent, a Mailbox server accepts unknown recipients and sends the NDR later, so addresses that slip past the gateway become backscatter. Keep the gateway's recipient data in sync with Active Directory.
• Gateway in front of Exchange Online instead? The counterpart there is Enhanced Filtering for Connectors; see hybrid mail flow connectors.
• Vendor guides: FortiMail, Barracuda, Mimecast, Proofpoint, Cisco and the rejection codes quick reference.

References#

• Receive connectors in Exchange Server (Microsoft Learn)
• Enable antispam functionality on Mailbox servers (Microsoft Learn)