Email security gateway in front of Exchange checklist

How it works

Collect first: the addresses the gateway delivers from (vendor documentation for cloud services; the delivering interface for appliances), its outbound relay hostnames, the public IP your Exchange servers appear from, and a certificate matching your connector FQDN.

• Inbound: MX points at the gateway, which delivers to the Front End Transport service on your Mailbox servers on port 25. Exchange picks the receive connector by the connecting IP; when remote IP ranges overlap, the most specific match wins.
• Outbound: a send connector uses the gateway as its smart host, and the gateway relays based on your egress IP or SMTP authentication.
• Filtering: the anti-spam agents on Mailbox servers aren't installed by default, and Microsoft notes you typically don't need them when another product filters inbound mail.

Checklist

1. 1

   Create a dedicated receive connector

   powershell

   New-ReceiveConnector -Name "Inbound from Gateway" -Server EX01 -TransportRole FrontendTransport -Custom `
       -Bindings 0.0.0.0:25 -RemoteIPRanges 203.0.113.0/28,198.51.100.20 `
       -PermissionGroups AnonymousUsers -AuthMechanism Tls -Fqdn mail.contoso.com -ProtocolLoggingLevel Verbose
   Set-ReceiveConnector "EX01\Inbound from Gateway" -MaxInboundConnectionPerSource 100 -MaxMessageSize 50MB

   Repeat on every Mailbox server the gateway delivers to. AnonymousUsers is deliberate: it's internet mail, so Exchange should treat it that way.

   Watch out: a scoped connector doesn't stop senders bypassing the gateway: Default Frontend <ServerName> still accepts anonymous mail from anywhere. Enforce "gateway only" on the firewall (TCP 25 from the gateway's ranges, plus Exchange Online if hybrid mail lands here) rather than narrowing the default connector, which devices and services may rely on.

2. 2

   Don't make internet mail "Externally secured"

   Externally secured (ExternalAuthoritative with the ExchangeServers permission group) is for links protected outside Exchange, such as IPsec or a physically controlled network.

   powershell

   Get-ReceiveConnector | Where-Object { $_.AuthMechanism -match 'ExternalAuthoritative' } |
       Format-Table Identity, RemoteIPRanges, PermissionGroups -AutoSize

3. 3

   Tell Exchange which hops are yours

   The Sender ID agent uses the Received headers to find the IP that handed the message to your organization and checks it against the sender's SPF record. Behind a gateway that's the gateway's IP, so legitimate mail can fail.

   powershell

   Set-TransportConfig -InternalSMTPServers @{Add="203.0.113.0/28","198.51.100.20","10.0.1.11","10.0.1.12"}
   Get-TransportConfig | Format-List InternalSMTPServers

4. 4

   Work with the header firewall

   Without the ms-Exch-Accept-Headers-Organization and -Forest permissions, which anonymous users don't have, Exchange removes every X-MS-Exchange-Organization-* and X-MS-Exchange-Forest-* header. So the gateway can't set the SCL by writing X-MS-Exchange-Organization-SCL, but its own verdict headers survive. Map one to an SCL (use your gateway's real header and value):

   powershell

   New-TransportRule -Name "Gateway spam verdict to Junk" -HeaderContainsMessageHeader "X-Gateway-Spam" `
       -HeaderContainsWords "yes" -SetSCL 6
   Get-OrganizationConfig | Format-List SCLJunkThreshold

5. 5

   TLS in both directions

   Inbound, Exchange offers STARTTLS only with a valid certificate for the connector's FQDN (event ID 12014 says it found none). Pin the certificate, and require TLS only once the gateway uses it; otherwise it gets 451 5.7.3 Must issue a STARTTLS command first.

   powershell

   $cert = Get-ExchangeCertificate -Thumbprint "<thumbprint>"
   Set-ReceiveConnector "EX01\Inbound from Gateway" -TlsCertificateName "<I>$($cert.Issuer)<S>$($cert.Subject)" -RequireTLS $true

6. 6

   Route outbound mail through the gateway

   powershell

   New-SendConnector -Name "Outbound via Gateway" -AddressSpaces "*" -DNSRoutingEnabled $false `
       -SmartHosts "relay1.gateway.example","relay2.gateway.example" -SourceTransportServers EX01,EX02 `
       -RequireTLS $true -TlsAuthLevel CertificateValidation
   Test-NetConnection relay1.gateway.example -Port 25

   Register your public egress IP (after NAT) with the gateway, or authenticate with -SmartHostAuthMechanism BasicAuthRequireTLS and -AuthenticationCredential; cloud services usually also want every sending domain registered. When the gateway refuses mail, the queue's LastError or the NDR quotes its response, so read that before changing Exchange.

7. 7

   Give every domain one final owner

   Loops start when each side thinks the other owns an address. Typical case: a domain is InternalRelay in Exchange, unknown recipients leave via the * connector, and the gateway routes them straight back, until the sender gets 554 5.4.6 Hop count exceeded - possible mail loop.

   powershell

   Get-AcceptedDomain | Format-Table DomainName, DomainType, Default -AutoSize

Verify

$log = Join-Path $env:ExchangeInstallPath "TransportRoles\Logs\FrontEnd\ProtocolLog\SmtpReceive"
Get-ChildItem $log -Filter "RECV*.LOG" | Sort-Object LastWriteTime | Select-Object -Last 1 |
    Select-String -Pattern "Inbound from Gateway" | Select-Object -Last 10
Get-TransportService | Get-Queue | Where-Object { $_.DeliveryType -eq 'SmartHostConnectorDelivery' } |
    Format-Table Identity, Status, MessageCount, LastError -AutoSize

• Gateway sessions show your connector name, a gateway IP and STARTTLS. Search for Default Frontend too: public IPs that aren't the gateway's mean senders can still bypass it.
• A test message from outside shows X-MS-Exchange-Organization-AuthAs: Anonymous (not Internal) and the gateway's Received header with the original sender's IP.
• Outbound, message tracking shows a SEND through Outbound via Gateway, and the smart host queue stays empty.

Tips & gotchas

• Keep one list of gateway addresses and update the connector, InternalSMTPServers and the firewall together; cloud services add ranges over time.
• Without the Recipient Filter agent, a Mailbox server accepts unknown recipients and sends the NDR later, so addresses that slip past the gateway become backscatter. Keep the gateway's recipient data in sync with Active Directory.
• Gateway in front of Exchange Online instead? The counterpart there is Enhanced Filtering for Connectors; see hybrid mail flow connectors.
• Vendor guides: FortiMail, Barracuda, Mimecast, Proofpoint, Cisco and the rejection codes quick reference.