Microsoft Exchange Writer failed or Retryable error: fixing Exchange Server VSS backups

Every Exchange-aware backup product, from Veeam to Windows Server Backup, goes through one component: the Microsoft Exchange Writer. When it fails, the job fails or ends with a warning and no log truncation. In this post I'll show how to read the writer's state, reset it safely, fix the DAG conflicts Microsoft documents, and prove which side is at fault.

Symptoms#

An elevated vssadmin list writers on the Exchange server shows the writer in a bad state:

Writer name: 'Microsoft Exchange Writer'
   Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
   Writer Instance Id: {...}
   State: [9] Failed
   Last error: Timed out

• State: [1] Stable with Last error: Retryable error: the writer has reset, but the last backup failed.
• Last error: Non-retryable error: the failure will recur until the cause is fixed.
• State: [5] Waiting for completion long after the job ended: the backup application never finished.
• Microsoft's BETEST tool reporting Status for writer Microsoft Exchange Writer: STABLE (0x800423f3 - VSS_E_WRITERERROR_RETRYABLE).
• On a DAG member backing up a passive copy, this Application log error:

Source: MSExchangeRepl   Event ID: 2153
The log copier was unable to communicate with server <active server FQDN>. The copy of database <database\server> is in a disconnected state.

Or quietly: the job reports success, but logs pile up and LastFullBackup never moves.

Why it happens#

Since Exchange 2013 there has been a single writer, Microsoft Exchange Writer, hosted by the Microsoft Exchange Replication service (MSExchangeRepl). It covers active and passive copies, but it's only advertised to VSS while the Information Store service (MSExchangeIS) is running too. A backup walks the writer through fixed phases, and Exchange truncates logs only after a successful full or incremental backup. The state number shows which phase broke:

The last error is the verdict: Retryable error (0x800423F3) is transient, Non-retryable error (0x800423F4) will recur, and Timed out (0x800423F2) means the freeze window expired.

In a DAG, a passive-copy backup makes the Replication service query the active server for the log range and hold an RPC channel to it for the whole backup. A busy active server, a network device dropping idle sessions or security software scanning that traffic fails the job as Retryable. So does backing up the active and passive copies of one database at the same time.

How to fix it#

1. Capture the state first#

vssadmin list writers | Select-String -Pattern "Microsoft Exchange Writer" -Context 0,4
Get-Service MSExchangeRepl, MSExchangeIS, VSS | Format-Table Name, Status, StartType
Get-MailboxDatabase -Server $env:COMPUTERNAME -Status | Format-List Name, *Backup*

Note the state, the last error and whether BackupInProgress is still True.

2. Reset the writer#

With no job running, restart the service that hosts the writer. Replication for copies on this server pauses briefly, so do it outside the backup window:

Restart-Service MSExchangeRepl
vssadmin list writers

Good is State: [1] Stable and Last error: No error. If the writer is missing, check that MSExchangeIS is running. Veeam's KB2215 documents an EnableVSSWriter value under HKLM\SOFTWARE\Microsoft\ExchangeServer\v15\Replay\Parameters that some older backup products required you to disable; set it to 1 and restart the Replication service. If the state won't reset, restart the Volume Shadow Copy and COM+ Event System services, or reboot.

3. Check VSS providers and leftovers#

vssadmin list providers
vssadmin list shadows
vssadmin list shadowstorage

Usually the only provider is Microsoft Software Shadow Copy provider 1.0; a SAN or agent provider must belong to the product you run today, not a retired one. Shadow copies stranded by failed jobs and full shadow storage also break snapshots; check for VSS errors in the Application log.

4. Fix DAG backup conflicts#

• Back up passive copies, and never back up the active copy of the same database in the same window.
• Don't switch databases over or start maintenance mode during a backup.
• For event 2153, fix the network path: Microsoft advises a TCP KeepAliveTime shorter than the idle-session timeout of the devices in between, and longer than 15 minutes.
• If the active server is busy or holds many logs, raise the log-range query timeout (30 seconds by default) on the server that runs the passive-copy backup:

New-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\ExchangeServer\v15\Replay\Parameters' `
  -Name QueryLogRangeTimeoutInMsec -PropertyType DWord -Value 120000
Restart-Service MSExchangeRepl

5. Use Windows Server Backup as a control test#

Windows Server Backup is Microsoft's own Exchange-aware requester (Exchange adds the WSBExchange plug-in). Pause your normal job and, on the server hosting the active copy, run a VSS full backup of the database and log volumes to a separate target. A full-server backup won't do: it's a VSS copy backup and never truncates logs (see Using Windows Server Backup with Exchange).

Install-WindowsFeature Windows-Server-Backup
wbadmin start backup -backupTarget:F: -include:D:,L: -vssFull -quiet

If it succeeds and logs truncate, Exchange and VSS are healthy: look at the backup product's job or agent. If it fails the same way, fix Exchange and Windows first. A VSS full backup changes the history other products rely on, so run a full backup with your normal product afterwards.

When the backup "succeeds" but logs don't truncate#

• It was a copy backup (copy-only job setting or full-server WSB): LastCopyBackup moves, LastFullBackup doesn't.
• A volume was left out: excluding any database or log volume (or virtual disk) blocks truncation.
• The writer failed at backup complete ([12] Failed): Exchange never got the success notification.
• A DAG copy holds the logs: logs go only when they're below the checkpoint, the other non-lagged copies agree and every lagged copy has inspected them; lagged copies keep theirs for ReplayLagTime plus TruncationLagTime. Microsoft also notes that Exchange 2019 keeps more logs before truncating.

Get-MailboxDatabaseCopyStatus * | Format-Table Name, Status, CopyQueueLength, ReplayQueueLength -AutoSize
Get-MailboxDatabase DB01 | Format-List Name, ReplayLagTimes, TruncationLagTimes
Test-ReplicationHealth

Fix unhealthy copies with high copy or replay queues or FailedAndSuspended copies; if the disk is filling, see transaction logs filling the disk.

Verify the fix#

• The writer shows State: [1] Stable and Last error: No error before and after the next job.
• Get-MailboxDatabase -Status | Format-List Name,*FullBackup shows today's LastFullBackup and SnapshotLastFullBackup : True.
• Log counts drop on every copy once queues are near zero.
• No new event 2153 or VSS errors in the backup window.

Prevent it next time#

• Alert on the age of LastFullBackup, not on job status alone.
• One backup product per server; remove old agents and their VSS providers.
• Back up DAGs from passive copies, never alongside an active-copy backup of the same database.
• Product guides: Veeam, Backup Exec, NetBackup and Commvault, and the VSS error codes quick reference.

References#

• Backup, restore, and disaster recovery in Exchange Server (Microsoft Learn)
• VSS_E_WRITERERROR_RETRYABLE error when backing up a passive database copy (Microsoft Learn)
• KB2215: No transaction logs truncated after backup of Exchange DAG passive node (Veeam)