Exchange Writer (VSS) backup failure checklist

Symptoms to confirm

An elevated vssadmin list writers on the Exchange server shows the writer in a bad state:

Writer name: 'Microsoft Exchange Writer'
   Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
   Writer Instance Id: {...}
   State: [9] Failed
   Last error: Timed out

• State: [1] Stable with Last error: Retryable error: the writer has reset, but the last backup failed.
• Last error: Non-retryable error: the failure will recur until the cause is fixed.
• State: [5] Waiting for completion long after the job ended: the backup application never finished.
• Microsoft's BETEST tool reporting Status for writer Microsoft Exchange Writer: STABLE (0x800423f3 - VSS_E_WRITERERROR_RETRYABLE).
• On a DAG member backing up a passive copy, this Application log error:

Source: MSExchangeRepl   Event ID: 2153
The log copier was unable to communicate with server <active server FQDN>. The copy of database <database\server> is in a disconnected state.

Likely causes

Since Exchange 2013 there has been a single writer, Microsoft Exchange Writer, hosted by the Microsoft Exchange Replication service (MSExchangeRepl).

• The last error is the verdict: Retryable error (0x800423F3) is transient, Non-retryable error (0x800423F4) will recur, and Timed out (0x800423F2) means the freeze window expired.
• In a DAG, a passive-copy backup makes the Replication service query the active server for the log range and hold an RPC channel to it for the whole backup. A busy active server, a network device dropping idle sessions or security software scanning that traffic fails the job as Retryable.

Checklist

1. 1

   Capture the state first

   powershell

   vssadmin list writers | Select-String -Pattern "Microsoft Exchange Writer" -Context 0,4
   Get-Service MSExchangeRepl, MSExchangeIS, VSS | Format-Table Name, Status, StartType
   Get-MailboxDatabase -Server $env:COMPUTERNAME -Status | Format-List Name, *Backup*

   Note the state, the last error and whether BackupInProgress is still True.

2. 2

   Reset the writer

   With no job running, restart the service that hosts the writer. Replication for copies on this server pauses briefly, so do it outside the backup window:

   powershell

   Restart-Service MSExchangeRepl
   vssadmin list writers

3. 3

   Check VSS providers and leftovers

   cmd

   vssadmin list providers
   vssadmin list shadows
   vssadmin list shadowstorage

   Usually the only provider is Microsoft Software Shadow Copy provider 1.0; a SAN or agent provider must belong to the product you run today, not a retired one. Shadow copies stranded by failed jobs and full shadow storage also break snapshots; check for VSS errors in the Application log.

4. 4

   Fix DAG backup conflicts

   • Back up passive copies, and never back up the active copy of the same database in the same window.
   • Don't switch databases over or start maintenance mode during a backup.
   • For event 2153, fix the network path: Microsoft advises a TCP KeepAliveTime shorter than the idle-session timeout of the devices in between, and longer than 15 minutes.
   • If the active server is busy or holds many logs, raise the log-range query timeout (30 seconds by default) on the server that runs the passive-copy backup:
   powershell

   New-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\ExchangeServer\v15\Replay\Parameters' `
     -Name QueryLogRangeTimeoutInMsec -PropertyType DWord -Value 120000
   Restart-Service MSExchangeRepl

5. 5

   Use Windows Server Backup as a control test

   Windows Server Backup is Microsoft's own Exchange-aware requester (Exchange adds the WSBExchange plug-in). Pause your normal job and, on the server hosting the active copy, run a VSS full backup of the database and log volumes to a separate target.

   powershell

   Install-WindowsFeature Windows-Server-Backup
   wbadmin start backup -backupTarget:F: -include:D:,L: -vssFull -quiet

Verify

• The writer shows State: [1] Stable and Last error: No error before and after the next job.
• Get-MailboxDatabase -Status | Format-List Name,*FullBackup shows today's LastFullBackup and SnapshotLastFullBackup : True.
• Log counts drop on every copy once queues are near zero.
• No new event 2153 or VSS errors in the backup window.

Prevent it next time

• Alert on the age of LastFullBackup, not on job status alone.
• One backup product per server; remove old agents and their VSS providers.
• Back up DAGs from passive copies, never alongside an active-copy backup of the same database.
• Product guides: Veeam, Backup Exec, NetBackup and Commvault, and the VSS error codes quick reference.