2022 Advisory: Exchange Server Mail Flow Outage

Occasion: New Year Eve | Holiday | December 31, 2021

Impacted Software: Exchange Server 2016 and 2019

Cause: Microsoft Exchange Server Antimalware Update

Symptoms:

1. All Emails will be Stuck in Submission Queue
   
2. Exchange Server – Application Log will receive the following events
   

   
   

   
   

   
   

   
   

   
   

   
   

3. Point to be noted is Exchange updates were applied then issue started
   

   
   

4. We have reported to the Exchange Team on Twitter
   

   
   

   
   

5. What Next:
   

   5a. Intermittent Resolution:
   

• Disabled Antimalware
  
• Step 1: Run the following commands on Exchange Management Shell:
  
  • cd $ExScripts
    
  • .\Disable-AntiMalwareScanning.ps1
    
• Step 2: Restart “Microsoft Exchange Transport” Service
  

5b. Permanent Resolution:

• Wait for Microsoft update and public announcement
  

More options can be checked here: https://docs.microsoft.com/en-us/archive/blogs/ehlro/exchange-2013-malware-engine-updates-troubleshooting

Update 1/2/2022:

On 1/1/2022 at 11:39 AM Microsoft Published this Issue and at 10:45 PM PST updated the resolution.

https://techcommunity.microsoft.com/t5/exchange-team-blog/email-stuck-in-transport-queues/bc-p/3049661#M31933

The following are the corrective actions which will stop email flow for estimated 10 minutes:

Automated resolution:

• Download the script Reset-ScanEngineVersion.ps1 from here
  
• Copy the script to C:\Program Files\Microsoft\Exchange Server\V15\Scripts>
  
• Run the script from Exchange Management Shell.
  

Manual resolution:

Remove existing engine and metadata
1. Stop the Microsoft Filtering Management service.  When prompted to also stop the Microsoft Exchange Transport service, click Yes.
2. Use Task Manager to ensure that updateservice.exe is not running.
3. Delete the following folder: %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\amd64\Microsoft.
4. Remove all files from the following folder: %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\metadata.

Update to latest engine
1. Start the Microsoft Filtering Management service and the Microsoft Exchange Transport service.
2. Open the Exchange Management Shell, navigate to the Scripts folder (%ProgramFiles%\Microsoft\Exchange Server\V15\Scripts), and run Update-MalwareFilteringServer.ps1 <server FQDN>.

Verify engine update info
1. In the Exchange Management Shell, run Add-PSSnapin Microsoft.Forefront.Filtering.Management.Powershell.
2. Run Get-EngineUpdateInformation and verify the UpdateVersion information is 2112330001.

Sorry if it has spoiled your New Year. I hope all relayed emails will be processed after this guidance.

Contact us at Golden Five Consulting for any further support or concerns at Support@GoldenFive.net.

Best Regards,

Prabhat Nigam | Twitter | LinkedIn

Global CTO | Golden Five

CEO | LAEXUG Foundation