2022 Advisory: Exchange Server Mail Flow Outage

Occasion: New Year Eve | Holiday | December 31, 2021

Impacted Software: Exchange Server 2016 and 2019

Cause: Microsoft Exchange Server Antimalware Update

Symptoms:

  1. All Emails will be Stuck in Submission Queue
  2. Exchange Server – Application Log will receive the following events







  3. Point to be noted is Exchange updates were applied then issue started


  4. We have reported to the Exchange Team on Twitter



  5. What Next:

    5a. Intermittent Resolution:

  • Disabled Antimalware
  • Step 1: Run the following commands on Exchange Management Shell:
    • cd $ExScripts
    • .\Disable-AntiMalwareScanning.ps1
  • Step 2: Restart “Microsoft Exchange Transport” Service

5b. Permanent Resolution:

  • Wait for Microsoft update and public announcement

More options can be checked here: https://docs.microsoft.com/en-us/archive/blogs/ehlro/exchange-2013-malware-engine-updates-troubleshooting

Update 1/2/2022:

On 1/1/2022 at 11:39 AM Microsoft Published this Issue and at 10:45 PM PST updated the resolution.

https://techcommunity.microsoft.com/t5/exchange-team-blog/email-stuck-in-transport-queues/bc-p/3049661#M31933

The following are the corrective actions which will stop email flow for estimated 10 minutes:

Automated resolution:

  • Download the script Reset-ScanEngineVersion.ps1 from here
  • Copy the script to C:\Program Files\Microsoft\Exchange Server\V15\Scripts>
  • Run the script from Exchange Management Shell.

Manual resolution:

Remove existing engine and metadata
1. Stop the Microsoft Filtering Management service.  When prompted to also stop the Microsoft Exchange Transport service, click Yes.
2. Use Task Manager to ensure that updateservice.exe is not running.
3. Delete the following folder: %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\amd64\Microsoft.
4. Remove all files from the following folder: %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\metadata.

Update to latest engine
1. Start the Microsoft Filtering Management service and the Microsoft Exchange Transport service.
2. Open the Exchange Management Shell, navigate to the Scripts folder (%ProgramFiles%\Microsoft\Exchange Server\V15\Scripts), and run Update-MalwareFilteringServer.ps1 <server FQDN>.

Verify engine update info
1. In the Exchange Management Shell, run Add-PSSnapin Microsoft.Forefront.Filtering.Management.Powershell.
2. Run Get-EngineUpdateInformation and verify the UpdateVersion information is 2112330001.

Sorry if it has spoiled your New Year. I hope all relayed emails will be processed after this guidance.

Contact us at Golden Five Consulting for any further support or concerns at Support@GoldenFive.net.

Best Regards,

Prabhat Nigam | Twitter | LinkedIn

Global CTO | Golden Five

CEO | LAEXUG Foundation

CTO @ Golden Five | CEO at LAExUG Foundation

Prabhat is 3 times Microsoft MVP Award winner. He is MBA in Information Technology and he is working as a CTO at Golden Five Consulting which is a Microsoft Gold Partner, MSP, T1CSP, and Education partner. He helps in designing, implementing, managing and supporting solutions for private messaging cloud, mergers, a collaboration between different messaging software and other migration & deployment projects for the following technologies Office 365, Azure, AWS, Exchange, SQL, ADFS, MFA, FIM, MIM and Directory services. He has worked for all big IT giants either as an employee or contractor where he has led the Global teams. He has started his career as Technical Consultant in Exchange 5.5 with Microsoft PSS and his exchange love never stopped & continued with 2000/2003/2007/2010/2013/2016/2019/O365. At one point in time, he was the only person to support EDS customers when Microsoft had closed all the supports for 5.5 and now for all old legacy Exchange versions.
He used to blog at MSExchangeGuru.com, manages multiple LinkedIn and Facebook Groups. He also Owns MSExchnageGuru YouTube channel where he uploads all his records technical sessions. Don’t forget to check his PowerShell scripts which are making admins life easier. Prabhat can be reached at PN@GoldenFive.net.

Leave Comment

Your email address will not be published.