Microsoft says it has fixed an Exchange Server bug that is causing emails for on-premises servers to get stuck in queues due to the change of the new year.
According to Microsoft, emails were stuck in transport queues of on-premises Exchange Server 2016 and 2019 due to a date check failure with the change of the new year. The company clarifies that the issue is not a security bug. A failure of the AV engine, a problem with malware scanning or a malware engine flaw.
According to BleepingComputer, the errors are caused by Microsoft Exchange checking the version of the FIP-FS antivirus scanning engine and attempting to store the data in a signed int32 variable.
However, the variable can store only a maximum value of 2,147,483,647, which is less than the new date value of 2,201,010,001 for January 1st, 2022, at midnight, the website reports.
Read: You Need To Look Out For These Software Vulns
Due to this, when Microsoft Exchange attempts to check the AV scanning version, it would generate a bug and cause the malware engine to crash, BleepingComputer reports.
“The version checking performed against the signature file is causing the malware engine to crash, resulting in messages being stuck in transport queues,” the Exchange Team said in a Tech Community blog on New Year’s Day.
To fix the issue, customer action is required, the company notes.
When the issue occurs, you’ll see errors in the Application event log on the Exchange Server, specifically event 5300 and 1106 (FIPFS), as illustrated below:
Log Name: Application
Source: FIPFS
Logged: 1/1/2022 1:03:42 AM
Event ID: 5300
Level: Error
Computer: server1.contoso.com
Description: The FIP-FS “Microsoft” Scan Engine failed to load. PID: 23092, Error Code: 0x80004005. Error Description: Can’t convert “2201010001” to long.Log Name: Application
Source: FIPFS
Logged: 1/1/2022 11:47:16 AM
Event ID: 1106
Level: Error
Computer: server1.contoso.com
Description: The FIP-FS Scan Process failed initialization. Error: 0x80004005. Error Details: Unspecified error.
Microsoft says customers can solve this via an automated or manual solution.
The Automated Solution
- Download this script
- Before running the script, change the execution policy for PowerShell scripts by running Set-ExecutionPolicy -ExecutionPolicy RemoteSigned.
- Using Elevated Exchange Management Shell, run the script on each Exchange mailbox server that downloads antimalware updates in your organization.
Microsoft says Edge Transport servers will be unaffected by the issue. The script can be run on multiple servers in parallels. After completed, admins will see this output:
[PS] C:\Program Files\Microsoft\Exchange Server\V15\Scripts>.\Reset-ScanEngineVersion.ps1
EXCH1 Stopping services…
EXCH1 Removing Microsoft engine folder…
EXCH1 Emptying metadata folder…
EXCH1 Starting services…
WARNING: Waiting for service ‘Microsoft Filtering Management Service (FMS)’ to start…
WARNING: Waiting for service ‘Microsoft Filtering Management Service (FMS)’ to start…
WARNING: Waiting for service ‘Microsoft Filtering Management Service (FMS)’ to start…
WARNING: Waiting for service ‘Microsoft Filtering Management Service (FMS)’ to start…
WARNING: Waiting for service ‘Microsoft Exchange Transport (MSExchangeTransport)’ to start…
EXCH1 Starting engine update…
Running as EXCH1-DOM\Administrator.
——–
Connecting to EXCH1.CONTOSO.com.
Dispatched remote command. Start-EngineUpdate -UpdatePath http://amupdatedl.microsoft.com/server/amupdate
——–
[PS] C:\Program Files\Microsoft\Exchange Server\V15\Scripts>Get-EngineUpdateInformationEngine : Microsoft
LastChecked : 01/01/2022 08:58:22 PM -08:00
LastUpdated : 01/01/2022 08:58:31 PM -08:00
EngineVersion : 1.1.18800.4
SignatureVersion : 1.355.1227.0
SignatureDateTime : 01/01/2022 03:29:06 AM -08:00
UpdateVersion : 2112330001
UpdateStatus : UpdateAttemptSuccessful
The Manual Solution
Customers can also perform manual steps to resolve the issue and restore email service. To do this, admins must perform the steps below on each Exchange mailbox server in the organization that downloads antimalware updates. Again, Edge Transport servers are unaffected, Microsoft notes.
Remove existing engine and metadata
1. Stop the Microsoft Filtering Management service. When prompted to also stop the Microsoft Exchange Transport service, click Yes.
2. Use Task Manager to ensure that updateservice.exe is not running.
3. Delete the following folder: %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\amd64\Microsoft.
4. Remove all files from the following folder: %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\metadata.Update to latest engine
1. Start the Microsoft Filtering Management service and the Microsoft Exchange Transport service.
2. Open the Exchange Management Shell, navigate to the Scripts folder (%ProgramFiles%\Microsoft\Exchange Server\V15\Scripts), and run Update-MalwareFilteringServer.ps1 <server FQDN>.Verify engine update info
1. In the Exchange Management Shell, run Add-PSSnapin Microsoft.Forefront.Filtering.Management.Powershell.
2. Run Get-EngineUpdateInformation and verify the UpdateVersion information is 2112330001.
After updating the engine, Microsoft recommends admins verify that mail flow is working and that FIPFS error events are not present in the Application log event.
For more information, read Microsoft’s Tech Community blog on this issue.
If you enjoyed this article and want to receive more valuable industry content like this, click here to sign up for our digital newsletters!
Leave a Reply