OnPrem Exchange and throttling with EXO

I was doing some checks the other day I noticed that lots of messages were queuing in the queues as "Active" but were not being sent, this was a little peculiar, I thought it was a good idea to dig a little bit deeper, I noticed that the messages is not being sent. had an email status code, which was very clear:

4.7.230 Connecting Exchange server version is out-of-date; connection to Exchange Online throttled for n mins/hr.

5.7.230 Connecting Exchange server version is out-of-date; connection to Exchange Online blocked for n mins/hr.

Microsoft have this to say about it:

For security reasons, messages sent from out-of-date on-premises Exchange servers over an inbound connector of type OnPremises would soon be subject to throttling and blocking. If your out-of-date on-premises connecting servers are getting throttled or blocked.

This would appear to be Microsoft throttling unpatched Exchange as it marked as vulnerable, this is the activation date of the version of Exchange, which means if you are reading this all Exchange versions now apply to this rule - well played Microsoft.

Reminder : Ensure local exchange is patched

This is a reminder of the importance of patching your local exchange environment to keep it secure and bulletproof against throttling policies like this, there is no reason why you are not patching your local Exchange - this particular policy only focuses on the Exchange Release version it goes without saying you should also be keeping on top of windows updates.

Check violation report for throttling

If you want a report of which Exchange servers are affected then you can run this command in Powershell with the session connection to Exchange Online:

Get-OnPremServerReportInfo

This will report all the servers that are not patched to the correct level and will give the recommended versions, here you can see I have one Exchange 2016 server not on the right version or patch level.

ServerFQDN : Exch1.bear.local
Build : 15.1.2507.16
ExchangeServer : ExchangeServer2016
FirstDetectedDate : 28/03/2024 00:15:08
RecommendedBuildVersions : 15.1.2507.39,15.1.2507.37,15.1.2507.36,15.1.2507.35,15.1.2507.34,15.1.2507.32,15.1.2507.31,15.01.2507.039,15.01.2507.037,15.01.2507.036,15.01.2507.035,15.01.2507.034,15.01.2507.032,15.01.2507.031
ThrottleEnabledDate : 27/04/2024 00:15:08
BlockingEnabledDate : 27/05/2024 00:15:08
NextStageThrottleRate : 20
NextStageBlockRate : 0
NextStageStartDate : 23/05/2024 00:15:08

If you are one for the graphical report you can use the link here however for a demo this is an example, here you can see a patched server and an unpatched server.

Check your Exemptions

If you are not up to date on your patching you need to check the status of the "throttle" this can be done with this command:

Get-TenantExemptionInfo -BlockingScenario UnpatchedOnPremServer

That will return the current status of your EXO "non patched" throttling as below:

TenantId : <tenant-id>
BlockingScenario : UnpatchedOnPremServer
StartTime : 09/05/2024 13:08:38
EndTime : 19/05/2024 13:08:38
Details :

Add a Tenant Exemption

If you are being throttled then you can ask for some extension which applies to future messages and the currently blocked messages will clear themselves on the next transport sending cycle with this command, this command extends the "throttle" for 10 days:

New-TenantExemptionInfo -BlockingScenario UnpatchedOnPremServer -NumberOfDays 10

Rules about extensions

These are the rules about applying in Powershell for an extension:

Check the version of Exchange - wait what ?

Well this is interesting as when you check the version installed with the usual command which is this:

Get-ExchangeServer | fl ame, Edition, AdminDisplayVersion

Edition : Enterprise
AdminDisplayVersion : Version 15.1 (Build 2507.6)

That is a good place to start but there is a problem with that approach, this proves it Exchange 2016 as per the version number and this proves it CU23 as per the build number, so if you reference this website for all the Exchnage version information:

https://learn.microsoft.com/en-us/exchange/new-features/build-numbers-and-release-dates?view=exchserver-2019

That will tell you that the build version should be updated when the update is indeed installed, but we see the Build version of 15.1.2507.6 from earlier (red box) but that version is not correct as we are on later (green box) - this means that this command is saying Exchange has not been patched since April 2022?

The answer to this riddle, is we should not be checking it with that Powershell command from eelier but with this command that gets the "installed" version not the "reported" version, which is very confusing when the Exchange management shell does not update the "Build version"

Check Build Version with Alternative Powershell script

To actually get the correct "installed" version of Powershell you need to use the script, ensure you are running this as a administrator with an elevated Powershell command window, then you need this:

$ExchangeServers = Get-ExchangeServer | Sort-Object Name

ForEach ($Server in $ExchangeServers)

{

Invoke-Command -ComputerName $Server.Name -ScriptBlock { Get-Command Exsetup.exe | ForEach-Object { $_.FileversionInfo } }

}

This will then return the correct data, based on the installed version of Exchange......I called this Version.ps1 so I ran this:

./Version.ps1

To get the correct report:

15.01.2507.039 15.01.2507.039 C:\Program Files\Microsoft\Exchange Server\V15\bin\ExSetup
15.01.2507.016 15.01.2507.016 C:\Program Files\Microsoft\Exchange Server\V15\bin\ExSetup
15.01.2507.035 15.01.2507.035 C:\Program Files\Microsoft\Exchange Server\V15\bin\ExSetup
15.01.2507.039 15.01.2507.039 C:\Program Files\Microsoft\Exchange Server\V15\bin\ExSetup

This then tells me that we have 1 x non-compliant server and one server that is compliant but is not patched to the level of the other servers, we are at the time of this post looking for v15.01.2507.39, so we need to get that update for Exchange 2016 April 24 HU update which for this post is this link:

https://support.microsoft.com/en-gb/topic/hotfix-update-for-exchange-server-2019-and-2016-april-23-2024-kb5037224-35eddea8-4828-4e38-b462-db89ea1100c9

Which will the in turn take you to this link:

https://www.microsoft.com/download/details.aspx?familyID=db6d864f-8ac9-44f0-bb05-45e1ef6fced4

Now you can get your Exchange server compliant and prevent delayed or blocked messages, once updated when you run "Version.ps1" it should look like this:

15.01.2507.039 15.01.2507.039 C:\Program Files\Microsoft\Exchange Server\V15\bin\ExSetup
15.01.2507.039 15.01.2507.039 C:\Program Files\Microsoft\Exchange Server\V15\bin\ExSetup
15.01.2507.039 15.01.2507.039 C:\Program Files\Microsoft\Exchange Server\V15\bin\ExSetup
15.01.2507.039 15.01.2507.039 C:\Program Files\Microsoft\Exchange Server\V15\bin\ExSetup

Excellent all patched for the moment, however this is a lesson about keeping on top of not just Windows updates, but all Microsoft updates.

OnPrem Exchange and throttling with EXO

نموذج الاتصال