Solarwinds is critical to the monitoring of your networking equipment if you use it and when a CVE is released with a new version to fix some RCE code exploits its always best to get them done as soon as you can.
However what you do not expect to see is this:
This reads as "configuring the message bus has FAILED" and you need to review logs as to why, excellent, without this Solarwinds cannot operate, which means your monitoring is down
Check rabbitmq Logs
You can check for logs in the directory below, but in this example we had no logs whatsoever.
C:\ProgramData\SolarWinds\Orion\RabbitMQ\log
In this folder you should see a log named : rabbit@<servername>.log
Check Other Logs
Without the RabbitMQ log, we need to look eslewhere so if you nagivate to this folder:
C:\ProgramData\SolarWinds\Logs\Installer
This will contain all the installer log files of which you are looking for files updated on the date of the upgrade
These logs were all normal, but it you looked for folders you have two interesting candidates:
The first one was all normal, but the bottom folder only contained one file and it was a goldmine for me anyway:
When you open this file towards the bottom I noticed this, and this is a stack trace, nice now we can get somewhere:
Server stack trace:
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at SolarWinds.Orion.ActiveDiagnostics.Engine.DiagnosticMethodInvoker.<>c__DisplayClass4_0.<RunDiagnostic>b__0()
at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Object[]& outArgs)
at System.Runtime.Remoting.Messaging.StackBuilderSink.AsyncProcessMessage(IMessage msg, IMessageSink replySink)
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.EndInvokeHelper(Message reqMsg, Boolean bProxyCase)
at System.Runtime.Remoting.Proxies.RemotingProxy.Invoke(Object NotUsed, MessageData& msgData)
at System.Func`1.EndInvoke(IAsyncResult result)
at SolarWinds.Orion.ActiveDiagnostics.Engine.DiagnosticMethodInvoker.RunDiagnostic(DiagnosticNode diagnosticNode, WaitHandle waitHandle, Object[] args)
2024-02-07 18:24:04,012 [9] ERROR (null) SolarWinds.Orion.Core.SharedCredentials.Utility.ReflectionHelper - Failed setting property
System.Security.Cryptography.CryptographicException: The data is invalid.
at System.Security.Cryptography.ProtectedData.Unprotect(Byte[] encryptedData, Byte[] optionalEntropy, DataProtectionScope scope)
at SolarWinds.Orion.Security.KeyManagement.ProtectedDataHandler.Unprotect(Byte[] encryptedData)
at SolarWinds.Orion.Security.KeyManagement.SingleKeyCache.GetKey(IProtectedDataHandler protectedDataHandler, Func`1 getUncachedRecord)
at SolarWinds.Orion.Security.KeyManagement.SingleKeyCache.GetKey(Int32 id, IProtectedDataHandler protectedDataHandler, Func`1 getUncachedRecord)
at SolarWinds.Orion.Security.KeyManagement.FileKeyManager.GetKey(Int32 id)
at SolarWinds.Orion.Security.CryptoHelper.DecryptAes(String encryptedText)
at SolarWinds.Orion.Security.CryptoHelper.Decrypt(String encryptedText)
at SolarWinds.Orion.Core.SharedCredentials.Utility.ReflectionHelper.DetermineInstanceValue(CredentialPropertyMetaData credentialPropertyMetaData, String databaseValue, Boolean dbValueEncrypted, Boolean decrypt, XmlStrippedSerializerCache serializerCache)
at SolarWinds.Orion.Core.SharedCredentials.Utility.ReflectionHelper.SetCredentialProperty[TCredential](CredentialPropertyEntity property, TCredential currentCredential, IEnumerator`1 iMetaDataProperty, XmlStrippedSerializerCache serializerCache, Boolean ignoreDecryptErrors, Boolean decrypt)
at SolarWinds.Orion.Core.SharedCredentials.Utility.ReflectionHelper.<ConstructCredentials>d__6`1.MoveNext()
2024-02-07 18:24:04,027 [9] DEBUG (null) SolarWinds.Orion.Swis.PubSub.MessageBus.MessageBusConnectionProvider - { Opened connection to ST1W10348 using type easynetq-direct and username orion entered
2024-02-07 18:24:04,027 [9] DEBUG (null) SolarWinds.Orion.Swis.PubSub.MessageBus.MessageBusConnectionProvider - } Opened connection to ST1W10348 using type easynetq-direct and username orion exited
2024-02-07 18:24:04,027 [9] DEBUG (null) SolarWinds.Orion.Swis.PubSub.MessageBus.MessageBusConnectionProvider - SolarWinds.Orion.Swis.PubSub.MessageBus.MessageBusConnectionProvider Dispose (True, True)
2024-02-07 18:24:04,607 [39] ERROR (null) SolarWinds.Orion.ActiveDiagnostics.Common.ConfigHelper - File DatabaseConnectionDiagnosticSuiteConfig.xml for SolarWinds.Orion.ActiveDiagnostics.Engine.Suites.DatabaseConnection.DatabaseConnectionDiagnosticSuiteConfig config isn't found.
The important bit for me is this, as it cannot find its database for RabbitMQ, or the database is corrupt and not useable:
SolarWinds.Orion.ActiveDiagnostics.Common.ConfigHelper - File DatabaseConnectionDiagnosticSuiteConfig.xml for SolarWinds.Orion.ActiveDiagnostics.Engine.Suites.DatabaseConnection.DatabaseConnectionDiagnosticSuiteConfig config isn't found.
Delete the RabbitMQ service
First, we need to delete the RabbitMQ service with this command, which will need to be done from a evelated command prompt:
sc delete rabbitmq
You will then get a "Success" confirmation from this so you can move on.
Delete the DB folder on for RabbitMQ
Now you need to delete the folder or copy if you like, but it is corrupt, the location is this:
C:\ProgramData\SolarWinds\Orion\RabbitMQ
You need to delete the "db" folder as shown below:
That will look like this
Run Solarwinds Configuration WizardNow you need to run the Solarwinds configuration wizard, use all the defaults from the previous install as you are not changing anything:
This will then rebuild the RabbitMQ service and the database and then Solarwinds will be back online.
