If you are getting errors with your remove a FE (front end server) from the topology, then there may be a simple reason for this.
When you add multiple FE servers to a pool, the roles are distributed across all server, which can cause an issue when the removal topology fails to drop these servers from the database.
You will see errors like this:
There was no response from the Front End to an OPTIONS request sent by the Mediation Server service.
The Front End, lyncpool.unicorn.local, is not responding to an OPTIONS request sent by the Mediation Server service.
DNS Resolution Failure: False
Exception: Microsoft.Rtc.Signaling.ConnectionFailureException:Unable to establish a connection. ---> System.Net.Sockets.SocketException (0x80004005): No connection could be made because the target machine actively refused it 10.242.1.197:5061
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at Microsoft.Rtc.Internal.Sip.TcpTransport.OnConnected(Object arg)
and this...
Local Pool Manager has been disconnected from Pool Fabric Manager.
Cause: This may happen if many Front-Ends in the cluster have gone down or become unreachable, causing loss of quorum.
and this....
Server startup is being delayed because fabric pool manager has not finished initial placement of users.
Currently waiting for routing group: {3521B7FF-0042-5853-8A8A-2C8166A2AE51}.
Number of groups potentially not yet placed: 7.
Total number of groups: 7.
Cause: This is normal during cold-start of a Pool and during server startup.
If you continue to see this message many times, it indicates that insufficient number of Front-Ends are available in the Pool.
This is because Skye is tring to talk to a server you have removed which uses the LB name, this requires a QuorumLossRecovery, to do this complete these steps:
Reset-CsPoolRegistrarState -ResetType QuorumLossRecovery
If you have more than one pool you will require:
Reset-CsPoolRegistrarState -Pool -ResetType QuorumLossRecovery
Once you see this, you should be back online:
Fabric primary was changed for the following fabric services by Fabric Pool Manager:
from to lync.unicorn.local for Fabric Service: fabric:/lync/storage/1673DC96160257E38E03FFBA7C877C14
from to lync.unicorn.local for Fabric Service: fabric:/lync/storage/3521B7FF004258538A8A2C8166A2AE51
from to lync.unicorn.local for Fabric Service: fabric:/lync/storage/9BF299CE3C405648A3E2AFEC36F1B72A
from to lync.unicorn.local for Fabric Service: fabric:/lync/storage/ACAB7EBF246B53C39976B0211969FA9A
from to lync.unicorn.local for Fabric Service: fabric:/lync/storage/C7D1BDE8FAB459559C047F9461128D8D
from to lync.unicorn.local for Fabric Service: fabric:/lync/storage/CE4827001B465A869D445F8C2AB382F2
from to lync.unicorn.local for Fabric Service: fabric:/lync/storage/D8E0B17820B75616BD744176120D8C59
.
When you add multiple FE servers to a pool, the roles are distributed across all server, which can cause an issue when the removal topology fails to drop these servers from the database.
You will see errors like this:
There was no response from the Front End to an OPTIONS request sent by the Mediation Server service.
The Front End, lyncpool.unicorn.local, is not responding to an OPTIONS request sent by the Mediation Server service.
DNS Resolution Failure: False
Exception: Microsoft.Rtc.Signaling.ConnectionFailureException:Unable to establish a connection. ---> System.Net.Sockets.SocketException (0x80004005): No connection could be made because the target machine actively refused it 10.242.1.197:5061
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at Microsoft.Rtc.Internal.Sip.TcpTransport.OnConnected(Object arg)
and this...
Local Pool Manager has been disconnected from Pool Fabric Manager.
Cause: This may happen if many Front-Ends in the cluster have gone down or become unreachable, causing loss of quorum.
and this....
Server startup is being delayed because fabric pool manager has not finished initial placement of users.
Currently waiting for routing group: {3521B7FF-0042-5853-8A8A-2C8166A2AE51}.
Number of groups potentially not yet placed: 7.
Total number of groups: 7.
Cause: This is normal during cold-start of a Pool and during server startup.
If you continue to see this message many times, it indicates that insufficient number of Front-Ends are available in the Pool.
This is because Skye is tring to talk to a server you have removed which uses the LB name, this requires a QuorumLossRecovery, to do this complete these steps:
Reset-CsPoolRegistrarState -ResetType QuorumLossRecovery
If you have more than one pool you will require:
Reset-CsPoolRegistrarState -Pool -ResetType QuorumLossRecovery
Once you see this, you should be back online:
Fabric primary was changed for the following fabric services by Fabric Pool Manager:
from to lync.unicorn.local for Fabric Service: fabric:/lync/storage/1673DC96160257E38E03FFBA7C877C14
from to lync.unicorn.local for Fabric Service: fabric:/lync/storage/3521B7FF004258538A8A2C8166A2AE51
from to lync.unicorn.local for Fabric Service: fabric:/lync/storage/9BF299CE3C405648A3E2AFEC36F1B72A
from to lync.unicorn.local for Fabric Service: fabric:/lync/storage/ACAB7EBF246B53C39976B0211969FA9A
from to lync.unicorn.local for Fabric Service: fabric:/lync/storage/C7D1BDE8FAB459559C047F9461128D8D
from to lync.unicorn.local for Fabric Service: fabric:/lync/storage/CE4827001B465A869D445F8C2AB382F2
from to lync.unicorn.local for Fabric Service: fabric:/lync/storage/D8E0B17820B75616BD744176120D8C59
.