SfB SE CMS Master failover success process

Background

You can start by reading this.  This is a tested path forward if you find yourself in the CMS split-brain scenario as described in that article.  After noodling through that process yesterday, and knowing that I have customers who need this to work so as to ETHICALLY meet their SLA/RTO/RTP type stuff, I got to thinking.  And then Josh Walters, a co-worker of mine, made the fateful comment “the server that gets failed over to is happy and functioning, why can’t we just leave it alone?”
In relative age terms, from the mouth of babes…I got to thinking – can I create a process that is repeatable, that comes before the Vale 19-step method, and allow me to confidently tell customers that “this works.”

Scenario

We are ignoring the RGS and the Edge changes necessary for the full site failovoer in this article.  We are totally focused on just the CMS, why it happens (theory on my part), and what to do to recover gracefully in a predictable manner (empirical on my part).

There is actually the option to not perform a CMS failover…I have had environments where the CMS was offline for extended periods of time with no ill effects.  Just don’t change anything.

Our environment is two SfB SE servers, pool paired.  Sfbse.tsoorad.net is the “old” master, sfbse2.tsoorad.net is the “new” master.

After making up the pool pair, we have simulated the datacenter outage by turning off sfbse.tsoorad.net, thereby making the surviving system components think that the CMS master is gone.  Power off is a state that pretty much assures that no-one is talking to that server anytime soon.










The initial CMS server failover goes just fine.  The problem comes up when the “old” master comes back online and thinks that IT is the master.  But the sfbse2 server, the “new” master is in charge, and suddenly, you cannot make changes.  Classic split-brain.  Replication is borked.  Attribute pointers don’t point.  See the blank in this example where the ActiveMasterFQDN might just be something we need to know about.



What is causing this

If my surmise/theory is correct, the split-brain starts when the second node assumes control of the CMS.  No problem.  As a domain member running with the proper authority/credentials, the AD gets changed, the topology gets changed, and the surviving servers in the environment start replicating from what they are told is the CMS.  At this point everything is fine; the split-brain has started, just not affecting us quite yet.
The split-brain posture really gets wound up when the failed server comes back online and it thinks that it is the CMS master.  Understandable.  Before whatever happened happened, that server was indeed the CMS master.  But another server is now designated, and the newly revived server never got re-written, and things are now just a tad stuck.  Again, see this article here, as well as the Mark Vale article here.

What to do about it

The obvious answer, of course, is the easiest.  We will wait right here as you locate your copy of last nights backup script and the ensuing copy of the export-csconfiguration and export-cslisconfiguration and carefully resolve NOT to use them. (they point to the OLD master, and the NEW master is up and running – and in the immortal words of  Josh Walters, “can’t we just leave it alone?)”.  Keep in mind that you don’t HAVE to move the CMS back.  To dovetail with the Savant Walters, we can further notice that the CMS has a failover cmdlet, but no failback cmdlet.
You will make new ones here in the next section and they will be better as they will not reference the original CMS master (pre failure) as the master or being “active”.

Fix me!

From the new master run:

• export-csconfiguration (we are just being thorough, you should not need this file for this exercise)
• export-csLISconfiguration (ditto)
• Place your new exports where you can use them in case you don’t already have them, and then throw them away after the next time your backup captures that data.  If you get the end of all this, and invoke a failback to the “old” master, you can throw the exports away in that case also.  You do have a plan, right? 
• stop services on “new” master:  FTA, LyncBackup, Master, Replica

Bring the “old” master back online.

• From the “old” master, stop services:  FTA, LyncBackup, Master, Replica

From the “new” master:

• install-csdatabase –centralmanagementdatabase –sqlserverfqdn sfbse.tsoorad.net –sqlinstancename RTC –clean –verbose

From the “old” master,

• start the SfB deployment wizard
• Run Step 1 (install Local Configuration Store)
• Run Step 2 (Setup or Remove
• Start the services stopped earlier BUT DO NOT START MASTER

From the “new” master:

• invoke-csbackupservicesync –poolfqdn sfbse2.tsoorad.net

Wait a bit, then run through:
Get-CsManagementConnection (should show “new” master)
Get-CsService –CentralManagement
Get-CsManagementStoreReplicationStatus –CentralManagementStoreStatus
Here we are, fixed.  Note that the “new” master is still the master, but now the “old” master thinks it is no longer the master, but subordinate to the “new” master.  All this is just fine.  We don’t care WHERE or WHO holds CMS master, as long as we have a posture where we can read/write topology.

At this point, you could do another Invoke-CsmMnagementServerFailover and get the CMS back over to the “old” master…if you are into consistency like me, then that is what you will do.  If you are like others, you can leave the CMS on the “new” master, and everything will be fine.

Summary

Seeing as how there is no failback cmdlet, could it possibly be that this is all by design, and was never properly documented on the way out of Microsoft-land?
Empirically, as long as both SE pool pair members are up, the CMS failover process is just fine.  If the “old” master is down, things go bad quick and the prudent admin will be prepared to handle that scenario – however remote the possibility may be.
If your CMS fails, then you could be failing also.  Invoke-CsManagementServerFailover is wonderful, provided all the players are still running.  Not so hot when the the existing master is no longer available.  This process will get you in a posture of success; is repeatable, and is not too onerous.  Ergo, we have something i can feel somewhat good about taking to the customer.

YMMV

SE Pool Pair w/CMS still borked

Background

SfB pool pair is a method by which SfB attains either local (inside the DC), or potentially globally, but at least to a second DC, disaster recovery.  I say a “measure” because the end result is not totally automatic, there is an established procedure to complete the failover process.  Overall, considering the vagaries of networks and host environments, pool pair is a very viable option.

The problem that surfaced for me is that I have been heads down on two projects since 4Q 2016.  Outside of some side trips for pre-sales, and minor involvement with long-term customers, two Enterprise Pool projects have held my attention for the last 12 months or so.  And during that time frame, I swear I saw traffic that indicated the following issue/bug/problem/glaring oversight was fixed in a CU.  Apparently, my myopic world of EE servers and pool pair worked just fine while the SE world floundered on.

Scenario

Brand new set of SE servers, one per DC.  2012R2 patched up to current, SfB from RTM followed by May 2017 CU (on a tangential note – has anyone else noticed that there have been no on-premises updates for 9 months?).  Change topology to create an SE-SE pool pair.  CMS was on the DC1 SE.  This of course, is now present on both SE’s.  All of the pool pair setup seemed to go normally.

At this point, we need to test things for failover/failback. Document that procedure for our specific environment and we have  accomplished a set of deliverables.  Except that it don’t work.  At this point we will wait while everyone brushes up on Invoke-CsManagementServerFailover and Invoke-CsPoolFailOver.

What we did and observed

First, we failed the sfbse.tsoorad.net server so as to simulate the DC failure.  No datacenter, no CMS host. We need to come live in the second data center on the pool pair partner (sfbse2.tsoorad.net!.  Note that the invoke-csmanagementserverfailover command line has gotten a bit longer than just the cmdlet.  You can use just the cmdlet from the new target, and it will work… but with the existing CMS master down, the cmdlet needs to have some other parameters.

We start the entire thing by doing a CMS failover.  Everything goes as advertised.  Replication occurs after the move as expected.  “Get-CsManagementStoreReplicationStatus –CentralManagementStoreStatus” contains the anticipated values in the anticipated attributes.  So far, so good.

Screeching halt on trying to move the CMS back.  Uh oh.  Not having CMS is a bad thing.  Lucky me, I ALWAYS  have a copy of the CMS handy.  You should have some knowledge about export-csonfiguration and export-cslisconfiguration and what those files are good for.  Here is an example of what they are good for!  (while you are at it, you may want to refresh yourself on export-csrgsconfiguration and import-csrgsconfiguration (seeing as how we are talking about doing pool failover)).

Having resolved the outage that caused us to failover, we need/want to failback to the original CMS holder…

While we have a good starting point, that is where the goodness stops.

This is scenario #3 as shown here. Not good.  Especially as how the stated cause for scenario 3 is #1 or #2, and we match neither.  Oh joy.

Fast forward through a lot of verifying and google-fu. Rather than write up what fellow MVP Mark Vale has already done so well… Let’s just wait while you read all of his write up.  You can access that right here. 

Recovery

It is entirely correct to observe that I did not have the exact SAME symptoms, but enough matched up that I tried his 19 (!) step fix.  While my errors did not match the KB article, my symptoms and outcomes come close to matching the Mark Vale article.  And his fix worked for me.

As a follow-up test, once I recovered, I stupidly was Mr. Clever and redid the entire test.  Guess what?  Different results.  Nice and consistent inconsistency.  And then, in a fit of sheer madness, I did it again.  And got yet a different set of outcomes.  However, I will note that in each case, Mr. Vale’s 19 step program worked.  Sometimes I had to work through the steps more than once to resolve things. 

Conclusion

As Mr. Vale opines, make sure you have a good copy of the CMS via the export-csconfiguration method.  Saves your bacon, less filling, and works.  If all else fails, you can go full DR mode and just restore the CMS from scratch (albeit with a CMS that has details :) ).  If you read the Vale process, what is being done there is a manual recovery of the CMS.

Also, note that doing the CMS failover with the existing master still available is no issue.  Things work as expected.  The problem starts when you try an actual failure scenario and the original master never gets updated and the environment ends up in a quasi-split-brain posture.  You may get lucky and have the new master be intact.  At least one of my tests ended up with the new master being orphaned also.  At which point you need to be ready for serious surgery.

I think the next test will be inserting portions of the 19 steps into the failover scenario BEFORE trying to get back to the original master. And remember, your backup/recovery process is only as good as the last time you tested successfully.

I think this whole thing falls squarely into the “you must be kidding me” category; Alas!  I am not.

YMMV

Skype for Business and Teams features Comparison

Fellow MVP Luca Vitali crreated a great feature comparison chart.  You can see it right here.

YMMV