SQL 2014 AlwaysOn and Skype for Business Server 2015

Let's do SQL AlwaysOn Availability Groups for Skype for Business Server 2015

If your SfB project is heading into the "I need SfB to be highly available" realm, then you need to start investigating what it is going to take to bring just ONE three node pool of SfB front end servers into being.  Beyond the obvious need for three servers to form the SfB Enterprise Pool, there is no point in doing an SFB EE Pool and have only one SQL server behind it - you will have created a single point of failure in a critical system component - to whit, the supporting backend SQL databases that the pool members use for re-hydration.

SfB is not totally different from Lync Server 2013 in this regard; however, one key difference is that now SfB officially supports using SQL Server AlwaysOn Availability Group (AG) as a supported SQL backend - this in addition to standalones, clusters, and mirrors.  If you have a SQL team that wants to use AG as the standard build, then that is what you wil do, eh?  The point to this article is to walk through a simple EE pool with AG installation into an existing Lync 2013 environment to highlight some of the lessons learned over the past month or so.

I am not including screen caps of every step, there are plenty of guides out there in blog-land to walk you through SQL and SfB installs.

Planning
Before cranking up some VM space and mounting the ISO's, you may want to consider some needed data points.  DNS, IP space, and construction details are always nice to know ahead of time. Aside from five host servers, their FQDN's and appropriate VM host space, here are a few items to ponder:

• Identify the Windows Failover Cluster (WFC) cluster name and IP - do you need to read up on WFC before you start?
• Identify FSW location - do you need to read up on File Share Witness before you start?
• Identify a SQL file share for enabling the AG itself.  SQL is going to want a share that it can use to shuttle the initial database backups into so that it can copy them onto (into?) the target secondary node.  Why it just don't do it direct is beyond my ken; I just do what I'm told! 
• Identify the SQL AG group listener FQDN and IP - maybe you should do some background on this subject too.  It certainly would have helped me a bit.
• Identify the SQL service account - don't try this with "network services" or you will be assigning certificates to logins
• Firewalls on the SQL servers need to opened for inbound traffic to flow properly.  1433 and 5022 TCP; 1434 UDP.
• SQL Database location - these must be IDENTICAL between SQL AG nodes.

To quickly summarize, we are talking about:

• 3 FQDNs and IP to match for the SfB EE pool (maybe more!)
• 2 FQDNs and IP to match for our SQL 2014 Enterprise Edition nodes
• 1 FQDN and IP for the WFC cluster
• 1 FQDN and IP for the SQL AG listener
• Service accounts
• File locations
• Firewalls
• Database location

Setup

Prepare two servers for SQL.  I used 2 cores and 8GB RAM.  Because I am only hosting SfB databases on these servers, I used 200GB for drive C and will put everything on the same drive. You may wish to follow a more esoteric construction with separate drives, perhaps SAN-based, and you may need way more space than that especially if your database team is using these servers for other purposes.  You may even have to live with the database team telling you what and where, and by whom.  If you are having a SQL team provide you an instance and space, then make SURE of the instance name, your permissions to that instance, and the space your environment will need. Permissions will be important.
And of course, you will need three servers for your SfB pool.

Install SQLAlwaysOn-A operating system (I used server 2012 R2)
Install WFC via server manager
Install .net 3.5
Lather, rinse, repeat for SQLAlwaysOn-B

Patch and then patch again. Dang. You would think doing updates ONCE would be sufficient.  But...no.

Install SQL 2014 Enterprise  - use the SQL Service account for all services.  And you might as well make sure the SQL agent is running.  If you don't, the ensuing SfB install will complain about it.

FWIW, I also installed SSRS on both nodes.  No, SSRS cannot cluster or failover, but you CAN AG the databases, and install SfB templates to each node, and then, if needed, use the second node for your reports.

Configure WFC (see http://stevenpoitras.com/2014/02/microsoft-failover-cluster-configuration-nutanix/)
Configure FSW quorum - you will need FSW from above.

Configure WFC cluster listener with static IP.

Move resources (like change the active) between nodes - this verifies that both nodes can r/w both DNS and FSW and that either node can be listener.

Leave the WFC cluster active on whichever node you want to be the primary - I use SQLAlwaysOn-A for this.
Ensure that Windows firewall sql inbound rules are done:  tcp 1433, 5022, udp 1434

Some Lync 2013 work

Official Lync 2013 documentation on TechNet

From 2013:

Export-csconfiguration
Export-cslisconfiguration
Open topology builder from 2013 and save the tbxml, twice, just to be really sure.

Hey, we get to SfB yet?

In case you need to brush up on all of this…

Install SfB admin tools on something other than an existing Lync 2013 server
Open topology from SfB tools.  Save the tbxml.  Twice.  And not with the same file names you used for saving the 2013 version. No turning back now unless you have the tbxml files and the exports.
You've just upgraded the topology.  You did the 2013 tbxml saves and the configuration exports, right?

Configure topology.  In my case, a net new EE pool.
Make sure that your primary node SQL server has the database folder already defined, and that you remember what you called it; no point in having the Topology Builder choose defaults if you don't have to.  Or worse, use the SQL defaults and have your databases buried about 15 levels down.  I used c:\sfbdata.

What you need to do is ensure that the AG listener is defined up top, but the SQLAlwaysOn-A node is defined down below.  You go back and change this later... But here is the reason:  There is no AG yet, there cannot be an AG created until there are databases.  And the databases don’t get created until topology publishes.  So put the proposed listener at the top entry, and the specific first node down at the bottom.

Publish topology which will install the databases on sqlalwayson-A.tsoorad.net, which you need to do to get the AG to work - cannot make an AG without having a Database to work with!
When the topology publishes, it will flip up a SQL database configure screen.  I ALWAYS put my databases somewhere defined by my project. I NEVER let SQL just throw things around.  In this case, we are using c:\sfbdata for all SfB database work.  This will include the future CMS and the Persistent Chat.

At this point, I paused and did

"install-csdatabase -centralmanagementdatabase -sqlserverfqdn SQLAlwaysOn-A.tsoorad.net -databasepaths c:\sfbdata"

- because I know I am going to be moving the CMS to the new pool at some point and I want those two databases (xds and lis) to be part of the AG.  If you wait until later, you will be doing parts of this all over again. Then I paused again for persistent chat databases:

"Install-CsDatabase -DatabaseType persistentchat -SqlServerFqdn SQLAlwaysOn-A.tsoorad.net -DatabasePaths c:\sfbdata -v"

and

"Install-CsDatabase -DatabaseType persistentchatcompliance -SqlServerFqdn SQLAlwaysOn-A.tsoorad.net -DatabasePaths c:\sfbdata -v"

- same reasoning as before.  Tastes great, less filling.

Back to SQL...
OK, now we have SfB databases!

Set all SfB databases to FULL backup – some SQL BrightBoy probably has some zippy tsql to do this; being archaic, I do it onesy-twosy.

Backup all databases using SQL Management Studio - just accept the default location - these are not really your backups, this is just a step to ensure that the AG forms properly. AG requires the databases to be backed up first.  Yes, same comment here for the backups.  I am sure there is some zippy method that I have never bothered learning.

Robocopy the database file structure  - robocopy is your friend -  the file structure must be exact between servers
If you installed SSRS on both nodes, so go remove reportserver db and reportservertempdb from SQLAlwaysOn-B.tsoorad.net or you won't be able to add those databases to the AG as the database locations will have files in them and that is a no-no.  Luckily, you can just delete the databases from the Management Studio.
Configure AG (see http://stevenpoitras.com/2014/02/configure-sql-db-availability-group/)

You may need to add your cluster nodes by NETBIOS to sys.server by doing

sp_addlinkedserver @server=”serverNETBIOS’'

I don’t know that this is an absolute requirement, but adding my nodes before trying to create the AG seemed to make some errors go away.  At any rate, I have done it ever since as a matter of rote.  YMMV.

SQL Permissions on Node 2
I have the topology already published, so now go look at the databases that you have already made members of the AG.  The AG retains the security logins on the database when it establishes the secondary copy; however the same mechanism does NOT replicate the Master.do database logins;  Read this as your logins to the secondary (now primary) will fail for the various RTC and CS groups.  You will need to work out a method to get the security logins AND their respective SIDS along with lining up those SIDs to the respective database permissions.  One option to perform this work (and have it done right the first time) is to run a script such as this (http://www.sqlsoldier.com/wp/sqlserver/transferring-logins-to-a-database-mirror).

SfB Install
Initial install for the first EE pool members.
Had a nasty time with the prereq script.  Had to remount the original O/S ISO before the install-windowsfeature -source would work :(  I had a copy of the \sxs local to the machine, but the installer did not like it for some reason.  Do Windows updates until it don’t wupdate no mo!  Don’t forget kb2982006 is a "hotfix available" special. Until you run the script to install all the Skype for Business operating system prerequisites, the hotfix will refuse to install, so you need to run windows updates AFTER the prerequisite script.
Continued with install.  Started seeing issues across the new pool members of speech files not installing.    Nothing clears it out. Arbitrary reboot of all EE pool members fixed it.  (This install is starting to turn into a nightmare)(I never had these odd issues with Lync 2013, or with pre-GA SfB either!)
Finally have EE members installing as expected.  Why me? All three EE members acted differently during install.  Huh?  The servers are as close to identical as we can get them.  Installed from same source.  Used all default locations.  Patched from WSUS source.  Each server had the same number of updates in the same order.  WTFO.

Persistent Chat
We chose to install Pchat to a single server, but use the existing SQL BE and collocate with the FE Pool databases. See above, eh?

FINALLY
Go back to the topology builder and change the SQL definition for the EE pool  - remember up there where we set SQL to the AG AND the single node?  Now that we have the pool up we need to tell the pool to talk to the SQL AG listener, not just the one node...

Well, we got done. IMHO, way too much manual effort, but apparently with SQL being the way it is, the product group was forced into a corner.  Maybe the future holds an automated version of this, but until then,

YMMV

Start-CSpool fails

Are you getting this nasty failure when you attempt to run Start-CsPool on your squeaky new SfB pool?

Did you create your topology and publish it before creating your pool’s host servers?

Did you get a little message about having to re-run Enable-CsTopology after creating the servers?

Did you ignore that little tidbit?

So, before you go much further, open a PowerShell session as administrator and run

Enable-CsTopology

and then, on each FE server:

Enable-CsComputer

YMMV

Skype for Business Server 2015 Training

Do you have users clamoring for some training resources to help get them started?

Take a gander at this:  Skype for Business client awareness and readiness resources

672Mb of goodness.

 

YMMV

CSAnalogDevice Dialing

Scenario

Analog devices live on.  Face it. There are some states that REQUIRE analog circuits outside of the VOIP configuration.  New Jersey, for instance, requires that elevators be hardwired. 

So, I was doing this project that had a need for analog devices to do things like open gates, be a parking lot phone, and other nefarious duties.  Suffice it to say, we had to put in several 24 port FXS gateways. 

We got them all configured, ran through the requisite new-csanalogdevice and associated all the devices with their respective gateways, assigned (carefully) DIDs, contact objects, and Dial Plans.  You can see some background on how to setup csanalogdevices here.

The desired dialing action was to use a complete 10-digit dial pattern.  No four, no five, and none of that seven digit stuff.  10 digits was the plan moving forward.  During testing, we noticed that we had to dial 11 digits to get the call to complete. 

Troubleshooting

I immediately went to the configuration of the gateway (AudioCodes MP124) but I found that I already had the “Max Digits in Phone Num” set properly.

As I was performing that useless verification task, I BFO’d on the trace log from the gateway, where we noticed that Lync was refusing the call because there was no normalization!  Huh?

Uh oh

1. The gateways were sending 10 digits as expected, but LYNC was not accepting them.
2. The Get-CSAnalogDevice configuration showed that we had assigned a USER Dial Plan to the objects, which was the correct action.
3. The User Dial Plan had a rule that accepted 10 digits and normalized to e.164.
4. Therefore the CSAnalogDevice had the right set of rules, was sending the correct number of digits as it was told, but was not working.
5. The CSAnalogDevice had the correct dial plan assigned, but the CSAnalogDevice did not like the dial plan scope, so the CSAnalogDevice fell back to using the Global Dial Plan ruleset, which did NOT include a 10 digit normalization rule
6. It turns out that the CSAnalogDevice objects only respect DEFAULT dial plans, not user dial plans.  Our assigned Dial Plan is a user scope dial plan. CSAnalogDevice wants something that can be a default – so pool, site global. 

No, I don’t know why.  I assume it is coding issue, and there is SOME reason that makes no sense to you and me.  I know that you and I consider it a bug, but the developers might well come back with “by design.”  The Lync documentation indicates that a user-level policy will work.  (note that the same documentation indicates that a voice policy needs to be assigned, and I agree, but dang, you can’t make a call without normalization, and the documentation says squatoosh about the dial plan scope level.)

The Fix

The short term fix was to add a 10 digit normalization rule to the Global Dial Plan.  That fixed the analog devices not dialing out correctly. There was joy in Mudville.

For the future, create a site or pool dial plan that has all requisite rules for that site. SBA installations are considered a site for this purpose.

This is current as of the December 2014 Lync 2013 CU.  I hope the SfB release addresses this, but I am not going to hold my breath.  What I intend to do is make sure that every site or pool has a dial plan, and that a user level dial plan is never assigned to a generic device. 

That should hold the barbarians at the gate, eh?

YMMV

SfB/Lync audio failures

 

Adios Audio!

The Issue

I don’t know about anyone else, but I have noticed that since I ran updates into my Windows 8.1 laptop the other week, my audio on my laptop has failed completely at least twice.  Today it quit very annoyingly right before I tried to make a Skype call.  Right off, I thought I had a device (headset) failure, or a mute button pushed, or something like that.  But no.

When this happens, ALL audio is gone.  Not just SfB/Lync.  System sounds, dings, dongs, and other helpful bleeps are simply not happening.  This is a Windows 8.1 operating system thing.

I have a Lenovo T530.  But, some google-fu and some University of Bing show that this is not just my Lenovo.

http://www.lmgtfy.com/?q=windows+8.1+audio+not+working

http://www.bing.com/search?FORM=UWDFTU&PC=SK2F&q=windows+8.1+audio+not+working&src=IE-TopResult&conversationid=

How to fix this? 

Well, rebooting did not seem to work.  But this did:

Start services.msc, go to Windows Audio Endpoint Builder, and restart it.  This will prompt you to accept a restart of the Windows Audio also.  Do that.  On restart, your audio will be back. 

then

then

And you should be good.

Keep in mind that I have now had to do this several times (or at least twice) and while this fixes the issue, the restart does not seem to completely fix the issue, as it has come back at me.

YMMV

Upgrade Lync 2013 SE to SfB SE

So the magic time has arrived.  We have a new Skype for Business 2015 server release.  After downloading the new ISO from your licensing site, or if you have MSDN for your lab, you can upgrade your existing Lync 2013 servers directly!  What a great feature.  I have been waiting for this since … well, a long time.

What needs doing?

I am going to upgrade my Lync 2013 SE.  You may want to review this here first. Then, I think prudence will dictate that you read this next as this is the official documentation on upgrading an existing Lync 2013 server to Skype for Business 2015.

Before I started, I did the February 2015 Lync CU updates to the entire environment, and also, based on our reading, we need the Lync server to be at SQL2012 sp2.  You can get that little download here.

If you are doing an SE like I am, then you need these three commands to accomplish the task:

SQLEXPR_x64_ENU.exe /ACTION=Patch /INSTANCENAME=RTCLOCAL /QS /HIDECONSOLE /IAcceptSQLServerLicenseTerms
SQLEXPR_x64_ENU.exe /ACTION=Patch /INSTANCENAME=LYNCLOCAL /QS /HIDECONSOLE /IAcceptSQLServerLicenseTerms
SQLEXPR_x64_ENU.exe /ACTION=Patch /INSTANCENAME=RTC /QS /HIDECONSOLE /IAcceptSQLServerLicenseTerms

BTW, if you don’t see this window during the second and third command lines, the upgrade did not take… reboot and start with number two…I don’t know why this happens, and rather than burn time figuring it out, I just booted between steps. FWIW, I seem to have “issues” with doing service packs on SQL Express, so I just always reboot between steps.  I am sure there is some elegant way around this that smarter people have figured out, but not this guy.

You can check your success by running this little scriplet: (I got it here)

$inst = (get-itemproperty 'HKLM:\SOFTWARE\Microsoft\Microsoft SQL Server').InstalledInstances
foreach ($i in $inst)
{
   $p = (Get-ItemProperty 'HKLM:\SOFTWARE\Microsoft\Microsoft SQL Server\Instance Names\SQL').$i
   (Get-ItemProperty "HKLM:\SOFTWARE\Microsoft\Microsoft SQL Server\$p\Setup").Edition
   (Get-ItemProperty "HKLM:\SOFTWARE\Microsoft\Microsoft SQL Server\$p\Setup").Version
}

Once you get done with that, you will need to pick a spot to install the SfB Admin tools.  We need to get the topology updated, and once you update topology, the Lync topology builder is going to useless to you; and there is no rolling back.  Unless you export-csconfiguration and save yourself a tbxml and then do a force restore.  I recommend that you take a pause and figure out those two steps and do them now before proceeding.  The last thing you want is to hang yourself out to dry.  A career limiting move to be sure.

OK, so choose a valid source onto which you will install the SfB admin tools.  The prerequisites for this are the same as Lync.  So you can read up on those requirements here. I don’t know about you, but I would pick a Server 2012 R2 instance or possibly an x64 Windows 8 or higher as the prerequisites will already be met.  YMMV.

With the SfB admin tools installed, and an export-csconfiguration and an export-cslisconfiguration AND a pre-SfB topology TBXML saved, we can proceed.

One of the things the official documentation mentions is to ensure that all the services are running on the pool you are upgrading.  You can see the status here (ignore the LYNCBACKUP you see here, I am doing something I am not supposed to…)

With that verified, we can now open Topology Builder using the SfB toolset. And here we are!

Note that the layout looks very similar, and in fact follows along with the 2010-2013 upgrade process – except this time we get to upgrade directly!  Yes!  At this point, Microsoft recommends making a copy of the TBXML you just saved and why you should do it…

If, like me, you have a SQL setup for Archiving and Monitoring that also supports the EE pool, you might want to be safe and update that SQL to SQL 2012 SP2 also.  I did, and had zero issues during this next part. I note this because the pool I am upgrading is doing Archiving and Monitoring, and those databases are being used for the EE pool, and I know that the upgrade process is going to touch those databases, and the minimum to get this done cleanly is SQL 2012 SP2….

Select the pool, select “Upgrade to Skype for Business Server 2015…”

Say yes to the nasty gram…keeping in mind that if you say yes, and things go sideway, you will be discovering the wonderfulness of restoring a topology from your backups…

Note that your ex-Lync 2013 pool is no longer there, but your topology now has a SE under the Skype for Business Server 2015 node…

Publish your new topology using the time-honored, traditional method…

Finally, the magic moment has arrived… we have worked down to Step 5 of the official guidance, and we are looking at this:

Being brave, we will select setup.exe and live with the consequences! Say yes to this first one..

Accept the license agreement…and no, I am not going to screen cap that thing.  Oddly enough, there were no updates to be found.

And voila!  A hidden requirement that is not listed anywhere.  Turns out you need 32 GB of uncompressed NTFS space.  Please wait while I go fix this.  Shouldn’t be too long.  OK, I am back with 25 more GB of virtual drive space…

Apparently there was a prerequisite that I missed (I swear I did them all), but!

As advertised, on reboot the installer just continued forward.

almost done… about 12 minutes…

es finite!

It don’t get easier than this.  Thank you Product Team!

Note that there is a new cmdlet to run to start the pool.  If you are doing EE pools, you should read up on this one.  For maintenance and CU work, you are going to have to change some of your process to include this new cmdlet.

And we are complete…I had a few scary-looking warnings go flying by, but I also understood that I was upgrading the CMS holder, and I sort of expected replication to be borked until the server was fully started.  In the end, patience!

The EE pool, still on 2013, thinks that the CMS is good to go!  Note the version difference between the EE and Edge pool members and the SE…

We even have a new SfB Control Panel!

As in-place upgrades go, this was fairly painless and very straightforward. All functions expecting to be functioning are in fact, working just fine.

YMMV