About Me

My photo
TsooRad is a blog for John Weber. John is a Skype for Business MVP (2015-2016) - before that, a Lync Server MVP (2010-2014). My day job is titled "Technical Lead, MS UC" - I work with an awesome group of people at CDW, LLC. I’ve been at this gig in one fashion or another since 1988 - starting with desktops (remember Z-248’s?) and now I am in Portland, Oregon. I focus on collaboration and infrastructure. This means Exchange of all flavors, Skype, LCS/OCS/Lync, Windows, business process, and learning new stuff. I have a variety of interests - some of which may rear their ugly head in this forum. I have a variety of certifications dating back to Novell CNE and working up through the Microsoft MCP stack to MCITP multiple times. FWIW, I am on my third career - ex-USMC, retired US Army. I have a fancy MBA. One of these days, I intend to start teaching. The opinions expressed on this blog are mine and mine alone.


New SfB SE won’t start

Usually I see this problem with EE pools, but in this case I have now seen it with two different SE installs.

The Problem

RTCSRV won’t start.  It just sits there for a bit, like 10 minutes and claims it is “Starting”.  Then the service status goes to “Stopped”.

Nothing in the event log, nothing shows in Powershell.  If you try to start from services.msc, it just sits there.  Nothing.

How did we get here?

A fairly locked down environment.  OK.  A severely locked down environment.  The tin-foil hat types have found a home in this place.  New install – new as in greenfield deployment.  Standard Edition installs, and before starting services for the first time, we ran the February 2017 CU into place.  All of that seemed fairly normal.

But the service wont’ start.

The Fix

Again, I usually see this with EE pools, but here is what fixed it:

Reset-CsPoolRegistrarState –PoolFqdn poolfqdn.domain.com -ResetType FullReset



Skype Test Matrix

As part of a project, Thaddeus Kurowski (CDW) and I put together a Skype test matrix to ensure that the implementation worked as designed/expected.

You may find it useful as well.



Skype Edge Server and 2:1 NAT

This morning, we resolved an issue that I have never seen before, and hope that I never do.

The Background

I tell customers during design sessions that if there are existing network issues, Skype (or Lync) is going to find them.  If there is something a bit wonky, we are going to discover the wonkiness.  And here we go.

Skype edge with 1:1 Nat.  Public IP is 71.16.x.x.  Edge server is doing the classic 3 IP thing.  Remote logins are fine.  Everything seems to be ducky.  Except we cannot talk outbound. 

Go check all the network again.  Looks good. Check the topology, servers, IP assignments, paths.  All good.  Certificates, the common culprit behind one-way federation and presence look good.  We are now scratching our heads.  We know now we are looking at something wonky, but what?

The Fix

I was under the impression that 1:1 NAT is 1:1.  But it turns out that a Watchguard Firebox is capable to doing 2:1 NAT.  Inbound to the Edge server worked because the firewall had 1:1 NAT from public to DMZ VLAN.  Edge trace logs showed subscriptions and connections timing out on the far side.  The connections were being made, just no return traffic.  No SYN.  Telnet client testing outbound from the edge server on 5061 ad 443 worked.  Clearly inbound connections were working or there would be no remote logins.

As long as the traffic originated from outside the organization, things worked fine and the Edge server, via the 1:1 NAT was responding as expected to the source IP.  But traffic originating from INSIDE the organization was failing.  One way presence, presence unknown, cannot send to user, etc.  Apparently…

…according to www.ipchicken, the Watchguard was sending all traffic from the DMZ external VLAN out via a completely separate set of addresses!  HUH?  Whaaaaat?  So inbound would work, but outbound went out on a separate address?

So their firewall guy fixed that, we are back to 1:1 NAT and all is good. Something to be aware of, eh? Go figure.



Inbound Call Failures due to TCP configuration

I will not attempt to embellish this content past commenting that this call failure is not common.  I have rarely seen it, most likely because my implementation practice for upgrades is to match system settings before testing.

Having said that, I think I would have thought the initial setup described here would have worked.  But apparently not.  Inbound calls follow the original port.  Something to be aware of.

Thanks to Josh Walters, CDW Senior Consulting Engineer for writing this up for us.



Customer is deploying a new 3-node Skype for Business Enterprise Pool to replace their existing 2-node Lync 2010 Enterprise pool.  Enterprise voice is enabled in Lync 2010 and Lync call traffic is directed inbound from their PRI and delivered to an Avaya Session Manager appliance, then it is delivered to Lync.  Internal call flow functions as below:

PRI --> Avaya Aura System Manager --> Lync 2010 Enterprise Pool

After deploying the new Skype for Business FE Enterprise Pool, Edge Pool, and Back-End we decided to migrate a test user who was enabled for Enterprise Voice to the new Skype for Business Pool and test call flow with the new infrastructure.  The new expected call flow should function as below:

PRI --> Avaya Aura System Manager --> Lync 2010 Enterprise Pool --> Skype for Business Enterprise Pool

After moving the user, the user was able to successfully place an outbound call to both internal and external recipients but was unable to receive an inbound call.  When attempting to dial the Line# for the migrated user we were being routed directly to Voicemail (Exchange 2010 Unified Messaging).  What gives? 

Inbound Traffic

Avaya Aura System Manager --TCP 5060--> Lync 2010 Enterprise --TCP 5060--> Skype for Business Enterprise

Outbound Traffic

Skype for Business --TCP 5060 or TLS 5067--> Lync 2010 Enterprise --TCP 5060 or TLS 5067--> Avaya Aura System Manager

Well, what we found was that Avaya was routing SIP traffic to Lync 2010 using TCP port 5060 only (as seen above).  When Lync 2010 received the SIP request it attempted to route the traffic to the Skype for Business pool where the user is homed and it tried to use the same port it received the traffic on, but we had not yet activated TCP on the Skype for Business pool for Mediation.  The Skype for Business pool was therefore rejecting the traffic and then sending the call to Voicemail. 

The fix:  Enable TCP (and make sure to use the correct port for YOUR environment) so that the Skype for Business pool is listening for traffic on said port.   After enabling TCP 5060 on the Mediation Server (Collocated) all inbound call routing for the user started working. 




Reverse O365 SfBO Migration Failure

The Scenario

Existing Office 365 tenant successfully using SfBO. Exchange on-premises.  Azure AD Connect version unknown, but up and functional  PBX with voice mail on-premises. We extended schema and installed SfB on-premises with Edge.  Modified the firewall to specification and attempted to get into hybrid. 

DNS mods we easy. Creating a test user and synching up to O365 went fine.  Enabling the test user for SfB went fine.  Another AAD sync and we were in business.  Moving the test user to O365 (so we could test moving back to on-premises) went just fine. And there the problems began.  Attempts to move the user back to on-premises failed with the following non-help message:

PS C:\Source\scripts> move-csuser -Identity sfb.test3@domain.com -Target domain-sfbfe01.domain.com -Credential $cred –HostedMigrationOverrideUrl https://admin0a.online.lync.com/HostedMigration/hostedmigrationservice.svc -Verbose
VERBOSE: CN=sfb test3,OU=hometown_Users,OU=domain_Users,DC=domain,DC=com

[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"): Y
VERBOSE: Validating parameters for move operation.
VERBOSE: Calculating new server information for user [domain-sfbfe01.domain.com].
VERBOSE: Moving user [sip:sfbtest3@domain.com] across deployments.
VERBOSE: Creating source external move endpoint.
VERBOSE: Validating the hosted migration override URL provided:
VERBOSE: Retrieving web ticket URL.
VERBOSE: Retrieving live id token.
VERBOSE: Initializing source external move endpoint.
VERBOSE: Creating target external move endpoint.
VERBOSE: Initializing source external move endpoint.
VERBOSE: Validating user [sip:sfbtest3@domain.com] online, for on premises to online move.
move-csuser : I
ndex was outside the bounds of the array.
At line:1 char:1
+ move-csuser -Identity sfb.test3@domain.com -Target domain-sfbfe01.domain.com -Credenti ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (CN=sfb test3,OU...,DC=domain,DC=com:OCSADUser) [Move-CsUser], IndexOutO
    + FullyQualifiedErrorId : MoveError,Microsoft.Rtc.Management.AD.Cmdlets.MoveOcsUserCmdlet

"Index was outside the bounds of the array."

You know how many hits googlepedia produces for that?  None of them helpful.  So we triple-checked our work.  Reviewing the overall picture, it was apparent that there was some issue with the on-premises environment, but everything we looked at came up good.

The Root Cause

The root cause was that Azure AD Connect was installed and configured BEFORE the extending schema for SfB.  As it turns out in the end, Azure AD Connect does not refresh schema very well, if at all, unless you tell it to. 

And even then, maybe not. There is a button inside the missclient (Synchronization Service Manager) that SAYS it will do it.  I mean, it clearly says “refresh schema”


…and the following message sure says it will…


But, guess what, that is not the case.

As you can probably guess, the root issue causing our migration failure was that the AAD Connect had no knowledge of the SfB attributes coming in with the online user.  Now, I would have thought they would have seeing as how we were successful in installing SfB, creating a good on-premises user, and moving that user up into the tenant.  But no.

Interesting side note is that once we twigged onto the schema concept, using the button on AAD connector populated SOME valiues – we could see them.  But still moving back to on-premises failed.

The Fix

It seems that if you run "C:\Program Files\Microsoft Azure Active Directory Connect\AzureADConnect.exe", you get a few options. Specifically, take a look at the third option from the top.


I do not pretend to know the difference between “refresh schema” in one location as opposed to the other, but I do know that running the “refresh directory schema” from this location, followed by a full synch on both connectors resolved our failed user moves.

Keeping your Azure AD Connect up to date might be helpful also and in theory the reinstallation process will trigger a schema refresh.  You can get a clean copy of that installer right here.

Of course, once you know what to look for, there is this also.



AudioCodes UC 3.0.x Office 365 MFA support


I feel a bit like Steve Martin

AudioCodes is due, very shortly I hope, to publish new firmware for the 440HD and 450HD phones (dare I hope for the 430HD also? 405? 405HD? 420HD?) that enables the device to do a web sign-in to an MFA-enabled Office 365 tenant account.  Wow, that was one long sentence.  My English prof at St Thomas Aquinas would beat me about the head and shoulders.  However, there it is.

Let’s walk through this process.

Update the firmware on the phone device. How you do that is up to you.  Personally, I used my IP Phone Manager Express.

My 440HD at, my 450HD is at  After getting the firmware updated, both devices appeared to be the same.  I am sure there is some detail that I did not notice, but they look the same to me.

Open either the web interface, or the phone screen, and start the sign-in process.



phone web interface:


select the web sign-in option….


What results from either method is this:




Inside the red box (which you will not get on your phone or browser screen) is the two critical pieces of information to complete the login process.  First is the URL http://aka.ms/sphone.  Go there with your code.

The code is not case sensitive.

So you go to the indicated URL and follow the prompts, then enter your code.  You will see where I did lower case while the phone and the browser GUI both indicated caps.

What follows is a bit round-about – but you get thrown into the office portal login…


and a redirect to the corporate AD sign-in…


and after working my way through the MFA routine, I get this:


after entering the requisite code… remember, not case sensitive, the page magically morphs to this:


Select continue, because I am assuming you WANT to get the device to work…and you get this


For you eagle-eye readers, you will note that now this page, which appears to look just like a few steps before now says I am signed in.  How nice.  Observing the device, I note that it SAYS it is logged in, but you know, it still looks pretty unusable at this point.  So, click on your account that was signed in…



and now the device itself looks like the following – well, it will in a bit – patience padiwan!


BTW, you have 15 minutes to complete the web sign-in gymkhana.  If you blow the 15 minute limit, you will need to start over.


I am told, by an source who only spoke on the condition of anonymity (this makes me equal to all the reporters in any nation’s capitol), that we can expect this new firmware code to be out in the wild sometime around the end of Q1 2017.



Server 2012 R2 KB2919355–WTF?

Last week, I innocently decided to build myself a new Server 2012 R2 image – and then sysprep it so I could easily spin up a new host for whatever I needed.

Yes, I know I could use Server 2016 – but the vast majority of my customers are using 2012 R2 – and what good is a lab exercise if it does not reflect what you will be doing in production?  So, off I go to build myself my squeaky clean image.

The install went so easy.  And the then update nightmare begins.  I have no idea why it has to be so &^%#$@! difficult.  It’s not like I am trying to do something that is way out there.  I just want to get all the operating system updates applicable up to and including today.

As we should know by now, Server 2012 R2 will go through multiple iterations of updates for a variety of reasons.  One of them being what some people called SP1 to R2 – specifically KB2919355.  Roughly 800MB of (eh?) goodness.  After that is another 190+ updates.

For my new image, KB2919355 refused to be seen, let alone install.  Dang.  Last time this happened I had to throw the server away.  Oddly, and why I am ranting today, is that the next server build, like 5 minutes later, went right through with zero issues.  This time, I resolved to figure it out rather than give in. 

Here is what I found.  This may or may not work for you.  It may or may not trip your trigger – you may just wish to throw things away and start the Server 2012 R2 Update Roulette game over again.

After doing some reading about the well-known issue that is KB2919355, I downloaded the components of the KB separately.  https://www.microsoft.com/en-us/download/details.aspx?id=42334. I also downloaded KB2919442 separately from here: https://www.microsoft.com/en-us/download/details.aspx?id=42162.

Then I installed/ran them in the following order:

  • kb2932046
  • kb2934018
  • kb2937592
  • kb2938439
  • kb2959977
  • kb2919442
  • clearcompressionflag.exe
  • Chant, light the candles, and spatter the chicken blood.  Reboot
  • kb2919355

Oh joy.  Only 190 more to go.





SfB Persistent Chat ChannelService.exe high CPU

Twice in the last two weeks, I have seen an SfB Persistent Chat server go bonkers over a topology publish action.  Specifically, it would seem that the topology publish action caused channelservice.exe to peg the CPU at 100% with the predictable result of a very sluggish server.

Tangential input and possibly related data points:

The fix was easy enough, in one case I did a stop-cswindowsservice followed by start-cswindowsservice.  In the other case I had to boot the box because PowerShell opened but never responded to any input.

Such is life.



SfB Online and AudioCodes handsets

As part of another process, I was browsing through the Skype OIP and Lync OIP sites…and noticed that only the AudioCodes 440HD was qualified for SfB Online.

Odd, says I.  My 450HD just worked.  So, I commenced to testing with the 405 and the 420HD that I happened to have handy.  Here is my firmware load per phone:


I then proceeded to use the web interface on each handset to modify the login to the UPN of an SfB online user.

SfBO login with each model shown was successful. Note the firmware version per handset. Test calls worked.  Transfer worked.  Holds worked.  All the basic features that I use worked just fine.

While the 405,420HD, and 450HD do not show on the SfBO OIP, they clearly function as expected.

Nice to know, eh?


YADR–AudioCodes 450HD

AudioCodes has a new phone, the 450HD – complete with a touch color screen. I have been using the 450HD as my desk handset now for a few weeks, and I like it a lot.  Form, fit, function, the 450HD has it all.  I am not sure if you can actually lay YOUR hands on one of these gems, but I am sure you will be able to soon.  In the meantime, let me present my opinion, so you can start salivating.


What comes in the box?  I have an “optional” model number, because I got the AC adapter.  Other than that, this is what I got.


A wall mount?  Nice touch.  I did a project one time where the client had to go out and have all new wall mounts custom made for their new IP phones (different manufacturer).

Build Quality

As I have mentioned before, AudioCodes has great build quality.  The 450HD continues this tradition.  Very nice. The manual buttons feel good, and the touch screen responds well.  After I realized that I had to remove the screen protector thingy, the touch screen went from “responds well” to “most excellent.”

The Screen and Controls


Give it an extension + PIN or futz through getting your URI entered, and in you go. Once you are logged in you see this:


Notice the soft keys on the left.  There is four more available on the right side of the screen.  These soft keys are programmable via the web interface, the ini file for each phone, or right on the phone itself.  Using the phone itself also allows you to do a directory lookup and choose from that so no typing needed. choosing the BLF options gives you presence on the contact…


…works with SfBO users…Wiley Coyote is SfBO and offline, Martin Luther is SfBO and available, while Chicken Hawk is on-premises and has gone into away status.  Works with federated contacts also.


Oddly, or perhaps by design, the user cannot change the button assignments unless the admin gives them access to the web interface.  And if I was to logoff, give you the phone, and you login to your domain with your user, the soft key assignments are then available to you too.  That might be good, that might be bad.  Something to consider if you ever have to decommission one of these units. I have elevated this issue to AudioCodes as I feel that these soft-buttons should follow the user, not the phone itself.

Having said that, I like the soft keys.  One button dialing is right up my alley.

Skype Integration

We have to talk about Skype – that’s why we’re here!  SfB logins were totally painless.  Extension + PIN code flew right through. I already mentioned the programmable buttons that work so well – and clearly the 450HD is working in concert with SfB for presence, making calls, directory lookups, etc.

Login with username and PW forces using the keypad with multiple pushes of each number to scroll through letters and symbols, etc.  YUK.  Where is the QWERTY keyboard this unit is screaming for?  I am told that it is coming.  In the meantime, I suggest the web interface is mucho better if your organization does username login.  I always advise my customers to use ext= format in SfB/Lync for this very reason (Not workable if you are SfBO).

Other than that, the 450HD is ready for SfB right out of the box, logs right in, functions as expected in a totally flawless manner.  The 450HD picked up the DHCP options, discovered the environment, and asked for an extension and PIN.  And connected.  Perfect.

With the current firmware (, the 450HD will also log straight into an O365 account with zero squabbles.   Martin.Luther@tsoorad.net is a synced account to O365 enabled for mail and SfBO with a PSTN number assigned from Microsoft.  Logs in. Perfect.


Note that the phone did not get calendar connected.  I am assured that this will be resolved by EOM Jan 2017.

Also note that with a different user, the soft keys remain the same…

Calls out, calls in, audio quality with speaker or handset is most excellent.  I am about 1/2 deaf and I had no issues with volume or clarity. 

I just realized I used the word “perfect” twice in this section.  I was going to change that, but then I realized, it is the right word.  Live with it.


Download the BToE client from AudioCodes…extract and install..


…and then get your pairing code from the phone itself.


BToE integration went very smoothly, as expected.  In addition, I used a virtual machine that is guest on a VMware host that has no audio, but with BToE that VM cranked right up to using the 450HD as an audio source.  Mo’ perfect.


The 450HD web interface is standard, totally functional AudioCodes fare.  You can probably figure it all out by just ratting through it without reading a thing.  AudioCodes has not yet published 450HD specific admin or user guides, but I am told that they are mere weeks away from providing a lovely document telling you just how to configure each and every nuance of this new product.

IPP Manager

The 450HD is fully supported by the IPP Manager.  If it works in the IPP Express version, then it will work in the full IPP version also.  So nice.

What’s Missing

I have already pointed out the QWERTY keyboard and the calendar connection to the O365 tenant account. On the phone, select the “MENU” button, select “settings” and then scroll down a bit, and the LCD Contrast and Backlight Timeout are “not implemented” (yet) – but other than that, the 450HD I have is ready for prime time.

And considering that I was shipped a preview beta unit, gees, only three things?  And I am told both are coming before GA.


AudioCodes has a color phone – with some very nice features – ready to go with SfB on-premises and also with SfB online.  Clean, functional, well-built, great audio.  POE or wall power, pass through switch for your desktop. USB for headset or hockey puck.Did I mention the color screen?  Did I mention it worked OOBE with me doing nothing?  Works with SfB Online (yes, I mention that twice in one paragraph).

Considering that this unit is not in GA yet – what was that word up above?  Oh yes, it starts with a “P”.

You can get one here.