About Me

My Photo
TsooRad is a blog for John Weber. John is a Skype for Business MVP (2015) - before that, a Lync Server MVP (2010-2014). My day job is titled "Principal Consulting Engineer" - I work with an awesome group of people at CDW, LLC. I’ve been at this gig in one fashion or another since 1988 - starting with desktops (remember Z-248’s?) and now I am in Portland, Oregon. I focus on collaboration and infrastructure. This means Exchange of all flavors, Skype, LCS/OCS/Lync, Windows, business process, and learning new stuff. I have a variety of interests - some of which may rear their ugly head in this forum. I have a variety of certifications dating back to Novell CNE and working up through the Microsoft MCP stack to MCITP multiple times. FWIW, I am on my third career - ex-USMC, retired US Army. I have a fancy MBA. One of these days, I intend to start teaching. The opinions expressed on this blog are mine and mine alone.

2016/06/17

Array Networks vAPV Review

Have you looked at the Lync Server/Skype for Business Server Open Interoperability Program (OIP)  and wondered who some of the qualified providers are?  I know I have.  And then with the advent of virtualization a few years back, I started to ponder whether or not a vendor who is qualified on the hardware list but not on the VM list would function the same.

For the last two months I have been putting the Array Networks vAPV through it’s paces.  For the above reason, I have the virtual edition rather than the physical appliance.  Just so we are on a level playing field, I have been working with this right here. Array has been on the OIP since the Lync 2010 days, and the qualified firmware version for the appliance is 8.x.  The vAPV with which I have been working is 8.5.1.6. I also have a running version of APV.8.6.0.14.  Gee, lots of periods there!

image

One of the things I have discovered is that Array is aimed a tad higher up the food chain than the SMB market.  This is not a bad thing, it just is.  In fact, I can easily make a case for this being a great thing. Price-wise, I would say that they are very competitive even for the SMB customer. Picture a company that is a bit above the SMB space, but below the “enterprise” space.  The needs are the same -  in fact even an SMB can have enterprise needs.  Array Networks has feature set in spades.  Very comprehensive – including some I had not thought were useful until I played with them.

Feature Set

Market-speak

:

The vAPV runs as an entry, small, medium or large virtual application delivery controller on Array’s AVX virtualized appliance to flexibly enable on-demand, full-featured load balancing and application delivery with guaranteed performance.

More Array Networks market speak: 

Local server load balancing, as well as both global server load balancing (GSLB)and link load balancing (LLB) ensure application uptime in the event that servers, ISP links, network segments or data centers become overburdened or unresponsive.

Upfront, this is the list of features that are available:

image

Specifications may be important to you also.  The capacity ramps up quickly – and you can see where I say that the APV is aimed at a higher market segment than the SMB (however, my previous comments still hold).

image

If you don’t like virtual load balancers, then guess what?  There is a range of physicals as indicated.  Bon appetit!  Bottom line, you need to read through this Array Networks information to get the full list of things the vAPV (or APV) can do for you.

What about me?

Well, what about me?  You’ve read the marketing material, I have presented some opinions, but what did I experience during the install and configuration and operation of the vAPV?

Installation

Because I am using VMWare workstation for my lab, I had to convert the OVA download into vmdx format.  No biggie once you have done it.  Here is the list of supported hypervisors.

image

Once that was done, there is a defined install guide that walks through the initial base configuration and then it is on to the good stuff.  Simple.  Worked first time through. What could be better?

In my time with this exercise I screwed up the entire mess.  So the support engineer had me check a box and restart.  The end result is that the box recovered with the previous known good.  So nice.

image

Speed

Speed is not an issue here.  The web UI is very snappy. Content delivery was noticeably faster than competing products.  I like that – I have zero patience for slow stuff when the resources are not being overtaxed and something is slow just because.  Array does not seem to have that problem.  Fast fast fast.  Did I mention content delivery was zippy?  And it got better with compression enabled.  I like it.  “nuff said.

Configuration

One of the issues administrators run into is configuration.  Sometimes just reading the documentation is enough, sometimes formal classroom training is almost a prerequisite to success.  The vAPV fits somewhere in between the two extremes.  Getting into the GUI and poking around was easy enough, and seeing the basic administrative function was clear also.  At that point, neither of my brain cells could figure out what was needed.  Maybe I am just a bit dense.

Luckily, there is this built-in “quick start” feature where there is an outline of the steps needed to do tasks.

image

But still, Array has so much to offer that the granularity gets in the way of the KISS method I like.  Even when using the list shown above, I was at a loss to divine the steps to get things working.  See below for “SUPPORT.”  Configuration will be much like using an AlphabetNumber product.

For you CLI types, Array also supports a full CLI that will allow you to script your configuration and work that way.

VS and Reals and Groups

Here is what we need Mr. vAPV to do for us:

image

Granular is the word of the day here.  Think of each service you need for your system.  You will need by IP by port.  So, 1.1.1.71:443 is NOT the same as 1.1.1.71:8080 – each of those needs to be separate.  And the common name is going to need to be unique also.  <sigh>  Each of these services, just for a DNSLB setup in SfB required the following.  Yes, I have my SE web services going through here also as I wanted to play with the content redirects.  Figure out your naming convention per layer before you start.

image

After you make the real servers up, you then need to lump them into groups.  Reals into groups; groups into virtual services.  Think ahead.  Maybe some UML work might be in order before you start?  Oh yes, you cannot put TCP reals into an HTTP group and hence an HTTP service.  Or, at least I could not figure out a way to do so.

image

Once you get past the real server and group setup, then you need to worry about the virtual services.  For my environment, here is what I came up with.

image

Logs

The APV has logs everywhere.  Which is right handy at times. For instance, you can drill into a group, and down at the bottom there are some basic stats listed for that group.

image

Included in the unusual plethora of admin tools is a copy of the running configuration for you CLI afficianados. And to get you really into it, the display is separated into startup config and running config.  So nice.

Monitoring of the entire mess gets granular as well.  Statistics exist for every layer of the construction.

image

Graphs?

Are you visually oriented?  You want pretty pictures to show that your virtually shiny appliance is in fact doing something?  Well, APV has you covered.

imageimage

And you can choose from the following pre-defined graphs…notice how the pre-defined collection has our configured real, virtual, and policy connections listed.  *I* did not do that – the system saved me my lunch break.

imageimage

You can also make your own definitions.  I put this beauty together in about 30 seconds. Such an artist am I.

image

Content Redirects

Near and dear to my heart is content redirects.  Not every customer of mine has entire class B subnets to work with in their public space – so we try to conserve IPV4 space as much as possible. If you read the link there, you will note that there was syntax involved, and you had to know what you wanted before you started.  Not a problem for ME…might be for an un-initiated hard-charging techie.  APV has you covered here also.  The content re-direct policy stuff in the vAPV was done on the fly, with no syntax needed other than knowing the called URL from the client perspective.  Sa-WEEET!  What you see here took about 10 minutes from start to finish.

image

Routing

Would you like to route stuff around your network?  Would you like some content to go via certain routes?  APV has you covered

(I just realized I have been using that phrase a lot.  But, in truth, APV does have all the bases covered, and then some.  Every time I look at it, there is more to appreciate.  Simply a very well done product that is continuing to evolve and get better.)

Adelante!

There is more in that networking section (basic and advanced) than my little pea-brain comprehends, but I showed this to a few networky-techy-nerd buddies, and then had to clean up drool from the monitor.  Their excitement over the possibilities was palpable.

image

Compression

Would you like some of your content delivery to get compressed and some not?  FWIW, this makes OWA 2016 pop on screen rather than ooze up there. As in a LOT faster.  I did not measure as I have no facility to do accurate measurements – Array claims 500% improvement over non-compressed.  I don’t know about that, but I know OWA flies up on screen.  OOS and OWAS scream into being rather than just oozing.  According to my setup, there has been an “87% compression ratio of compressible data” – whatever that works out into improvement percentage I do not know.  But seat-of-the-pants – mucho mejor.

IPv6 support – NAT64

When enabled, the APV can translate ipv6 to ipv4.  Or ipv4 to ipv6. You can’t mix the two in a group, but you can have both inside and both outside – you just can’t mix the group. I can think where this will come in handy down the road just a bit as (supposedly) the IPV4 pool is now exhausted.

SSL

Yes Matilda, the vAPV does SSL.  My configuration is decrypting and inspecting, then re-encrypting and sending to the real servers.  All faster than you can type about it.  The certificate import process was easy as it took .cer format directly.  I had some moments with the configuration, but read below in “support”  - we got through it.  After having it explained to me in kindergarten terms, even I grasped the simplicity (when you think about it) of how the SSL is handled.

image

image

image

 

Support

Take a look at the wealth of deployment guides here.  The only problem I see on the deployment guide page is that the Lync 2013 guide is for the full load balance solution, whereas I only deploy in that fashion when I am pushed into a corner for some business or technical reason – otherwise I am going to advocate and deploy DNSLB.  In working with Array support engineers, I am told that the SfB documentation will include both methods.

I had some difficulties due to the extreme levels of granularity of the APV. My friendly (he never cursed at me – not even once!)(and I gave him plenty of reasons – there are times I am just stupid beyond belief…) support engineer showed me how to get multiple ports into the virtual service so that you could theoretically define a real service with port 0 and then create virtual services with any ports you want.  so if you had some generic needs, like RPC Endpoint mapper and port 80, you could handle that with one assembly.  Not the most obvious solution set, but when you look at the granularity model, it makes sense.

So they get a frowny, a smiley, AND a straighty.

Sad smileSmileDisappointed smile

I will reiterate, even in the midst of my personal issues, my assigned Array Networks support engineer was extremely helpful and patient. I am not the easiest person to coexist with; whoever that guy is deserves a medal.  If the rest of the folks at Array are anywhere close to this guy, it speaks well for them as a company/staff.

The SfB / Lync Connection

I would not be doing this homework if it was not for wanting to make my customer’s Skype deployments better.  That’s the bottom line.  After configuring the APV as shown above to match the environment shown below, SfB was happy as can be.  Internal and external web services were flawless. No issues.  OWAS as mentioned popped up on screen.  LWA worked perfectly.  Mobile clients went tearing through. I saw no issues whatsoever – let alone anything that could be attributed to the compression.  Web services with the compression were “seat of the pants” faster.

lab

The Array Networks installation/deployment guide does a fine job of laying out the requirements and the “how to” part of the vAPV deployment to support Lync/SfB.  I have not yet had the chance to convert to a full load balance solution (nor do I really want to), but I would imagine that the results would be the same.

Conclusion

Let’s face it.  If you have an organization that is big enough, or perhaps small but needing the services of a load balancer – be it application delivery or just simple reverse proxy, then almost anything will work.  However; should you want to control the beast, and use your deployment for something other than just a one off, you need something more sophisticated.  As your traffic load grows and expands to cover more than just one workload, the underlying network devices become more and more important.  Enter Array Networks. The Array vAPV (and the physical APV for that matter) presents some very interesting feature sets for discussion.  Do you want simple or do you want granular control?  Are you willing to accept some sluggish performance or do you want screen-popping speed? Local load balancing is needed and you want global load balancing options for the future? If you went the caviar route on those questions, then Array Networks needs examination.

For a load balancer/application controller that offers a great feature set, is granular (seriously granular!), along with being wicked fast, then Array networks vAPV should be on your short list. 

You can get your very own vAPV here.

YMMV

2016/06/06

WebConf modalities not working for internal users after server patching

This falls into the “oh wonderful” category…

https://technet.microsoft.com/en-us/library/security/ms16-065.aspx breaks Office Web Apps for internal users.  External users seem to be unaffected.

Conferencing modalities no longer function in Lync Server 2010, Lync Server 2013, or Skype for Business Server 2015 after you install Security Bulletin MS16-065Here is a fix workaround:

https://support.microsoft.com/en-us/kb/3165438

And people wonder why I always advise waiting 90 days or so before patching Lync and SfB host servers.

The documented update in the article is KB3156757, but the actual KB installed was KB3156756.  Which also is associated with MS16-065.

YMMV

2016/04/30

Yealink SIP Phone Review

Edited 2016-05-02

Yealink has already answered one of the login issues highlighted below – login from outside the edge server.  It would seem that the PKI is not trusted – I assumed it was not as I never put a copy of the trusted root on the device.  So disabling that security requirement allows the device to login from outside the edge server.

clip_image001

end 2016-05-02 edit

Yealink, in their infinite wisdom, shipped me a few units to play with.  When the box arrived, I found the following three SIP phones:  T42G, T46G, and a T48G.  Oddness is the different labeling depending on where you look.  On the boxes for each unit it says “Ultra-elegant Gigabit IP Phone SIP-T4xG” whilst on the web screen for the phone it says “Gigabit Color IP Phone SIP-T48G”; the T42G claims it is a “Enterprise IP phone SIP-T42G”; the T48G and the T46G claim “Gigabit Color IP Phone SIP-T46G” – I love the consistency between the box labels and the internal programmed labels eh wot?

Here are the three devices in the hot seat for this round of testing:  From the left:  T46, T42, and T48.  The 46 and 48 have a color screen, the 42 is a bit smaller with a monochrome display.  The 46 has buttons around the screen, the 48 is completely touch on the screen.  The 42 also has buttons around the screen.  Like the 46, some of the buttons on the 42 work, some do not.

image

Initial Impressions

I want to get on the record with one thing:  I like these phones.  A lot. There are some issues as you will see; but do not let that detract from that first concept – these are nice pieces.

Coming out of the box, all three devices feel solid, well-constructed, and the various ports are well laid out, marked legibly, and everything fit together as expected.  I am sure glad I have a PoE switch handy, or I would have been hurting as none of the devices came with a power brick.

The buttons push as expected, the screens are crisp and have a good layout.  As a Skype-compatible device, it would seem that Yealink has engineered their own GUI interface for the phones for Lync.  As I got the phones, they were SIP Phones, generic.  I had to flash the firmware and upload a license to enable them for Skype.  I expected this, as these units are pre-release, and the fine folks at Yealink had sent me instructions in advance.

Initial setup and login went about as expected. Attaching to internal worked perfectly.  The phone unlock code is nice touch.  User SIP and UPN and PIN login works as expected. 

image

 

On dial out, the phone does not start the call until pressing OK. I am used to seeing a Lync phone take 10 digits and start the call.  Or take 4 digits (or whatever) and start the call.  Other devices in the Tsoorad Test Lab do exactly that.  Mashing the ‘#’ key sends the call as expected. Something to do with a configuration perhaps?

Yes!  Found it.  Settings | Preference| Live Dialpad –> set to enabled.  Yay!  I am told that the documentation for the newest firmware is coming with the GTM which is supposed to be June-ish 2016.  Having documentation will make this sort of thing easier to sort.

image

Choosing your ringtone could be onerous.  When using the web interface, you have to choose one, then save it, then make a phone call to determine what is what.  Driving into the touch screen on the phone itself plays the ringtone for you in real time.  So, choose your management interface and learn it.  When doing ringtones, 6, 7, and 8 are interesting.  And according to choices in the web interface you can upload your own ringtones. 

OTOH, you can login to the phone as admin while someone is logged into the phone as a user.  I like that.  A lot.  A flip side to that is the web interface times out on a very short cycle, and I could never figure out where to lengthen that out to like several hours.

To get the phone to be the correct time, I had to set it as shown.  Using DHCP time did not work, it came up an hour off.  A competitor phone got the correct time from DHCP. 

image 

T48G

Sample calls both inside and outside of the Tsoorad Test Labs facility were flawless.  Audio quality is really very nice impressive. The 7 (seven SEVEN) inch screen is really nice.  Touch.  Color.  I like it.  Here is the T48G.  Check out the color screen.  In phone terms, huge.  My old eyeballs have no issues reading the screen on the T48G.

image

T46G

The T46 has a smaller color screen than the T48 and the layout is different.  The functionality is the same, just things are in a different spot due to the interface being different.  Call quality, materials, and look and feel remain in the “dang this is a pretty nice unit” category. The T48 is all touch-screen and the only buttons are the dial pad and buttons down there; The T46 has buttons all the way around the screen – but only some of them are operational with the Skype firmware.  If you broke out Mr. Tape Measure, the T46 is also somewhat smaller overall.

image

T42G

The T42 is the smallest of the three, with a monochrome display.  Other than Skype sign-in, this unit it pretty much just punch the numbers and make calls.  I never did figure out how to do a conference call with the T42 even though the option is clearly presented.  The T42 is smaller than the T46, but the audio quality and build quality seem to be just as good as the larger units.

image

The Skype Connection

Obviously, or maybe not so obviously, I have only one reason to use a phone handset device like these.  To whit, I work with customers and their Skype projects and I get asked what handsets to use so I have to know.  For myself, I would not use a handset; I have my headset and I am good with that. But for others, a handset is requirement of life.  Therefore, we have Skype phones.  And they need to work to MY satisfaction. 

With the Yealink units, after flashing the firmware on all three phones, the Skype connection was entirely painless.  I inputted my user name and password and the phone signed right into the pool.  You cannot ask for more than that.  PIN login is equally painless. 

I could not get these phones to login to my lab from outside of my firewall.  Claims they cannot find the web server.  Funny.  Other vendors phones work just fine when attached outside of my Edge server. 

A lingering bug in the Yealink software will also prevent you from connecting to your organization from outside your domain and when your account is actually in a resource domain.  For instance, in my real world work, my account is actually in a subdomain of the larger forest.  And the Yealink phones don’t particularly like this arrangement.  I am told that the fix is a firmware revision that is coming with the anticipated Microsoft approval of these phones for use with Skype.  But on this list here the T48G is already listed…so I am now confused.  Which is really unfortunate, as these phones rock with Skype.

BToE

Better Together over Ethernet.  A little splice that goes onto your desktop and joins your phone at the hip to your full Skype client.  Pretty doggone handy for those without a CX phone.  Like me.  I installed the software on a Win8.1 machine and was up and running with the phone is less than 5 minutes.

image

After installing, all you need to know is the phones’ IP and the pairing PIN…

image

…voila!…

image

…and we are magically transported to EV-land, where the Skype client can operate as a softphone, a full web conference client, a desktop share client, a consumer, a producer, and a traditional telephony device user. 

image

So sweet. The Yealink BToE software is clean and well thought out and did not give me any hint of trouble.

Summary

Aside from a software/firmware thing that gets in the way of the phones being successful in two login scenarios – these Yealink Ultra-elegant GIgabit IP Phone units are easily on par with any other vendor device.  Provisioning via FTP is available.  Fellow MVP Grieg Sheridan seems to think that you can update these phones via CSCP tools…I could never find a ucupdates.exe for Yealink – but I also admit I did not look very hard (in my defense, I have one of those pesky customers that expects me to actually do things for them and not sit around dreaming up things to say in a blog article).

I found the materials, construction, and overall quality to be at least on par with all the other vendors out there in phone-land.  And two of these are COLOR.  Squirrel!

Documentation on these devices is extensive.

T48G firmware, docs, user guides, admin guides

T46G firmware, docs, user guides, admin guides

T42G – I am told by my “SFB Sales Engineer” who must remain nameless, that June 2016 will see complete SfB related documentation.  Which I hope applies to the T48 and the T46 also!

I really want to like these phones.  The market needs the competition.  Yealink has done a credible job on producing Skype versions of their existing (beautiful) phones.  So kudos to them for jumping into the fray.  Let’s hope they can iron out a few niggling firmware items and then they will have solid winners for the Skype environment.

You can get your Yealink phones right here:

T48G

T46G

T42G

YMMV

2016/04/18

AudioCodes 405 IP Phone

I used to think that the AudioCodes 420HD was my leading candidate for a low-cost, high-value SIP phone.  Now I think there is a new leader. 

Enter the AudioCodes 405 IP Phone.  I am getting some mixed signals as to how long this model has been on the market, but I just got one the other day and have been putting it to the Tsoorad Test Lab Experience for the last week or so.

image

Here’s a stock photo just to show the difference between a professional marketing photographer and yours truly with a cell phone camera.

image

You can get all the official AudioCodes material here.

Build Quality

Ho Hum.  I wish AudioCodes would ship a unit that would give me something for this section other than “excellent.” If I have to make some comment, the button feel is, IMHO, much better than the 420HD in a side-by-side.  Oh, and the display, while smaller, is crisper with a tad better contrast. While I did not use the handset to hammer nails, the materials appear to be a touch above some other competing vendor’s products.

Voice Quality

Again.  See previous paragraph.  BORING.  Same high standards as before.  Speaker phone volume is nicely controllable.

BToE (Better together over Ethernet) is available should you so desire. Second line, call transfers, conference calls, etc.  All that works as expected without doing anything other than maybe a quick study through the various documentation bits.  Here is the official market-speak pertaining to features on the 405 IP Phone:

The 405 SIP IP Phone is a cost-effective, entry-level IP phone designed to offer the essential everyday features that the modern business environment demands.

  • Graphical, backlit multi-lingual LCD (132 X 64)
  • 4 programmable soft keys
  • AudioCodes Auto-provisioning
  • Full SIP protocol support with extensive interoperability
  • Robust security mechanisms
  • Power over Ethernet (PoE)
  • Multiple language support
  • Integration with voice quality monitoring
  • Full duplex speakerphone and headset connectivity
400HD SIP IP Phone Series Shared Features
  • High voice quality
  • Full duplex speaker phone
  • Robust security mechanisms
  • PoE
  • Out of the box global redirection server support
  • Multi-language user interface
  • Centralized management with AudioCodes EMS

Skype for Business Connection

Seeing as how I make market with Skype, you know I had to connect this to a Skype system and kick the tires, right?  Here is the connections I made:  Ethernet and handset.  Highly technical, eh?

image

After that, it was a matter of a minute or so to input my login information and sign-in.  My lab worked perfectly first time through.  My work account worked perfectly the first time through also.  I literally did NOTHING to this device to make it Lync/SfB compatible; it just worked.  At that point, I had access to all applicable SfB Enterprise Voice functionality.  Without knowing anything more than my account and password.  This would seem to lend itself very nicely to providing a phone to a telecommuter.

You can get your very own AudioCodes 405 IP Phone right here.

Conclusions

A first rate telephony device for your SIP (I sure hope you are running Skype) system.  Excellent build quality, excellent audio, good feature set for the money, able to be centrally managed, part of a larger eco-system of VOIP solutions, etc.  You can’t do wrong choosing this direction.

YMMV.

2016/04/11

Skype for Business Cloud Connector Edition

This is highly reminiscent of watching this Steve Martin classic.

But for those of us who have been waiting for lo! these many months, as of 0900 PST, the CCE is here!

The Skype for Business Cloud Connector Edition is a set of packaged virtual machines for deployment on-premises which connect a customer’s existing Public Switched Telephone Network (PSTN) service provider circuits with Skype for Business Cloud PBX operating in Office 365. This allows for the user’s phone capability to be managed out of Office 365 while their phone calls continue to use their existing phone number, circuits and PSTN provider contract.

One caveat – the existing documentation calls for Windows 2012 R2 Datacenter –

image

There is some light at the end of that particular tunnel in that Windows 2012 R2 Standard Edition appears to be valid also using a secondary methodology.  More clarification to the Technet documentation is pending.  Specifically, there is this statement in the reference:

Before you deploy Cloud Connector Edition, make sure you have the following for your environment:

  • A Windows Server 2012 R2 ISO image (.iso). Both Standard edition and Data Center edition are supported. The ISO will be converted to VHDs for the virtual machines that will run Skype for Business Cloud Connector Edition.

You can get started right here:  https://www.microsoft.com/en-us/download/details.aspx?id=51693

YMMV

2016/02/17

1008;reason=Unable to resolve DNS SRV record

ms-diagnostics: 1008;reason="Unable to resolve DNS SRV record";domain="domain.com";dns-srv-result="NegativeResult";dns-source="InternalCache";source="SfBSIP.domain.com"

Scenario Outline

SFB on-premises patched to November 2015. Split-DNS. Firewalls, networks, and even VLANs are all highly segregated.  Classic DMZ in operation with outside firewall, inside firewall, no internet browser access from DMZ servers.  Port 53 outbound from DMZ servers is not allowed.

The edge servers are using internal DNS resolution (hello InfoSec!).  Everything is testing perfectly.  IM/P, WebCon, media flow; the mobile clients are working, and PPT publishing internally and externally is perfect.  After working through the expected HLB and firewall issues, we are looking right successful.  First time through.  Nailed it.  But wait!

Organization moves from closed federation to open federation.  About a week later we notice that federation is suddenly borked – and one-way presence rears it’s ugly head – it would appear that federated partner –> internal org can start things, but the opposite does not work so well.  However, everything except presence works AFTER the inside person responds to the outside –> inside toast.  Screen sharing fails also – unless the outside person starts the screen share, then the inside person can share.  This is a hint for you troubleshooting mavens – we’ll wait while you digest all this information.

TShoot

We traced the above client side errors and see the following:

Subscribe attempt…

image

…and the resulting 504.

image

We traced the same errors from the server-side (thank you centralzed logging) and see the same set of outcomes.  Here is a simple subscribe request from the inside to a federated partner…

image

…and you can see the 504 – I cannot find out who I am because I cannot resolve my federation SRV record.  This is not good.

image

A side symptom was that we were seeing similar 504 errors on test-csfederatedpartner and test-csmcxpushnotification.

Hmmm.  Does this look like the Edge server cannot find itself?  Like there is no _sipfederationtls._tcp.domain.com record?  Consider the lock-down environment, and the requirement that all DNS come from the inside…and the inside is going to be authoritative for the zone.  Hmm.  Lync 2013 documentation (essentially the same for SfB) indicates the SRV record for _sipfederationtls._tcp.domain.com needs to be on the external DNS server. So, go double check that.  Yes.  We got that part right. 

The Fix

Simple.  We put the _sipfederationtls._tcp.domain.com SRV record into place on the internal DNS, with the proper target.  And then modified the host file on each Edge server to have the public IP for themselves.  We did a TTL of 5 minutes on the SRV record.  Almost immediate relief.  It was like watching Bones cure the planetwide plague with a simple shot of his hyper-injector and you get watch the horrible disease be cured before the next commercial break.

But WHY?

Why did the transition from closed federation to open federation cause this?  And why did “this” take 7 days to manifest itself in failures?  Why didn’t the issue show up immediately?

Summary

I can guess at the first, as to the second and third, I am clueless. I am not willing to guess in a public forum, so you will have to draw your own conclusions. But I do know what fixed this issue – the federation SRV record being added to the internal DNS zone and modification of the Edge Server host files so that they can find the SRV target by IP.

YMMV

2016/02/09

SfB Patch/Upgrade Outline

Skype for Business (SfB) Server 2015 embodies several server-side enhancements beyond Lync Server 2013. The patching cycle for the SfB environment will need to be modified to allow for these enhancements. SfB contains five layers of servers, each of which will need to have separate handling:

  • Front End Pool Servers
  • Persistent Chat
  • Edge Servers
  • OWAS (Office Web Apps Servers) Servers
  • SQL Server and File Shares.

Host Server updates also need consideration – primarily because rebooting SfB servers can cause Windows Fabric errors that can affect the ability of the SfB server to recover into a running state.

Host Servers need to be patched to corporate standards; however, the application host servers cannot just be rebooted at will. Rebooting servers that host SfB services will result service outages and potentially in service failures where the servers may not recover services after rebooting.

Accordingly, phase one in the entire setup for patching SfB and related servers is to set the Windows Update to download but require administrator to install. For ORGNAME this may require moving servers away from containers to which GPO applies and controls WUPDATE settings.

clip_image001

This guidance will not apply to the SfB Edge servers as they are not domain members. However, the SfB Edge servers should be checked to ensure that the WUPDATE is set as shown.

Locate and download the latest SfB server updater from this site: https://technet.microsoft.com/en-us/office/dn788954 - as of this writing, November 2015 is the latest SfB 2015 update.  The consolidated server update installer is preferred over the individual updates. 

Note that the file name show is mostly correct, but that I rename them to help me keep track of what is what.

clip_image002

Place the update file in a separate folder on each front end, persistent chat, and edge server.  The update process generates log files which are kept in the origination location. Having a separate folder for each updater constrains the log file location and makes the entire thing easy to delete or verify later.

SfB Front End Servers

Reference:

  1. https://support.microsoft.com/en-us/kb/3061064
    1. Find the section labeled: “Upgrade or update the Enterprise Edition pool that has at least three front-end servers” and READ IT.
  2. Read the following TechNet guidance: https://technet.microsoft.com/en-us/library/jj204736.aspx
  3. Then execute those instructions ONE SERVER AT A TIME.
  4. After each server reboots wait until ALL indicated services are running before moving to the next server in the pool. Keep in mind that these services are on delayed startup, and there could be a significant (10-15 minutes) delay before the SfB Front-End service starts.

clip_image003

SfB Persistent Chat Server

SfB Persistent Chat requires only that the services be running after the persistent chat server is patched and rebooted.

clip_image004

SfB Edge Servers

Edge servers are easy. Execute the serverupdateinstaller.exe on one Edge server at a time. Reboot if requested. If a reboot is needed, monitor the reboot process until the SfB services are restarted (about 10 minutes). Otherwise, verify the following services are running.

clip_image005

Do the next edge server.

Office Web Apps Server (OWAS)

The OWAS requires different handling from the other servers. See the following articles:

Assuming the two OWAS servers are hlbwowasp101 and hlbwowasp102, the following commands will recreate the OWAS farm when the time comes:

1. From server hlbwowasp101.corp.domain.com, open PowerShell as administrator, and execute the following (command wrapped):

    • new-officewebappsfarm -internalurl https://hlbsfbowas.domain.com -externalurl https://hlbsfbowas.domain.com -certificatename sfbwebext

2. From server hlbwowasp102.corp.domain.com, open powershell as administrator, and execute the following command AFTER the previous command on the other server:

    • new-officewebappsmachine -machinetojoin hlbwowasp101.corp.domain.com

After patching, reboot, and recreation of the WebAppsFarm, verify the following service is running on each server:

clip_image006

SfB File Shares

ORGNAME runs the SfB file share (\\corp.domain.com\sfb-fileshare ) on the OWAS servers. Care must be given to handling the DFS in that the entire environment is relying on the sfb-fileshare for various functions and downtime on the OWAS servers will affect all other servers. Other than the update process shown above, the OWAS servers should only be updated one at a time.

SfB SQL

ORGNAME is using a single SQL server. This server should be patched along with the other SfB infrastructure with the following caveat: The SQL needs to be back online within 30 minutes or there will be impact to the users. The impact will be the clients entering “resiliency mode” due to the SQL server not being available to the front end servers. For more information, see this: https://technet.microsoft.com/en-us/library/jj205184.aspx.

If you have mirrored SQL or perhaps Availability Groups in SQL, then you will need to investigate the SQL patching process from a slighlty different aspect – namely, keeping the active node where you want it.

Summary

SfB has changed the patching process from how it was done in Lync 2010 and Lync 2013.  Each layer of the system needs something that is just a little different from the other layers of the system.

YMMV