Users unable to join LM session, MEET NOW button not working, LM functions grayed out.

In the middle of a deployment - an OCS R1 to R2 migration - we noticed that the LM functions were not working.  LM worked for the individual workstations when connecting to a remote session initiated by a federated partner.  The meeting policy at the global level was created, edited, and assigned correctly.

We noticed that we had neglected to run the Web Conferencing validation tests.  We had a tick box on our checklist, we had just overlooked it.  Running the validation wizard revealed that the validation connection checks were failing to successfully contact the MCU on either of the Front End servers.  We double-checked everything, and concluded that everything on the OCS side was correct.

“Aha,” says we.  “Must be a certificate issue.”  Except that we were using good public certificates on all the interfaces for the FE servers.  Bummer.  Except, I then read this blog article.

Oddly, this was so on target it floored me.  I used the solution that changed the usage on the Trusted Root Certificate to “ALL” - voila!  problem resolved.  I don’t know who exactly wrote this, and what follows is a cut ‘n paste and edit of the relevant parts of that blog that fixed my issue. Many thanks to the unknown CSS engineer (Dave) who took the time to write this up.

Event Type:    Error
Event Source:    OCS MCU Infrastructure
Event ID:    61013
User:        N/A
Computer:    OCS1
Description:
The process DataMCUSvc(2596) failed to send health notifications to the MCU factory at https://OCS1.contoso.com:444/LiveServer/MCUFactory/.
Failure occurrences: 3491, since 3/24/2009 10:05:18 PM.

If you run the Web Conferencing validation wizard from the OCS Pool, you may find the following error in the output log:

MCU Type: meeting
URL: https://OCS1.contoso.com:444/LiveServer/MCUFactory/
HTTP Connectivity Error : ReceiveFailure
HTTP Connectivity Error : Receive failure typically indicates that the connection was closed by
the remote host. This can happen if the remote server does not trust the certificate presented by the
Local Server.

HTTP Connectivity Error : Ensure that the certificate of the local server and remote server are both
valid, have not expired, and contain valid subject name. In addition, ensure that the certificate chain
of both Server(s) are valid. Ensure that the certificate chain of the local server is installed
on the remote server and vice-versa. The most up-to date certificate chain that was used to issue
the server certificate must be present.

When you see errors like these, it usually indicates that a certificate-related authentication problem exists with the OCS Pool (or with a particular OCS Front End server).  Most of the time, this turns out to be a problem with the certificate from an issuing Certification Authority.  To troubleshoot this issue, you would typically perform the following steps:

1. Log in to the affected OCS 2007 Front End server either locally or remotely using Remote Desktops.
2. If the issuing CA is a Root CA (the top of the list), expand Trusted Root Certification Authorities > Certificates
1. If the issuing CA is an Intermediate CA (not the top of the list), expand Intermediate Certification Authorities > Certificates
2. From the list of CA certificates, right click on the certificate and choose Properties
3. Under the General tab, verify that Enable all purposes for this certificate is selected (or, if Enable only the following purposes is selected, verify that both Server Authentication and Client Authentication are enabled)
4. Click OK to close the properties of the CA certificate.
5. If this was an Intermediate CA certificate, repeat steps 6 through 10 until these settings from all certificates in the trusted certification chain are verified
6. Close the Certificates Management Console (be sure to restart services if you made any changes)

Why this occurred on a brand new R2 installation on server 2008 SP2 is beyond me.  The OCS R1 system (on Server 2003 SP2 R2) did not have this issue, but the brand new setup did.  Go figure.

Edge Server Certs and blank Communicator message windows

A client recently changed their certificates on the edge server.  They put together a certificate that handled everything with one certificate.  However, the SAN construction on the cert was a little wrong.

Symptoms: 

Presence worked, but federated contacts could not fully establish an IM session.  LM and AV did not work as expected either.  If a federated user initiated an IM, the internal user would get the toast and then when the toast was opened, there was nothing but a blank….but the toast had the initial message…but a blank content pane.  If the internal user attempted to initiate an IM session, the reverse would occur.  After the blank IM window appeared, any subsequent efforts at IM resulted in a timeout with a 504 error.

What caused this?

Logging on both edges revealed that the initial IM invite was addressed to the proper SIP SRV record, but after the initial ACK, the client system packets were being directed at a different FQDN.  Digging into the client’s edge server revealed that the FQDN was the actual server FQDN.  It seems the cert had been issued for the FQDN and that SIP, AV, and LM were on the cert, but that SIP was not the FIRST SAN name.

So, what happens is that the federated contact can get to SIP.domain.com (via _sipfederatedtls._tcp.domain.com SRV) for the initial invite, but after that the packet sourcing of the remainder of the conversation looked like it came from FQDN of the client’s edge because, according to the certificate on the Access Edge, that is exactly where it came from. The initial SIP invite worked because the traffic arrived at the edge.domain.com access edge interface IP address, and the SAN on the existing (new) cert did indeed have SIP.domain.com as a valid domain.  However, certificate’s common name and first SAN entry is what drives that particular NIC FQDN name when it comes to transmitting vice receiving. The end result is the federated side of the conversation gets started just fine, but then tries to communicate to an FQDN that is not accessible from the internet.

Clear as the bottom of a well on a dark night, eh?

The Fix:

Changing the certificates back to the originally installed set fixed the issue….

• SIP.domain.com
• AV.domain.com
• LM.domain.com

This will also work if you use ONE certificate with those three names (or whatever name you choose for each) as long as the SIP.domain.com is both the common name of the cert as well as the first SAN entry.  The FQDN of the actual server should only show on the internally-facing Edge interface.  For even MORE confusion, see this:

Bon Appetit!

MS KB 974571 and OCS/LCS

http://communicationsserverteam.com/archive/2009/10/14/632.aspx  outlines and issue with applying this security patch to your OCS/LCS servers.  The only fix if this is happening to you is to uninstall the KB fix.

Microsoft delivers zero license cost XMPP Gateway for OCS

Today, Microsoft delivers a new gateway.  Read about it here.  This is GREAT news.

Now you can federate/PIC with Microsoft Live, Google Talk, and Jabber.

OCS 2007 R2 and Server Role Virtualization

I get asked in every engagement, “…can we virtualize OCS R2?”  This is a great question.  Using virtualization has clear benefits - and also clear drawbacks.  On the benefit side, using virtualization offers more efficient hardware utilization; on the drawback side, applications that are time sensitive or are CPU sensitive may suffer degraded performance.  OCS 2007 R2 falls into both categories.  OCS has some roles that lend themselves to virtualization, but OCS also has server roles that Microsoft does not support in a VM environment (presumably because of the performance degradation).  Specifically, any server role that handles media (read A/V) is not supported in a virtualized environment.

For reference see the Office Communications Server team blog on this subject.  Also, I encourage you to read this document that lays out the situation in more detail.

Now that we know what is supported, what is practical and not practical?  We know that some organizations will weigh support issues against the advantages of an unsupported deployment and decide the benefits outweigh the risks. The documents I just pointed you to are definitive, yet the scale numbers in the second document are for large environments.  What about the smaller enterprises with only a few hundred users?  How about a company with less than 5000 users (the implied limit for a Standard Edition server)?  What about the ancillary OCS server roles (archiving, monitoring, directors)? What follows is purely my opinion and experience.  If you are concerned with being in a supported status (from the Microsoft CSS viewpoint), stop here and follow the guidance in the references to the letter.

Let’s look at the VM host environment and then see what we can do. When you plan your VM host server, make sure it has PLENTY of CPU cycles - read many fast cores.  RAM is an important item on your VM host server.  Do not scrimp on RAM.  The guest VM must have at LEAST the minimum number of CPU cores and amount RAM as that server role is required to have on a physical server.  If the recommended minimum is 2 CPU cores and 4GB RAM, then that is the minimum for the guest VM instance also.  As to drive speed on the VM host, resist using SATA and go with 15k RPM SCSI.  And do not try to cram too many guests VM’s onto one host.  As always, leave enough RAM and CPU for the host OS to function properly.

Plan your VM host server network support carefully.  Good resources for this critical task when using Microsoft's Hyper-V are here and here.  I highly recommend using at least two physical Network Interface Cards - one for the host server and one for the guests. Resist the urge to get fancy and do NOT use wireless.  The combination of VM, OCS, and wireless creates performance issues.  Another item of note is that you can adjust the performance of both the VM host and the VM guest to maximize performance.  Face it, GUI is nice, bells and whistles are cool, but do you really need that stuff on your servers?  You will never miss these fancy features…they are just fluff. In a virtualized environment, where cost reduction through hardware consolidation is generally a main goal, performance is king.This is what I do for every VM host and guest I touch - the performance gain is noticeable:

Now that we have looked at the VM host and guest environment we can consider which OCS roles are candidates for virtualization, and in my experience, what is practical. The following table outlines the OCS 2007 R2 servers roles and their respective VM possibilities:

Server role	VM Yes/No	Notes
Consolidated FE (SE)	No	Just don’t
Consolidated FE (EE)	Yes for IM&P only	No for anything A/V, desktop share, or Live Meeting
CWA	Yes	As the user numbers climb, performance will drop off.  IE; You will see some screen-draw issues as the user count climbs.
Group Chat	Yes	Works well
Group Chat Archiving	Yes	Works well.
Mediation	No	Just don’t
Director	Yes	For the numbers we quoted above, it works
Consolidated Edge	No	Just don’t
Archiving	Yes	For the numbers we quoted above, it works
Monitoring	Yes	For the number we quoted above, it works
Web Components	Yes	This will depend on your LM work load, how many remote users are expanding DL’s etc.
SQL	Yes	SQL in VM is supported; but not SQL cluster.  However, I have done SQL cluster on two separate VM hosts.  Do not do VMotion or equivalent on the cluster.

While you are at this, you may want to take a look at SQL databases.  Microsoft says you need a separate SQL instance for each database.  This is for performance reasons.  However, I know that one SQL server can host the backend OCS database, the Group Chat Archive Database, the archiving database, and the monitoring database and do just fine.  Again, this is a numbers game.  As the number of users climbs, the load on the SQL will climb also and eventually you will see a drop in performance.  Specifically, our second reference document says that the SQL backend is the limiting factor to how many users can be attached to a virtualized enterprise pool.  But, our target user numbers are only 1/8th of that limit.  Therefore, I feel confident in stating that one SQL server (with the proper resources) will support your OCS needs.

Let’s take look at this table from a deployment perspective.  As an example, say you have 1000 users. Your usage projections are for moderate use of Live Meeting, about 100 users maximum for CWA, and about 200 or so of your users are remote, and federation/PIC will be used by 30% of your users (just picking numbers here for this example).  In this case, I would think that you could explore the possibilities of using VM for everything but your SE and your Edge.  A single SQL server, deployed in VM, will be able to host all of your OCS databases. Should you deploy Enterprise Voice, you would place your mediation server on physical as well.

Before you move ahead with a project such as this, remember this is just my opinion, based on my experience, and that virtualizing OCS 2007 R2 roles that are not supported as virtualized by Microsoft places you in an unsupported configuration.

Exchange 2010 OWA and IE

It appears that there are significant differences between the IE versions and how OWA displays and operates with Exchange 2010 RC.

IE7 is the minimum to get the premium OWA; IE6 gets you OWA lite.

:(

polite drivers

Hello Wenatchee Washington.

I have had the opportunity to cross the street recently, and I have to say; I wish my home town had drivers who stopped for pedestrians.  And I am not talking about just one or two, I mean everybody! 

I am simply not used to drivers stopping 50-60 meters short of where I am waiting to allow me to walk across the street.  And these are four lane main streets, not some dinky-ass side street.

In Portland (Oregon), and more specifically Gresham, where I live, crossing the street is an adventure - I fully expect some psycho to attempt to run me over should I have the temerity to use a cross-walk.  In Wenatchee, even crossing in the middle of the block worked just the way it should.